Inference and deferred buffers

Before each operation, a kamping::v2 wrapper calls infer() with the operation’s tag, its buffers and the communicator. For plain buffers it does nothing. For deferred buffers, which do not know their receive count(s) yet, it obtains them: by probing the message, by a count exchange, or locally where the count is already determined. See Receive buffers that size themselves.

Deferred buffers

template<typename T>
concept deferred_recv_buf

template<typename T>
concept deferred_recv_buf = requires(T& t, std::ptrdiff_t n) { t.set_recv_count(n); }

Recv buffer whose size is not known upfront and is determined by infer(). The collective calls set_recv_count(n) after inferring n; the view then lazily resizes on first mpi_ptr() access.

Defined in header <kamping/v2/views/concepts.hpp>

template<typename T>
concept deferred_recv_buf_v

template<typename T>
concept deferred_recv_buf_v =
    has_mpi_counts_mutable<T> && has_commit_counts<T> && has_set_comm_size<T>

Recv buffer for variadic collectives whose per-rank counts are not known upfront. The infer() protocol:

1. set_comm_size(comm_size) — pre-allocate the counts buffer

2. MPI writes into mpi_counts() — fill per-rank counts directly

3. commit_counts() — signal counts are ready; invalidate cached state

Defined in header <kamping/v2/views/concepts.hpp>

template<typename T>
concept deferred_send_buf_v

template<typename T>
concept deferred_send_buf_v = send_buffer_v<T> && has_set_comm_size<T>

Send buffer for variadic collectives whose per-rank counts depend on the communicator size (e.g. a sparse or flattened source where ranks not present in the input must receive a count of 0). The infer() protocol calls set_comm_size(comm_size) before reading mpi_counts()/mpi_displs(), giving the buffer a chance to lay out its counts/displs/data across all comm_size ranks.

Unlike deferred_recv_buf_v, the counts are produced by the buffer rather than written by MPI, so no mutable mpi_counts()/commit_counts() is required.

Defined in header <kamping/v2/views/concepts.hpp>

template<typename T>
concept has_monotonic_displs

template<typename T>
concept has_monotonic_displs = requires(T const& t) {
    { t.displs_monotonic() } -> std::convertible_to<bool>;
}

A buffer whose mpi_displs() are guaranteed to be monotonically non-decreasing (e.g. computed via exclusive_scan). When true, resize_v_view can use the O(1) tight bound displs.back()+counts.back() instead of the O(p) max formula.

Defined in header <kamping/v2/views/concepts.hpp>

template<typename T>
void materialize(T &&buf)

Forces a deferred receive buffer to realize its backing storage now.

Deferred receive buffers (those built with views::resize / views::resize_v / views::auto_recv / views::auto_recv_v) postpone resizing the underlying container and computing displacements until their data pointer is first accessed during the MPI call. Custom collectives sometimes need the buffer fully realized before the actual exchange — for instance to read back the final displacements in order to lay out a multi-phase transfer. materialize makes that an explicit, public operation instead of relying on the side effect of a ptr() access.

Contract: after materialize(buf) returns, count(), ptr(), counts() and displs() on buf yield final, stable values and the backing storage is sized. The operation is idempotent — calling it repeatedly, or letting the MPI wrapper trigger the lazy realization afterwards, is harmless.

The materialize() member is only invoked on buffers that are both materializable and an actual deferred receive buffer; on anything else (a plain, already-concrete buffer, or an unrelated type that merely happens to expose a materialize() member) this is a no-op, so it is safe to call unconditionally in generic code.

Propagates automatically through kamping view layers: an outer wrapper (with_type, GPU/serialization adaptors, …) around a deferred buffer forwards materialize() — see view_interface.hpp — so a nested deferred buffer is realized regardless of how many layers wrap it.

Parameters:

• buf – The buffer to realize. Mutated in place.

Defined in header <kamping/v2/views/concepts.hpp>

bool supports_matched_probe

Trait controlling whether infer() uses matched probing (MPI_Mprobe / MPI_Mrecv) for a deferred recv buffer. Defaults to true. Set to false for buffer types whose MPI implementation does not support GPU-aware matched receives (e.g. thrust::device_vector with Intel MPI); infer() then falls back to a plain MPI_Probe followed by a regular MPI_Recv.

Propagates automatically through kamping view layers — see view_interface.hpp.

To opt out a custom buffer type:

template <>
inline constexpr bool kamping::v2::supports_matched_probe<MyGpuBuffer> = false;

Defined in header <kamping/v2/views/concepts.hpp>

The infer() customization point

Add an overload of infer() for your own buffer types in their namespace; it is found by argument-dependent lookup. The overloads below are the defaults.

template<recv_buffer RBuf>
auto infer(comm_op::recv, RBuf &rbuf, int source, int tag, MPI_Comm comm)
(1)

template<recv_buffer RBuf>
void infer(comm_op::mrecv, RBuf &rbuf, v2::probe_result const &probe)
(2)

template<send_recv_buffer SRBuf>
auto infer(comm_op::bcast, SRBuf &srbuf, int root, MPI_Comm comm)
(3)

template<send_buffer SBuf, recv_buffer RBuf>
void infer(comm_op::allgather, SBuf const &sbuf, RBuf &rbuf, MPI_Comm comm)
(4)

template<send_buffer SBuf, recv_buffer_v RBuf>
void infer(comm_op::allgatherv, SBuf const &sbuf, RBuf &rbuf, MPI_Comm comm)
(5)

template<send_buffer SBuf, recv_buffer RBuf>
void infer(comm_op::alltoall, SBuf const &sbuf, RBuf &rbuf, MPI_Comm)
(6)

template<send_buffer_v SBuf, recv_buffer_v RBuf>
void infer(comm_op::alltoallv, SBuf &sbuf, RBuf &rbuf, MPI_Comm comm)
(7)

template<send_buffer SBuf, recv_buffer RBuf>
void infer(comm_op::sendrecv, SBuf const &sbuf, RBuf &rbuf, int dest, int send_tag, int source, int recv_tag, MPI_Comm comm)
(8)

template<send_recv_buffer SRBuf>
void infer(comm_op::sendrecv_replace, SRBuf&, int, int, int, int, MPI_Comm)
(9)

template<send_buffer SBuf, recv_buffer RBuf>
void infer(comm_op::reduce, SBuf const &sbuf, RBuf &rbuf, MPI_Op, int root, MPI_Comm comm)
(10)

template<send_buffer SBuf, recv_buffer RBuf>
void infer(comm_op::allreduce, SBuf const &sbuf, RBuf &rbuf, MPI_Op, MPI_Comm)
(11)

template<send_buffer SBuf, recv_buffer RBuf>
void infer(comm_op::gather, SBuf const &sbuf, RBuf &rbuf, int root, MPI_Comm comm)
(12)

template<send_buffer SBuf, recv_buffer_v RBuf>
void infer(comm_op::gatherv, SBuf const &sbuf, RBuf &rbuf, int root, MPI_Comm comm)
(13)

template<send_buffer SBuf, recv_buffer RBuf>
void infer(comm_op::scatter, SBuf const &sbuf, RBuf &rbuf, int root, MPI_Comm comm)
(14)

template<send_buffer_v SBuf, recv_buffer RBuf>
void infer(comm_op::scatterv, SBuf &sbuf, RBuf &rbuf, int root, MPI_Comm comm)
(15)

template<send_buffer SBuf, recv_buffer RBuf>
void infer(comm_op::scan, SBuf const &sbuf, RBuf &rbuf, MPI_Op, MPI_Comm)
(16)

template<send_buffer SBuf, recv_buffer RBuf>
void infer(comm_op::exscan, SBuf const &sbuf, RBuf &rbuf, MPI_Op, MPI_Comm)
(17)

(1) recv / irecv: probes the message (MPI_Mprobe, or MPI_Probe if the buffer does not support matched probes) to size a deferred receive buffer. Returns the matched MPI_Message, or the resolved {source, tag} after a plain probe, which the caller then receives.

(2) mrecv / imrecv: the message has already been probed, so a deferred receive buffer is sized from the probed count.

(3) bcast: broadcasts root’s count to size a deferred buffer on the other ranks.

(4) allgather: sizes a deferred receive buffer to comm_size * count(sbuf).

(5) allgatherv: allgathers the send counts into the deferred counts of the receive buffer.

(6) alltoall: sizes a deferred receive buffer to count(sbuf).

(7) alltoallv: tells a deferred send buffer (e.g. flatten_v_view) the communicator size so it can lay out its counts, then alltoalls the send counts into the deferred counts of the receive buffer. sbuf is non-const for the former.

(8) sendrecv / isendrecv: exchanges the send counts to size a deferred receive buffer.

(9) sendrecv_replace: nothing to infer for a buffer of known size.

A deferred buffer is rejected: MPI_Sendrecv_replace uses one count for both directions, so the buffer cannot be sized to the incoming message without changing what is sent. Buffer types that can handle this themselves, e.g. by exchanging additional metadata, can provide their own overload.

(10) reduce: sizes a deferred receive buffer on root to count(sbuf).

(11) allreduce: sizes a deferred receive buffer to count(sbuf).

(12) gather: sizes a deferred receive buffer on root to comm_size * count(sbuf).

(13) gatherv: gathers the send counts into the deferred counts of the receive buffer on root.

(14) scatter: broadcasts root’s per-rank count to size a deferred receive buffer.

(15) scatterv: tells a deferred send buffer on root the communicator size, then scatters root’s send counts to size a deferred receive buffer. sbuf is non-const for the former; non-root ranks pass null_buf_v.

(16) scan: sizes a deferred receive buffer to count(sbuf).

(17) exscan: sizes a deferred receive buffer to count(sbuf).

Defined in header <kamping/v2/infer.hpp>

Operation tags

Each operation passes an empty tag type from kamping::v2::comm_op as first argument, e.g. comm_op::allgatherv{}, so that infer() overloads can be written per operation: recv, mrecv, sendrecv, bcast, gather, gatherv, scatter, scatterv, allgather, allgatherv, alltoall, alltoallv, reduce, allreduce, scan, exscan.