Collective operations

Collectives follow the same conventions as point-to-point operations: the communicator comes last, the root defaults to rank 0 and the reduction operation to std::plus<>. Rooted collectives also have a shorthand for non-root ranks that leaves out the buffer only the root uses, e.g. gather(sbuf, comm) on a non-root rank. See Calling conventions.

Variable-size collectives (gatherv, scatterv, allgatherv, alltoallv) take buffers with per-rank counts and displacements. If a receive buffer defers its counts (views::auto_counts, auto_recv_v<T>()), they are exchanged before the operation; see Receive buffers that size themselves.

Operations with a send and a receive buffer return a kamping::v2::result, which can be unpacked with a structured binding: auto [sbuf, rbuf] = v2::allgather(...).

Synchronization

template<convertible_to_mpi_handle<MPI_Comm> Comm>
void barrier(Comm const &comm)

Blocks until all ranks of comm have called it (MPI_Barrier).

Defined in header <kamping/v2/collectives/barrier.hpp>

template<convertible_to_mpi_handle<MPI_Comm> Comm, convertible_to_mpi_handle_ptr<MPI_Request> Request>
void ibarrier(Comm const &comm, Request &&request)

Starts a non-blocking barrier on comm (MPI_Ibarrier).

The operation completes once all ranks of comm have started the barrier.

Parameters:

  • request – Receives the request of the barrier; the caller must complete it.

Defined in header <kamping/v2/collectives/ibarrier.hpp>

Broadcast

template<send_recv_buffer SRBuf, rank Root, convertible_to_mpi_handle<MPI_Comm> Comm>
SRBuf bcast(SRBuf &&send_recv_buf, Root root, Comm const &comm)
(1)
template<send_recv_buffer SRBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
SRBuf bcast(SRBuf &&send_recv_buf, Comm const &comm)
(2)

(1) Broadcasts send_recv_buf from root to all ranks (MPI_Bcast).

If the buffer is a deferred receive buffer (e.g. views::resize), root’s count is broadcast first and the buffer is resized on the other ranks.

(2) Broadcasts send_recv_buf from rank 0 to all ranks; see the overload above.

Returns:

The buffer (owned if passed as rvalue, borrowed otherwise).

Defined in header <kamping/v2/collectives/bcast.hpp>

Gather and scatter

template<send_buffer SBuf, recv_buffer RBuf, rank Root, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> gather(SBuf &&sbuf, RBuf &&rbuf, Root root, Comm const &comm)
(1)
template<send_buffer SBuf, recv_buffer RBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> gather(SBuf &&sbuf, RBuf &&rbuf, Comm const &comm)
(2)
template<send_buffer SBuf, rank Root, convertible_to_mpi_handle<MPI_Comm> Comm>
SBuf gather(SBuf &&sbuf, Root root, Comm const &comm)
(3)
template<send_buffer SBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
SBuf gather(SBuf &&sbuf, Comm const &comm)
(4)

(1) Gathers sbuf from all ranks into rbuf on root (MPI_Gather).

On root, rbuf may be a deferred receive buffer (e.g. views::resize); on the other ranks it is ignored, so pass null_buf or use the overloads without receive buffer.

(2) Gathers sbuf from all ranks into rbuf on rank 0; see the overload above.

(3) Non-root shorthand of gather: contributes sbuf to the gather on root, without a receive buffer.

(4) Non-root shorthand of gather: contributes sbuf to the gather on rank 0, without a receive buffer.

Returns:

  • (1) The send and receive buffer (owned if passed as rvalue, borrowed otherwise).

  • (3, 4) The send buffer.

Defined in header <kamping/v2/collectives/gather.hpp>

template<send_buffer SBuf, recv_buffer_v RBuf, rank Root, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> gatherv(SBuf &&sbuf, RBuf &&rbuf, Root root, Comm const &comm)
(1)
template<send_buffer SBuf, recv_buffer_v RBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> gatherv(SBuf &&sbuf, RBuf &&rbuf, Comm const &comm)
(2)

(1) Gathers variable amounts of data from all ranks into the variadic buffer rbuf on root (MPI_Gatherv).

On root, rbuf may have deferred counts (e.g. auto_recv_v<T>()); the counts are then gathered first. The other ranks must then pass auto_null_recv_v() to take part in that exchange, or null_buf_v if root’s counts are given explicitly. There is no non-root shorthand, because which of the two is right depends on root.

(2) Gathers into rbuf on rank 0; see the overload above.

Returns:

The send and receive buffer (owned if passed as rvalue, borrowed otherwise).

Defined in header <kamping/v2/collectives/gatherv.hpp>

template<send_buffer SBuf, recv_buffer RBuf, rank Root, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> scatter(SBuf &&sbuf, RBuf &&rbuf, Root root, Comm const &comm)
(1)
template<send_buffer SBuf, recv_buffer RBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> scatter(SBuf &&sbuf, RBuf &&rbuf, Comm const &comm)
(2)
template<recv_buffer RBuf, rank Root, convertible_to_mpi_handle<MPI_Comm> Comm>
RBuf scatter(RBuf &&rbuf, Root root, Comm const &comm)
(3)
template<recv_buffer RBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
RBuf scatter(RBuf &&rbuf, Comm const &comm)
(4)

(1) Distributes equal shares of sbuf on root to all ranks, receiving into rbuf (MPI_Scatter).

On root, the count of sbuf must be divisible by the communicator size, and rbuf may be inplace to keep root’s share in sbuf. The other ranks pass null_buf as sbuf or use the overloads without send buffer. rbuf may be a deferred receive buffer (e.g. views::resize).

(2) Scatters from rank 0; see the overload above.

(3) Non-root shorthand of scatter: receives this rank’s share from root into rbuf, without a send buffer.

(4) Non-root shorthand of scatter: receives this rank’s share from rank 0 into rbuf, without a send buffer.

Returns:

  • (1) The send and receive buffer (owned if passed as rvalue, borrowed otherwise).

  • (3, 4) The receive buffer.

Defined in header <kamping/v2/collectives/scatter.hpp>

template<send_buffer_v SBuf, recv_buffer RBuf, rank Root, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> scatterv(SBuf &&sbuf, RBuf &&rbuf, Root root, Comm const &comm)
(1)
template<send_buffer_v SBuf, recv_buffer RBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> scatterv(SBuf &&sbuf, RBuf &&rbuf, Comm const &comm)
(2)
template<recv_buffer RBuf, rank Root, convertible_to_mpi_handle<MPI_Comm> Comm>
RBuf scatterv(RBuf &&rbuf, Root root, Comm const &comm)
(3)
template<recv_buffer RBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
RBuf scatterv(RBuf &&rbuf, Comm const &comm)
(4)

(1) Distributes parts of the variadic buffer sbuf on root to all ranks, receiving into rbuf (MPI_Scatterv).

The other ranks pass null_buf_v as sbuf or use the overloads without send buffer. rbuf may be a deferred receive buffer (e.g. views::resize); root’s counts are then scattered first.

(2) Scatters from rank 0; see the overload above.

(3) Non-root shorthand of scatterv: receives this rank’s part from root into rbuf, without a send buffer.

(4) Non-root shorthand of scatterv: receives this rank’s part from rank 0 into rbuf, without a send buffer.

Returns:

  • (1) The send and receive buffer (owned if passed as rvalue, borrowed otherwise).

  • (3, 4) The receive buffer.

Defined in header <kamping/v2/collectives/scatterv.hpp>

template<send_buffer SBuf, recv_buffer RBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> allgather(SBuf &&sbuf, RBuf &&rbuf, Comm const &comm)

Gathers sbuf from all ranks into rbuf on every rank (MPI_Allgather).

rbuf may be a deferred receive buffer (e.g. views::resize), which is resized to comm_size * count(sbuf).

Returns:

The send and receive buffer (owned if passed as rvalue, borrowed otherwise).

Defined in header <kamping/v2/collectives/allgather.hpp>

template<send_buffer SBuf, recv_buffer_v RBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> allgatherv(SBuf &&sbuf, RBuf &&rbuf, Comm const &comm)

Gathers a variable amount of data from all ranks into the variadic buffer rbuf on every rank (MPI_Allgatherv).

If rbuf has deferred counts (e.g. auto_recv_v<T>()), the counts are exchanged first and rbuf is sized to fit.

Returns:

The send and receive buffer (owned if passed as rvalue, borrowed otherwise).

Defined in header <kamping/v2/collectives/allgatherv.hpp>

All-to-all

template<send_buffer SBuf, recv_buffer RBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> alltoall(SBuf &&sbuf, RBuf &&rbuf, Comm const &comm)

Sends an equal share of sbuf to every rank and receives one share from every rank into rbuf (MPI_Alltoall).

rbuf may be a deferred receive buffer (e.g. views::resize), which is resized to count(sbuf).

Returns:

The send and receive buffer (owned if passed as rvalue, borrowed otherwise).

Defined in header <kamping/v2/collectives/alltoall.hpp>

template<send_buffer_v SBuf, recv_buffer_v RBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> alltoallv(SBuf &&sbuf, RBuf &&rbuf, Comm const &comm)

Exchanges variable amounts of data between all ranks (MPI_Alltoallv).

sbuf is a variadic buffer, e.g. data | views::with_counts(c) | views::auto_displs() or nested | views::flatten_v(). If rbuf has deferred counts (e.g. auto_recv_v<T>()), the counts are exchanged first and rbuf is sized to fit.

Returns:

The send and receive buffer (owned if passed as rvalue, borrowed otherwise).

Defined in header <kamping/v2/collectives/alltoallv.hpp>

Reductions

template<send_buffer SBuf, recv_buffer RBuf, valid_op<SBuf, RBuf> Op, rank Root, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> reduce(SBuf &&sbuf, RBuf &&rbuf, Op const &op, Root root, Comm const &comm)
(1)
template<send_buffer SBuf, recv_buffer RBuf, valid_op<SBuf, RBuf> Op, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> reduce(SBuf &&sbuf, RBuf &&rbuf, Op const &op, Comm const &comm)
(2)
template<send_buffer SBuf, recv_buffer RBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
requires valid_op<std::plus<>, SBuf, RBuf>
result<SBuf, RBuf> reduce(SBuf &&sbuf, RBuf &&rbuf, Comm const &comm)
(3)
template<send_buffer SBuf, valid_op<SBuf, null_buf_t> Op, rank Root, convertible_to_mpi_handle<MPI_Comm> Comm>
SBuf reduce(SBuf &&sbuf, Op const &op, Root root, Comm const &comm)
(4)
template<send_buffer SBuf, valid_op<SBuf, null_buf_t> Op, convertible_to_mpi_handle<MPI_Comm> Comm>
SBuf reduce(SBuf &&sbuf, Op const &op, Comm const &comm)
(5)
template<send_buffer SBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
requires valid_op<std::plus<>, SBuf, null_buf_t>
SBuf reduce(SBuf &&sbuf, Comm const &comm)
(6)

(1) Reduces sbuf across all ranks with op and stores the result in rbuf on root (MPI_Reduce).

Pass inplace as sbuf on root to reduce rbuf in place. On root, rbuf may be a deferred receive buffer (e.g. views::resize); on the other ranks it is ignored, so pass null_buf or use the overloads without receive buffer.

(2) Reduces with op to rank 0; see the overload above.

(3) Sums to rank 0; see the overload above.

(4) Non-root shorthand of reduce: contributes sbuf to the reduction with op on root, without a receive buffer.

(5) Non-root shorthand of reduce: contributes sbuf to the reduction with op on rank 0.

(6) Non-root shorthand of reduce: contributes sbuf to the sum on rank 0.

Parameters:

  • op – Reduction operation: an MPI_Op, a standard functor such as std::plus<>, or the result of make_op.

Returns:

  • (1) The send and receive buffer (owned if passed as rvalue, borrowed otherwise).

  • (4-6) The send buffer.

Defined in header <kamping/v2/collectives/reduce.hpp>

template<send_buffer SBuf, recv_buffer RBuf, valid_op<SBuf, RBuf> Op, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> allreduce(SBuf &&sbuf, RBuf &&rbuf, Op const &op, Comm const &comm)
(1)
template<send_buffer SBuf, recv_buffer RBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
requires valid_op<std::plus<>, SBuf, RBuf>
result<SBuf, RBuf> allreduce(SBuf &&sbuf, RBuf &&rbuf, Comm const &comm)
(2)

(1) Reduces sbuf across all ranks with op and stores the result in rbuf on every rank (MPI_Allreduce).

Pass inplace as sbuf to use rbuf for both input and output. rbuf may be a deferred receive buffer (e.g. views::resize).

(2) Like the overload above, with std::plus<> as operation.

Parameters:

  • op – Reduction operation: an MPI_Op, a standard functor such as std::plus<>, or the result of make_op.

Returns:

The send and receive buffer (owned if passed as rvalue, borrowed otherwise).

Defined in header <kamping/v2/collectives/allreduce.hpp>

template<send_buffer SBuf, recv_buffer RBuf, valid_op<SBuf, RBuf> Op, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> scan(SBuf &&sbuf, RBuf &&rbuf, Op const &op, Comm const &comm)
(1)
template<send_buffer SBuf, recv_buffer RBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
requires valid_op<std::plus<>, SBuf, RBuf>
result<SBuf, RBuf> scan(SBuf &&sbuf, RBuf &&rbuf, Comm const &comm)
(2)

(1) Computes the inclusive prefix reduction of sbuf over the ranks with op into rbuf (MPI_Scan).

Pass inplace as sbuf to use rbuf for both input and output. rbuf may be a deferred receive buffer (e.g. views::resize).

(2) Like the overload above, with std::plus<> as operation.

Parameters:

  • op – Reduction operation: an MPI_Op, a standard functor such as std::plus<>, or the result of make_op.

Returns:

The send and receive buffer (owned if passed as rvalue, borrowed otherwise).

Defined in header <kamping/v2/collectives/scan.hpp>

template<send_buffer SBuf, recv_buffer RBuf, valid_op<SBuf, RBuf> Op, convertible_to_mpi_handle<MPI_Comm> Comm>
result<SBuf, RBuf> exscan(SBuf &&sbuf, RBuf &&rbuf, Op const &op, Comm const &comm)
(1)
template<send_buffer SBuf, recv_buffer RBuf, convertible_to_mpi_handle<MPI_Comm> Comm>
requires valid_op<std::plus<>, SBuf, RBuf>
result<SBuf, RBuf> exscan(SBuf &&sbuf, RBuf &&rbuf, Comm const &comm)
(2)

(1) Computes the exclusive prefix reduction of sbuf over the ranks with op into rbuf (MPI_Exscan).

Pass inplace as sbuf to use rbuf for both input and output. rbuf may be a deferred receive buffer (e.g. views::resize).

The content of rbuf on rank 0 is undefined after the call, as in MPI.

(2) Like the overload above, with std::plus<> as operation.

Parameters:

  • op – Reduction operation: an MPI_Op, a standard functor such as std::plus<>, or the result of make_op.

Returns:

The send and receive buffer (owned if passed as rvalue, borrowed otherwise).

Defined in header <kamping/v2/collectives/exscan.hpp>

Reduction operations

An operation is an MPI_Op, a standard function object such as std::plus<> or std::multiplies<> (mapped to the builtin operation where one exists), or any callable wrapped with make_op.

template<typename Op, typename SBuf, typename RBuf>
concept valid_op
template<typename Op, typename SBuf, typename RBuf>
concept valid_op =
    requires(Op const& op, SBuf const& sbuf, RBuf const& rbuf) { mpi::experimental::as_mpi_op(op, sbuf, rbuf); }

Op can be used as reduction operation on buffers of types SBuf and RBuf, i.e. as_mpi_op is defined.

Defined in header <kamping/v2/ops.hpp>

template<commutativity_tag Commutativity = non_commutative_tag>
auto make_op(detail::op_callback_t callback, Commutativity = {})
(1)
template<typename T, typename Op, commutativity_tag Commutativity = non_commutative_tag>
requires std::is_default_constructible_v<std::remove_cvref_t<Op>> && std::is_invocable_r_v<T, std::remove_cvref_t<Op>, T const&, T const&>
auto make_op(Op &&op, Commutativity = {})
(2)

(1) Creates an MPI_Op from a raw MPI reduction callback.

Use this when you already have a function with MPI’s callback signature void(void* invec, void* inoutvec, int* len, MPI_Datatype* datatype). No element type is required because the callback is already type-erased.

(2) Creates an MPI_Op from a default-constructible binary functor.

Handles stateless functors and captureless lambdas (which are default-constructible in C++20). The element-wise application is provided by kamping::types::ScopedFunctorOp; the functor must be callable as T(T const&, T const&).

Stateful (capturing) lambdas are intentionally not supported here — they are neither default-constructible nor convertible to a raw callback. Wrap the required state in a default-constructible functor, or create the callback yourself and pass it to the overload above.

Template parameters:

  • Commutativity – Commutativity tag type; defaults to non-commutative.

  • T – The element type the operation is applied to. Must be specified explicitly, e.g. make_op<int>(...).

  • Op – Functor type (deduced).

Parameters:

  • callback – Non-null MPI callback function pointer.

  • op – The binary operation.

Returns:

  • (1) A kamping::types::ScopedCallbackOp owning the freshly created MPI_Op.

  • (2) A kamping::types::ScopedFunctorOp owning the freshly created MPI_Op.

Defined in header <kamping/v2/ops.hpp>

commutative_tag commutative

Tag value marking an operation as commutative (re-exported from kamping::ops).

Defined in header <kamping/v2/ops.hpp>

non_commutative_tag non_commutative

Tag value marking an operation as non-commutative (re-exported from kamping::ops).

Defined in header <kamping/v2/ops.hpp>

Special buffers

inplace_t inplace

MPI_IN_PLACE: passed as the send buffer of a collective, the receive buffer holds the input and receives the result.

The receive buffer must already have the right size; it cannot be resized for an in-place operation.

int sum = local;
v2::allreduce(v2::inplace, views::ref_single(sum), comm);

Defined in header <kamping/v2/sentinels.hpp>

null_buf_t null_buf

An empty buffer: for the buffer of a rooted collective that only the root uses, or as an empty message.

The non-root shorthands of gather, scatter and reduce pass it for you.

v2::gather(local_data, v2::null_buf, root, comm);  // on a non-root rank
v2::send(v2::null_buf, dest, tag, comm);           // a message without data

Defined in header <kamping/v2/sentinels.hpp>

null_buf_v_t null_buf_v

An empty variable-size buffer, for the buffer of gatherv or scatterv that only the root uses, if the root provides its counts. If the root’s receive buffer obtains its counts from the others (e.g. auto_recv_v), non-root ranks pass auto_null_recv_v() instead.

v2::gatherv(local_data, v2::null_buf_v, root, comm);  // on a non-root rank

Defined in header <kamping/v2/sentinels.hpp>

auto_null_recv_v_t auto_null_recv_v() noexcept

An empty receive buffer for gatherv on non-root ranks, if the root’s receive buffer obtains its counts from the others, e.g. auto_recv_v: the non-root ranks then still take part in the count exchange.

if (comm.rank() == root) {
    auto [_, all] = v2::gatherv(local_data, v2::auto_recv_v<int>(), root, comm);
} else {
    v2::gatherv(local_data, v2::auto_null_recv_v(), root, comm);
}

Pass the result directly, as here; it cannot be used through a const variable.

Defined in header <kamping/v2/sentinels.hpp>

bottom_t bottom

A buffer at MPI_BOTTOM, for derived datatypes that contain absolute addresses.

Not a complete buffer on its own: add the datatype and the count with views.

v2::send(v2::bottom | views::with_type(absolute_type) | views::with_count(1), dest, comm);

Defined in header <kamping/v2/sentinels.hpp>