Sparse all-to-all

template<sparse_alltoall_send_buffer SBuf, std::invocable<kamping::v2::probe_result> OnMessage, kamping::v2::tag Tag = int, kamping::v2::convertible_to_mpi_handle<MPI_Comm> Comm = MPI_Comm>
void sparse_alltoall(sparse_alltoall_algorithm::nbx_t, SBuf &&sbuf, OnMessage &&on_message, Tag tag, Comm const &comm = MPI_COMM_WORLD)
(1)
template<sparse_alltoall_send_buffer SBuf, std::invocable<kamping::v2::probe_result> OnMessage, kamping::v2::tag Tag = int, kamping::v2::convertible_to_mpi_handle<MPI_Comm> Comm = MPI_Comm>
void sparse_alltoall(sparse_alltoall_algorithm::pcx_t, SBuf &&sbuf, OnMessage &&on_message, Tag tag, Comm const &comm = MPI_COMM_WORLD)
(2)
template<sparse_alltoall_send_buffer SBuf, std::invocable<kamping::v2::probe_result> OnMessage, kamping::v2::tag Tag = int, kamping::v2::convertible_to_mpi_handle<MPI_Comm> Comm = MPI_Comm>
void sparse_alltoall(sparse_alltoall_algorithm::rsx_t, SBuf &&sbuf, OnMessage &&on_message, Tag tag, Comm const &comm, rsx_window_cache &windows)
(3)
template<sparse_alltoall_send_buffer SBuf, std::invocable<kamping::v2::probe_result> OnMessage, kamping::v2::tag Tag = int, kamping::v2::convertible_to_mpi_handle<MPI_Comm> Comm = MPI_Comm>
void sparse_alltoall(sparse_alltoall_algorithm::rsx_t, SBuf &&sbuf, OnMessage &&on_message, Tag tag, Comm const &comm = MPI_COMM_WORLD)
(4)
template<sparse_alltoall_send_buffer SBuf, std::invocable<kamping::v2::probe_result> OnMessage, kamping::v2::tag Tag = int, kamping::v2::convertible_to_mpi_handle<MPI_Comm> Comm = MPI_Comm>
void sparse_alltoall(SBuf &&sbuf, OnMessage &&on_message, Tag tag, Comm const &comm = MPI_COMM_WORLD)
(5)
template<is_sparse_alltoall_algorithm Algorithm, sparse_alltoall_send_buffer SBuf, sparse_alltoall_recv_buffer RBuf, typename ...Args>
std::size_t sparse_alltoall(Algorithm algorithm, SBuf &&sbuf, RBuf &&rbuf, Args&&... args)
(6)
template<sparse_alltoall_send_buffer SBuf, sparse_alltoall_recv_buffer RBuf, typename ...Args>
std::size_t sparse_alltoall(SBuf &&sbuf, RBuf &&rbuf, Args&&... args)
(7)

(1) Sparse personalized all-to-all with the NBX protocol.

Sends every non-empty message in sbuf and calls on_message with a probe_result for every message received. Only processes that exchange messages communicate point-to-point; completion is detected with a nonblocking barrier, so there is no collective on the message counts.

Input (see sparse_alltoall_send_buffer). sbuf is either a range of (destination, message) pairs, in which a destination may appear more than once, or a range of messages indexed by destination (at most one entry per process). Each message is a buffer (e.g. a contiguous range), sent in place; empty messages are not sent, so only processes that exchange data communicate. The MPI datatype of the elements is the payload datatype of sbuf: attach it with views::with_value_type(dt) or views::with_value_pool(pool), or rely on the builtin datatype of the element type. The dense form takes O(p) local time to scan; the sparse form only touches the messages actually sent.

Callback. on_message must start receiving the message (kamping::v2::mrecv / imrecv) before it returns, and must not block on anything outside MPI that depends on other processes. Messages from the same source arrive in the order they were sent.

Tags. All messages are sent and probed with tag on comm. Consecutive calls on the same communicator must use different tags, or be separated by a synchronization; and no other messages may use tag on comm during the call.

(2) Sparse personalized all-to-all with the PCX protocol.

Like the NBX overload, but each process learns the number of messages addressed to it from a reduce-scatter of per-destination message counts (O(p) memory and communication per process), then probes for exactly that many.

(3) Sparse personalized all-to-all with the RSX protocol, using cached counter windows.

Like the NBX overload, but each sender adds to a counter window on each of its destinations, which then probe for exactly as many messages as were counted. The counter window for comm is taken from windows, and allocated there on the first exchange on comm.

(4) Sparse personalized all-to-all with the RSX protocol.

Like the cached RSX overload, but allocates and frees a counter window on every call (both collective). Prefer the cached overload for repeated exchanges on the same communicator.

(5) Sparse personalized all-to-all with the default protocol (NBX); see the NBX overload.

(6) Sparse personalized all-to-all receiving all messages into rbuf, one after the other in arrival order.

Like the callback overloads (with any protocol), but each message is received at the next free position of rbuf, in the datatype of rbuf. args are the arguments that follow the callback in the callback overloads: the tag, optionally the communicator and, for RSX, optionally the cached counter windows. A flat variadic buffer becomes an input via views::chunked: 

auto received = dstl::sparse_alltoall(
    nbx, data | views::with_counts(counts) | views::auto_displs() | views::chunked, out | views::resize, tag, comm);

  • Presized rbuf: must have room for all received elements (asserted). Each message is received with imrecv as soon as it is matched, so receives overlap; all are waited for before returning. Elements after the received ones are left untouched.

  • views::resize: each message is received with a blocking mrecv after growing rbuf to fit it (growing keeps the elements already received, see resize_for_receive), so no receive is in flight when the buffer moves. Finally rbuf is sized to exactly the received elements.

The order is the arrival order; messages from the same source keep their send order. To know which data came from where, use the callback overloads.

(7) The recv-buffer overload with the default protocol (NBX).

Throws:

  • mpi_error – if an MPI call fails.

Returns:

The number of elements received.

Defined in header <dstl/sparse_alltoall.hpp>

Protocols

nbx_t nbx

Nonblocking consensus: synchronous sends, probing until a nonblocking barrier completes. No collective on the message counts; the default.

Defined in header <dstl/sparse_alltoall.hpp>

pcx_t pcx

Personalized census: a reduce-scatter of per-destination message counts tells each process how many messages to probe for.

Defined in header <dstl/sparse_alltoall.hpp>

rsx_t rsx

Remote summing: each sender adds to a counter window on its destinations, which then know how many messages to probe for.

Defined in header <dstl/sparse_alltoall.hpp>

using rsx_window_cache = kamping::v2::cache<kamping::v2::attribute::comm, detail::rsx_counter>

Reusable RSX counter windows, one per communicator.

Pass one to the RSX overload of sparse_alltoall to allocate the counter window only on the first exchange on each communicator, instead of on every exchange. Destroying the cache frees the windows (collective).

Defined in header <dstl/sparse_alltoall.hpp>

Buffers

template<typename SBuf>
concept sparse_alltoall_send_buffer
template<typename SBuf>
concept sparse_alltoall_send_buffer =
    (kamping::v2::nested_send_buffer<SBuf> || kamping::v2::sparse_nested_send_buffer<SBuf>)
    && kamping::v2::buffer_message<kamping::v2::nested_message_t<SBuf>>

Input of sparse_alltoall, in one of two forms:

Messages are sent in place, so they must be accessible through the buffer protocol (kamping::v2::buffer_message: a pointer and a count). Contiguous ranges qualify, as do custom buffers adapted via buffer_traits or mpi_ptr() / mpi_count(). Messages need no datatype of their own: all messages share the payload datatype of sbuf (see views::with_value_type).

Defined in header <dstl/sparse_alltoall.hpp>

template<typename RBuf>
concept sparse_alltoall_recv_buffer
template<typename RBuf>
concept sparse_alltoall_recv_buffer =
    kamping::v2::recv_buffer<RBuf> && !std::invocable<RBuf, kamping::v2::probe_result>

A recv buffer for the recv-buffer overloads of sparse_alltoall: any recv_buffer, presized or deferred (e.g. views::resize), that cannot be mistaken for a callback.

Defined in header <dstl/sparse_alltoall.hpp>

template<typename SBuf>
concept messages_outlive_iteration
template<typename SBuf>
concept messages_outlive_iteration =
    std::is_lvalue_reference_v<std::ranges::range_reference_t<std::remove_reference_t<SBuf> const>>
    || std::is_lvalue_reference_v<kamping::v2::nested_message_t<SBuf>>
    || kamping::v2::borrowed_buffer<kamping::v2::nested_message_t<SBuf>>

Whether the messages of SBuf stay valid while they are being sent.

Messages are sent nonblocking while the input is iterated, so their data must outlive the loop iteration that posts them. That holds when the input yields references into storage that outlives the call (e.g. a std::vector of pairs), or when the messages do not own their data (e.g. std::span). It fails for messages that are produced by value and own their data, e.g. std::views::transform returning std::vectors: each one is destroyed at the end of its iteration while its send may still be in flight.

Defined in header <dstl/sparse_alltoall.hpp>