Distributions and redistribution

class distribution

A monotone partition of a globally-indexed contiguous range across a communicator.

Defined in header <dstl/distribution.hpp>

template<kamping::v2::convertible_to_mpi_handle<MPI_Comm> Comm = MPI_Comm>
explicit distribution(std::size_t local_size, Comm const &comm = MPI_COMM_WORLD)
(1)

template<std::ranges::forward_range Range, kamping::v2::convertible_to_mpi_handle<MPI_Comm> Comm = MPI_Comm>
requires std::ranges::sized_range<Range> && std::convertible_to<std::ranges::range_value_t<Range>, std::size_t>
distribution(Range &&offsets, Comm const &comm = MPI_COMM_WORLD)
(2)

(1) Builds the distribution by allgathering local_size from every rank (one collective).

(2) Builds the distribution from an explicit, already globally-agreed offset table (size comm.size() + 1, monotonically increasing). No communication is performed.

int get_owner(std::size_t idx) const

Rank owning global index idx. O(1) when local sizes are uniform, else binary search.

auto counts() const

Lazy view of each rank’s local size, in rank order.

auto displs() const

Lazy view of each rank’s displacement (== index_range_begin(rank)), in rank order. Monotonically increasing by construction — pairs with counts() to back a variadic buffer.

std::size_t index_range_begin(int rank) const

First global index owned by rank.

std::size_t index_range_end(int rank) const

One past the last global index owned by rank.

bool is_local(std::size_t idx, int rank) const

Whether global index idx is owned by rank.

std::size_t global_to_local(std::size_t global_idx, int rank) const

Global -> local index conversion for an index owned by rank.

std::size_t local_to_global(std::size_t local_idx, int rank) const

Local -> global index conversion for rank.

std::size_t global_size() const

Total element count across all ranks.

std::size_t local_size(int rank) const

Element count owned by rank.

int size() const

Number of ranks the distribution spans.

std::ranges::iota_view<std::size_t, std::size_t> local_indices(int rank) const

Local indices [0, local_size(rank)).

std::ranges::iota_view<std::size_t, std::size_t> global_indices(int rank) const

Global indices [index_range_begin(rank), index_range_end(rank)).

template<kamping::v2::send_buffer SBuf, kamping::v2::recv_buffer RBuf, kamping::v2::convertible_to_mpi_handle<MPI_Comm> Comm = MPI_Comm, kamping::v2::tag Tag = int>
void redistribute(SBuf &&sbuf, RBuf &&rbuf, Comm const &comm = MPI_COMM_WORLD, Tag tag = 0)

Redistributes data between ranks according to globally agreed send/recv windows.

Each rank owns a contiguous slice of a conceptual global send buffer (sbuf) and wants to receive a contiguous slice of a conceptual global recv buffer (rbuf). The two global arrays are the same data in flight: what rank i sends at global offset [s_i, s_i+k) lands in whoever’s recv window covers that range.

Algorithm:

1. Allgather all per-rank send/recv counts to derive global displacement arrays.

2. For each rank, binary-search for the set of remote ranks whose send (or recv) window overlaps this rank’s recv (or send) window, and issue one irecv/isend per overlap.

3. Wait for all transfers (kamping::v2::waitall).

No additional data copies beyond the MPI transfers are performed.

Defined in header <dstl/redistribute.hpp>

template<kamping::v2::convertible_to_mpi_handle<MPI_Comm> Comm, std::invocable<kamping::v2::comm_view> Func>
void with_subset(Comm const &comm, bool cond, Func &&f)

Calls f with a communicator of the processes for which cond is true (collective).

If cond holds on all processes, f is called with comm itself. Otherwise a subcommunicator of the processes with cond == true is split off, and only those call f with it.

Defined in header <dstl/with_subset.hpp>