gko::experimental::mpi::
CollectiveCommunicator#

Abstract interface for the collective communication patterns used by the distributed SpMV. A CollectiveCommunicator wraps a base communicator; the concrete implementations also take an index_map that defines the communication pattern. It exposes a single nonblocking variable all-to-all primitive that ships locally-owned values to the ranks that need them as ghost entries.

The distributed Matrix gathers, through its RowGatherer, the halo values that the off-diagonal SpMV consumes; the RowGatherer holds a CollectiveCommunicator and invokes i_all_to_all_v. The exchange overlaps the diagonal SpMV (on host executors only when the exchange is non-blocking; see the Open MPI < 4.1 fallback of DenseCommunicator) and completes before the off-diagonal SpMV starts. Two concrete implementations ship with Ginkgo: DenseCommunicator and NeighborhoodCommunicator; applications can supply their own by deriving from this interface.

class CollectiveCommunicator#

Interface for a collective communicator.

A collective communicator only provides routines for collective communications. At the moment this is restricted to the variable all-to-all.

Subclassed by

Public Types

using index_map_ptr = std::variant<const distributed::index_map<int32, int32>*, const distributed::index_map<int32, int64>*, const distributed::index_map<int64, int64>*>#

All allowed index_map types (as const *)

Public Functions

template<typename SendType, typename RecvType>
request i_all_to_all_v(
std::shared_ptr<const Executor> exec,
const SendType *send_buffer,
RecvType *recv_buffer,
) const#

Non-blocking all-to-all communication.

The send_buffer must have allocated at least get_send_size number of elements, and the recv_buffer must have allocated at least get_recv_size number of elements.

Template Parameters:

  • SendType – the type of the elements to send

  • RecvType – the type of the elements to receive

Parameters:

  • exec – the executor for the communication

  • send_buffer – the send buffer

  • recv_buffer – the receive buffer

Returns:

a request handle

virtual std::unique_ptr<CollectiveCommunicator> create_with_same_type(
communicator base,
index_map_ptr imap,
) const = 0#

Creates a new CollectiveCommunicator with the same dynamic type.

Parameters:

  • base – The base communicator

  • imap – The index_map that defines the communication pattern

Returns:

a CollectiveCommunicator with the same dynamic type

virtual std::unique_ptr<CollectiveCommunicator> create_inverse(
) const = 0#

Creates a CollectiveCommunicator with the inverse communication pattern than this object.

Returns:

a CollectiveCommunicator with the inverse communication pattern.

virtual std::pair<std::unique_ptr<CollectiveCommunicator>, std::vector<comm_index_type>> resize(
std::shared_ptr<const Executor> exec,
const std::vector<comm_index_type> &send_factors,
) const = 0#

Create new communicator that sends and receives a different number of elements to/from each rank in its current pattern.

The number of elements to send is deduces from the send_factors. If this communicator is used to send 2 elements to rank 0, and 3 elements to rank 1, then send_factors needs to have size 5, which is equal to get_send_size(). An send_factors = { 4, 2, 1, 3, 2} means that instead of the first element to send to rank 0, four elements are sent, instead of the second element to send to rank 0, two elements are sent, and so on. A visual representation of the transformation is: 

[ 1 1 | 1 1 1 ] -> [ 4 2 | 1 3 2] -> [ 1 1 1 1 1 1 | 1 1 1 1 1 1 ]
The communication pattern will not change. Even if send_factors is zero for all elements of one rank, the rank will stay in the new pattern.

Note

This is a collective call.

Parameters:

  • exec – the executor to use for the new communicator

  • send_factors – the new number of elements to send.

Returns:

The new communicator and the recv_factors implied by the send_factors

virtual comm_index_type get_recv_size() const = 0#

Get the number of elements received by this process within this communication pattern.

Returns:

number of received elements.

virtual comm_index_type get_send_size() const = 0#

Get the number of elements sent by this process within this communication pattern.

Returns:

number of sent elements.