gko::experimental::mpi::
CollectiveCommunicator#

Abstract interface for the collective communication patterns used by the distributed SpMV. A CollectiveCommunicator wraps a base communicator (the concrete implementations also take an index_map that defines the pattern) and exposes a single nonblocking variable all-to-all primitive that ships locally-owned values to the ranks that need them as ghost entries.

The distributed Matrix gathers, through its RowGatherer, the halo values that the non-local SpMV consumes; the RowGatherer holds a CollectiveCommunicator and invokes i_all_to_all_v. The exchange completes before the non-local SpMV starts. The default DenseCommunicator is blocking, so on the host executor the exchange has already completed when the local SpMV starts; on device executors the local kernel is queued ahead of the exchange and the two overlap. Two concrete implementations ship with Ginkgo: DenseCommunicator and NeighborhoodCommunicator; applications can supply their own by deriving from this interface.

class CollectiveCommunicator#

Interface for a collective communicator.

A collective communicator only provides routines for collective communications. At the moment this is restricted to the variable all-to-all.

Subclassed by

Public Types

using index_map_ptr = std::variant<const distributed::index_map<int32, int32>*, const distributed::index_map<int32, int64>*, const distributed::index_map<int64, int64>*>#

All allowed index_map types (as const *)

Public Functions

template<typename SendType, typename RecvType>
request i_all_to_all_v(
std::shared_ptr<const Executor> exec,
const SendType *send_buffer,
RecvType *recv_buffer,
) const#

Non-blocking all-to-all communication.

The send_buffer must have allocated at least get_send_size number of elements, and the recv_buffer must have allocated at least get_recv_size number of elements.

Template Parameters:

  • SendType – the type of the elements to send

  • RecvType – the type of the elements to receive

Parameters:

  • exec – the executor for the communication

  • send_buffer – the send buffer

  • recv_buffer – the receive buffer

Returns:

a request handle

virtual std::unique_ptr<CollectiveCommunicator> create_with_same_type(
communicator base,
index_map_ptr imap,
) const = 0#

Creates a new CollectiveCommunicator with the same dynamic type.

Parameters:

  • base – The base communicator

  • imap – The index_map that defines the communication pattern

Returns:

a CollectiveCommunicator with the same dynamic type

virtual std::unique_ptr<CollectiveCommunicator> create_inverse(
) const = 0#

Creates a CollectiveCommunicator with the inverse communication pattern than this object.

Returns:

a CollectiveCommunicator with the inverse communication pattern.

virtual comm_index_type get_recv_size() const = 0#

Get the number of elements received by this process within this communication pattern.

Returns:

number of received elements.

virtual comm_index_type get_send_size() const = 0#

Get the number of elements sent by this process within this communication pattern.

Returns:

number of sent elements.