gko::experimental::mpi#

Thin C++ wrappers around the MPI runtime. The gko::experimental::distributed types are built on top of these primitives — a Matrix or Vector carries an mpi::communicator, nonblocking operations return an mpi::request, and the collective-communicator hierarchy abstracts the actual data exchange patterns used by the distributed SpMV.

Most application code only ever touches environment (RAII wrapper around MPI_Init / MPI_Finalize) and communicator (lifetime holder for MPI_Comm). The remaining types are the lower-level building blocks that the distributed kernels use internally.

Runtime#

  • environment — RAII wrapper around MPI_Init and MPI_Finalize; construct once at the top of main.

  • communicator — owning C++ wrapper around MPI_Comm. Carries the rank and size, and is the value that every distributed Ginkgo type holds onto.

Asynchronous operations#

  • request — move-only handle representing an outstanding nonblocking MPI operation; analogous to MPI_Request.

  • status — light wrapper around MPI_Status, returned by completion calls.

Type system#

  • contiguous_type — RAII wrapper that builds and frees a contiguous MPI_Datatype for sending fixed-size blocks of an element type.

Collective communicators#

The distributed SpMV’s halo exchange is abstracted as a CollectiveCommunicator. Ginkgo ships two concrete implementations; applications can plug in their own.

  • CollectiveCommunicator — abstract base: gather/scatter on a RowGatherer-style index pattern.

  • DenseCommunicator — implementation using MPI_Alltoallv-style dense collectives. Default everywhere; works on any communicator topology.

  • NeighborhoodCommunicator — implementation using MPI-3 neighborhood collectives over a graph-topology communicator. Faster on large rank counts where each rank only talks to a small set of neighbours.