gko::experimental::distributed::Matrix#

Distributed sparse matrix. Stores the rank’s owned rows as two CSR blocks: a local block (get_local_matrix()) holding the columns the rank also owns, and a non-local block (get_non_local_matrix()) holding the remote columns (compressed so only the columns actually referenced are stored). apply gathers the remote vector entries that the non-local SpMV needs through a RowGatherer. On device executors the local SpMV is queued before the exchange is issued and runs while it is in flight; on the host executor the default, blocking DenseCommunicator completes the exchange before the local SpMV starts.

template<typename ValueType = default_precision, typename LocalIndexType = int32, typename GlobalIndexType = int64>
class Matrix #

Inherits from

  • public gko::EnableLinOp<Matrix<default_precision, int32, int64>>

  • public gko::ConvertibleTo<Matrix<next_precision<default_precision>, int32, int64>>

  • public gko::ConvertibleTo<Matrix<next_precision<default_precision, 2>, int32, int64>>

  • public gko::ConvertibleTo<Matrix<next_precision<default_precision, 3>, int32, int64>>

  • public gko::experimental::distributed::DistributedBase

The Matrix class defines a (MPI-)distributed matrix.

The matrix is stored in a row-wise distributed format. Each process owns a specific set of rows, where the assignment of rows is defined by a row Partition. The following depicts the distribution of global rows according to their assigned part-id (which will usually be the owning process id): 

Part-Id  Global Rows                   Part-Id  Local Rows
0        | .. 1  2  .. .. .. |         0        | .. 1  2  .. .. .. |
1        | 3  4  .. .. .. .. |                  | 13 .. .. .. 14 .. |
2        | .. 5  6  ..  7 .. |  ---->  1        | 3  4  .. .. .. .. |
2        | .. .. .. 8  ..  9 |  ---->           | .. .. .. 10 11 12 |
1        | .. .. .. 10 11 12 |         2        | .. 5  6  ..  7 .. |
0        | 13 .. .. .. 14 .. |                  | .. .. .. 8  ..  9 |
The local rows are further split into two matrices on each process. One matrix, called local, contains only entries from columns that are also owned by the process, while the other one, called non_local, contains entries from columns that are not owned by the process. The non-local matrix is stored in a compressed format, where empty columns are discarded and the remaining columns are renumbered. This splitting is depicted in the following: 
Part-Id  Global                            Local      Non-Local
0        | .. 1  ! 2  .. ! .. .. |         | .. 1  |  | 2  |
0        | 3  4  ! .. .. ! .. .. |         | 3  4  |  | .. |
         |-----------------------|
1        | .. 5  ! 6  .. ! 7  .. |  ---->  | 6  .. |  | 5  7  .. |
1        | .. .. ! .. 8  ! ..  9 |  ---->  | 8  .. |  | .. .. 9  |
         |-----------------------|
2        | .. .. ! .. 10 ! 11 12 |         | 11 12 |  | .. 10 |
2        | 13 .. ! .. .. ! 14 .. |         | 14 .. |  | 13 .. |
This uses the same ownership of the columns as for the rows. Additionally, the ownership of the columns may be explicitly defined with an second column partition. If that is not provided, the same row partition will be used for the columns. Using a column partition also allows to create non-square matrices, like the one below: 
Part-Id  Global                  Local      Non-Local
P_R/P_C    2  2  0  1
0        | .. 1  2  .. |         | 2  |     | 1  .. |
0        | 3  4  .. .. |         | .. |     | 3  4  |
         |-------------|
1        | .. 5  6  .. |  ---->  | .. |     | 6  5  |
1        | .. .. .. 8  |  ---->  | 8  |     | .. .. |
         |-------------|
2        | .. .. .. 10 |         | .. .. |  | 10 |
2        | 13 .. .. .. |         | 13 .. |  | .. |
Here P_R denotes the row partition and P_C denotes the column partition.

The Matrix should be filled using the read_distributed method, e.g. 

auto part = Partition<...>::build_from_mapping(...);
auto mat = Matrix<...>::create(exec, comm);
mat->read_distributed(matrix_data, part);
or if different partitions for the rows and columns are used: 
auto row_part = Partition<...>::build_from_mapping(...);
auto col_part = Partition<...>::build_from_mapping(...);
auto mat = Matrix<...>::create(exec, comm);
mat->read_distributed(matrix_data, row_part, col_part);
This will set the dimensions of the global and local matrices automatically by deducing the sizes from the partitions.

By default the Matrix type uses Csr for both stored matrices. It is possible to explicitly change the datatype for the stored matrices, with the constraint that the new type should implement the LinOp and ReadableFromMatrixData interface. The type can be set by: 

auto mat = Matrix<ValueType, LocalIndexType[, ...]>::create(
  exec, comm,
  Ell<ValueType, LocalIndexType>::create(exec).get(),
  Coo<ValueType, LocalIndexType>::create(exec).get());
Alternatively, the helper function with_matrix_type can be used: 
auto mat = Matrix<ValueType, LocalIndexType>::create(
  exec, comm,
with_matrix_type<Ell>(),
with_matrix_type<Coo>());
The Matrix LinOp supports the following operations: 
experimental::distributed::Matrix *A;       // distributed matrix
experimental::distributed::Vector *b, *x;   // distributed multi-vectors
matrix::Dense *alpha, *beta;  // scalars of dimension 1x1

// Applying to distributed multi-vectors computes an SpMV/SpMM product
A->apply(b, x)              // x = A*b
A->apply(alpha, b, beta, x) // x = alpha*A*b + beta*x

See also

with_matrix_type

Template Parameters:

  • ValueType – The underlying value type.

  • LocalIndexType – The index type used by the local matrices.

  • GlobalIndexType – The type for global indices.

Public Functions

void read_distributed(
const device_matrix_data<value_type, global_index_type> &data,
std::shared_ptr<const Partition<local_index_type, global_index_type>> partition,
assembly_mode assembly_type = assembly_mode::local_only,
)#

Reads a square matrix from the device_matrix_data structure and a global partition.

The global size of the final matrix is inferred from the size of the partition. Both the number of rows and columns of the device_matrix_data are ignored.

Note

The matrix data can contain entries for rows other than those owned by the process. Entries for those rows are discarded.

Parameters:

  • data – The device_matrix_data structure.

  • partition – The global row and column partition.

  • x – The mode of assembly.

Returns:

the index_map induced by the partitions and the matrix structure

void read_distributed(
const matrix_data<value_type, global_index_type> &data,
std::shared_ptr<const Partition<local_index_type, global_index_type>> partition,
assembly_mode assembly_type = assembly_mode::local_only,
)#

Reads a square matrix from the matrix_data structure and a global partition.

See also

read_distributed

Note

For efficiency it is advised to use the device_matrix_data overload.

void read_distributed(
const device_matrix_data<value_type, global_index_type> &data,
std::shared_ptr<const Partition<local_index_type, global_index_type>> row_partition,
std::shared_ptr<const Partition<local_index_type, global_index_type>> col_partition,
assembly_mode assembly_type = assembly_mode::local_only,
)#

Reads a matrix from the device_matrix_data structure, a global row partition, and a global column partition.

The global size of the final matrix is inferred from the size of the row partition and the size of the column partition. Both the number of rows and columns of the device_matrix_data are ignored.

Note

The matrix data can contain entries for rows other than those owned by the process. Entries for those rows are discarded.

Parameters:

  • data – The device_matrix_data structure.

  • row_partition – The global row partition.

  • col_partition – The global col partition.

  • assembly_type – The mode of assembly.

Returns:

the index_map induced by the partitions and the matrix structure

void read_distributed(
const matrix_data<value_type, global_index_type> &data,
std::shared_ptr<const Partition<local_index_type, global_index_type>> row_partition,
std::shared_ptr<const Partition<local_index_type, global_index_type>> col_partition,
assembly_mode assembly_type = assembly_mode::local_only,
)#

Reads a matrix from the matrix_data structure, a global row partition, and a global column partition.

See also

read_distributed

Note

For efficiency it is advised to use the device_matrix_data overload.

inline std::shared_ptr<const LinOp> get_local_matrix() const#

Get read access to the stored local matrix.

Returns:

Shared pointer to the stored local matrix

inline std::shared_ptr<const LinOp> get_non_local_matrix() const#

Get read access to the stored non-local matrix.

Returns:

Shared pointer to the stored non-local matrix

Matrix(const Matrix &other)#

Copy constructs a Matrix.

Parameters:

other – Matrix to copy from.

Matrix(Matrix &&other) noexcept#

Move constructs a Matrix.

Parameters:

other – Matrix to move from.

Matrix &operator=(const Matrix &other)#

Copy assigns a Matrix.

Parameters:

other – Matrix to copy from, has to have a communicator of the same size as this.

Returns:

this.

Matrix &operator=(Matrix &&other)#

Move assigns a Matrix.

Parameters:

other – Matrix to move from, has to have a communicator of the same size as this.

Returns:

this.

void col_scale(ptr_param<const global_vector_type> scaling_factors)#

Scales the columns of the matrix by the respective entries of the vector. The vector’s row partition has to be the same as the matrix’s column partition. The scaling is done in-place.

Parameters:

scaling_factors – The vector containing the scaling factors.

void row_scale(ptr_param<const global_vector_type> scaling_factors)#

Scales the rows of the matrix by the respective entries of the vector. The vector and the matrix have to have the same row partition. The scaling is done in-place.

Parameters:

scaling_factors – The vector containing the scaling factors.

Public Static Functions

static std::unique_ptr<Matrix> create(
std::shared_ptr<const Executor> exec,
mpi::communicator comm,
)#

Creates an empty distributed matrix.

Parameters:

  • exec – Executor associated with this matrix.

  • comm – Communicator associated with this matrix. The default is the MPI_COMM_WORLD.

Returns:

A smart pointer to the newly created matrix.

static std::unique_ptr<Matrix> create(
std::shared_ptr<const Executor> exec,
std::shared_ptr<const RowGatherer<LocalIndexType>> row_gatherer_template,
)#

Creates an empty distributed matrix with a specified implementation of the row gather operation.

Parameters:

  • exec – Executor associated with this matrix.

  • row_gatherer_template – A template for the used row gather operation. This is only used to create a new row gatherer during the read_distributed.

Returns:

A smart pointer to the newly created matrix.

template<typename MatrixType, typename = std::enable_if_t<gko::detail::is_matrix_type_builder<MatrixType, ValueType, LocalIndexType>::value>>
static inline std::unique_ptr<Matrix> create(
std::shared_ptr<const Executor> exec,
mpi::communicator comm,
MatrixType matrix_template,
)#

Creates an empty distributed matrix with specified type for local matrices.

See also

with_matrix_type

Note

This is mainly a convenience wrapper for Matrix(std::shared_ptr<const Executor>, mpi::communicator, const

LinOp*)

Template Parameters:

MatrixType – A type that has a create<ValueType, IndexType>(exec) function to create a smart pointer of a type derived from LinOp and ReadableFromMatrixData.

Parameters:

  • exec – Executor associated with this matrix.

  • comm – Communicator associated with this matrix.

  • matrix_template – the local matrices will be constructed with the same type as create returns. It should be the return value of make_matrix_template.

Returns:

A smart pointer to the newly created matrix.

template<typename LocalMatrixType, typename NonLocalMatrixType, typename = std::enable_if_t<gko::detail::is_matrix_type_builder<LocalMatrixType, ValueType, LocalIndexType>::value && gko::detail::is_matrix_type_builder<NonLocalMatrixType, ValueType, LocalIndexType>::value>>
static inline std::unique_ptr<Matrix> create(
std::shared_ptr<const Executor> exec,
mpi::communicator comm,
LocalMatrixType local_matrix_template,
NonLocalMatrixType non_local_matrix_template,
)#

Creates an empty distributed matrix with specified types for the local matrix and the non-local matrix.

See also

with_matrix_type

Note

This is mainly a convenience wrapper for Matrix(std::shared_ptr<const Executor>, mpi::communicator,

const LinOp*, const LinOp*)

Template Parameters:

  • LocalMatrixType – A type that has a create<ValueType, IndexType>(exec) function to create a smart pointer of a type derived from LinOp and ReadableFromMatrixData.

  • NonLocalMatrixType – A (possible different) type with the same constraints as LocalMatrixType.

Parameters:

  • exec – Executor associated with this matrix.

  • comm – Communicator associated with this matrix.

  • local_matrix_template – the local matrix will be constructed with the same type as create returns. It should be the return value of make_matrix_template.

  • non_local_matrix_template – the non-local matrix will be constructed with the same type as create returns. It should be the return value of make_matrix_template.

Returns:

A smart pointer to the newly created matrix.

static std::unique_ptr<Matrix> create(
std::shared_ptr<const Executor> exec,
mpi::communicator comm,
ptr_param<const LinOp> matrix_template,
)#

Creates an empty distributed matrix with specified type for local matrices.

Note

It internally clones the passed in matrix_template. Therefore, the LinOp should be empty.

Parameters:

  • exec – Executor associated with this matrix.

  • comm – Communicator associated with this matrix.

  • matrix_template – the local matrices will be constructed with the same runtime type.

Returns:

A smart pointer to the newly created matrix.

static std::unique_ptr<Matrix> create(
std::shared_ptr<const Executor> exec,
mpi::communicator comm,
ptr_param<const LinOp> local_matrix_template,
ptr_param<const LinOp> non_local_matrix_template,
)#

Creates an empty distributed matrix with specified types for the local matrix and the non-local matrix.

Note

It internally clones the passed in local_matrix_template and non_local_matrix_template. Therefore, those LinOps should be empty.

Parameters:

  • exec – Executor associated with this matrix.

  • comm – Communicator associated with this matrix.

  • local_matrix_template – the local matrix will be constructed with the same runtime type.

  • non_local_matrix_template – the non-local matrix will be constructed with the same runtime type.

Returns:

A smart pointer to the newly created matrix.

static std::unique_ptr<Matrix> create(
std::shared_ptr<const Executor> exec,
mpi::communicator comm,
dim<2> size,
std::shared_ptr<LinOp> local_linop,
)#

Creates a local-only distributed matrix with existent LinOp

Note

It use the input to build up the distributed matrix

Parameters:

  • exec – Executor associated with this matrix.

  • comm – Communicator associated with this matrix.

  • size – the global size

  • local_linop – the local linop

Returns:

A smart pointer to the newly created matrix.

static std::unique_ptr<Matrix> create(
std::shared_ptr<const Executor> exec,
mpi::communicator comm,
dim<2> size,
std::shared_ptr<LinOp> local_linop,
std::shared_ptr<LinOp> non_local_linop,
std::vector<comm_index_type> recv_sizes,
std::vector<comm_index_type> recv_offsets,
array<local_index_type> recv_gather_idxs,
)#

Creates distributed matrix with existent local and non-local LinOp and the corresponding mapping to collect the non-local data from the other ranks.

Note

It use the input to build up the distributed matrix

Parameters:

  • exec – Executor associated with this matrix.

  • comm – Communicator associated with this matrix.

  • size – the global size

  • local_linop – the local linop

  • non_local_linop – the non-local linop

  • recv_sizes – the size of non-local receiver

  • recv_offsets – the offset of non-local receiver

  • recv_gather_idxs – the gathering index of non-local receiver

Returns:

A smart pointer to the newly created matrix.

static std::unique_ptr<Matrix> create(
std::shared_ptr<const Executor> exec,
mpi::communicator comm,
index_map<local_index_type, global_index_type> imap,
std::shared_ptr<LinOp> local_linop,
std::shared_ptr<LinOp> non_local_linop,
)#

Creates distributed matrix with existent local and non-local LinOp and the corresponding mapping to collect the non-local data from the other ranks.

Parameters:

  • exec – Executor associated with this matrix.

  • comm – Communicator associated with this matrix.

  • imap – The index map to define the communication pattern

  • local_linop – the local linop

  • non_local_linop – the non-local linop

Returns:

A smart pointer to the newly created matrix.

gko::with_matrix_type#

Helper for selecting the local-block storage type used by a distributed::Matrix without having to commit to the value and index types up front. Matrix::create accepts one or two of these selectors to override the default Csr storage of the local and non-local blocks:

auto mat = Matrix<ValueType, LocalIndexType>::create(
    exec, comm,
    gko::with_matrix_type<gko::matrix::Ell>(),   // local block
    gko::with_matrix_type<gko::matrix::Coo>());  // non-local block

with_matrix_type<MatrixType>(args...) returns a deferred-creation object; the distributed Matrix constructor invokes its create<value_type, index_type>(exec) once the value and index types are known, forwarding args... to MatrixType::create. Any type that satisfies LinOp and ReadableFromMatrixData is acceptable.

template<template<typename, typename> class MatrixType, typename ...Args>
auto gko::with_matrix_type(
Args&&... create_args,
)#

This function returns a type that delays a call to MatrixType::create.

It can be used to set the used value and index type, as well as the executor at a later stage.

For example, the following code creates first a temporary object, which is then used later to construct an operator of the previously defined base type: 

auto type = gko::with_matrix_type<gko::matrix::Csr>();
...
std::unique_ptr<LinOp> concrete_op
if(flag1){
  concrete_op = type.template create<double, int>(exec);
} else {
  concrete_op = type.template create<float, int>(exec);
}

Note

This is mainly a helper function to specify the local matrix type for a gko::experimental::distributed::Matrix more easily.

Template Parameters:

  • MatrixType – A template type that accepts two types, the first one will be set to the value type, the second one to the index type.

  • Args – Types of the arguments passed to MatrixType::create.

Parameters:

create_args – arguments that will be forwarded to MatrixType::create

Returns:

A type with a function create<value_type, index_type>(executor).