Tpetra: how to copy a matrix with a different range/domain map without invalidating the original

[bathmatt]

Question

@trilinos/tpetra
@jhux2 @cgcgcg @kddevin

So, I;m removing UVM from zoltan2 and I ran into a use case in Z2 that is problematic for UVM removal, They have two matrices

JBlock and JCyclic where the only difference is the rance and domain maps. They construct the JCyclic via the call below

   JCyclic = rcp(new matrix_t(JBlock->getLocalMatrixDevice(),
                               JBlock->getRowMap(), JBlock->getColMap(),
                              vMapCyclic, wMapCyclic));

This does a shallow copy of the device matrix and then sets the use counts for the graph and values to 8 for the device side and only 4 for the host side.

I tried the following, make a deep copy of the matrix since this is just in a test and who cares about memory

    {
      { auto fff = JBlock->getLocalMatrixHost();}
      matrix_t JBlockCopy(*JBlock, Teuchos::Copy);
      { auto fff = JBlock->getLocalMatrixHost();}
      JCyclic = rcp(new matrix_t(JBlockCopy.getLocalMatrixDevice(),
                                 JBlock->getRowMap(), JBlock->getColMap(),
                                 vMapCyclic, wMapCyclic));
    }
      { auto fff = JBlock->getLocalMatrixHost();}

and the 3rd getLocalMatrixHost fails cuz the graph has a mismatched use count. So, the graph is always a shallow copy it looks like

The auto fff = ... is there just for use count checking, just FYI

So, there has to be a way to do what I want, I looked there is no setRangeMap and only getters or I would just copy the matrix raising the counts to 8 but evenly.

Thanks

[cgcgcg]

Not sure about this, but what happens if you use the CrsMatrix copy constructor with copyOrView=Teuchos::View and then do a fillResume and a fillComplete with the different domain and range maps?

[bathmatt]

Not sure about this, but what happens if you use the CrsMatrix copy constructor with copyOrView=Teuchos::View and then do a fillResume and a fillComplete with the different domain and range maps?

Look at you with all the answers :)
TEST PASSED

[bathmatt]

Not sure about this, but what happens if you use the CrsMatrix copy constructor with copyOrView=Teuchos::View and then do a fillResume and a fillComplete with the different domain and range maps?

Look at you with all the answers :)
TEST PASSED

Wait @cgcgcg you're not so smart as of yet, when I run in parallel I get failures in the fill complete call

Throw test that evaluated to true: (! domainMapsMatch)

Tpetra::CrsMatrix<double, int, long long, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Cuda, Kokkos::CudaSpace> >::fillComplete: The CrsMatrix's domain Map
 does not match the graph's domain Map.  The graph cannot be changed because it was given to the CrsMatrix constructor as const.  You can fix this by passing 
in the graph's domain Map and range Map to the matrix's fillComplete call.
terminate called after throwing an instance of 'std::runtime_error'
  what():  /scratch/mbetten/Trilinos/Trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp:4881:

looking into why

[cgcgcg]

Hm. There is also a replaceDomainMapAndImporter in case it's just the domain map that needs to be changed.

[bathmatt]

Here is the code I replaced it with

     // Make JCyclic:  same matrix with different Domain and Range maps
-    JCyclic = rcp(new matrix_t(JBlock->getLocalMatrixDevice(),
-                               JBlock->getRowMap(), JBlock->getColMap(),
-                               vMapCyclic, wMapCyclic));
+
+    JCyclic = rcp(new matrix_t(*JBlock,Teuchos::View));
+    JCyclic->resumeFill();
+    JCyclic->fillComplete(vMapCyclic, wMapCyclic);

[cgcgcg]

Another attempt: Construct a new CrsGraph object using the local graph of the first matrix and different domain and range maps. Then construct a new CrsMatrix using the new graph and the values.

[bathmatt]

Another attempt: Construct a new CrsGraph object using the local graph of the first matrix and different domain and range maps. Then construct a new CrsMatrix using the new graph and the values.

Yeah, i thought about all that crap, i can't use a local graph but will look into some other way to rebuild the matrix.

Hoping for a good idea from someone hoping not to have to do all that crap

[kddevin]

@trilinos/tpetra should look at this case. Either tpetra should correctly handle the use-case or it should deprecate that constructor.

We had to do some workaround for FECrsMatrix; let us take a look.

[bathmatt]

@JacobDomagala

[bathmatt]

@trilinos/tpetra any comment on the comment by @kddevin ??? I can make a copy of the graph and fill in the values and do all the stuff to recreate the matrix. It is in a test so it isn't a big deal that we duplicate the entire matrix but if this has a real application we will need a better path forward

[bathmatt]

@cgcgcg or anyone else, is there an easy way to copy one matrix values to another matrix with the same graph

Was hoping for a getValues and a deep copy of some sort.

If I was to use an importer, do I just create an importer with the row maps?

[rppawlo]

getLocalMatrixDevice() returns the KokkosSparse::CrsMatrix. The matrix values in this object are stored in a Kokkos::View. I don't see an accessor to get the entire view (there are row view accessors), so you may need to add a new method for that. Since the object is snapshotted into trilinos you would have to get the changes into kokkos kernels first.

[cgcgcg]

I think the values view in the local matrix is accessible.

[rppawlo]

@cgcgcg is correct. The class members are public. Just grab the "values" member.

[bathmatt]

This is what I've done to replace that simple single line

   // Make JCyclic:  same matrix with different Domain and Range maps                                                                                        
    RCP<const graph_t> block_graph = JBlock->getCrsGraph();
    RCP<graph_t> cyclic_graph = rcp(new graph_t(*block_graph));
    cyclic_graph->resumeFill();
    cyclic_graph->fillComplete(vMapCyclic, wMapCyclic);
    JCyclic = rcp(new matrix_t(cyclic_graph));
    JCyclic->resumeFill();
    TEUCHOS_ASSERT(block_graph->getNodeNumRows() == cyclic_graph->getNodeNumRows());
    {
      auto val_s = JBlock->getLocalMatrixHost().values;
      auto val_d = JCyclic->getLocalMatrixHost().values;
      TEUCHOS_ASSERT(val_s.extent(0) == val_d.extent(0));
      Kokkos::deep_copy(val_d, val_s);
    }
    JCyclic->fillComplete();
  }

[kddevin]

We decided to add methods to replace the Domain and Range map. (Epetra has these functions, but Tpetra does not.) Then one would use the shallow copy constructor followed by the methods for replacing the Domain and Range map. See #9451

[bathmatt]

Graph will need those as well maybe. Just saying they don't have that function either.

…

On Mon, Jul 19, 2021 at 12:37 PM K Devine ***@***.***> wrote: We decided to add methods to replace the Domain and Range map. (Epetra has these functions, but Tpetra does not.) Then one would use the shallow copy constructor followed by the methods for replacing the Domain and Range map. See #9451 <#9451> — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#9391 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADR4GIA7FNLBQE3BN5QO7WDTYRIGBANCNFSM475JCI3A> .

[kddevin]

In branch tpetra_9391, there is a test that attempts to copy a matrix four different ways. Two work -- repeating all the inserts to build the new matrix from scratch

Trilinos/packages/tpetra/core/test/CrsMatrix/Bug9391.cpp

Line 203 in 6f74de8

TEUCHOS_UNIT_TEST_TEMPLATE_4_DECL(CrsMatrix, Bug9391_Scratch, SC, LO, GO, NT)

and @bathmatt 's method above

Trilinos/packages/tpetra/core/test/CrsMatrix/Bug9391.cpp

Line 386 in 6f74de8

TEUCHOS_UNIT_TEST_TEMPLATE_4_DECL(CrsMatrix, Bug9391_CopyGraph, SC, LO, GO, NT)

Two do not work: building the new matrix from the original's local device matrix gets the counts wrong so that one cannot take a host view

Trilinos/packages/tpetra/core/test/CrsMatrix/Bug9391.cpp

Line 275 in 6f74de8

TEUCHOS_UNIT_TEST_TEMPLATE_4_DECL(CrsMatrix, Bug9391_LocalMat, SC, LO, GO, NT)

and calling the copy constructor for a "deep" copy followed by replaceDomainMap, replaceRangeMap creates a static graph for which the maps cannot be changed

Trilinos/packages/tpetra/core/test/CrsMatrix/Bug9391.cpp

Line 328 in 6f74de8

TEUCHOS_UNIT_TEST_TEMPLATE_4_DECL(CrsMatrix, Bug9391_Replace, SC, LO, GO, NT)

That is, our proposed solution above will not work, as the "deep" copy constructor actually deep-copies only the values of the matrix, not the graph.