tpetra vector creation creates 6 modified flags causing a ton of launch overhead

[bathmatt]

Bug Report

@trilinos/tpetra

Description

I've figured out that this call here

RCP<vector_type> B_vec = B.getVectorNonConst (0);

creates 6 views of modified flags

|   |   |   |   |   |   |   |   |   |   |   |   |   compute: 5.93946 - 99.1017% [2148]
|   |   |   |   |   |   |   |   |   |   |   |   |   |   fused: 5.93083 - 99.8546% [2148]
|   |   |   |   |   |   |   |   |   |   |   |   |   |   |   Kokkos View Allocation - DualView::modified_flags: 0.00229072 - 0.0386239% [3866\

It should be 1. Fixes in the non-deprecated branch make it 2,

Currently a 0 length view takes 60us to create and ifpack2 makes 1000s of them per solve... That is free money

There are 4500 calls in trilinos to this, that is a lot of places to reduce launch overhead of initializing these.

Steps to Reproduce

1. SHA1: [insert here]
2. Configure script: [attach here]
3. Configure log: [attach here]
4. Build log: [attach here]
5. Input deck: [attach here]
6. Do this.
7. Do that.
8. Shake fist angrily at computer.
9. Run log: [attach here]

[kddevin]

The modified flags are in Kokkos::DualView. There are three now, but when applications set KOKKOS_ENABLE_DEPRECATED_CODE=OFF, there is one.

getVectorNonConst(0) creates a new Tpetra::MultiVector containing the 0th vector of the "this" multivector (not a zero-length view). New flags are created when subviews of the DualView are taken; the subview code is in Kokkos (Kokkos_DualView.hpp). Subviews of both the host and device memory are also created in the DualView subview constructor. Subviews of the import and export buffers are also created on host and device.

In Tpetra, perhaps we can add logic to test the number of vectors in a MultiVector, just reusing the views (new = old) instead of subviewing (new = subview(old, ...)) when the multivector has only one vector. @mhoemmen, what do you think?

@cgcgcg

[mhoemmen]

Hi @kddevin ! I'm looking at the implementations of getVectorNonConst and the DualView subview constructor, and I'm not seeing where the subviews create new flags. (This matters because people don't always realize that subviews of a DualView share flags with the parent.)

• Here is the relevant MultiVector constructor that implements getVectorNonConst.
• Here is the Kokkos::DualView subview constructor.

Notice how the Kokkos::DualView subview constructor invokes the flags' copy constructor(s). (Whether one or two copy constructors are invoked depends on whether deprecated code is enabled, but it's the copy constructor each time.) The copy constructor does not allocate; it just shares the allocation of its input, like the std::shared_ptr copy constructor.

[kddevin]

@crtrott Is there launch overhead in copying the modify flags' view?
There will be two copies (after the user can run with KOKKOS_ENABLE_DEPRECATED_CODE=OFF) because the newly created Vector has (one) a subview of the DualView and (two) a view of the original DualView. The modify flags are touched for each. But isn't each touch inexpensive? Thanks!

[kddevin]

With Kokkos' deprecated code removed, the number of flags is reduced.
I will close this issue; if the problem persists, please re-open.