Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Teko_testdriver_tpetra_MPI_4 randomly timing out/hanging in all CUDA 'waterman' builds starting 2019-11-14 #6463

Closed
bartlettroscoe opened this issue Dec 17, 2019 · 7 comments
Closed

Teko_testdriver_tpetra_MPI_4 randomly timing out/hanging in all CUDA 'waterman' builds starting 2019-11-14 #6463

bartlettroscoe opened this issue Dec 17, 2019 · 7 comments
Labels
ATDM Sev: Blocker Problems that make Trilinos unfit to be adopted by one or more ATDM APPs client: ATDM Any issue primarily impacting the ATDM project client: EMPIRE All issues that most directly target the ATDM EMPIRE code CLOSED_DUE_TO_INACTIVITY Issue or PR has been closed by the GitHub Actions bot due to inactivity. Disabled Tests Issue has been partially addressed by disabling *all* of the failing tests related to the issue MARKED_FOR_CLOSURE Issue or PR is marked for auto-closure by the GitHub Actions bot. PA: Linear Solvers Issues that fall under the Trilinos Linear Solvers Product Area pkg: Teko type: bug The primary issue is a bug in Trilinos code or tests
Milestone
Keep promoted "AT...

Comments

@bartlettroscoe
Copy link
Member

CC: @trilinos/teko, @srajama1 (Trilinos Linear Solvers Product Lead), @bartlettroscoe, @fryeguy52

Next Action Status

Description

As shown in this query and this query over the time period [2019-08-01, 2019-12-17], the test:

  • Teko_testdriver_tpetra_MPI_4

in the builds:

  • Trilinos-atdm-waterman_cuda-9.2_fpic_static_opt
  • Trilinos-atdm-waterman_cuda-9.2_shared_opt
  • Trilinos-atdm-waterman-cuda-9.2-debug
  • Trilinos-atdm-waterman-cuda-9.2-opt
  • Trilinos-atdm-waterman-cuda-9.2-rdc-release-debug
  • Trilinos-atdm-waterman-cuda-9.2-release-debug

started randomly timing out (or hanging) on testing day 2019-11-14. It randomly timed out (or hung) a total of 29 times between 2019-11-14 and 2019-12-17 in those 6 builds. (Therefore, it times out only occasionally).

As shown in this query, the runtimes over the time period [2019-11-10, 2019-12-17] for this test are very erratic. When it passes, it can pass in just over a minute and take between the lowest time and timing out at 10 minutes. Since these CUAA builds now all use ctest -j4 (see #6051) and since this is an MPI-4 test, this test should be running by itself on its own node on its own GPU. So the big differences may be due to non-determinism in the code.

There is no sense in looking for commits that could have triggered this as this is a rarely randomly failing test.

Current Status on CDash

Steps to Reproduce

One should be able to reproduce this failure on the machine 'waterman' as described in:

More specifically, the commands given for the system 'waterman' are provided at:

The exact commands to reproduce this issue should be:

$ cd <some_build_dir>/

$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh \
    Trilinos-atdm-waterman-cuda-9.2-debug

$ cmake \
 -GNinja \
 -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
 -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Teko=ON \
 $TRILINOS_DIR

$ make NP=16

$ bsub -x -Is -n 20  ctest -j4

However, given this is a rarely randomly failing test, this could be hard to reproduce.
@bartlettroscoe bartlettroscoe added type: bug The primary issue is a bug in Trilinos code or tests pkg: Teko client: ATDM Any issue primarily impacting the ATDM project client: EMPIRE All issues that most directly target the ATDM EMPIRE code ATDM Sev: Blocker Problems that make Trilinos unfit to be adopted by one or more ATDM APPs PA: Linear Solvers Issues that fall under the Trilinos Linear Solvers Product Area labels Dec 17, 2019
@bartlettroscoe
Copy link
Member Author

For example, for the build Trilinos-atdm-waterman-cuda-9.2-opt over the time period [2019-11-10, 2019-12-17], the runtimes varied between 1m 19s to 9m 25s and timing out once at 10m a shown in this query showing:

Test Name Status Time Details Build Time
Teko_testdriver_tpetra_MPI_4 Passed 2m 52s 190ms Completed 2019-12-17T05:42:06 MST
Teko_testdriver_tpetra_MPI_4 Passed 3m 27s 500ms Completed 2019-12-16T05:41:44 MST
Teko_testdriver_tpetra_MPI_4 Passed 1m 58s 260ms Completed 2019-12-15T05:40:58 MST
Teko_testdriver_tpetra_MPI_4 Passed 2m 8s 590ms Completed 2019-12-14T07:51:59 MST
Teko_testdriver_tpetra_MPI_4 Passed 1m 25s 420ms Completed 2019-12-13T08:02:43 MST
Teko_testdriver_tpetra_MPI_4 Passed 1m 49s 940ms Completed 2019-12-12T11:56:11 MST
Teko_testdriver_tpetra_MPI_4 Passed 1m 49s 940ms Completed 2019-12-12T11:56:11 MST
Teko_testdriver_tpetra_MPI_4 Passed 1m 49s 940ms Completed 2019-12-12T11:56:11 MST
Teko_testdriver_tpetra_MPI_4 Passed 3m 27s 910ms Completed 2019-12-11T15:37:39 MST
Teko_testdriver_tpetra_MPI_4 Passed 6m 27s 930ms Completed 2019-12-10T06:55:50 MST
Teko_testdriver_tpetra_MPI_4 Passed 1m 29s 580ms Completed 2019-12-09T11:53:35 MST
Teko_testdriver_tpetra_MPI_4 Passed 1m 29s 580ms Completed 2019-12-09T11:53:35 MST
Teko_testdriver_tpetra_MPI_4 Failed 10m 170ms Completed (Timeout) 2019-12-08T16:04:17 MST
Teko_testdriver_tpetra_MPI_4 Passed 4m 41s 560ms Completed 2019-12-07T12:20:47 MST
Teko_testdriver_tpetra_MPI_4 Passed 3m 24s 180ms Completed 2019-12-06T12:23:17 MST
Teko_testdriver_tpetra_MPI_4 Passed 2m 20s 200ms Completed 2019-12-01T05:51:45 MST
Teko_testdriver_tpetra_MPI_4 Passed 2m 8s 190ms Completed 2019-11-30T05:42:14 MST
Teko_testdriver_tpetra_MPI_4 Passed 5m 14s 390ms Completed 2019-11-29T05:53:42 MST
Teko_testdriver_tpetra_MPI_4 Passed 1m 19s 560ms Completed 2019-11-28T07:40:39 MST
Teko_testdriver_tpetra_MPI_4 Passed 6m 5s 180ms Completed 2019-11-27T08:01:25 MST
Teko_testdriver_tpetra_MPI_4 Passed 3m 7s 630ms Completed 2019-11-26T08:10:31 MST
Teko_testdriver_tpetra_MPI_4 Passed 2m 27s 620ms Completed 2019-11-25T05:37:06 MST
Teko_testdriver_tpetra_MPI_4 Passed 6m 32s 150ms Completed 2019-11-24T06:19:52 MST
Teko_testdriver_tpetra_MPI_4 Passed 7m 30s 680ms Completed 2019-11-23T08:49:22 MST
Teko_testdriver_tpetra_MPI_4 Passed 4m 49s 450ms Completed 2019-11-22T06:04:22 MST
Teko_testdriver_tpetra_MPI_4 Passed 1m 42s 400ms Completed 2019-11-21T06:05:21 MST
Teko_testdriver_tpetra_MPI_4 Passed 4m 5s 480ms Completed 2019-11-20T08:26:50 MST
Teko_testdriver_tpetra_MPI_4 Passed 3m 56s 690ms Completed 2019-11-19T07:48:22 MST
Teko_testdriver_tpetra_MPI_4 Passed 8m 23s 930ms Completed 2019-11-18T05:49:06 MST
Teko_testdriver_tpetra_MPI_4 Passed 9m 25s 50ms Completed 2019-11-17T05:50:44 MST
Teko_testdriver_tpetra_MPI_4 Passed 3m 19s 200ms Completed 2019-11-16T16:48:54 MST
Teko_testdriver_tpetra_MPI_4 Passed 2m 14s 750ms Completed 2019-11-15T09:17:58 MST
Teko_testdriver_tpetra_MPI_4 Passed 5m 19s 500ms Completed 2019-11-14T07:44:48 MST

But compare this to the runtimes for the test Teko_StridedEpetraOperator_test_MPI_4 in that same build Trilinos-atdm-waterman-cuda-9.2-opt over that same time period which shows times between 4s and 19s in this query showing the times:

Test Name Status Time Details Build Time
Teko_StridedEpetraOperator_test_MPI_4 Passed 7s 200ms Completed 2019-12-17T05:42:06 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 5s 270ms Completed 2019-12-16T05:41:44 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 5s 60ms Completed 2019-12-15T05:40:58 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 5s 170ms Completed 2019-12-14T07:51:59 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 10s 190ms Completed 2019-12-13T08:02:43 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 4s 510ms Completed 2019-12-12T11:56:11 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 4s 510ms Completed 2019-12-12T11:56:11 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 4s 510ms Completed 2019-12-12T11:56:11 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 9s 60ms Completed 2019-12-11T15:37:39 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 5s 870ms Completed 2019-12-10T06:55:50 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 7s 260ms Completed 2019-12-09T11:53:35 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 7s 260ms Completed 2019-12-09T11:53:35 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 5s 440ms Completed 2019-12-08T16:04:17 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 19s 670ms Completed 2019-12-07T12:20:47 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 4s 770ms Completed 2019-12-06T12:23:17 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 4s 930ms Completed 2019-12-01T05:51:45 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 8s Completed 2019-11-30T05:42:14 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 5s 170ms Completed 2019-11-29T05:53:42 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 5s 30ms Completed 2019-11-28T07:40:39 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 5s 200ms Completed 2019-11-27T08:01:25 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 4s 650ms Completed 2019-11-26T08:10:31 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 4s 770ms Completed 2019-11-25T05:37:06 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 5s 30ms Completed 2019-11-24T06:19:52 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 4s 910ms Completed 2019-11-23T08:49:22 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 5s 40ms Completed 2019-11-22T06:04:22 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 4s 950ms Completed 2019-11-21T06:05:21 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 4s 600ms Completed 2019-11-20T08:26:50 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 17s 850ms Completed 2019-11-19T07:48:22 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 5s Completed 2019-11-18T05:49:06 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 17s 500ms Completed 2019-11-17T05:50:44 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 5s 810ms Completed 2019-11-16T16:48:54 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 13s 700ms Completed 2019-11-15T09:17:58 MST
Teko_StridedEpetraOperator_test_MPI_4 Passed 4s 730ms Completed 2019-11-14T07:44:48 MST

I wonder if this is due to the CPU and/or GPUs getting hot and throttling back speed? Steven Oliver suggested this is happening on these types of machines and it is mentioned in:

Can such factors of 5x or more differences in runtime be caused by such issues on these machines? Is this just par-for-the-course when testing on these types of systems?

@bartlettroscoe
Copy link
Member Author

Unless there is an objection, I will just disable this randomly timing out test in all of the ATDM Trilinos builds on 'waterman'. This test will still run on the builds on 'ride' and will also run on 'vortex' once we get those builds up and running.

@eric-c-cyr
Copy link
Contributor

eric-c-cyr commented Jan 6, 2020 via email

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Jan 17, 2020
@bartlettroscoe
Disable Teko_testdriver_tpetra_MPI_4 in all atdm 'waterman' builds (t…
557db4d 
…rilinos#6463)
@bartlettroscoe bartlettroscoe mentioned this issue Jan 17, 2020
Merged
bartlettroscoe added a commit that referenced this issue Jan 17, 2020
@bartlettroscoe
Merge branch '6463-atdm-waterman-disable-teko-test' into atdm-nightly…
2e26557 
…-manual-updates (#6463, #6600)
@bartlettroscoe
Copy link
Member Author

Unless there is an objection, I will just disable this randomly timing out test in all of the ATDM Trilinos builds on 'waterman'.

No objection

@eric-c-cyr, I created PR #6600 for this disable. Can you approve that PR?

In any case, I manaully merged that branch to the 'atdm-nightly' branch for tomorrow in commit
2e26557 so this test will be disabled in ATDM Trilinos builds going forward in case PR #6600 can't be merged before tonight for some reason.

trilinos-autotester added a commit that referenced this issue Jan 17, 2020
@trilinos-autotester
Merge Pull Request #6600 from bartlettroscoe/Trilinos/6463-atdm-water…
464684e 
…man-disable-teko-test

Automatically Merged using Trilinos Pull Request AutoTester
PR Title: Disable Teko_testdriver_tpetra_MPI_4 in all atdm 'waterman' builds (#6463)
PR Author: bartlettroscoe
@bartlettroscoe
Copy link
Member Author

With the merge of #6600, we can add the "Disabled Tests" label and remove this issue from our main list of issues.

@trilinos/teko, if there is no interest in trying to get this test running better on 'waterman' (and I don't know why there would be since it runs fine everywhere else including 'ride' and on 'sems-rhel7' builds with CUDA and GPUs), then can we just close this as "WONTFIX"?

@bartlettroscoe bartlettroscoe added the Disabled Tests Issue has been partially addressed by disabling *all* of the failing tests related to the issue label Jan 17, 2020
jmgate pushed a commit to tcad-charon/Trilinos that referenced this issue Jan 18, 2020
Merge 'trilinos/Trilinos:develop' (630dfd2) into 'tcad-charon/Trilino…
5a0c8b9 
…s:develop' (d17489d).

* trilinos-develop:
  SEACAS: go back to lib:fmt 6.0.0 until fix issue on vortex xl/cuda build
  Disable Teko_testdriver_tpetra_MPI_4 in all atdm 'waterman' builds (trilinos#6463)
  zoltan2:  add missing include file for non-ETI builds
  Tpetra: Missed ifdef guard
  zoltan2:  name change to prevent shadow warnings
  zoltan2:  Change logic for determining gno types to use in tests Simplified now that Trilinos builds only one gno_t
  Tpetra: More stacked timer fixes
  Tpetra: Fix overflow for TpetraCore_MatrixMatrix_UnitTests
  tpetra:  removed unused field from FixedHashTable This removes some warnings about calling host functions from host device functions trilinos#5698; E.g., warning: calling a __host__ function("std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string") from a __host__ __device__ function("Tpetra::Details::LocalMap<int, long long,  ::Kokkos::Device< ::Kokkos::Serial,  ::Kokkos::HostSpace> > ::~LocalMap [subobject]") is not allowed
  tpetra:  when looking at trilinos#6158, I saw that, when creating a Vector from a MultiVector, we created subviews for comm buffers, but did not store them. This commit stores them.  It also offsets the buffers by the vector j requested from the MultiVector.
jmgate pushed a commit to tcad-charon/Trilinos that referenced this issue Jan 19, 2020
Merge 'trilinos/Trilinos:develop' (630dfd2) into 'tcad-charon/Trilino…
88c01b4 
…s:develop' (d17489d).

* trilinos-develop:
  SEACAS: go back to lib:fmt 6.0.0 until fix issue on vortex xl/cuda build
  Disable Teko_testdriver_tpetra_MPI_4 in all atdm 'waterman' builds (trilinos#6463)
  zoltan2:  add missing include file for non-ETI builds
  Tpetra: Missed ifdef guard
  zoltan2:  name change to prevent shadow warnings
  zoltan2:  Change logic for determining gno types to use in tests Simplified now that Trilinos builds only one gno_t
  Tpetra: More stacked timer fixes
  Tpetra: Fix overflow for TpetraCore_MatrixMatrix_UnitTests
  tpetra:  removed unused field from FixedHashTable This removes some warnings about calling host functions from host device functions trilinos#5698; E.g., warning: calling a __host__ function("std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string") from a __host__ __device__ function("Tpetra::Details::LocalMap<int, long long,  ::Kokkos::Device< ::Kokkos::Serial,  ::Kokkos::HostSpace> > ::~LocalMap [subobject]") is not allowed
  tpetra:  when looking at trilinos#6158, I saw that, when creating a Vector from a MultiVector, we created subviews for comm buffers, but did not store them. This commit stores them.  It also offsets the buffers by the vector j requested from the MultiVector.
jmgate pushed a commit to tcad-charon/Trilinos that referenced this issue Jan 20, 2020
Merge 'trilinos/Trilinos:develop' (ebdb4cb) into 'tcad-charon/Trilino…
56f3102 
…s:develop' (d17489d).

* trilinos-develop:
  tpetra:  In trilinos#6598, @mhoemmen recommended this change of offset
  SEACAS: go back to lib:fmt 6.0.0 until fix issue on vortex xl/cuda build
  Disable Teko_testdriver_tpetra_MPI_4 in all atdm 'waterman' builds (trilinos#6463)
  zoltan2:  add missing include file for non-ETI builds
  Tpetra: Missed ifdef guard
  zoltan2:  name change to prevent shadow warnings
  zoltan2:  Change logic for determining gno types to use in tests Simplified now that Trilinos builds only one gno_t
  Tpetra: More stacked timer fixes
  Tpetra: Fix overflow for TpetraCore_MatrixMatrix_UnitTests
  tpetra:  removed unused field from FixedHashTable This removes some warnings about calling host functions from host device functions trilinos#5698; E.g., warning: calling a __host__ function("std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string") from a __host__ __device__ function("Tpetra::Details::LocalMap<int, long long,  ::Kokkos::Device< ::Kokkos::Serial,  ::Kokkos::HostSpace> > ::~LocalMap [subobject]") is not allowed
  tpetra:  when looking at trilinos#6158, I saw that, when creating a Vector from a MultiVector, we created subviews for comm buffers, but did not store them. This commit stores them.  It also offsets the buffers by the vector j requested from the MultiVector.
jmgate pushed a commit to tcad-charon/Trilinos that referenced this issue Jan 21, 2020
Merge 'trilinos/Trilinos:develop' (5ee3160) into 'tcad-charon/Trilino…
0b9b598 
…s:develop' (d17489d).

* trilinos-develop:
  tpetra:  In trilinos#6598, @mhoemmen recommended this change of offset
  Tpetra::CrsMatrix: Add Kokkos kernel labels; expose debug code
  Tpetra::CrsMatrix: Remove values2D_
  Tpetra::CrsGraph: Remove gblInds2D_
  Tpetra::CrsGraph: Remove lclInds2D_
  Tpetra::CrsMatrix: Remove unused method allocateValues2D
  Tpetra: Use verbosePrintCountThreshold in copyOffsets
  Tpetra::Details::Behavior: Add longRowMinNumEntries
  Tpetra::Details::Behavior: Factor out size_t reading
  SEACAS: go back to lib:fmt 6.0.0 until fix issue on vortex xl/cuda build
  Disable Teko_testdriver_tpetra_MPI_4 in all atdm 'waterman' builds (trilinos#6463)
  zoltan2:  add missing include file for non-ETI builds
  Tpetra: Missed ifdef guard
  zoltan2:  name change to prevent shadow warnings
  zoltan2:  Change logic for determining gno types to use in tests Simplified now that Trilinos builds only one gno_t
  Tpetra: More stacked timer fixes
  Tpetra: Fix overflow for TpetraCore_MatrixMatrix_UnitTests
  tpetra:  removed unused field from FixedHashTable This removes some warnings about calling host functions from host device functions trilinos#5698; E.g., warning: calling a __host__ function("std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string") from a __host__ __device__ function("Tpetra::Details::LocalMap<int, long long,  ::Kokkos::Device< ::Kokkos::Serial,  ::Kokkos::HostSpace> > ::~LocalMap [subobject]") is not allowed
  Framework: updating the autotester env to remove dependency on atdm-env
  tpetra:  when looking at trilinos#6158, I saw that, when creating a Vector from a MultiVector, we created subviews for comm buffers, but did not store them. This commit stores them.  It also offsets the buffers by the vector j requested from the MultiVector.
jmgate pushed a commit to tcad-charon/Trilinos that referenced this issue Jan 22, 2020
Merge 'trilinos/Trilinos:develop' (c610d9d) into 'tcad-charon/Trilino…
38181ab 
…s:develop' (d17489d).

* trilinos-develop: (29 commits)
  Tpetra: Correcting unecessary extraction of remotes for A matrix in Jacobi
  Tpetra: MMM Modifications to avoid remotemap construction in serial
  Ifpack2: pass correct "symmetric" flag to MTSGS setup
  MueLu: Adding timer granularity to SaP
  Fix const_cast in Tpetra, Amesos and Ifpack2
  SEACAS: Remove locale setting
  Teuchos utils: fix another stacked timer plotting bug
  tpetra:  In trilinos#6598, @mhoemmen recommended this change of offset
  Tpetra::CrsMatrix: Add Kokkos kernel labels; expose debug code
  Tpetra::CrsMatrix: Remove values2D_
  Tpetra::CrsGraph: Remove gblInds2D_
  Tpetra::CrsGraph: Remove lclInds2D_
  Tpetra::CrsMatrix: Remove unused method allocateValues2D
  Tpetra: Use verbosePrintCountThreshold in copyOffsets
  Tpetra::Details::Behavior: Add longRowMinNumEntries
  Tpetra::Details::Behavior: Factor out size_t reading
  SEACAS: go back to lib:fmt 6.0.0 until fix issue on vortex xl/cuda build
  Disable Teko_testdriver_tpetra_MPI_4 in all atdm 'waterman' builds (trilinos#6463)
  zoltan2:  add missing include file for non-ETI builds
  Tpetra: Missed ifdef guard
  ...
jmgate pushed a commit to tcad-charon/Trilinos that referenced this issue Jan 22, 2020
Merge 'trilinos/Trilinos:develop' (c610d9d) into 'tcad-charon/Trilino…
aeb20a9 
…s:develop' (d17489d).

* trilinos-develop: (29 commits)
  Tpetra: Correcting unecessary extraction of remotes for A matrix in Jacobi
  Tpetra: MMM Modifications to avoid remotemap construction in serial
  Ifpack2: pass correct "symmetric" flag to MTSGS setup
  MueLu: Adding timer granularity to SaP
  Fix const_cast in Tpetra, Amesos and Ifpack2
  SEACAS: Remove locale setting
  Teuchos utils: fix another stacked timer plotting bug
  tpetra:  In trilinos#6598, @mhoemmen recommended this change of offset
  Tpetra::CrsMatrix: Add Kokkos kernel labels; expose debug code
  Tpetra::CrsMatrix: Remove values2D_
  Tpetra::CrsGraph: Remove gblInds2D_
  Tpetra::CrsGraph: Remove lclInds2D_
  Tpetra::CrsMatrix: Remove unused method allocateValues2D
  Tpetra: Use verbosePrintCountThreshold in copyOffsets
  Tpetra::Details::Behavior: Add longRowMinNumEntries
  Tpetra::Details::Behavior: Factor out size_t reading
  SEACAS: go back to lib:fmt 6.0.0 until fix issue on vortex xl/cuda build
  Disable Teko_testdriver_tpetra_MPI_4 in all atdm 'waterman' builds (trilinos#6463)
  zoltan2:  add missing include file for non-ETI builds
  Tpetra: Missed ifdef guard
  ...
@github-actions
Copy link

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

@github-actions github-actions bot added the MARKED_FOR_CLOSURE Issue or PR is marked for auto-closure by the GitHub Actions bot. label Nov 10, 2021
@github-actions
Copy link

This issue was closed due to inactivity for 395 days.

@github-actions github-actions bot added the CLOSED_DUE_TO_INACTIVITY Issue or PR has been closed by the GitHub Actions bot due to inactivity. label Dec 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
ATDM Sev: Blocker Problems that make Trilinos unfit to be adopted by one or more ATDM APPs client: ATDM Any issue primarily impacting the ATDM project client: EMPIRE All issues that most directly target the ATDM EMPIRE code CLOSED_DUE_TO_INACTIVITY Issue or PR has been closed by the GitHub Actions bot due to inactivity. Disabled Tests Issue has been partially addressed by disabling *all* of the failing tests related to the issue MARKED_FOR_CLOSURE Issue or PR is marked for auto-closure by the GitHub Actions bot. PA: Linear Solvers Issues that fall under the Trilinos Linear Solvers Product Area pkg: Teko type: bug The primary issue is a bug in Trilinos code or tests
Projects
None yet
Milestone
Keep promoted "ATDM" builds of Trilin...
Development

No branches or pull requests
2 participants
@bartlettroscoe @eric-c-cyr