PanzerAdaptersSTK_[Mixed]CurlLaplacianExample tests failing in Trilinos-atdm-waterman-cuda-9.2-release-debug build #3939

fryeguy52 · 2018-11-27T18:17:41Z

CC: @trilinos/panzer, @mperego (Trilinos Discretizations Product Lead), @bartlettroscoe, @fryeguy52

Next Action Status

With the merge of PR #4079 to 'develop' on 12/19/2018, these tests should now be disabled in this build Trilinos-atdm-waterman-cuda-9.2-release-debug. All tests that should be disabled were disabled on 12/19/2018 and all of the Panzer tests in this build passed on 12/19/2018 and 12/20/2018.

Description

As shown in this query the tests:

are failing in the build:

Trilinos-atdm-waterman-cuda-9.2-release-debug

Test names above link to the test output

Current Status on CDash

The current status of these tests/builds for the current testing day can be found here

Steps to Reproduce

One should be able to reproduce this failure on waterman as described in:

https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md

More specifically, the commands given for waterman are provided at:

https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#waterman

The exact commands to reproduce this issue should be:

$ cd <some_build_dir>/
$ source $TRILINOS_DIR/cmake/std/atdm/load-env.sh Trilinos-atdm-waterman-cuda-9.2-release-debug
$ cmake \
 -GNinja \
 -DTrilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake \
 -DTrilinos_ENABLE_TESTS=ON -DTrilinos_ENABLE_Panzer=ON \
 $TRILINOS_DIR
$ make NP=16
$ bsub -x -Is -n 20 ctest -j20

The text was updated successfully, but these errors were encountered:

bartlettroscoe · 2018-11-27T19:12:12Z

@mperego and @rppawlo, as of now, these are the only failures stopping us from promoting the build Trilinos-atdm-waterman-cuda-9.2-release-debug.

Also note that this is the only ATDM CUDA build of Trilinos that enables optimized compiler options and turns on some Kokkos debug-mode runtime checking. We are seeing failures like shown here:

terminate called after throwing an instance of 'std::length_error'
  what():  vector::_M_fill_insert
[waterman2:06914] *** Process received signal ***
[waterman2:06914] Signal: Aborted (6)
[waterman2:06914] Signal code:  (-6)

and here showing:

:0: : block: [0,0,0], thread: [0,192,0] Assertion `View bounds error of view FixedHashTable::pairs` failed.
:0: : block: [0,0,0], thread: [0,193,0] Assertion `View bounds error of view FixedHashTable::pairs` failed.
:0: : block: [0,0,0], thread: [0,194,0] Assertion `View bounds error of view FixedHashTable::pairs` failed.
:0: : block: [0,0,0], thread: [0,195,0] Assertion `View bounds error of view FixedHashTable::pairs` failed.
...

and here showing:

:0: : block: [0,0,0], thread: [0,225,0] Assertion `View bounds error of view FixedHashTable::pairs` failed.
:0: : block: [0,0,0], thread: [0,229,0] Assertion `View bounds error of view FixedHashTable::pairs` failed.
:0: : block: [2,0,0], thread: [0,226,0] Assertion `View bounds error of view FixedHashTable::pairs` failed.
:0: : block: [2,0,0], thread: [0,228,0] Assertion `View bounds error of view FixedHashTable::pairs` failed.
...

and here showing:

TEST_3

Running: "python" "/home/jenkins/waterman/workspace/Trilinos-atdm-waterman-cuda-9.2-release-debug/SRC_AND_BUILD/Trilinos/packages/panzer/adapters-stk/example/MixedCurlLaplacianExample/convergence_rate.py" "2" "MPE-ConvTest-Tri-2-" "4" "8" "16"

--------------------------------------------------------------------------------

opening "MPE-ConvTest-Tri-2-04"
opening "MPE-ConvTest-Tri-2-08"
MPE-ConvTest-Tri-2-08: Convergence order of 1.833883 is within 10 percent of 2.000000: PASSED
opening "MPE-ConvTest-Tri-2-16"
MPE-ConvTest-Tri-2-16: Convergence order of -8.615104 is not within 10 percent of 2.000000: FAILED
Test Failed: 
Exception

So, several different types of failures.

I will try setting up a 'release-debug' build on 'white'/'ride' to see what happens.

rppawlo · 2018-11-27T19:14:27Z

So did this just start happening recently? I thought we had things cleaned up. There was a push to panzer this morning but I'm guessing these tests are from last night?

bartlettroscoe · 2018-11-27T19:39:10Z

So did this just start happening recently? I thought we had things cleaned up. There was a push to panzer this morning but I'm guessing these tests are from last night?

@rppawlo, no, these have been failing for several days as shown in the table below. (Click the numbers and see the history.)

As I noted above, this is a bit of a special build in that it uses optimized compiler options (CMAKE_BUILD_TYPE=RELEASE) but it turns on some runtime debug-mode checking in Kokkos and other packages. The pure optimized builds don't turn on any runtime debug-mode checking.

Site	Build Name	Test Name	Status	Details	Consecutive Days Nopass	Nopass last 30 Days	Pass last 30 Days
waterman	Trilinos-atdm-waterman-cuda-9.2-release-debug	PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-2	Failed	Completed (Failed)	4	4	22
waterman	Trilinos-atdm-waterman-cuda-9.2-release-debug	PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-3	Failed	Completed (Failed)	2	3	1
waterman	Trilinos-atdm-waterman-cuda-9.2-release-debug	PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1	Failed	Completed (Failed)	4	4	0
waterman	Trilinos-atdm-waterman-cuda-9.2-release-debug	PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-2	Failed	Completed (Failed)	3	3	1
waterman	Trilinos-atdm-waterman-cuda-9.2-release-debug	PanzerAdaptersSTK_MixedCurlLaplacianExample	Failed	Completed (Failed)	4	4	0

rppawlo · 2018-11-28T20:32:31Z

@egphill - this looks like the new hvol stuff from commit bf5b85c has broken the cuda builds. Would you mind taking a look?

rppawlo · 2018-11-28T21:17:49Z

Just had a quick talk with @egphill . We think the best thing to do is disable these failing tests for all cuda builds for now. The same tests are passing for lower order and it looks like high order has a memory issue. We will try to address soon.

bartlettroscoe · 2018-11-28T21:24:16Z

Just had a quick talk with @egphill . We think the best thing to do is disable these failing tests for all cuda builds for now. The same tests are passing for lower order and it looks like high order has a memory issue. We will try to address soon.

@rppawlo, okay. So can you disable these on your end or do you want us to disable these for the ATDM CUDA builds as per:

https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#disable-a-test-for-builds-on-all-platforms

Or you can use that approach and post a PR.

bartlettroscoe · 2018-11-29T13:14:41Z

@rppawlo,

I talked with @fryeguy52 and he is going to create a PR to disable these tests in all CUDA builds for now using the approach documented at:

https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/README.md#disable-a-test-for-builds-on-all-platforms

rppawlo · 2018-11-29T14:18:44Z

Thanks @bartlettroscoe and @fryeguy52 !

issue trilinos#3939 Disabled: PanzerAdaptersSTK_MixedCurlLaplacianExample PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1 PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-2 PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-3 PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-2

bartlettroscoe · 2018-12-03T20:55:43Z

PR #3962 was merged to 'develop' on 11/29/2018 that should disable these tests in all CUDA builds. But due to TRIL-237, we have not gotten any results from CDash since 11/27/2018. But, we can see from other builds on 11/30/2018 on CDash like here that show these four tests as missing:

PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-2 | Missing | | |
PanzerAdaptersSTK_MixedCurlLaplacianExample | Missing | | |
PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-3 | Missing | | |
PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1 | Missing | | |
PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-2 | Missing | |

Therefore, I think we can assume these tests will be disabled in this CUDA build on waterman as well once we get that back running again.

Therefore, I think we can add the "Disabled Tests" label.

@rppawlo, but I would assume that we should not close this issue because you will want to fix these tests and re-enable them?

rppawlo · 2018-12-03T21:04:01Z

yes - but there's no rush (or manpower) at the moment. Do you want to take it off the ATDM board?

bartlettroscoe · 2018-12-03T22:28:01Z

@rppawlo said:

Do you want to take it off the ATDM board?

I added the "Disabled Tests" label so this issue has disappeared from the links in the "Unresolved ATDM Trilinos GitHub Issues in your Trilinos Product Area" email that goes out every Monday morning. You can see the full set of ATDM Trilinos issues that have been addressed by disabling all of the tests related to that issue and have the label "Disabled Tests" in the link Only "Disabled Tests" at the ATDM Trilinos Wiki Page.

bartlettroscoe · 2018-12-03T22:28:58Z

@rppawlo, is the fact these tests are disabled mean that functionality used by EMPIRE is not being tested?

rppawlo · 2018-12-04T13:27:56Z

This was done for empire - the commit that added these tests was actually done by an empire developer. However, it is experimental and they are just evaluating at this point. They don't plan to use on the gpu any time soon. So disabling for gpu builds is fine. If this pans out and they decide to use in production empire, then we will make fixing for gpu a high priority.

bartlettroscoe · 2018-12-04T14:18:46Z

This was done for empire - the commit that added these tests was actually done by an empire developer. However, it is experimental and they are just evaluating at this point. They don't plan to use on the gpu any time soon. So disabling for gpu builds is fine. If this pans out and they decide to use in production empire, then we will make fixing for gpu a high priority.

@rppawlo, thanks for the explanation. Based on this I added the "ATDM Nonblocker" label as per ATDM Severity/Criticality Labels.

…ebug-pt build (trilinos#2464, trilinos#3633) We really need to switch most of these 'debug' builds to 'release-debug' builds (see trilinos#3633). Also, the Trilinos CUDA PR build really needs to be a cuda-9.2-release-debug build since that runs more tests and catches more issues than either a cuda-9.2-opt or cuda-9.2-debug build (see trilinos#3939).

issue trilinos#3939 Disabled: PanzerAdaptersSTK_MixedCurlLaplacianExample PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1 PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-2 PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-3 PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-2

bartlettroscoe · 2018-12-17T14:34:26Z

@fryeguy52 - yes - please disable.

@rppawlo, for all CUDA builds like we did the others or just for this CUDA build?

rppawlo · 2018-12-18T12:41:50Z

all cuda builds is fine for now.

bartlettroscoe · 2018-12-18T17:45:21Z

@rppawlo said:

all cuda builds is fine for now.

@fryeguy52, I think you need to disable all of the Panzer tests in all CUDA builds on all platforms shown in this query.

rppawlo · 2018-12-18T17:56:00Z

you could just disable in debug-release for now and leave opt-release enabled. The failures look to be triggered from debug checks. I'm hoping to get to this before the break.

bartlettroscoe · 2018-12-18T18:26:25Z

you could just disable in debug-release for now and leave opt-release enabled. The failures look to be triggered from debug checks. I'm hoping to get to this before the break.

@rppawlo, does the mean that the code is fine and the debug checks are defective or does it mean that the code has a defect but passes anyway when not running the debug checks?

rppawlo · 2018-12-18T18:48:33Z

If I could answer that, the fix would be done :) Hope to get to this soon.

bartlettroscoe · 2018-12-18T18:51:04Z

If I could answer that, the fix would be done :) Hope to get to this soon.

Indeed :-)

@fryeguy52, for now please disable the new failing tests shown in this query just for this one 'release-debug' build.

see issue trilinos#3939 This disables tests: * PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-1 * PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-3 * PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4 * PanzerAdaptersSTK_CurlLaplacianMultiblockExample-ConvTest-Quad-Order-1 * PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-1 * PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-2 * PanzerAdaptersSTK_MixedCurlLaplacianMultiblockExample-ConvTest-Quad-Order-1 in the build `Trilinos-atdm-waterman-cuda-9.2-release-debug`

This moves some panzer test disables that were effecting all cuda builds to only effect the waterman release-debug build issue trilinos#3939

see issue trilinos#3939 This disables tests: * PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-1 * PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-3 * PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4 * PanzerAdaptersSTK_CurlLaplacianMultiblockExample-ConvTest-Quad-Order-1 * PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-1 * PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-2 * PanzerAdaptersSTK_MixedCurlLaplacianMultiblockExample-ConvTest-Quad-Order-1 in the build `Trilinos-atdm-waterman-cuda-9.2-release-debug` This also moves some panzer test disables that were effecting all cuda builds to only effect the waterman release-debug build

bartlettroscoe · 2018-12-19T14:09:16Z

With the merge of PR #4079 to 'develop' on 12/19/2018, these tests should now be disabled in this build Trilinos-atdm-waterman-cuda-9.2-release-debug. Next: Confirm these tests are disabled in this build on 12/20/2018 ...

Actually, since we are still running the ATDM Trilinos builds against raw 'develop', the build Trilinos-atdm-waterman-cuda-9.2-release-debug running right now should show these tests disabled. We will wait to see that these tests are indeed disabled and all of the Panzer tests pass in this build ...

bartlettroscoe · 2018-12-21T02:43:25Z

The Panzer tests in the build Trilinos-atdm-waterman-cuda-9.2-release-debug all passed over the last two days with results for today show here. Many of the tests were shown newly missing yesterday as shown here:

PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-1 | Missing
PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-3 | Missing
PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-4 | Missing
PanzerAdaptersSTK_CurlLaplacianMultiblockExample-ConvTest-Quad-Order-1 | Missing
PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-1 | Missing
PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-2 | Missing
PanzerAdaptersSTK_MixedCurlLaplacianMultiblockExample-ConvTest-Quad-Order-1 | Missing

These were disabled along with the tests already disabled shown in the table below.

Since all of the Panzer tests have passed for the last two days with all of these disables, I have added the "Disabled Tests" label to get this off of our main list of ATDM Trilinos bug issues. But I will leave this open as a reminder that these still need to be fixed.

Tests with issue trackers Missing: twim=5 (On 2018-12-20)

Site	Build Name	Test Name	Status	Details	Consecutive Missing Days	Non-pass Last 30 Days	Pass Last 30 Days	Tracker
waterman	Trilinos-atdm-waterman-cuda-9.2-release-debug	PanzerAdaptersSTK_CurlLaplacianExample-ConvTest-Quad-Order-2	Missing	Missing	23	5	1	#3939
waterman	Trilinos-atdm-waterman-cuda-9.2-release-debug	PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-3	Missing	Missing	23	3	2	#3939
waterman	Trilinos-atdm-waterman-cuda-9.2-release-debug	PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1	Missing	Missing	23	5	0	#3939
waterman	Trilinos-atdm-waterman-cuda-9.2-release-debug	PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-2	Missing	Missing	23	4	1	#3939
waterman	Trilinos-atdm-waterman-cuda-9.2-release-debug	PanzerAdaptersSTK_MixedCurlLaplacianExample	Missing	Missing	23	4	1	#3939

rppawlo · 2018-12-21T15:49:21Z

So I spent some time on this and will need some help from the kokkos or testbeds teams. I can run the tests manually and they all consistently pass. I can run the tests with ctest -j1 and they consistently all pass. The only way I can trigger the failure is to get multiple executables running at the same time by using ctest -j16. This seems like multiple executables are corrupting memory on the cuda. @crtrott @nmhamster

bash-4.2$ ctest
Test project /ascldap/users/rppawlo/Trilinos/BUILD_WATERMAN_3939/packages/panzer/adapters-stk/example/MixedCurlLaplacianExample
    Start 1: PanzerAdaptersSTK_MixedCurlLaplacianExample
1/7 Test #1: PanzerAdaptersSTK_MixedCurlLaplacianExample ...................................   Passed   13.18 sec
    Start 2: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-1
2/7 Test #2: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-1 .............   Passed   25.46 sec
    Start 3: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-2
3/7 Test #3: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-2 .............   Passed   42.03 sec
    Start 4: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-3
4/7 Test #4: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-3 .............   Passed   97.54 sec
    Start 5: PanzerAdaptersSTK_MixedCurlLaplacianMultiblockExample-ConvTest-Quad-Order-1
5/7 Test #5: PanzerAdaptersSTK_MixedCurlLaplacianMultiblockExample-ConvTest-Quad-Order-1 ...   Passed   21.86 sec
    Start 6: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1
6/7 Test #6: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1 ..............   Passed  183.65 sec
    Start 7: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-2
7/7 Test #7: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-2 ..............   Passed   86.17 sec

100% tests passed, 0 tests failed out of 7

Label Time Summary:
Panzer    = 1879.60 sec*proc (7 tests)

Total Test time (real) = 469.99 sec
bash-4.2$ ctest
Test project /ascldap/users/rppawlo/Trilinos/BUILD_WATERMAN_3939/packages/panzer/adapters-stk/example/MixedCurlLaplacianExample
    Start 1: PanzerAdaptersSTK_MixedCurlLaplacianExample
1/7 Test #1: PanzerAdaptersSTK_MixedCurlLaplacianExample ...................................   Passed   14.15 sec
    Start 2: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-1
2/7 Test #2: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-1 .............   Passed   25.35 sec
    Start 3: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-2
3/7 Test #3: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-2 .............   Passed   42.78 sec
    Start 4: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-3
4/7 Test #4: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-3 .............   Passed   96.91 sec
    Start 5: PanzerAdaptersSTK_MixedCurlLaplacianMultiblockExample-ConvTest-Quad-Order-1
5/7 Test #5: PanzerAdaptersSTK_MixedCurlLaplacianMultiblockExample-ConvTest-Quad-Order-1 ...   Passed   21.84 sec
    Start 6: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1
6/7 Test #6: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1 ..............   Passed  184.42 sec
    Start 7: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-2
7/7 Test #7: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-2 ..............   Passed   86.06 sec

100% tests passed, 0 tests failed out of 7

Label Time Summary:
Panzer    = 1886.08 sec*proc (7 tests)

Total Test time (real) = 471.60 sec
bash-4.2$ 
bash-4.2$ ctest -j8
Test project /ascldap/users/rppawlo/Trilinos/BUILD_WATERMAN_3939/packages/panzer/adapters-stk/example/MixedCurlLaplacianExample
    Start 6: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1
    Start 4: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-3
1/7 Test #4: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-3 .............   Passed  199.11 sec
    Start 7: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-2
2/7 Test #6: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1 ..............***Failed  Required regular expression not found.Regex=[OVERALL FINAL RESULT: TEST PASSED .PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1.
]265.64 sec
    Start 3: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-2
3/7 Test #3: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-2 .............   Passed   68.89 sec
    Start 2: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-1
4/7 Test #7: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-2 ..............   Passed  159.49 sec
    Start 5: PanzerAdaptersSTK_MixedCurlLaplacianMultiblockExample-ConvTest-Quad-Order-1
5/7 Test #2: PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Quad-Order-1 .............   Passed   31.01 sec
    Start 1: PanzerAdaptersSTK_MixedCurlLaplacianExample
6/7 Test #1: PanzerAdaptersSTK_MixedCurlLaplacianExample ...................................***Failed  Required regular expression not found.Regex=[OVERALL FINAL RESULT: TEST PASSED .PanzerAdaptersSTK_MixedCurlLaplacianExample.
] 16.66 sec
7/7 Test #5: PanzerAdaptersSTK_MixedCurlLaplacianMultiblockExample-ConvTest-Quad-Order-1 ...***Failed  Required regular expression not found.Regex=[OVERALL FINAL RESULT: TEST PASSED .PanzerAdaptersSTK_MixedCurlLaplacianMultiblockExample-ConvTest-Quad-Order-1.
] 25.24 sec

57% tests passed, 3 tests failed out of 7

Label Time Summary:
Panzer    = 3064.18 sec*proc (7 tests)

Total Test time (real) = 383.92 sec

The following tests FAILED:
	  1 - PanzerAdaptersSTK_MixedCurlLaplacianExample (Failed)
	  5 - PanzerAdaptersSTK_MixedCurlLaplacianMultiblockExample-ConvTest-Quad-Order-1 (Failed)
	  6 - PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1 (Failed)
Errors while running CTest
bash-4.2$

rppawlo · 2018-12-21T16:45:21Z

Should also comment that cuda memcheck comes back clean when I run manually.

bartlettroscoe · 2018-12-22T19:45:49Z

@rppawlo, this might be a good use case to experiment with the improved binding approach being discussed in:

https://sems-atlassian-son.sandia.gov/jira/browse/TRIL-231?

In particular, see this comment. That looks very promising.

…s:develop' (7db7806). * trilinos-develop: (23 commits) Fix cmake-file error in stk_balance that was making the m2n exe be a test. tpetra: minor fix; return the values Fix incorrect line length in copy_string change Automatic snapshot commit from seacas at f9bf59a SEACAS: cgns - support self-looping models Disable failing ROL test already known to fail in CUA builds (trilinos#3543) Disable known failing Panzer tests (trilinos#3632) Small formatting change to comment (trilinos#3939) Enable SPARC TPLs and packages on 'waterman' (ATDV-151) ShyLU/FROSch: Correct use of booleans for interface components Don't allow Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt to run on 'ride7' (ATDV-155) tpetra: minor additional deprecations trilinos#4839 MiniEM: Fix discrete gradient tpetra: changes to address Mark's comments on trilinos#4839 Xpetra: MueLu: fix issue 4038 ShyLU/FROSch: Use insertGlobalValues instead of insertLocalValues for GlobalCoarseMatrix stokhos: fix compilation error due to tpetra deprecation changes Thyra: fixed compilation error due to deprecation changes tpetra: More deprecations of function arguments involving Node. create*MapWithNode generate_miniFM_* Tpetra: removing Node from argument lists of functions Completed MatrixMarket_Tpetra functions (readSparse, readDense, etc.) Also removed a few compiler warnings reported in clang ...

…s:develop' (7db7806). * trilinos-develop: (30 commits) Fix cmake-file error in stk_balance that was making the m2n exe be a test. Tpetra: Global Ordinal validation tpetra: minor fix; return the values Fix incorrect line length in copy_string change Tpetra: Moved GORDS logic to right file this time, really. Tpetra: GORDS Deprecation Cleanup Tpetra: Relocated # GORDS validation logic to packages/tpetra/core/CMakeLists.txt Tpetra: clean up deprecation WIP tags Tpetra: Add deprecations for global ordinal types Automatic snapshot commit from seacas at f9bf59a SEACAS: cgns - support self-looping models Disable failing ROL test already known to fail in CUA builds (trilinos#3543) Disable known failing Panzer tests (trilinos#3632) Small formatting change to comment (trilinos#3939) Enable SPARC TPLs and packages on 'waterman' (ATDV-151) ShyLU/FROSch: Correct use of booleans for interface components Don't allow Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-rdc-release-debug-pt to run on 'ride7' (ATDV-155) tpetra: minor additional deprecations trilinos#4839 Ifpack2 - fix issue 4858 MiniEM: Fix discrete gradient ...

github-actions · 2021-08-18T12:25:21Z

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

github-actions · 2021-09-18T12:23:36Z

This issue was closed due to inactivity for 395 days.

fryeguy52 added type: bug The primary issue is a bug in Trilinos code or tests pkg: Panzer client: ATDM Any issue primarily impacting the ATDM project labels Nov 27, 2018

fryeguy52 added this to the Keep promoted "ATDM" builds of Trilinos clean milestone Nov 27, 2018

bartlettroscoe changed the title ~~PanzerAdaptersSTK_[Mixed]CurlLaplacianExample tests failing in ATDM waterman build~~ PanzerAdaptersSTK_[Mixed]CurlLaplacianExample tests failing in ATDM waterman cuda-9 2-release-debug build Nov 28, 2018

bartlettroscoe changed the title ~~PanzerAdaptersSTK_[Mixed]CurlLaplacianExample tests failing in ATDM waterman cuda-9 2-release-debug build~~ PanzerAdaptersSTK_[Mixed]CurlLaplacianExample tests failing in ATDM waterman cuda-9.2-release-debug build Nov 28, 2018

bartlettroscoe changed the title ~~PanzerAdaptersSTK_[Mixed]CurlLaplacianExample tests failing in ATDM waterman cuda-9.2-release-debug build~~ PanzerAdaptersSTK_[Mixed]CurlLaplacianExample tests failing in Trilinos-atdm-waterman-cuda-9.2-release-debug build Nov 28, 2018

bartlettroscoe mentioned this issue Nov 28, 2018

PanzerAdaptersSTK_MixedCurlLaplacianExample-ConvTest-Tri-Order-1 timeout in Trilinos-atdm-white-ride-cuda-9.2-opt #3950

Closed

fryeguy52 mentioned this issue Nov 29, 2018

Atdm ride panzer disable #3962

Merged

bartlettroscoe added the PA: Discretizations Issues that fall under the Trilinos Discretizations Product Area label Nov 29, 2018

bartlettroscoe added the Disabled Tests Issue has been partially addressed by disabling *all* of the failing tests related to the issue label Dec 3, 2018

rppawlo self-assigned this Dec 3, 2018

bartlettroscoe added the ATDM Sev: Nonblocker Problems with Trilinos that should not block ATDM APPs from getting updates label Dec 4, 2018

fryeguy52 mentioned this issue Dec 18, 2018

2018-12-18 ATDM test disables #4079

Merged

bartlettroscoe added the stage: in review Primary work is completed and now is just waiting for human review and/or test feedback label Dec 19, 2018

bartlettroscoe added the Disabled Tests Issue has been partially addressed by disabling *all* of the failing tests related to the issue label Dec 21, 2018

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 10, 2019

Small formatting change to comment (trilinos#3939)

b0ad2c9

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 11, 2019

Small formatting change to comment (trilinos#3939)

e2c00ad

bartlettroscoe added a commit to bartlettroscoe/Trilinos that referenced this issue Apr 12, 2019

Small formatting change to comment (trilinos#3939)

406a8a5

github-actions bot added the MARKED_FOR_CLOSURE Issue or PR is marked for auto-closure by the GitHub Actions bot. label Aug 18, 2021

github-actions bot added the CLOSED_DUE_TO_INACTIVITY Issue or PR has been closed by the GitHub Actions bot due to inactivity. label Sep 18, 2021

github-actions bot closed this as completed Sep 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PanzerAdaptersSTK_[Mixed]CurlLaplacianExample tests failing in Trilinos-atdm-waterman-cuda-9.2-release-debug build #3939

PanzerAdaptersSTK_[Mixed]CurlLaplacianExample tests failing in Trilinos-atdm-waterman-cuda-9.2-release-debug build #3939

fryeguy52 commented Nov 27, 2018 •

edited by bartlettroscoe

Loading

bartlettroscoe commented Nov 27, 2018

rppawlo commented Nov 27, 2018

bartlettroscoe commented Nov 27, 2018

rppawlo commented Nov 28, 2018

rppawlo commented Nov 28, 2018

bartlettroscoe commented Nov 28, 2018

bartlettroscoe commented Nov 29, 2018

rppawlo commented Nov 29, 2018

bartlettroscoe commented Dec 3, 2018

rppawlo commented Dec 3, 2018

bartlettroscoe commented Dec 3, 2018

bartlettroscoe commented Dec 3, 2018

rppawlo commented Dec 4, 2018

bartlettroscoe commented Dec 4, 2018

bartlettroscoe commented Dec 17, 2018

rppawlo commented Dec 18, 2018

bartlettroscoe commented Dec 18, 2018

rppawlo commented Dec 18, 2018

bartlettroscoe commented Dec 18, 2018

rppawlo commented Dec 18, 2018

bartlettroscoe commented Dec 18, 2018

bartlettroscoe commented Dec 19, 2018

bartlettroscoe commented Dec 21, 2018

rppawlo commented Dec 21, 2018

rppawlo commented Dec 21, 2018

bartlettroscoe commented Dec 22, 2018

github-actions bot commented Aug 18, 2021

github-actions bot commented Sep 18, 2021

PanzerAdaptersSTK_[Mixed]CurlLaplacianExample tests failing in Trilinos-atdm-waterman-cuda-9.2-release-debug build #3939

PanzerAdaptersSTK_[Mixed]CurlLaplacianExample tests failing in Trilinos-atdm-waterman-cuda-9.2-release-debug build #3939

Comments

fryeguy52 commented Nov 27, 2018 • edited by bartlettroscoe Loading

Next Action Status

Description

Current Status on CDash

Steps to Reproduce

bartlettroscoe commented Nov 27, 2018

rppawlo commented Nov 27, 2018

bartlettroscoe commented Nov 27, 2018

rppawlo commented Nov 28, 2018

rppawlo commented Nov 28, 2018

bartlettroscoe commented Nov 28, 2018

bartlettroscoe commented Nov 29, 2018

rppawlo commented Nov 29, 2018

bartlettroscoe commented Dec 3, 2018

rppawlo commented Dec 3, 2018

bartlettroscoe commented Dec 3, 2018

bartlettroscoe commented Dec 3, 2018

rppawlo commented Dec 4, 2018

bartlettroscoe commented Dec 4, 2018

bartlettroscoe commented Dec 17, 2018

rppawlo commented Dec 18, 2018

bartlettroscoe commented Dec 18, 2018

rppawlo commented Dec 18, 2018

bartlettroscoe commented Dec 18, 2018

rppawlo commented Dec 18, 2018

bartlettroscoe commented Dec 18, 2018

bartlettroscoe commented Dec 19, 2018

bartlettroscoe commented Dec 21, 2018

Tests with issue trackers Missing: twim=5 (On 2018-12-20)

rppawlo commented Dec 21, 2018

rppawlo commented Dec 21, 2018

bartlettroscoe commented Dec 22, 2018

github-actions bot commented Aug 18, 2021

github-actions bot commented Sep 18, 2021

fryeguy52 commented Nov 27, 2018 •

edited by bartlettroscoe

Loading