Panzer: CurlLaplacianExample-ConvTest randomly failing

[rppawlo]

On my workstation, this test occasionally fails. Probably a race condition on the output check.

[rppawlo]

adding @eric-c-cyr

[rppawlo]

Looks like it's a race condition in the parsing of output. Below is the test output:

opening "MPE-ConvTest-05"
Test Failed:
invalid literal for float(): 0.00970975ALL PASSED: EpetraALL PASSED: Epetra

---

[rppawlo]

We can't rely on screen output for checking test convergence when multiple MPI processes are writing at the same time. We need to change this to write to a file and have the convergence tester read that file. Since Panzer will be changed from ST to PT in Trilinos soon this needs to be fixed soon.

@eric-c-cyr @jmgate

[jmgate]

Oh good, I was seeing this test randomly fail too but could never reproduce the error when I tried so I thought I was just losing my mind :)

Forgive my ignorance, but what are ST and PT?

@trilinos/panzer

[mhoemmen]

@rppawlo wrote:

We can't rely on screen output for checking test convergence when multiple MPI processes are writing at the same time. We need to change this to write to a file and have the convergence tester read that file.

You may find Tpetra::Details::gathervPrint handy. It lives in tpetra/core/src/Tpetra_Details_gathervPrint.hpp. It takes a string on each MPI process in a communicator, and outputs the result of each in turn (deterministically) to an output stream only valid on Process 0 in the communicator. It's also memory scalable :-D

[mhoemmen]

@jmgate wrote:

Forgive my ignorance, but what are ST and PT?

PT ("primary tested") is a label for packages that Trilinos builds and tests by default. In the check-in test script, only PT packages get enabled, even if you changed something in a non-PT package. Historically, they may not depend on any TPLs other than MPI (optionally), the BLAS, and LAPACK.

ST ("secondary tested") is a label for packages that claim to require TPLs other than MPI, the BLAS, or LAPACK, but which Trilinos nevertheless wants to test regularly. If you want to test ST packages in the check-in test script, you must use --st-extra-builds or --extra-builds to identify those packages explicitly.

I wrote "claim to" above, because the ST label does not actually check the packages' TPL dependencies. Packages get to decide whether they want to stay ST or even EX ("experimental"). However, a PT package may not depend on TPLs other than those three.

[jmgate]

Thanks @mhoemmen. @rppawlo, when is the planned transition from ST to PT for Panzer?

[rppawlo]

not sure of the exact timeline. it is being tracked in #410 @bartlettroscoe has commented on it the past few days so it is being actively worked. @bartlettroscoe - do you know when you will be changing the PT code?

[rppawlo]

Also - in #410, PT is being expanded to include more TPLs than just MPI, BLAS and LAPACK.

[bartlettroscoe]

I just need to test on a few platforms and then it is ready to push. I will ask for review before first. I will send out an email.

[rppawlo]

Should be fixed now. Moving to "In Review". Will watch the test over the next few days to see if issues clear up.