Trilinos build failures using Sierra build process

[ajpowelsnl]

Hello,

Sierra-related Trilinos builds are failing on some engineering workstations (ews), and we have not been able to discern the cause. We have performed bake distributed builds with both gcc and intel compilers. I can provide complete logs. Errors are of this type:

make[2]: Leaving directory /sierra/dev/ajpowel/code_032619/objs/tpls/Trilinos/20c07de8996d4f55' [ 94%] Built target muelu make[1]: Leaving directory /sierra/dev/ajpowel/code_032619/objs/tpls/Trilinos/20c07de8996d4f55'
make: *** [all] Error 2
WARNING: Trilinos build failed!
Rebuilding Makefile (Option change(s): ['--installdir=/sierra/dev/ajpowel/code_032619/objs/tpls/trilinos_tpls/20c07de8996d4f55', '/tpl/trilinos//install-trilinos-tpls']->['--bin-dir=/sierra/dev/ajpowel/code_032619/bin'])
Building bjam...
Using dependency graph: /sierra/dev/ajpowel/code_032619/bakefiles/bakefile_e3b0c44298fc1c14_20c07de8996d4f55_deps
Using Trilinos out of: /sierra/dev/ajpowel/code_032619/objs/tpls/trilinos_tpls/20c07de8996d4f55
INFO: Changing version for trilinos from dev to external
INFO: Changing version for trilinos-kokkoscore from dev to external
INFO: Changing version for trilinos-kokkoscontainers from dev to external
INFO: Changing version for trilinos-kokkosalgorithms from dev to external
INFO: Changing version for trilinos-tpetraclassic from dev to external
INFO: Changing version for trilinos-tpetracore from dev to external
INFO: Changing version for trilinos-kokkoskernels from dev to external
INFO: Changing version for trilinos-tpetratsqr from dev to external
error: Unable to find file or target named
error: '/sierra/dev/ajpowel/code_032619/objs/tpls/trilinos_tpls/20c07de8996d4f55/lib/libkokkoscore.a'
error: referred from project at
error: '/sierra/dev/ajpowel/code_032619/TPLs_src/Nbtools/Trilinos/external/KokkosCore'

Bjam failed!

Many thanks!

Best,

AJP

@trilinos/

Expectations

Current Behavior

Motivation and Context

Definition of Done

Possible Solution

Steps to Reproduce

Your Environment

• Relevant repo SHA1s:
• Relevant configure flags or configure script:
• Operating system and version:
• Compiler and TPL versions:

Related Issues

• Blocks
• Is blocked by
• Follows
• Precedes
• Related to
• Part of
• Composed of

Additional Information

[jhux2]

@srajama1 Not sure who to notify for this.

[srajama1]

Adding the other product leads @jwillenbring @kddevin @rppawlo @mperego and Sierra developers @prwolfe @mhoemmen to see if some one has access to this environment.

[srajama1]

@ajpowelsnl I am trying to evaluate this so we can help you. Is this a new failure on a build/configuration that used to work or is this is a new setup ? The error is not very descriptive. Is there a verbose mode in bjam that you can use so we can see more output on where things are going wrong ?

[srajama1]

Adding @rrdrake as well.

[ajpowelsnl]

Hi there, Many thanks for your response. This is a new problem, apparently affecting only Mark Hamilton and me (on our EWS) for “bake” (default) distributed builds. I have complete log files, but I’m having trouble emailing them, but they’re being bounced b/c of line count, I think. How can I get them to you? From: Siva Rajamanickam <[email protected]> Reply-To: trilinos/Trilinos <[email protected]> Date: Wednesday, March 27, 2019 at 1:54 PM To: trilinos/Trilinos <[email protected]> Cc: Amy Jo Powell <[email protected]>, Mention <[email protected]> Subject: [EXTERNAL] Re: [trilinos/Trilinos] Trilinos build failures using Sierra build process (#4742) @ajpowelsnl<https://github.com/ajpowelsnl> I am trying to evaluate this so we can help you. Is this a new failure on a build/configuration that used to work or is this is a new setup ? The error is not very descriptive. Is there a verbose mode in bjam that you can use so we can see more output on where things are going wrong ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#4742 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AuuumcGRlN8sKn9cBtZvJS26ZsMzdYxzks5va8x4gaJpZM4cOeSw>.

[mhoemmen]

Sierra DevOps has been working on this for a day or two.

[srajama1]

@mhoemmen : Can we wait on the Trilinos side for Sierra DevOps to give us more info ? We could also see if a Trilinos framework person can help (I am assuming this is framework related), but I doubt they want to deal with bjam. @jwillenbring : What do you think ?

@ajpowelsnl : I see you are in SNL. You can e-mail it the logs to me srajama@...

[mhoemmen]

Correction: @ajpowelsnl and the other Sierra DevOps people have been working on this for a day or two :-) (sorry I didn't recognize your handle!).

[prwolfe]

Hmmm, this is short on details, but I just ran into an issue when using a recent master checkout of Trilinos.

Basically lines 2472 and 2478 in the file packages/ifpack2/src/Ifpack2_BlockTriDiContainer_impl.hpp are using a const instance of a class (Kokkos::TeamPolicy) to call a const function for a non-const instance. Being named a "set" function I tried removing the "const" from the call in Ifpack2_BlockTriDiContainer_impl.hpp and got a clean build.

The $6 question is how did this get through Trilinos testing and how do we fix that. This is a very simple issue which should have been caught early.

[srajama1]

@prwolfe : This issue is different. We took it offline from this github issue.

That said, I am surprised by the Ifpack2 issue. Is there a seprate issue for this. If not can you please post one ?

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

[mhoemmen]

revive

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

[github-actions]

This issue was closed due to inactivity for 395 days.