Kokkos: Not rebuilding libkokkoscore correctly on rebuilds?

[bartlettroscoe]

As part of debugging a SPARC Trilinos Integration Build error (see SPAR-767), it was discovered that libkokkoscore was not getting correctly rebuilt after the Kokkos 2.99 promotion after the merging of the PR #6671

This issue is to try to characterize the problem and see if it can be reproduced so it can be fixed. We really need to maintain the ability to rebuild Trilinos reliability since it saves a massive amount of CPU cycles in the ATDM Trilinos builds and saves a lot developer time if they can just rebuild and not have to blow away their build directories all the time.

[ndellingwood]

Adding @jjwilke

[bartlettroscoe]

I will post back where what I find and try to characterize this problem. I have build directories that have been rebuilding for months that show this problem. When I blew away the build directory and let it build from scratch, the correct libkokkoscore.a got built (see SPAR-767). Hopefully I can simulate what happened in a new source and build directory so we can reproduce this.

Stay tuned ...

[ndellingwood]

Linking a couple kokkos issues that may help investigating:

kokkos/kokkos#1902
PR (cmake overhaul) kokkos/kokkos#2104
PR (remove KOKKOS_SEPARATE_LIBS) kokkos/kokkos#2447

[bartlettroscoe]

kokkos/kokkos#1902

That is not the problem. When you build Trilinos from scratch you get 'libkokkoscore.a'.

kokkos/kokkos#2104

Of course that big refactoring that is what I am suspecting is the cause here. I will see.

[bartlettroscoe]

FYI: I have been able to create a local reproducer. I will post a detailed comment with instructions on how to run the simulation and prove that the Kokkos_Serial.cpp.o file is not getting rebuilt correctly (but when you build from scratch it does). Using ninja debugging aids it should hopefully be pretty easy to figure out what the problem is and how to fix this so this does not happen in the future with rebuilds of Trilinos.

[bartlettroscoe]

@trilinos/kokkos,

FYI: More evidence in #7195 that Kokkos is not rebuilding correctly after the big CMake refactor.

I will add detailed instructions on how to reproduce the problem.

[jjwilke]

Do we have reproducer instructions yet?

[bartlettroscoe]

Do we have reproducer instructions yet?

I will get this updated when I can. Just have a bunch of other stuff right now.

[bartlettroscoe]

Another example of rebuild problems in #7341.

I have to deal with #6840 then I will put in the reproducability instructions for the case I know it is repeatable on every platform.

[bartlettroscoe]

We have not seen this problem in a long time. Perhaps it has been fixed? I could still provide reproducibility instructions but not sure what the point would be at this point.

[bartlettroscoe]

CC: @crtrott

I think another example of this problem is documented in #8638 (comment) (expand TEST DETAILS and look at section ride). Trying to rebuild the build Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-debug on 'ride' showed the build error:

/ascldap/users/rabartl/Trilinos.base/BUILDS/RIDE/CTEST_S/Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-debug/SRC_AND_BUILD/Trilinos/packages/kokkos/core/src/impl/Kokkos_Core.cpp(330): error: identifier "KOKKOS_VERSION" is undefined

1 error detected in the compilation of "/tmp/tmpxft_0000912b_00000000-6_Kokkos_Core.cpp1.ii".

I had to delete the build Trilinos-atdm-white-ride-cuda-9.2-gnu-7.2.0-debug and configure and build from scratch and the Kokkos-related errors went away. That is another pretty strong sign that there may be a problem with the Kokkos CMakeLists.txt files things don't always rebuild correctly, even when you run rm -r CMake* and configure from scratch. There is likely some bad logic there.

[bartlettroscoe]

FYI: I just his this error after a local rebuild attempt of Trilinos:

[32/27570] Building CXX object packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Core.cpp.o
FAILED: packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Core.cpp.o 
/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/5.3.0/openmpi/1.10.1/bin/mpicxx  -Dkokkoscore_EXPORTS -I. -Ipackages/kokkos/core/src -I/home/rabartl/Trilinos.base/Trilinos/packages/kokkos/core/src -Ipackages/kokkos -isystem /projects/sems/install/rhel7-x86_64/sems/compiler/gcc/5.3.0/openmpi/1.10.1/include -pedantic -Wall -Wno-long-long -Wwrite-strings  -fopenmp  -O3 -DNDEBUG -fPIC   -std=c++14 -MD -MT packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Core.cpp.o -MF packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Core.cpp.o.d -o packages/kokkos/core/src/CMakeFiles/kokkoscore.dir/impl/Kokkos_Core.cpp.o -c /home/rabartl/Trilinos.base/Trilinos/packages/kokkos/core/src/impl/Kokkos_Core.cpp
/home/rabartl/Trilinos.base/Trilinos/packages/kokkos/core/src/impl/Kokkos_Core.cpp: In function ‘void Kokkos::Impl::{anonymous}::pre_initialize_internal(const Kokkos::InitArguments&)’:
/home/rabartl/Trilinos.base/Trilinos/packages/kokkos/core/src/impl/Kokkos_Core.cpp:330:58: error: ‘KOKKOS_VERSION’ was not declared in this scope
                                  version_string_from_int(KOKKOS_VERSION));

I had to blow away the entire build dir and configure from scratch to make this error go away. (Just deleting the packages/ directory was not enough which is interesting.)

After the big Kokkos CMake build system refactoring 1.5 years ago something went wrong that has broken rebuilds in some cases. Some dependency rule must not have been set correctly as part of that refactoring.

At some point, someone needs to take some time to debug what is happening and fix this. Before this, I can't remember the last time I had a build error with Trilinos doing a rebuild after reconfiguring from scratch (i.e. rm -r CMake* ; cmake ...). So much time has passed that it may be hard to come up with a rock solid reproducer now.

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

[github-actions]

This issue was closed due to inactivity for 395 days.