Tribits: make package_source creates tarballs with missing BuildStatsWrappers.cmake #11976
Comments
ndellingwood
added
type: bug
|
Adding @dmdunla , he initially reported this on project we work on |
|
@ndellingwood and @dmdunla, I don't think anyone has created a source tarball for Trilinos in many years and there is no automated testing for this, so I am not surprised this failed. With that being the case, I can see what the problem is and put in a fix and set up an automated test as part of the Trilinos test suite for this if that is okay with @sebrowne and @ccober6. Otherwise, if no automated testing is going to be set up and run to protect these feature, then Trilinos should announce that creating source tarballs is not supported. @sebrowne and @ccober6, what say you? Is Trilinos going to support creating source tarballs going forward? |
|
Thanks @bartlettroscoe ! |
@ndellingwood, I am well aware of the advantages. The TriBITS automatic support for trimming down the source tallball based on what packages are enabled (or not enabled for the tarball excludes) was extensively used in CASL VERA project (for removing sensitive packages for some reduced releases as well as reducing source size). The question is not if this feature is useful (it clearly is for some customers); the question is if Trilinos is willing to support it going forward. But supporting this is not that hard and I could write a basic set of tests to protect this that could run as part of Trilinos PR and nightly testing. (The test would create the source tarball for whatever set of Trilinos packages was enabled and would untar and try to configure the reduced Trilinos source as part of the test.) |
|
The issue is that the directory The fix is to guard this code based on the existence of the directory |
This commit was just to streamline local testing to address #11976.
This is also a TriBITS Trilinos package so if it is not enabled, then it gets excluded. I am not sure this is the right thing to do but for now, it fixes the immediate problem reported in #11976.
This is a temp fix the TriBITS that makes building from a reduced tarball work.
This is also a TriBITS Trilinos package so if it is not enabled, then it gets excluded. I am not sure this is the right thing to do but for now, it fixes the immediate problem reported in trilinos#11976.
This is a temp fix the TriBITS that makes building from a reduced tarball work.
|
@ndellingwood and @dmdunla, this should be fixed in PR #11977. The local testing I did seems to suggest this should work. But if you would like to test this for your exact use case before this PR gets merged, that would be greatly appreciated :-) |
|
This sounds like it is a useful feature, and the support cost appears to be low. I would be in favor in setting up testing for this and see how it goes. If it starts becoming a burden, we should reevaluate. @sebrowne what are your thoughts? |
|
Seems fine to me |
|
@ndellingwood, @sebrowne, @ccober6, I added a Tasks section above for the remaining work to put some reasonable checks in place for the ability to create and configure/build/test/install from reduced tarballs. But note that even though you can create a reduced tarball and configure from it, that does not guarantee that the enabled packages from the reduced source tree will build, pass the tests, and install successfully. The issue is that some Trilinos packages might have have hooks into other non-enabled packages (like grabbing tests files from another package) that may be missing from the untarred reduced source tree. The only way to know for sure would be to build, test, and install from the reduced tarball. (We did that on the CASL VERA project but it requires some more testing infrastructure to do that. It is actually not that hard to set up but that involves a lot more build and test time.) |
|
We configure first so that all of the dependencies are resolved. Then, when we call `make package_source` we (seem) to get everything that we need across the packages that we configured for. If there is a different approach we should use, please clarify. Thanks!
From: Roscoe A. Bartlett ***@***.***>
Date: Thursday, June 15, 2023 at 8:57 AM
To: trilinos/Trilinos ***@***.***>
Cc: Dunlavy, Daniel M ***@***.***>, Mention ***@***.***>
Subject: [EXTERNAL] Re: [trilinos/Trilinos] Tribits: make package_source creates tarballs with missing BuildStatsWrappers.cmake (Issue #11976)
@ndellingwood<https://github.com/ndellingwood>, @sebrowne<https://github.com/sebrowne>, @ccober6<https://github.com/ccober6>,
I added a Tasks section above<#11976 (comment)> for the remaining work to put some reasonable checks in place for the ability to create and configure/build/test/install from reduced tarballs.
But note that even though you can create a reduced tarball and configure from it, that does not guarantee that the enabled packages from the reduced source tree will build, pass the tests, and install successfully. The issue is that some Trilinos packages might have have hooks into other non-enabled packages (like grabbing tests files from another package) that may be missing from the untarred reduced source tree. The only way to know for sure would be to build, test, and install from the reduced tarball. (We did that on the CASL VERA project but it requires some more testing infrastructure to do that. It is actually not that hard to set up but that involves a lot more build and test time.)
—
Reply to this email directly, view it on GitHub<#11976 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHIY2IUAOWC6EL32OUYTMEDXLMPFJANCNFSM6AAAAAAZHBC7RA>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
|
That is it which is consistent with: Just make sure you follow that carefully (especially the part about running |
…-fix Automatically Merged using Trilinos Pull Request AutoTester PR Title: Fix reduced tarball configuration (#11976) PR Author: bartlettroscoe
…s:develop' (0f5e2d6). * trilinos-develop: TriBITS: Temp fix for configuring from reduced tarball (trilinos#11976) Make work with missing commonTools/build_stats (trilinos#11976) Add a spec for all packages + complex<float> Update MueLu_FactoryManager_def.hpp KokkosKernels: Don't list include for non-existant 'batched' build dir (trilinos#11966) MueLu: Enabling better dumps for Maxwell1 MueLu RefMaxwell: Accept "aggregation: phase3 avoid singletons" Tpetra: Carl's fixes Tpetra: Improving std::move()-ability of BlockMultiVector Tpetra: Improving std::move()-ability of BlockMultiVector
…s:develop' (0f5e2d6). * trilinos-develop: TriBITS: Temp fix for configuring from reduced tarball (trilinos#11976) Make work with missing commonTools/build_stats (trilinos#11976) Add a spec for all packages + complex<float> Update MueLu_FactoryManager_def.hpp KokkosKernels: Don't list include for non-existant 'batched' build dir (trilinos#11966) MueLu: Enabling better dumps for Maxwell1 MueLu RefMaxwell: Accept "aggregation: phase3 avoid singletons" Tpetra: Carl's fixes Tpetra: Improving std::move()-ability of BlockMultiVector Tpetra: Improving std::move()-ability of BlockMultiVector
|
@ndellingwood and @dmdunla, With the merge of PR #11977 to 'develop', this should be working again. If it is not, please let me know. |
…ilinos#11976) This failing test demonstrates a defect that got added with the recent TriBITS refactorings.
…ke file (trilinos/Trilinos#11976) This fixes cases where a dirty/incomplete source directory for a package remains but does not contain the Dependencies.cmake file. This was a use case I hit while experimenting with how to fix trilinos/Trilinos#11976. This replaces a temp commit made to the Trilinos 'develop' branch.
…ke file (trilinos/Trilinos#11976) This fixes cases where a dirty/incomplete source directory for a package remains but does not contain the Dependencies.cmake file. This was a use case I hit while experimenting with how to fix trilinos/Trilinos#11976. This replaces a temp commit made to the Trilinos 'develop' branch.
…ke file (trilinos/Trilinos#11976) This fixes cases where a dirty/incomplete source directory for a package remains but does not contain the Dependencies.cmake file. This was a use case I hit while experimenting with how to fix trilinos/Trilinos#11976. This replaces a temp commit made to the Trilinos 'develop' branch.
Origin repo remote tracking branch: 'github/master' Origin repo remote repo URL: 'github = [email protected]:TriBITSPub/TriBITS.git' Git describe: vera-release-3.5-start-1806-ge88b8af8 At commit: commit 1a9b519a5cd6005d299be5d07c8823a6a87c974e Author: Roscoe A. Bartlett <[email protected]> Date: Tue Jun 27 17:28:15 2023 -0600 Summary: Only submit warning for existing ignored package dir in WARNING mode (#11976)
…nos#11976)" This reverts commit 40e37d9. There is a better fix in TriBITS 'master' that will be merged in from the snapshot branch. I needed to revert this or it would be a conflict.
Brings in a better fix to the issue reported in trilinos#11976 on the TriBITS side (not the Trilinos side). Also brings in some other TriBITS updates.
This reverts commit 7c630d7. I am going to remove the exclude entry instead and put in excludes for the wrapper TriBITS package CMakeLists.txt and cmake/Dependencies.cmake files instead so that it will not show up as a TriBITS package.
…s#11976) This allows the TrilinosBuildStats test package to be removed but still keep the build_stats files to provide that functionality to tarball customers.
We removed autotools files from the Trilinos tarball 14 years ago because those we kept those around but they were out of date and we did not want customers using them. But since they have been stripped out of every Trilinos package (except Zoltan), there is no value in including this. Also, the latest TriBITS snapshot removed this macro. The only package that still has old autotools files in Zoltan. These have been stripped out of every other Trilinos package. For Zolan, we now leave the autotools files in the tarball. However, I did not remove the usage TRIBITS_EXCLUDE_AUTOTOOLS_FILES() from Kokkos or KokkosKernels so I could avoid the proces of having to get these changes into the latest Kokkos 4.1 release. I will post PRs to pull out the usage of TRIBITS_EXCLUDE_AUTOTOOLS_FILES() from the 'develop' branches of the native repos so this can be cleaned up later. For now, I will create an empty macro TRIBITS_EXCLUDE_AUTOTOOLS_FILES() to allow these packages to be unchanged.
…nos#11976)" This reverts commit 40e37d9. There is a better fix in TriBITS 'master' that will be merged in from the snapshot branch. I needed to revert this or it would be a conflict.
This reverts commit 7c630d7. I am going to remove the exclude entry instead and put in excludes for the wrapper TriBITS package CMakeLists.txt and cmake/Dependencies.cmake files instead so that it will not show up as a TriBITS package.
…s#11976) This allows the TrilinosBuildStats test package to be removed but still keep the build_stats files to provide that functionality to tarball customers.
We removed autotools files from the Trilinos tarball 14 years ago because those we kept those around but they were out of date and we did not want customers using them. But since they have been stripped out of every Trilinos package (except Zoltan), there is no value in including this. Also, the latest TriBITS snapshot removed this macro. The only package that still has old autotools files in Zoltan. These have been stripped out of every other Trilinos package. For Zolan, we now leave the autotools files in the tarball. However, I did not remove the usage TRIBITS_EXCLUDE_AUTOTOOLS_FILES() from Kokkos or KokkosKernels so I could avoid the proces of having to get these changes into the latest Kokkos 4.1 release. I will post PRs to pull out the usage of TRIBITS_EXCLUDE_AUTOTOOLS_FILES() from the 'develop' branches of the native repos so this can be cleaned up later. For now, I will create an empty macro TRIBITS_EXCLUDE_AUTOTOOLS_FILES() to allow these packages to be unchanged.
…rilinos#12024) This sets up a test that would have caught the problem in trilinos#11976 and will provide some protection for reduced tarball creation going forward. NOTE: Currently, this test will only be added if configure settings are passed through `-D${CMAKE_PROJECT_NAME}_CONFIGURE_OPTIONS_FILE=<cache-file>.cmake`. There is no way to pass that file (or thoses files) through `-C <cache-file>.make`.
…rilinos#12024) This sets up a test that would have caught the problem in trilinos#11976 and will provide some protection for reduced tarball creation going forward. NOTE: Currently, this test will only be added if configure settings are passed through `-D${CMAKE_PROJECT_NAME}_CONFIGURE_OPTIONS_FILE=<cache-file>.cmake`. There is no way to pass that file (or thoses files) through `-C <cache-file>.make`.
…rilinos#12024) This sets up a test that would have caught the problem in trilinos#11976 and will provide some protection for reduced tarball creation going forward. NOTE: Currently, this test will only be added if configure settings are passed through `-D${CMAKE_PROJECT_NAME}_CONFIGURE_OPTIONS_FILE=<cache-file>.cmake`. There is no way to pass that file (or thoses files) through `-C <cache-file>.make`.
…nos#11976)" This reverts commit 40e37d9. There is a better fix in TriBITS 'master' that will be merged in from the snapshot branch. I needed to revert this or it would be a conflict.
This reverts commit 7c630d7. I am going to remove the exclude entry instead and put in excludes for the wrapper TriBITS package CMakeLists.txt and cmake/Dependencies.cmake files instead so that it will not show up as a TriBITS package.
…s#11976) This allows the TrilinosBuildStats test package to be removed but still keep the build_stats files to provide that functionality to tarball customers.
We removed autotools files from the Trilinos tarball 14 years ago because those we kept those around but they were out of date and we did not want customers using them. But since they have been stripped out of every Trilinos package (except Zoltan), there is no value in including this. Also, the latest TriBITS snapshot removed this macro. The only package that still has old autotools files in Zoltan. These have been stripped out of every other Trilinos package. For Zolan, we now leave the autotools files in the tarball. However, I did not remove the usage TRIBITS_EXCLUDE_AUTOTOOLS_FILES() from Kokkos or KokkosKernels so I could avoid the proces of having to get these changes into the latest Kokkos 4.1 release. I will post PRs to pull out the usage of TRIBITS_EXCLUDE_AUTOTOOLS_FILES() from the 'develop' branches of the native repos so this can be cleaned up later. For now, I will create an empty macro TRIBITS_EXCLUDE_AUTOTOOLS_FILES() to allow these packages to be unchanged.
Origin repo remote tracking branch: 'github/master' Origin repo remote repo URL: 'github = [email protected]:TriBITSPub/TriBITS.git' Git describe: vera-release-3.5-start-1806-ge88b8af8 At commit: commit 1a9b519a5cd6005d299be5d07c8823a6a87c974e Author: Roscoe A. Bartlett <[email protected]> Date: Tue Jun 27 17:28:15 2023 -0600 Summary: Only submit warning for existing ignored package dir in WARNING mode (trilinos#11976)
…rilinos#12024) This sets up a test that would have caught the problem in trilinos#11976 and will provide some protection for reduced tarball creation going forward. NOTE: Currently, this test will only be added if configure settings are passed through `-D${CMAKE_PROJECT_NAME}_CONFIGURE_OPTIONS_FILE=<cache-file>.cmake`. There is no way to pass that file (or thoses files) through `-C <cache-file>.make`.
…bits This brings in the following TriBITS PRs: * TriBITSPub/TriBITS#591: Allow 100% raw CMake to be used in a TriBITS-compliant package (TriBITSPub/TriBITS#582) * TriBITSPub/TriBITS#588: gitdist: Pass in '-c color.status=always' when --dist-no-color is not added * TriBITSPub/TriBITS#590: Fix: gitdist: dist-repo-status: Display tag or SHA1 instead of 'HEAD' * TriBITSPub/TriBITS#587: gitdist: dist-repo-status: Display tag or SHA1 instead of 'HEAD' * TriBITSPub/TriBITS#586: More cleanup of packaging support (trilinos#11976) This goes back to TriBITS commits from June 2023.
Bug Report
@trilinos/tribits @bartlettroscoe
After using the
make package_sourceoption to generate packaged source code tarballs for a configuration of Trilinos, if I try to configure the packaged source with the same configuration I encounter errors due to thebuild_statsdirectory missing from thecommonToolsdirectory:Configure error output (click to expand)
Copying the missing
commonTools/build_statsfrom the original Trilinos source to the packaged source directory allows for configuration and compilationSteps to Reproduce
Reproduction details (click to expand)
Tasks
commonTools/build_statseven if theTrilinosBuildStatspackage is not enabled (e.g. by removing the exclude fromCPACK_SOURCE_IGNORE_FILESin the filecmake/CallbackDefineRepositoryPackaging.cmake)[ ] Add new Trilinos tests to the... Deferred to Test and support creation of reduced Trilinos tarballs #12024TrilinosInstallTestspackage to test tarball creation and configure from expanded tarball source treeThe text was updated successfully, but these errors were encountered: