Panzer: Panzer broke Albany nightly builds

[ikalash]

It looks like Panzer broke the Albany nightly builds. Here is the error:

Trilinos/packages/panzer/dof-mgr/src/Panzer_Filtered_GlobalIndexer.cpp:98:17: error: ‘using Vector = class Tpetra::Vector<long long int, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> > {aka class Tpetra::Vector<long long int, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >}’ has no member named ‘sync_host’; did you mean ‘need_sync_host’?
   ownedFiltered.sync_host();
                 ^~~~~~~~~
                 need_sync_host

https://sems-cdash-son.sandia.gov/cdash/viewBuildError.php?buildid=15633

Could someone please have a look?

@trilinos/panzer

[rppawlo]

Panzer has not changed recently. This looks like a non-backwards compatible change to Tpetra - they pushed UVM removal from Tpetra multivector yesterday. Do you have any flags enabled to disable deprecated code in trilinos? That would probably trigger this as only Tpetra was changed and not all downstream packages have been updated for the deprecated code yet. @kddevin

[ikalash]

Thanks for looking at it @rppawlo - interesting. Adding the @trilinos/tpetra tag.

[rppawlo]

@ikalash - if you are using the atdm build scripts, they are disabling deprecated code. We will probably have to reenable deprecated code in those scripts to get empire and charon passing again as well. They are both reporting similar issues.

s1057859:atdm rppawlo$ git grep DEPRECA
ATDMDevEnvSettings.cmake:ATDM_SET_CACHE(Phalanx_SHOW_DEPRECATED_WARNINGS OFF CACHE BOOL)
ATDMDevEnvSettings.cmake:ATDM_SET_CACHE(Kokkos_ENABLE_DEPRECATED_CODE OFF CACHE BOOL)
ATDMDevEnvSettings.cmake:ATDM_SET_CACHE(Tpetra_ENABLE_DEPRECATED_CODE OFF CACHE BOOL)
ATDMDevEnvSettings.cmake:ATDM_SET_CACHE(Belos_HIDE_DEPRECATED_CODE ON CACHE BOOL)
ATDMDevEnvSettings.cmake:ATDM_SET_CACHE(Epetra_HIDE_DEPRECATED_CODE ON CACHE BOOL)
ATDMDevEnvSettings.cmake:ATDM_SET_CACHE(Ifpack2_HIDE_DEPRECATED_CODE ON CACHE BOOL)
ATDMDevEnvSettings.cmake:ATDM_SET_CACHE(Ifpack2_ENABLE_DEPRECATED_CODE OFF CACHE BOOL)
ATDMDevEnvSettings.cmake:ATDM_SET_CACHE(MueLu_ENABLE_DEPRECATED_CODE OFF CACHE BOOL)
ATDMDevEnvSettings.cmake:ATDM_SET_CACHE(Panzer_HIDE_DEPRECATED_CODE ON CACHE BOOL)
ATDMDevEnvSettings.cmake:ATDM_SET_CACHE(Phalanx_HIDE_DEPRECATED_CODE ON CACHE BOOL)
ATDMDevEnvSettings.cmake:ATDM_SET_CACHE(STK_HIDE_DEPRECATED_CODE ON CACHE BOOL)
ATDMDevEnvSettings.cmake:ATDM_SET_CACHE(Teuchos_HIDE_DEPRECATED_CODE ON CACHE BOOL)
ATDMDevEnvSettings.cmake:ATDM_SET_CACHE(Thyra_HIDE_DEPRECATED_CODE ON CACHE BOOL)
ATDMDevEnvSettings.cmake:ATDM_SET_CACHE(Trios_HIDE_DEPRECATED_CODE ON CACHE BOOL)
s1057859:atdm rppawlo$ 

[ikalash]

No, I don't use ATDM build scripts. Would setting "Tpetra_ENABLE_DEPRECATED_CODE:BOOL=ON" in my Trilinos configure script disable the problematic code? What is the timeframe for getting the changes needed into develop? I'd prefer not to change the configuration if possible.

[kddevin]

Yes, Tpetra's changes to disable the UVM requirement were merged yesterday. We are deprecating some functions, but the rest of Trilinos is not yet updated to remove deprecated code. For now, deprecated code is ENABLED by default, so you can either specify Tpetra_ENABLE_DEPRECATED_CODE=ON or just remove any specification of Tpetra_ENABLE_DEPRECATED_CODE.

Propagating the UVM removal to all of Trilinos is not as simple as only changing the function call arguments. Care has to be taken to correctly access memory on device or host, not relying on UVM. Some interfaces will have to be rewritten to avoid holding pointers to host or device memory. I suspect the effort will take two months.

Details of the UVM-removal changes that were merged are at https://github.com/trilinos/Trilinos/wiki/Tpetra_Multivector_NoUVM.pdf

[kddevin]

@rppawlo I submitted a PR to change the setting in the ATDM scripts. #9071

[rppawlo]

Thanks @kddevin !

[ikalash]

Unfortunately I get a bunch of other Tpetra-related errors when I set: Tpetra_ENABLE_DEPRECATED_CODE=ON .

[ 68%] Building CXX object packages/xpetra/src/CMakeFiles/xpetra.dir/BlockedCrsMatrix/Xpetra_BlockReorderManager.cpp.o
In file included from /home/projects/albany/nightlyCDashTrilinosBlake/build/TriBuildSerialGccNoWarn/packages/xpetra/src/Utils/ExplicitInstantiation/Xpetra_TpetraMultiVector.cpp:55:0:
/home/projects/albany/nightlyCDashTrilinosBlake/repos/Trilinos/packages/xpetra/src/MultiVector/Xpetra_TpetraMultiVector_def.hpp: In instantiation of ‘typename Xpetra::MultiVector<Scalar, LocalOrdinal, GlobalOrdinal, Node>::dual_view_type::t_host_um Xpetra::TpetraMultiVector<Scalar, LocalOrdinal, GlobalOrdinal, Node>::getHostLocalView() const [with Scalar = double; LocalOrdinal = int; GlobalOrdinal = long long int; Node = Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial>; typename Xpetra::MultiVector<Scalar, LocalOrdinal, GlobalOrdinal, Node>::dual_view_type::t_host_um = Kokkos::View<double**, Kokkos::LayoutStride, Kokkos::Device<Kokkos::Serial, Kokkos::HostSpace>, Kokkos::MemoryTraits<1> >]’:
/home/projects/albany/nightlyCDashTrilinosBlake/repos/Trilinos/packages/xpetra/src/Utils/Xpetra_ETI_4arg_mv.hpp:42:3:   required from here
/home/projects/albany/nightlyCDashTrilinosBlake/repos/Trilinos/packages/xpetra/src/MultiVector/Xpetra_TpetraMultiVector_def.hpp:400:19: error: no matching function for call to ‘Tpetra::MultiVector<double, int, long long int, Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial> >::getLocalView<Kokkos::DualView<double**, Kokkos::LayoutStride, Kokkos::Serial, Kokkos::MemoryTraits<1> >::host_mirror_space>()’
     return subview(vec_->template getLocalView<typename TpetraMultiVector<Scalar,LocalOrdinal,GlobalOrdinal,Node>::dual_view_type::host_mirror_space> (),
            ~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                    Kokkos::ALL(), Kokkos::ALL());
                    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/projects/albany/nightlyCDashTrilinosBlake/build/TriBuildSerialGccNoWarn/packages/tpetra/core/src/Tpetra_MultiVector.hpp:1:0,
                 from /home/projects/albany/nightlyCDashTrilinosBlake/repos/Trilinos/packages/xpetra/src/MultiVector/Xpetra_TpetraMultiVector_decl.hpp:55,
                 from /home/projects/albany/nightlyCDashTrilinosBlake/repos/Trilinos/packages/xpetra/src/MultiVector/Xpetra_TpetraMultiVector_def.hpp:55,
                 from /home/projects/albany/nightlyCDashTrilinosBlake/build/TriBuildSerialGccNoWarn/packages/xpetra/src/Utils/ExplicitInstantiation/Xpetra_TpetraMultiVector.cpp:55:
/home/projects/albany/nightlyCDashTrilinosBlake/repos/Trilinos/packages/tpetra/core/src/Tpetra_MultiVector_decl.hpp:1538:5: note: candidate: template<class TargetDeviceType> typename std::remove_reference<decltype (declval<Kokkos::DualView<typename Kokkos::Details::ArithTraits<RealFloatType>::val_type**, Kokkos::LayoutLeft, typename Tpetra::Map<LocalOrdinal, GlobalOrdinal, Node>::device_type, void> >().view<TargetDeviceType>())>::type::const_type Tpetra::MultiVector<Scalar, LocalOrdinal, GlobalOrdinal, Node>::getLocalView(Tpetra::Access::ReadOnlyStruct) const [with TargetDeviceType = TargetDeviceType; Scalar = double; LocalOrdinal = int; GlobalOrdinal = long long int; Node = Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::Serial>]
     getLocalView (Access::ReadOnlyStruct) const
     ^~~~~~~~~~~~

Am I supposed to set the other package deprecated code flags on as well, not just Tpetra?

[jewatkins]

@ikalash I just came across this issue while building. It looks like you need the xpetra one too:

    -D Tpetra_ENABLE_DEPRECATED_CODE:BOOL=ON \
    -D Xpetra_ENABLE_DEPRECATED_CODE:BOOL=ON \

I also had to turn off FastILU but I don't think that's in the nightlies:

-DTrilinos_ENABLE_ShyLU_NodeFastILU=OFF \

[kddevin]

I am unable to reproduce this error.
I use the cmake configuration from the Albany test, except I set Tpetra_ENABLE_DEPRECATED_CODE=ON
The compilation on blake goes through fine.
Can you try a clean pull and build?

[ikalash]

Thanks @jewatkins ! Interestingly it looks like the error above is not happening in the Intel build on blake, just the gcc one. Anyway, I'll add the Xpetra option and see if it gets resolved. May need to tweak things in the next few days.

@kddevin : what is the timeframe for not needing to enable the deprecated code? Will you notify us when we need to remove those new configure options (disable the deprecated code)?

[mperego]

@ikalash It seems we are turning off the deprecated code for most packages:
https://github.com/SNLComputation/Albany/blob/master/doc/dashboards/blake.sandia.gov/do-cmake-trilinos-gcc-serial#L23-#L34
I suspect we'll have to remove many of those lines for a while, until Trilinos is consistently UVM free (which will likely take months).

@kddevin should we (Trilinos) use new cmake options to deprecate code internally to Trilinos?

[mperego]

@kddevin to clarify, I was wondering if we should use different cmake options, like
Tpetra_ENABLE_DEPRECATED_UVM_CODE=ON
enabled by default, until Trilinos packages use those deprecated functions. In this way applications like Albany that turn off deprecation code using the usual options won't be affected.
It would require extra work, I know..

[ikalash]

It looks like things are good if I set:

-D Tpetra_ENABLE_DEPRECATED_CODE:BOOL=ON \
    -D Xpetra_ENABLE_DEPRECATED_CODE:BOOL=ON \

Thanks!