Panzer: build fails with Epetra disabled

[tjfulle]

Bug Report

@trilinos/panzer

When Trilinos_ENABLE_Epetra:BOOL=OFF is defined, I get the following build error:

packages/panzer/dof-mgr/src/Panzer_GlobalIndexer_EpetraUtilities.hpp:52:10: fatal error: Epetra_Vector.h: No such file or directory
 #include <Epetra_Vector.h>

Full configure:

/projects/sems/install/rhel6-x86_64/sems/utility/cmake/3.12.2/bin/cmake \
  -G "Ninja" \
  -D Trilinos_CONFIGURE_OPTIONS_FILE:FILEPATH=$TRILINOS_DIR/cmake/std/PullRequestLinuxCommonTestingSettings.cmake \
  -D Trilinos_ENABLE_Tpetra:BOOL=ON \
  -D Trilinos_ENABLE_Epetra:BOOL=OFF \
  -D Trilinos_ENABLE_Moertel:BOOL=OFF \
  -D Trilinos_ENABLE_ShyLU_DDFROSch:BOOL=OFF \
  -D Trilinos_ENABLE_Stokhos:BOOL=OFF \
  -D ROL_ENABLE_EXAMPLES:BOOL=OFF \
  -D Tpetra_ENABLE_DEPRECATED_CODE:BOOL=OFF \
  -D Xpetra_ENABLE_DEPRECATED_CODE:BOOL=OFF \
  -D Trilinos_ENABLE_Galeri:BOOL=OFF \
  -D Rythmos_ENABLE_TESTS:BOOL=OFF \
  -D Trilinos_ENABLE_FEI:BOOL=OFF \
  -D MueLu_ENABLE_TESTS:BOOL=OFF \
  $TRILINOS_DIR

[rppawlo]

yes - panzer currently requires both the type 1 and type 2 stacks to be enabled. It's on the list of things to fix in FY20. What is the timeline/priority for this?

[tjfulle]

@rppawlo, by type 1 and 2, are you referring to Epetra and Tpetra? If so, I disabled Epetra because many packages assume that if Epetra is enabled then Tpetra’s GO=int - which is not true by default. Those build failures will eventually be addressed in the PR. So long as Panzer is ok with different GO in [ET]petra, or Trilinos is built with (non-default) GO=int in Tpetra, it should be fine.

[tjfulle]

Just noticed I’m commenting in this issue and not PR #5590 - which was the genesis for this issue

[rppawlo]

@tjfulle - yes - type 1 is epetra/amesos/ifpack/ml and type 2 is tpetra/amesos2/ifpack2/muelu. Panzer can use any GO for tpetra, however we always assume epetra is 32-bit. Unfortunately, almost all of our testing is based on epetra, so it will be a lot of work to disable complete type 1 stack and still have adequate test converage of panzer. Most of panzer is agnostic to epetra/tpetra but uses epetra for assembly. Its not hard to change but is a lot of busy work. I can move up priority if needed. Was targeting FY20/Q2

[tjfulle]

I don’t know that it is a priority for Tpetra. I just wanted to build Tpetra+downstream with deprecated code off (using default ordinal types). Many packages downstream of Tpetra fail to build in this case, so disabling Epetra and any other packages that fail to build with Epetra disabled was my first pass.

Perhaps Panzer can add some CMake logic to catch this at configure, rather than build, time?

[rppawlo]

It should catch this at configure. Epetra is a required package in panzer's disc-fe subpackage. I'll take a look at this.

[mhoemmen]

Panzer can use any GO for tpetra, however we always assume epetra is 32-bit.

This is a bit of a problem for @trilinos/xpetra and @trilinos/muelu. The current plan is that if you enable Epetra support in Xpetra and MueLu, then you must enable GO=int. This excludes enabling both Epetra and Tpetra with deprecated code disabled in Tpetra.

[tjfulle]

@mhoemmen does it? Can’t we disable deprecated code and explicitly set Tpetra_INST_INT_INT?

[rppawlo]

This is a bit of a problem for @trilinos/xpetra and @trilinos/muelu.

We have been disabling Epetra support in Xpetra and Muelu in our builds. We only use xpetra/muelu for type 2 stack runs. We never call that code with epetra. I think ATDM might have adopted these settings already. we should check with @bartlettroscoe .

[bartlettroscoe]

The settings ATDM is using can be seen in:

• https://github.com/trilinos/Trilinos/blob/develop/cmake/std/atdm/ATDMDevEnvSettings.cmake

which includes:

ATDM_SET_CACHE(Tpetra_INST_INT_INT OFF CACHE BOOL)

Any deviation from the ATDM settings is may not work so my advice is to just use the ATDM Trilinos configuration if you want to be productive.

[tjfulle]

@bartlettroscoe said

Any deviation from the ATDM settings is may not work so my advice is to just use the ATDM Trilinos configuration if you want to be productive.

That is probably true, but @trilinos/tpetra plans to remove deprecated code soon and many packages don’t yet build if deprecated code is not enabled. This, and many other issues, were opened when build failures occurred with deprecated code disabled but otherwise default test settings.

[rppawlo]

So to follow up - we are able to build using a single GO type and are able to jump between the type 1 and type 2 stacks at runtime. It is really useful in certain applications to do performance testing comparisons of the two stacks from the same executable. This would be a painful change for some applications that currently assume both stacks must be built built, but we could do it.

[bartlettroscoe]

NOTE: several non-ATDM Trilinos packages don't currently build with Tpetra_INST_INT_INT=OFF. See, for example this build (which @fryeguy52 will create a GitHub issue for when he has the time as per ATDV-188).

[tjfulle]

@rppawlo, when the dust settles, I hope to have things fixed to the point where (with deprecated code disabled and eventually removed) the current default test settings will just work. To get there many packages need to be updated to work with any Tpetra GO and Epetra - or fail at configure time if they can’t. Direct comparison between the two stacks may require building Tpetra with GO=int (which will not be built by default in the coming weeks)

[mhoemmen]

Direct comparison between the two stacks may require building Tpetra with GO=int

Right, this is the only way that MueLu plans to permit enabling both Epetra and Tpetra support. However, one could always enable Epetra but disable support for Epetra in Amesos2, Xpetra, and MueLu.

[tjfulle]

... but disable support for Epetra in Amesos2, Xpetra, and MueLu.

Not quite yet, I still have 3-4 packages that won’t build with deprecated code disabled - but we’re getting there!