Errors compiling Trilinos with TriKota

[ikalash]

I am trying to revive our Albany tests that use TriKota, which we have not run for awhile (on the order of a year or two). I am getting the following compilation error when I try to compile it:

[ 30%] Building CXX object packages/TriKota/Dakota/src/CMakeFiles/dakota_src.dir/ParallelLibrary.cpp.o
/home/ikalash/nightlyAlbanyTests/Results/Trilinos/packages/TriKota/Dakota/packages/pecos/util/src/linear_algebra.cpp: In function ‘void Pecos::util::GEMV(Teuchos::ETransp, bool, ScalarType, const Teuchos::SerialDenseMatrix<OrdinalType, ScalarType>&, const Teuchos::SerialDenseVector<OrdinalType, ScalarType>&, ScalarType, Teuchos::SerialDenseVector<OrdinalType, ScalarType>&) [with OrdinalType = int; ScalarType = std::complex<double>]’:
/home/ikalash/nightlyAlbanyTests/Results/Trilinos/packages/TriKota/Dakota/packages/pecos/util/src/linear_algebra.cpp:84:3: error: ‘ZGEMV_F77’ was not declared in this scope
   ZGEMV_F77( &T, &M, &N, &alpha,
   ^~~~~~~~~
/home/ikalash/nightlyAlbanyTests/Results/Trilinos/packages/TriKota/Dakota/packages/pecos/util/src/linear_algebra.cpp:84:3: note: suggested alternative: ‘SGEMV_F77’
   ZGEMV_F77( &T, &M, &N, &alpha,
   ^~~~~~~~~
   SGEMV_F77

I am using dakota 6.9 public release, and this is on a Fedora 29, with gcc-8.3.1. Could someone please help me resolve the issue? It's possible I am not using the right Dakota version, or there is some option I'm missing in my configure script (as I said, I haven't tried to build with Dakota in awhile).

@trilinos/trikota

[mhoemmen]

I'm just guessing, but could it be that some option enabling complex arithmetic was turned off? I'm a bit surprised since I had thought that Dakota used Teuchos::BLAS etc. directly. @briadam might know more.

[briadam]

Dakota 6.9 has a dependency on Teuchos complex blas. (Which actually we realized isn't needed and we need to turn off in our next release, as it's only used in some experimental code for now.) I'd guess that you can get around this by something like this (or corresponding -D) in your configure:

set(Teuchos_ENABLE_COMPLEX TRUE CACHE BOOL "Pecos requires some complex blas wrappers")

[mhoemmen]

CORRECTION: It should be possible to set Teuchos_ENABLE_COMPLEX=ON but Trilinos_ENABLE_COMPLEX_DOUBLE=OFF; this is important to avoid Tpetra instantiations for complex Scalar types, if you don't need them.

Users are supposed to set Trilinos_ENABLE_COMPLEX_DOUBLE=ON now. That sets Teuchos_ENABLE_COMPLEX=ON by default.

[mhoemmen]

@briadam wrote:

Dakota 6.9 has a dependency on Teuchos complex blas.

It looks like Dakota is using some kind of internal Teuchos BLAS macros, instead of using Teuchos::BLAS -- is that right?

[briadam]

Users are supposed to set Trilinos_ENABLE_COMPLEX_DOUBLE=ON now. That sets Teuchos_ENABLE_COMPLEX=ON by default.

Thanks @mhoemmen, I had discovered and then forgotten that. We're doing it wrong in our Dakota configure and I meant to fix it; this is a helpful reminder!

[briadam]

Sorry I failed to answer that question. Between Dakota and our TPLs, I believe we have a mix of calls to *_F77 macros and use of Teuchos::BLAS<int, Real>. I suspect this is either for historical reasons and/or lack of awareness or completeness of the templated interface at the time.

[mhoemmen]

@briadam That's OK, I just didn't realize that people were using the macros in Teuchos. I'll keep that in mind if we need to change them.

[ikalash]

@briadam : thanks for the replies. If I turn on complex using Trilinos_ENABLE_COMPLEX_DOUBLE=ON the compilation gets further but I get another error:

[ 21%] Building CXX object packages/TriKota/Dakota/packages/external/VPISparseGrid/test/CMakeFiles/sandia_sgmg_size_table.dir/sandia_rules2.cpp.o
/home/ikalash/nightlyAlbanyTests/Results/Trilinos/packages/TriKota/Dakota/packages/external/JEGA/Utilities/src/DiscreteDesignVariableNature.cpp: In function ‘double JEGA::Utilities::relative_difference(double, double)’:
/home/ikalash/nightlyAlbanyTests/Results/Trilinos/packages/TriKota/Dakota/packages/external/JEGA/Utilities/src/DiscreteDesignVariableNature.cpp:106:13: error: ‘isnan’ is not a member of ‘std’
     if(std::isnan(arg_a) || std::isnan(arg_b)) return max_val;

The fix is very easy - just add #include<cmath> to Trilinos/packages/TriKota/Dakota/packages/external/JEGA/Utilities/src/DiscreteDesignVariableNature.cpp (though getting this fix in given that I am using a released version of Dakota will be hacky).

Unfortunately there is another compilation error later on:

[ 99%] Building CXX object packages/piro/src/CMakeFiles/piro.dir/Piro_Epetra_NECoupledModelEvaluator.cpp.o
In file included from /home/ikalash/nightlyAlbanyTests/Results/Trilinos/packages/stokhos/src/Stokhos_ReducedQuadratureFactory.hpp:224,
                 from /home/ikalash/nightlyAlbanyTests/Results/Trilinos/packages/stokhos/src/Stokhos_GSReducedPCEBasisBaseImp.hpp:42,
                 from /home/ikalash/nightlyAlbanyTests/Results/Trilinos/packages/stokhos/src/Stokhos_GSReducedPCEBasisBase.hpp:260,
                 from /home/ikalash/nightlyAlbanyTests/Results/Trilinos/packages/stokhos/src/Stokhos_MonomialProjGramSchmidtPCEBasis.hpp:45,
                 from /home/ikalash/nightlyAlbanyTests/Results/Trilinos/packages/stokhos/src/Stokhos_ReducedBasisFactoryImp.hpp:43,
                 from /home/ikalash/nightlyAlbanyTests/Results/Trilinos/packages/stokhos/src/Stokhos_ReducedBasisFactory.hpp:101,
                 from /home/ikalash/nightlyAlbanyTests/Results/Trilinos/packages/piro/src/Piro_Epetra_NECoupledModelEvaluator.cpp:56:
/home/ikalash/nightlyAlbanyTests/Results/Trilinos/packages/stokhos/src/Stokhos_ReducedQuadratureFactoryImp.hpp:66:10: fatal error: LinearSolver.hpp: No such file or directory
 #include "LinearSolver.hpp"
          ^~~~~~~~~~~~~~~~~~

It's interesting that the error only happens when TriKota is on. I guess resolving this is a question for @trilinos/stokhos developers.

[briadam]

My apologies Irina, I forgot about that JEGA patch; it was added to Dakota's devel and master branches a few months ago and will be in a planned patch release coming in May. In case a helpful data point, for Dakota we often maintain patch files for inbound TPLs and apply them after extracting the last released version of the code. When the inbound TPL integrates the patch we delete them from the process.

And yeah, no clue about the Stokhos error...

[etphipp]

I don't think Albany cares about Stokhos, so they should be able to turn that off. Regardless, LinearSolver.hpp used to be where the compressive sensing tools were in Pecos, so I needed them for this code in that file:

  Pecos::CompressedSensingTool CS;
  Teuchos::SerialDenseMatrix<ordinal_type, value_type> AA(A, transa);
  Teuchos::SerialDenseVector<ordinal_type, value_type> bb(b);
  Pecos::RealMatrixArray xx;
  Pecos::CompressedSensingOptions opts;
  Pecos::CompressedSensingOptionsList opts_list;
  CS.solve(AA, bb, xx, opts, opts_list);

I take it Pecos::CompressedSensingTool has moved somewhere else?

[briadam]

John Jakeman worked on refactoring out the Pecos solvers from the core Pecos orthogonal polynomial capabilities. However, we left the legacy tools around for now in pecos/src/LinearSolverPecosSrc.hpp (to distinguish from the newer, but not necessarily backward compatible pecos/util/src/ solvers). I suspect that just changing that include to LinearSolverPecosSrc.hpp will be easiest for now, but will break for Dakota <6.9, so let me know if we need to finesse this more...

[ikalash]

@etphipp : you're right, I can just turn off Stokhos. We don't use it explicitly in Albany anymore as you say, but I was somehow thinking we need it enabled for AD to work under the hood. Anyway, I was able to compile by turning off Stokhos.

@briadam , last question hopefully: did the file TriKota_MPDirectApplicInterface.hpp get renamed? I get a compilation error now when building Albany due to that file not being found:

[ 99%] Building CXX object src/CMakeFiles/AlbanyDakotaT.dir/Albany_DakotaT.cpp.o
/home/ikalash/nightlyCDash/repos/Albany/src/Albany_Dakota.cpp:14:10: fatal error: TriKota_MPDirectApplicInterface.hpp: No such file or directory
 #include "TriKota_MPDirectApplicInterface.hpp"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

[etphipp]

You should be able to remove the include of that file, since that was for the multi-point ensemble propagation, which was also removed from Albany.

[ikalash]

Thanks @etphipp , that fixes the compilation error. I'll go ahead and close the issue. Thanks everyone for your help!