tpetra with OneAPI 2023.1.0: Tpetra_Import_Util2.hpp:1033:55: error: SYCL kernel cannot call a variadic function #12277

eugeneswalker · 2023-09-18T17:19:03Z

Bug Report

Description

Trying to build Trilinos master branch (5aaae1a from Fri Sep 1 08:49:54 2023 -0500) with OneAPI 2023.1.0 for Intel DG1 is failing using the following CMake:

cmake \
    -DBLAS_LIBRARY_DIRS=/home/opt/ubuntu20.04/intel/oneapi/mkl/latest/lib/intel64 \
    -DBLAS_LIBRARY_NAMES=mkl_rt \
    -DBUILD_SHARED_LIBS=ON \
    -DCMAKE_C_COMPILER=mpiicc \
    -DCMAKE_CXX_COMPILER=mpiicpc \
    -DCMAKE_Fortran_COMPILER=mpiifort \
    -DCMAKE_INSTALL_PREFIX=./install \
    -DKokkos_ARCH_INTEL_XEHP=ON \
    -DKokkos_ENABLE_SYCL=ON \
    -DLAPACK_LIBRARY_DIRS=/home/opt/ubuntu20.04/intel/oneapi/mkl/latest/lib/intel64 \
    -DLAPACK_LIBRARY_NAMES=mkl_rt \
    -DMKL_INCLUDE_DIRS=/home/opt/ubuntu20.04/intel/oneapi/mkl/latest/include \
    -DMKL_LIBRARY_DIRS=/home/opt/ubuntu20.04/intel/oneapi/mkl/latest/lib/intel64 \
    -DMKL_LIBRARY_NAMES=mkl_rt \
    -DTeuchos_ENABLE_COMPLEX:BOOL=OFF \
    -DTeuchos_ENABLE_FLOAT:BOOL=OFF \
    -DTpetra_INST_COMPLEX_DOUBLE:BOOL=OFF \
    -DTpetra_INST_COMPLEX_FLOAT:BOOL=OFF \
    -DTpetra_INST_DOUBLE:BOOL=ON \
    -DTpetra_INST_FLOAT:BOOL=OFF \
    -DTpetra_INST_INT_INT:BOOL=OFF \
    -DTpetra_INST_INT_LONG_LONG:BOOL=ON \
    -DTpetra_INST_INT_LONG:BOOL=OFF \
    -DTpetra_INST_OPENMP:BOOL=OFF \
    -DTpetra_INST_SERIAL:BOOL=ON \
    -DTpetra_INST_SYCL=ON \
    -DTPL_ENABLE_BLAS=ON \
    -DTPL_ENABLE_MKL=ON \
    -DTPL_ENABLE_MPI=ON \
    -DTrilinos_ENABLE_AztecOO=ON \
    -DTrilinos_ENABLE_DEBUG_SYMBOLS=ON \
    -DTrilinos_ENABLE_Epetra=ON \
    -DTrilinos_ENABLE_Ifpack=ON \
    -DTrilinos_ENABLE_ML:BOOL=ON \
    -DTrilinos_ENABLE_OpenMP=ON \
    -DTrilinos_ENABLE_TESTS=ON \
    -DTrilinos_ENABLE_Tpetra:BOOL=ON \
    -DTrilinos_TEST_CATEGORIES="HEAVY;PERFORMANCE" \
    -DTrilinos_WARNINGS_AS_ERRORS_FLAGS="-Wno-error -Wno-implicit-function-declaration"

Build error:

$> make
...
[ 32%] Building CXX object packages/tpetra/core/src/CMakeFiles/tpetra.dir/Tpetra_CrsMatrix_LONG_LONG_INT_LONG_LONG_SYCL.cpp.o
In file included from /home/users/lpeyrala/trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_LONG_LONG_INT_LONG_LONG_SYCL.cpp:66:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_decl.hpp:47:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_fwd.hpp:45:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Details_DefaultTypes.hpp:45:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/compat/Tpetra_KokkosCompat_ClassicNodeAPI_Wrapper.hpp:4:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/kokkos/core/src/Kokkos_Core.hpp:53:
/home/users/lpeyrala/trilinos/Trilinos/packages/kokkos/core/src/Kokkos_MathematicalFunctions.hpp:470:34: warning: explicit comparison with infinity in fast floating point mode [-Wtautological-constant-compare]
KOKKOS_IMPL_MATH_UNARY_PREDICATE(isinf)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~
/home/users/lpeyrala/trilinos/Trilinos/packages/kokkos/core/src/Kokkos_MathematicalFunctions.hpp:169:12: note: expanded from macro 'KOKKOS_IMPL_MATH_UNARY_PREDICATE'
    return FUNC(x);                                                          \
           ^~~~~~~
/home/users/lpeyrala/trilinos/Trilinos/packages/kokkos/core/src/Kokkos_MathematicalFunctions.hpp:471:34: warning: explicit comparison with NaN in fast floating point mode [-Wtautological-constant-compare]
KOKKOS_IMPL_MATH_UNARY_PREDICATE(isnan)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~
/home/users/lpeyrala/trilinos/Trilinos/packages/kokkos/core/src/Kokkos_MathematicalFunctions.hpp:169:12: note: expanded from macro 'KOKKOS_IMPL_MATH_UNARY_PREDICATE'
    return FUNC(x);                                                          \
           ^~~~~~~
In file included from /home/users/lpeyrala/trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_LONG_LONG_INT_LONG_LONG_SYCL.cpp:66:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_decl.hpp:48:
In file included from /home/users/lpeyrala/trilinos/packages/tpetra/core/src/Tpetra_LocalCrsMatrixOperator.hpp:1:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_LocalCrsMatrixOperator_decl.hpp:44:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_LocalOperator.hpp:45:
/home/users/lpeyrala/trilinos/Trilinos/packages/kokkos-kernels/common/src/Kokkos_ArithTraits.hpp:940:12: warning: explicit comparison with infinity in fast floating point mode [-Wtautological-constant-compare]
    return isinf(Kokkos::Experimental::cast_from_half<float>(x));
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/users/lpeyrala/trilinos/Trilinos/packages/kokkos-kernels/common/src/Kokkos_ArithTraits.hpp:946:12: warning: explicit comparison with NaN in fast floating point mode [-Wtautological-constant-compare]
    return isnan(Kokkos::Experimental::cast_from_half<float>(x));
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/users/lpeyrala/trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_LONG_LONG_INT_LONG_LONG_SYCL.cpp:66:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_decl.hpp:49:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_RowMatrix_decl.hpp:45:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_ConfigDefs.hpp:78:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_Array.hpp:52:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_ArrayRCP.hpp:47:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_ArrayView.hpp:48:
/home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_as.hpp:1314:27: warning: implicit conversion from 'const long' to 'double' changes value from 9223372036854775807 to 9223372036854775808 [-Wimplicit-const-int-float-conversion]
        t < minVal || t > maxVal,
                        ~ ^~~~~~
/home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_TestForException.hpp:180:33: note: expanded from macro 'TEUCHOS_TEST_FOR_EXCEPTION'
  const bool throw_exception = (throw_exception_test); \
                                ^~~~~~~~~~~~~~~~~~~~
In file included from /home/users/lpeyrala/trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_LONG_LONG_INT_LONG_LONG_SYCL.cpp:66:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_decl.hpp:49:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_RowMatrix_decl.hpp:45:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_ConfigDefs.hpp:78:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_Array.hpp:52:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_ArrayRCP.hpp:47:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_ArrayView.hpp:48:
/home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_as.hpp:1519:27: warning: implicit conversion from 'const int' to 'float' changes value from 2147483647 to 2147483648 [-Wimplicit-const-int-float-conversion]
        t < minVal || t > maxVal,
                        ~ ^~~~~~
/home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_TestForException.hpp:180:33: note: expanded from macro 'TEUCHOS_TEST_FOR_EXCEPTION'
  const bool throw_exception = (throw_exception_test); \
                                ^~~~~~~~~~~~~~~~~~~~
In file included from /home/users/lpeyrala/trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_LONG_LONG_INT_LONG_LONG_SYCL.cpp:66:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_decl.hpp:49:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_RowMatrix_decl.hpp:45:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_ConfigDefs.hpp:83:
/home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_ScalarTraits.hpp:130:7: warning: explicit comparison with NaN in fast floating point mode [-Wtautological-constant-compare]
  if (std::isnan(x)) return true;
      ^~~~~~~~~~~~~
/home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_ScalarTraits.hpp:642:12: note: in instantiation of function template specialization 'Teuchos::generic_real_isnaninf<float>' requested here
    return generic_real_isnaninf<float>(x);
           ^
/home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_ScalarTraits.hpp:131:7: warning: explicit comparison with infinity in fast floating point mode [-Wtautological-constant-compare]
  if (std::isinf(x)) return true;
      ^~~~~~~~~~~~~
/home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_ScalarTraits.hpp:130:7: warning: explicit comparison with NaN in fast floating point mode [-Wtautological-constant-compare]
  if (std::isnan(x)) return true;
      ^~~~~~~~~~~~~
/home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_ScalarTraits.hpp:755:12: note: in instantiation of function template specialization 'Teuchos::generic_real_isnaninf<double>' requested here
    return generic_real_isnaninf<double>(x);
           ^
/home/users/lpeyrala/trilinos/Trilinos/packages/teuchos/core/src/Teuchos_ScalarTraits.hpp:131:7: warning: explicit comparison with infinity in fast floating point mode [-Wtautological-constant-compare]
  if (std::isinf(x)) return true;
      ^~~~~~~~~~~~~
In file included from /home/users/lpeyrala/trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_LONG_LONG_INT_LONG_LONG_SYCL.cpp:67:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp:52:
/home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Import_Util2.hpp:1033:55: error: SYCL kernel cannot call a variadic function
          printf("Cannot figure out if ID is owned.\n");
                                                      ^
/home/users/lpeyrala/trilinos/Trilinos/packages/kokkos/core/src/SYCL/Kokkos_SYCL_Team.hpp:559:5: note: called by 'parallel_reduce<unsigned long, (lambda at /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Import_Util2.hpp:1017:88), unsigned long>'
    closure(i, val);
    ^
/home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Import_Util2.hpp:1017:13: note: called by 'operator()'
    Kokkos::parallel_reduce(Kokkos::TeamThreadRange(member, rowptr_start, rowptr_end), [&](const size_t j, size_t &innerUpdate) {
            ^
/home/users/lpeyrala/trilinos/Trilinos/packages/kokkos/core/src/SYCL/Kokkos_SYCL_Parallel_Team.hpp:613:19: note: called by 'operator()'
                  functor(team_member, update);
                  ^
/home/opt/ubuntu20.04/intel/oneapi/compiler/2023.1.0/linux/bin-llvm/../include/sycl/handler.hpp:1202:5: note: called by 'kernel_parallel_for<(lambda at /home/users/lpeyrala/trilinos/Trilinos/packages/kokkos/core/src/SYCL/Kokkos_SYCL_Parallel_Team.hpp:601:13), sycl::nd_item<2>, (lambda at /home/users/lpeyrala/trilinos/Trilinos/packages/kokkos/core/src/SYCL/Kokkos_SYCL_Parallel_Team.hpp:601:13)>'
    KernelFunc(detail::Builder::getElement(detail::declptr<ElementType>()));
    ^
10 warnings and 1 error generated.
make[2]: *** [packages/tpetra/core/src/CMakeFiles/tpetra.dir/build.make:1000: packages/tpetra/core/src/CMakeFiles/tpetra.dir/Tpetra_CrsMatrix_LONG_LONG_INT_LONG_LONG_SYCL.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:10569: packages/tpetra/core/src/CMakeFiles/tpetra.dir/all] Error 2
make: *** [Makefile:166: all] Error 2

The key error seems to be:

/home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Import_Util2.hpp:1033:55: error: SYCL kernel cannot call a variadic function
          printf("Cannot figure out if ID is owned.\n");

Any idea what is going on? CC @nchaimov

The text was updated successfully, but these errors were encountered:

csiefer2 · 2023-09-18T17:49:25Z

We do not do regular SYCL testing at present, but hope to resolve these issues once we do.

jhux2 · 2023-09-18T17:50:05Z

Possibly related: #11968

csiefer2 · 2023-09-18T18:41:06Z

@jhux2 Not really. #11968 is using the Serial backend, not the SYCL one (although it does use the same set of compilers).

Also #11650

jwillenbring · 2023-09-19T14:34:00Z

We do not do regular SYCL testing at present, but hope to resolve these issues once we do.

@csiefer2 This test runs regularly, and if you would like access to the machine for you or for someone on your team, we can help with that.

jhux2 · 2023-09-19T16:36:41Z

We do not do regular SYCL testing at present, but hope to resolve these issues once we do.

@csiefer2 This test runs regularly, and if you would like access to the machine for you or for someone on your team, we can help with that.

@jwillenbring Is this something that can be posted?

csiefer2 · 2023-09-19T21:17:46Z

@csiefer2 I second @jhux2's request. I don't see that test on any of our cdashes

eugeneswalker · 2023-09-20T14:07:26Z

@csiefer2 I second @jhux2's request. I don't see that test on any of our cdashes

We were running the SYCL test nightly on the Frank system at University of Oregon until #12016 in July, at which point we turned off the nightly SYCL test because it was broken. Now that #12016 is resolved, we are trying to turn it back on but are not able to do so until this issue can be resolved.

csiefer2 · 2023-09-20T14:23:29Z

@eugeneswalker Given that Trilinos doesn't work correctly with SYCL (even if you could build it), IMO there's no real harm in leaving testing broken. I just tried a SYCL build on one of our local PVC machines (we don't have any XEHPs) and the linker can't even link libkokkoscore.so correctly, so we have even less in the way of testing capability than you do. I can't reproduce your error, so I can't fix it.

If you think you know how to fix it, feel free to submit a PR which fixes it and then we can make sure it doesn't break anything else.

csiefer2 · 2023-09-20T15:50:30Z

Update: Our compiler install on our PVC machine is incomplete. Someone is working on it.

masterleinad · 2023-09-20T19:01:04Z

#12294 should fix this.

csiefer2 · 2023-09-21T14:17:13Z

#12294 just merged. @eugeneswalker did this fix your reported issue?

eugeneswalker · 2023-09-21T15:27:52Z

#12294 just merged. @eugeneswalker did this fix your reported issue?

I am verifying this now. Will post back today with results. Thank you

eugeneswalker · 2023-09-21T20:21:17Z

#12294 just merged. @eugeneswalker did this fix your reported issue?

I got the same error using Trilinos master branch from a couple hours ago:

commit 5aaae1ada6fe1ce777e671a0ff84fdc4f0779406 (HEAD -> master, origin/master, origin/HEAD)
Merge: f3ff0b54c51 8db46da6c04
Author: trilinos-autotester <[email protected]>
Date:   Fri Sep 1 08:49:54 2023 -0500

    Merge Pull Request #12170 from trilinos/Trilinos/master_merge_20230825_175835

    Automatically Merged using Trilinos Master Merge AutoTester
    PR Title: b'Trilinos Master Merge PR Generator: Auto PR created to promote from master_merge_20230825_175835 branch to master'
    PR Author: trilinos-autotester

I got the same error:

...
make[2]: *** [packages/tpetra/core/src/CMakeFiles/tpetra.dir/build.make:1000: packages/tpetra/core/src/CMakeFiles/tpetra.dir/Tpetra_CrsMatrix_LONG_LONG_INT_LONG_LONG_SYCL.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
In file included from /home/users/lpeyrala/trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_DOUBLE_INT_LONG_LONG_SYCL.cpp:67:
In file included from /home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_CrsMatrix_def.hpp:52:
/home/users/lpeyrala/trilinos/Trilinos/packages/tpetra/core/src/Tpetra_Import_Util2.hpp:1033:55: error: SYCL kernel cannot call a variadic function
          printf("Cannot figure out if ID is owned.\n");

Does this not include the merged fix?

csiefer2 · 2023-09-21T20:22:51Z

@eugeneswalker Master does not include the fix. Try the develop branch.

eugeneswalker · 2023-09-22T15:27:17Z

#12294 just merged. @eugeneswalker did this fix your reported issue?

This resolves the issue for me! Thanks!

eugeneswalker · 2023-09-22T15:27:54Z

When might this be merged into master @csiefer2 @masterleinad ?

masterleinad · 2023-09-22T15:56:17Z

When might this be merged into master @csiefer2 @masterleinad ?

Whenever Trilinos releases again. 🙂

eugeneswalker added the type: bug The primary issue is a bug in Trilinos code or tests label Sep 18, 2023

jhux2 added the pkg: Tpetra label Sep 18, 2023

masterleinad mentioned this issue Sep 20, 2023

Tpetra: Fix compiling with SYCL #12294

Merged

eugeneswalker closed this as completed Sep 22, 2023

jhux2 added this to Tpetra Aug 12, 2024

jhux2 moved this to Done in Tpetra Aug 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tpetra with OneAPI 2023.1.0: Tpetra_Import_Util2.hpp:1033:55: error: SYCL kernel cannot call a variadic function #12277

tpetra with OneAPI 2023.1.0: Tpetra_Import_Util2.hpp:1033:55: error: SYCL kernel cannot call a variadic function #12277

eugeneswalker commented Sep 18, 2023

csiefer2 commented Sep 18, 2023

jhux2 commented Sep 18, 2023

csiefer2 commented Sep 18, 2023 •

edited

Loading

jwillenbring commented Sep 19, 2023

jhux2 commented Sep 19, 2023

csiefer2 commented Sep 19, 2023

eugeneswalker commented Sep 20, 2023

csiefer2 commented Sep 20, 2023 •

edited

Loading

csiefer2 commented Sep 20, 2023

masterleinad commented Sep 20, 2023

csiefer2 commented Sep 21, 2023

eugeneswalker commented Sep 21, 2023

eugeneswalker commented Sep 21, 2023

csiefer2 commented Sep 21, 2023

eugeneswalker commented Sep 22, 2023

eugeneswalker commented Sep 22, 2023

masterleinad commented Sep 22, 2023

tpetra with OneAPI 2023.1.0: Tpetra_Import_Util2.hpp:1033:55: error: SYCL kernel cannot call a variadic function #12277

tpetra with OneAPI 2023.1.0: Tpetra_Import_Util2.hpp:1033:55: error: SYCL kernel cannot call a variadic function #12277

Comments

eugeneswalker commented Sep 18, 2023

Bug Report

Description

csiefer2 commented Sep 18, 2023

jhux2 commented Sep 18, 2023

csiefer2 commented Sep 18, 2023 • edited Loading

jwillenbring commented Sep 19, 2023

jhux2 commented Sep 19, 2023

csiefer2 commented Sep 19, 2023

eugeneswalker commented Sep 20, 2023

csiefer2 commented Sep 20, 2023 • edited Loading

csiefer2 commented Sep 20, 2023

masterleinad commented Sep 20, 2023

csiefer2 commented Sep 21, 2023

eugeneswalker commented Sep 21, 2023

eugeneswalker commented Sep 21, 2023

csiefer2 commented Sep 21, 2023

eugeneswalker commented Sep 22, 2023

eugeneswalker commented Sep 22, 2023

masterleinad commented Sep 22, 2023

csiefer2 commented Sep 18, 2023 •

edited

Loading

csiefer2 commented Sep 20, 2023 •

edited

Loading