openmp.gesv failures with gcc/10+armpl/21

[e10harvey]

@vqd8a, when standing up the A64FX CI testing I encountered these test failures. Can you investigate?

Snippet of ctest output

1: [ RUN      ] openmp.gesv_mrhs_double
1: /path/to/workspace/KokkosKernels_PullRequest_Tpls_ARMPL2110_Tpls_ARMPL2030_GCC1020/kokkos-kernels/unit_test/blas/Test_Blas_gesv.hpp:229: Failure
1: Value of: true
1: Expected: test_flag
1: Which is: false
1: /path/to/workspace/KokkosKernels_PullRequest_Tpls_ARMPL2110_Tpls_ARMPL2030_GCC1020/kokkos-kernels/unit_test/blas/Test_Blas_gesv.hpp:229: Failure
1: Value of: true
1: Expected: test_flag
1: Which is: false
1: /path/to/workspace/KokkosKernels_PullRequest_Tpls_ARMPL2110_Tpls_ARMPL2030_GCC1020/kokkos-kernels/unit_test/blas/Test_Blas_gesv.hpp:229: Failure
1: Value of: true
1: Expected: test_flag
1: Which is: false
1: [  FAILED  ] openmp.gesv_mrhs_double (50 ms)
1: [ RUN      ] openmp.gesv_complex_double
1: /path/to/workspace/KokkosKernels_PullRequest_Tpls_ARMPL2110_Tpls_ARMPL2030_GCC1020/kokkos-kernels/unit_test/blas/Test_Blas_gesv.hpp:121: Failure
1: Value of: true
1: Expected: test_flag
1: Which is: false
1: /path/to/workspace/KokkosKernels_PullRequest_Tpls_ARMPL2110_Tpls_ARMPL2030_GCC1020/kokkos-kernels/unit_test/blas/Test_Blas_gesv.hpp:121: Failure
1: Value of: true
1: Expected: test_flag
1: Which is: false
1: /path/to/workspace/KokkosKernels_PullRequest_Tpls_ARMPL2110_Tpls_ARMPL2030_GCC1020/kokkos-kernels/unit_test/blas/Test_Blas_gesv.hpp:121: Failure
1: Value of: true
1: Expected: test_flag
1: Which is: false
1: [  FAILED  ] openmp.gesv_complex_double (74 ms)
1: [ RUN      ] openmp.gesv_mrhs_complex_double
1: /path/to/workspace/KokkosKernels_PullRequest_Tpls_ARMPL2110_Tpls_ARMPL2030_GCC1020/kokkos-kernels/unit_test/blas/Test_Blas_gesv.hpp:229: Failure
1: Value of: true
1: Expected: test_flag
1: Which is: false
1: /path/to/workspace/KokkosKernels_PullRequest_Tpls_ARMPL2110_Tpls_ARMPL2030_GCC1020/kokkos-kernels/unit_test/blas/Test_Blas_gesv.hpp:229: Failure
1: Value of: true
1: Expected: test_flag
1: Which is: false
1: /path/to/workspace/KokkosKernels_PullRequest_Tpls_ARMPL2110_Tpls_ARMPL2030_GCC1020/kokkos-kernels/unit_test/blas/Test_Blas_gesv.hpp:229: Failure
1: Value of: true
1: Expected: test_flag
1: Which is: false
1: [  FAILED  ] openmp.gesv_mrhs_complex_double (91 ms)

Reproducer instructions

cd kokkos/
git checkout -f 0d19eebfa26d076f551d5b7a43230f627887df21
cd ../kokkos-kernels/
git checkout -f f5d7490dee7751a5a3cff8242e7de9f6ad6fe5b2
cd ../
mkdir testing
cd testing/
../kokkos-kernels/scripts/cm_test_all_sandia --spot-check-tpls armpl/21.1.0 --with-tpls=armpl --kokkos-path=../kokkos --kokkoskernels-path=../kokkos-kernels

[vqd8a]

@e10harvey I will look at this issue.

[vqd8a]

@e10harvey I have looked into this. I think these failures are only caused by ARMPL because when I tested with OpenBLAS TPL, these openmp gesv tests passed (Note that gesv currently can only work with either MAGMA TPL or BLAS TPL, we don't have a fall-back implementation yet, sorry for my lazziness).

An interesting thing is that these gesv tests fail with even numbers of OMP_NUM_THREADS. If we set OMP_NUM_THREADS to an odd number (ex: 47, 21, ...), these tests pass.
Because it might be an ARMPL issue, can we set the OMP_NUM_THREADS in A64FX CI testing to an odd number for now?

[ndellingwood]

The hip.gesv_double and hip.gesv_mrhs_double also fail with Hip when compiled with rocm/4.5.0 for MI100 arch, at least when building through Trilinos. I'll open a separate issue but wanted to comment here in case there is a common underlying issue to the hip case and this issue

[e10harvey]

@vqd8a : Are there any updates on this? Once this is resolved, we can enable Armpl CI checks and improve our code coverage.

[vqd8a]

@e10harvey Hi Evan, I do not have any update recently. I will find time to look at these tests on Hip.
But for ARMPL, I think this is an ARMPL issue. And as I mentioned above, can you set up an odd OMP_NUM_THREADS in A64FX CI testing to avoid these failures?

[e10harvey]

Thanks, @vqd8a. Yes, I will set OMP_NUM_THREADS. Do you already have a bug report filed for armpl?

[vqd8a]

@e10harvey I did not. I can do that.

[e10harvey]

Thank you, @vqd8a. Can this be closed?

[vqd8a]

Yes. I am closing it.