oneDNN 2 missing headerfiles #19690

leezu · 2020-12-17T16:24:34Z

#19667 breaks Horovod (cf horovod/horovod#2530) as some header files are missing in the pip wheel:/usr/local/lib/python3.6/dist-packages/mxnet/include/mkldnn/oneapi/dnnl/dnnl.hpp:23:10: fatal error: oneapi/dnnl/dnnl_config.h: No such file or directory

cc @bartekkuncer

The text was updated successfully, but these errors were encountered:

leezu · 2020-12-17T16:25:30Z

Related code on MXNet side is at https://github.com/apache/incubator-mxnet/blob/91503f71e17fca9151779503fc9f5edefe26f2ef/tools/pip/setup.py#L148-L150

sxjscience · 2020-12-18T05:48:21Z

I've met this error too. @bartekkuncer Is it possible to add the header to pip wheel? Also ping @szha

sxjscience · 2020-12-18T05:56:46Z

In terms of the wheel, I think the last wheel that works is 20201214.

bartekkuncer · 2020-12-18T13:13:38Z

@leezu I believe that changing L149 to: shutil.copytree(os.path.join(CURRENT_DIR, 'mxnet-build/3rdparty/mkldnn/include/oneapi/dnnl'), fixes the issue but as I am not familiar with horovod I do not know how to check if the fix works. Can you provide me with a way to reproduce the issue?

sxjscience · 2020-12-18T16:00:39Z

@bartekkuncer For me, I met this error when trying to install horovod (you may change the cuda version):

python3 -m pip install -U --pre "mxnet-cu102==2.0.0b20201217" -f https://dist.mxnet.io/python
HOROVOD_GPU_OPERATIONS=NCCL HOROVOD_WITHOUT_GLOO=1 HOROVOD_WITH_MPI=1 HOROVOD_WITH_MXNET=1 HOROVOD_WITHOUT_PYTORCH=1 HOROVOD_WITHOUT_TENSORFLOW=1 python3 -m pip install --no-cache-dir horovod

szha · 2020-12-18T16:04:21Z

@bartekkuncer I recommend the following for verification:

Download a wheel with the correct headers (e.g. the 20201214 one that @sxjscience mentioend) and one without (the ones after 1214)
Unzip them and examine the header content and see where the headers should be.
Correct the toolts/pip/setup.py and use the wheel build scripts to build correct wheel.
Verify if the headers are in the right location in the resulting wheel.

leezu · 2020-12-18T18:56:55Z

Thanks @bartekkuncer! I opened #19694 as weekend has started in your timezone

leezu · 2020-12-21T14:38:09Z

Horovod now fails with

�[0m�[91m    In file included from /tmp/pip-req-build-bhade3mm/horovod/mxnet/mpi_ops.h:24:0,
                     from /tmp/pip-req-build-bhade3mm/horovod/mxnet/mpi_ops.cc:21:
    /usr/local/lib/python3.6/dist-packages/mxnet/include/mxnet/ndarray.h:41:10: fatal error: mkldnn.hpp: No such file or directory
�[0m�[91m     #include <mkldnn.hpp>
�[0m�[91m              ^~~~~~~~~~~~
�[0m�[91m    compilation terminated.
�[0m�[91m    horovod/mxnet/CMakeFiles/mxnet.dir/build.make:758: recipe for target 'horovod/mxnet/CMakeFiles/mxnet.dir/mpi_ops.cc.o' failed

bartekkuncer · 2020-12-21T14:41:50Z

Horovod now fails with

�[0m�[91m    In file included from /tmp/pip-req-build-bhade3mm/horovod/mxnet/mpi_ops.h:24:0,
                     from /tmp/pip-req-build-bhade3mm/horovod/mxnet/mpi_ops.cc:21:
    /usr/local/lib/python3.6/dist-packages/mxnet/include/mxnet/ndarray.h:41:10: fatal error: mkldnn.hpp: No such file or directory
�[0m�[91m     #include <mkldnn.hpp>
�[0m�[91m              ^~~~~~~~~~~~
�[0m�[91m    compilation terminated.
�[0m�[91m    horovod/mxnet/CMakeFiles/mxnet.dir/build.make:758: recipe for target 'horovod/mxnet/CMakeFiles/mxnet.dir/mpi_ops.cc.o' failed

Yes, I saw that, working on the fix.

leezu · 2020-12-21T14:42:31Z

Thanks @bartekkuncer!

bartekkuncer · 2020-12-22T15:47:30Z

@leezu I think this PR #19706 should fix the problem.

leezu · 2020-12-30T14:13:42Z

@bartekkuncer looks like the dnnl_config.h is included in the wrong directory. It should be in oneapi/dnnl/dnnl_config.h at least horovod build fails with usr/local/lib/python3.6/dist-packages/mxnet/include/mkldnn/oneapi/dnnl/dnnl.hpp:23:10: fatal error: oneapi/dnnl/dnnl_config.h: No such file or directory #include "oneapi/dnnl/dnnl_config.h")

leezu · 2020-12-30T14:18:02Z

The files included in are

mxnet/include/mkldnn
mxnet/include/mkldnn/mkldnn_version.h
mxnet/include/mkldnn/dnnl_debug.h
mxnet/include/mkldnn/mkldnn_debug.h
mxnet/include/mkldnn/dnnl_ocl.h
mxnet/include/mkldnn/dnnl_sycl.h
mxnet/include/mkldnn/dnnl_ocl.hpp
mxnet/include/mkldnn/mkldnn_types.h
mxnet/include/mkldnn/dnnl_version.h
mxnet/include/mkldnn/oneapi
mxnet/include/mkldnn/oneapi/dnnl
mxnet/include/mkldnn/oneapi/dnnl/dnnl_debug.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_ocl.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_sycl.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_ocl.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_types.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_sycl_types.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_threadpool_iface.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_threadpool.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_sycl.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_threadpool.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl.h
mxnet/include/mkldnn/mkldnn_config.h
mxnet/include/mkldnn/dnnl_types.h
mxnet/include/mkldnn/dnnl.hpp
mxnet/include/mkldnn/dnnl_config.h
mxnet/include/mkldnn/mkldnn.hpp
mxnet/include/mkldnn/mkldnn_dnnl_mangling.h
mxnet/include/mkldnn/dnnl_sycl_types.h
mxnet/include/mkldnn/dnnl_threadpool_iface.hpp
mxnet/include/mkldnn/dnnl_threadpool.hpp
mxnet/include/mkldnn/dnnl_sycl.hpp
mxnet/include/mkldnn/dnnl_threadpool.h
mxnet/include/mkldnn/mkldnn.h
mxnet/include/mkldnn/dnnl.h

I think we may need to update https://github.com/apache/incubator-mxnet/blob/3c5beb3596b6bc01f77bc7ddd14ed90221c31950/cd/mxnet_lib/static/Jenkins_pipeline.groovy#L36 to ensure that the config files are stashed correctly on the CD

szha · 2020-12-31T01:04:09Z

we might also consider making it robust in setup.py by asserting the existence of these header files instead of only include when available.

bartekkuncer · 2021-01-04T09:34:01Z

The files included in are

mxnet/include/mkldnn
mxnet/include/mkldnn/mkldnn_version.h
mxnet/include/mkldnn/dnnl_debug.h
mxnet/include/mkldnn/mkldnn_debug.h
mxnet/include/mkldnn/dnnl_ocl.h
mxnet/include/mkldnn/dnnl_sycl.h
mxnet/include/mkldnn/dnnl_ocl.hpp
mxnet/include/mkldnn/mkldnn_types.h
mxnet/include/mkldnn/dnnl_version.h
mxnet/include/mkldnn/oneapi
mxnet/include/mkldnn/oneapi/dnnl
mxnet/include/mkldnn/oneapi/dnnl/dnnl_debug.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_ocl.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_sycl.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_ocl.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_types.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_sycl_types.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl_threadpool_iface.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_threadpool.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_sycl.hpp
mxnet/include/mkldnn/oneapi/dnnl/dnnl_threadpool.h
mxnet/include/mkldnn/oneapi/dnnl/dnnl.h
mxnet/include/mkldnn/mkldnn_config.h
mxnet/include/mkldnn/dnnl_types.h
mxnet/include/mkldnn/dnnl.hpp
mxnet/include/mkldnn/dnnl_config.h
mxnet/include/mkldnn/mkldnn.hpp
mxnet/include/mkldnn/mkldnn_dnnl_mangling.h
mxnet/include/mkldnn/dnnl_sycl_types.h
mxnet/include/mkldnn/dnnl_threadpool_iface.hpp
mxnet/include/mkldnn/dnnl_threadpool.hpp
mxnet/include/mkldnn/dnnl_sycl.hpp
mxnet/include/mkldnn/dnnl_threadpool.h
mxnet/include/mkldnn/mkldnn.h
mxnet/include/mkldnn/dnnl.h

I think we may need to update

https://github.com/apache/incubator-mxnet/blob/3c5beb3596b6bc01f77bc7ddd14ed90221c31950/cd/mxnet_lib/static/Jenkins_pipeline.groovy#L36

to ensure that the config files are stashed correctly on the CD

Thanks @leezu . I changed it in CI but must have overlooked it in CD.

leezu · 2021-01-04T14:51:07Z

It looks like there are still more issues with the CD. Horovod still fails with

�[0m�[91m    /usr/local/lib/python3.6/dist-packages/mxnet/include/mkldnn/oneapi/dnnl/dnnl.hpp:23:10: fatal error: oneapi/dnnl/dnnl_config.h: No such file or directory
�[0m�[91m     #include "oneapi/dnnl/dnnl_config.h"
�[0m�[91m              ^~~~~~~~~~~~~~~~~~~~~~~~~~~
�[0m�[91m    compilation terminated.
�[0m�[91m    horovod/mxnet/CMakeFiles/mxnet.dir/build.make:758: recipe for target 'horovod/mxnet/CMakeFiles/mxnet.dir/mpi_ops.cc.o' failed

bartekkuncer · 2021-01-04T15:37:00Z

It looks like there are still more issues with the CD. Horovod still fails with

�[0m�[91m    /usr/local/lib/python3.6/dist-packages/mxnet/include/mkldnn/oneapi/dnnl/dnnl.hpp:23:10: fatal error: oneapi/dnnl/dnnl_config.h: No such file or directory
�[0m�[91m     #include "oneapi/dnnl/dnnl_config.h"
�[0m�[91m              ^~~~~~~~~~~~~~~~~~~~~~~~~~~
�[0m�[91m    compilation terminated.
�[0m�[91m    horovod/mxnet/CMakeFiles/mxnet.dir/build.make:758: recipe for target 'horovod/mxnet/CMakeFiles/mxnet.dir/mpi_ops.cc.o' failed

Yes, I saw that. #19726 should fix the issue.

leezu · 2021-01-06T16:38:41Z

Thank you @bartekkuncer!

leezu added Bug needs triage and removed needs triage labels Dec 17, 2020

leezu mentioned this issue Dec 18, 2020

Update tools/pip/setup.py for oneDNN v2 #19694

Merged

leezu mentioned this issue Dec 20, 2020

Revert "Pin mxnet-nightly following missing file oneapi/dnnl/dnnl_config.h" horovod/horovod#2540

Merged

bartekkuncer added a commit to bartekkuncer/incubator-mxnet that referenced this issue Dec 22, 2020

Fix build problems with oneDNN v2.0 apache#19690

42b965c

bartekkuncer mentioned this issue Dec 22, 2020

Fix build problems with oneDNN v2.0 #19690 #19706

Merged

bartekkuncer added a commit to bartekkuncer/incubator-mxnet that referenced this issue Dec 22, 2020

Fix build problems with oneDNN v2.0 apache#19690

3ba2ad8

bartekkuncer added a commit to bartekkuncer/incubator-mxnet that referenced this issue Dec 22, 2020

Fix build problems with oneDNN v2.0 apache#19690

931bea9

bartekkuncer added a commit to bartekkuncer/incubator-mxnet that referenced this issue Dec 22, 2020

Fix build problems with oneDNN v2.0 apache#19690

6a1bbd7

bartekkuncer added a commit to bartekkuncer/incubator-mxnet that referenced this issue Dec 23, 2020

Fix build problems with oneDNN v2.0 apache#19690

7a3e00a

leezu closed this as completed in #19706 Dec 24, 2020

leezu reopened this Dec 30, 2020

leezu mentioned this issue Dec 30, 2020

Update CD Jenkins config for include/mkldnn/oneapi/dnnl #19725

Merged

bartekkuncer mentioned this issue Jan 4, 2021

Fix cd pip build #19726

Merged

leezu closed this as completed Jan 6, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

oneDNN 2 missing headerfiles #19690

oneDNN 2 missing headerfiles #19690

leezu commented Dec 17, 2020

leezu commented Dec 17, 2020

sxjscience commented Dec 18, 2020

sxjscience commented Dec 18, 2020

bartekkuncer commented Dec 18, 2020

sxjscience commented Dec 18, 2020

szha commented Dec 18, 2020

leezu commented Dec 18, 2020

leezu commented Dec 21, 2020

bartekkuncer commented Dec 21, 2020

leezu commented Dec 21, 2020

bartekkuncer commented Dec 22, 2020

leezu commented Dec 30, 2020

leezu commented Dec 30, 2020

szha commented Dec 31, 2020

bartekkuncer commented Jan 4, 2021 •

edited

Loading

leezu commented Jan 4, 2021

bartekkuncer commented Jan 4, 2021

leezu commented Jan 6, 2021

oneDNN 2 missing headerfiles #19690

oneDNN 2 missing headerfiles #19690

Comments

leezu commented Dec 17, 2020

leezu commented Dec 17, 2020

sxjscience commented Dec 18, 2020

sxjscience commented Dec 18, 2020

bartekkuncer commented Dec 18, 2020

sxjscience commented Dec 18, 2020

szha commented Dec 18, 2020

leezu commented Dec 18, 2020

leezu commented Dec 21, 2020

bartekkuncer commented Dec 21, 2020

leezu commented Dec 21, 2020

bartekkuncer commented Dec 22, 2020

leezu commented Dec 30, 2020

leezu commented Dec 30, 2020

szha commented Dec 31, 2020

bartekkuncer commented Jan 4, 2021 • edited Loading

leezu commented Jan 4, 2021

bartekkuncer commented Jan 4, 2021

leezu commented Jan 6, 2021

bartekkuncer commented Jan 4, 2021 •

edited

Loading