Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Training spaCy project file in Anaconda - OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized. #7664

Closed
EdmondStassen opened this issue Apr 6, 2021 · 6 comments
Closed

Training spaCy project file in Anaconda - OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized. #7664

EdmondStassen opened this issue Apr 6, 2021 · 6 comments
Labels
conda conda package manager windows Issues related to Windows

Comments

@EdmondStassen
Copy link

EdmondStassen commented Apr 6, 2021 •
edited by polm
Loading

How to reproduce the behaviour

Running Spacy project in Anaconda gives OpenMP error.

Happens in clean windows installation (reinstalled) and either in clean environment of conda or miniconda (both installed on clean environment) when using individual Anaconda build. For example, when running the ner_drugs tutorial.

OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

Tried to downgrade intel-openmp to 2020.2 (conda install -n spacy intel-openmp=2020.2) but didn't solve.

When installed in clean (non Anaconda) Python environment (in virtual environment via https://docs.python.org/3/tutorial/venv.html) it does not have this error!

Your Environment

  • spaCy version: 3.0.5
  • Platform: Windows-10-10.0.19041-SP0
  • Python version: 3.9.2
  • Pipelines: en_core_web_sm (3.0.0), nl_core_news_sm (3.0.0)

Using clean Anaconda individual installation on top of reinstalled windows version.

SpaCy project output

 python -m spacy project run all --force
ℹ Running workflow 'all'

================================= preprocess =================================
Running command: 'C:\Users\X\miniconda3\envs\spacy\python.exe' scripts/preprocess.py assets/drugs_training.jsonl corpus/drugs_training.spacy
Processed 1477 documents: drugs_training.spacy
Running command: 'C:\Users\X\miniconda3\envs\spacy\python.exe' scripts/preprocess.py assets/drugs_eval.jsonl corpus/drugs_eval.spacy
Processed 500 documents: drugs_eval.spacy

=================================== train ===================================
Running command: 'C:\Users\X\miniconda3\envs\spacy\python.exe' -m spacy train configs/config.cfg --output training/ --paths.train corpus/drugs_training.spacy --paths.dev corpus/drugs_eval.spacy
ℹ Using CPU

=========================== Initializing pipeline ===========================
[2021-04-06 12:56:42,990] [INFO] Set up nlp object from config
[2021-04-06 12:56:43,028] [INFO] Pipeline: ['tok2vec', 'ner']
[2021-04-06 12:56:43,028] [INFO] Created vocabulary
[2021-04-06 12:56:43,028] [INFO] Finished initializing nlp object
[2021-04-06 12:56:44,645] [INFO] Initialized pipeline components: ['tok2vec', 'ner']
✔ Initialized pipeline

============================= Training pipeline =============================
ℹ Pipeline: ['tok2vec', 'ner']
ℹ Initial learn rate: 0.0
E    #       LOSS TOK2VEC  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE
---  ------  ------------  --------  ------  ------  ------  ------
OMP: Error #15: Initializing libiomp5md.dll, but found libiomp5md.dll already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

List of packages in environment

# packages in environment at C:\Users\[...]\miniconda3\envs\spacy:
#
# Name                    Version                   Build  Channel
aiohttp                   3.7.4            py39hb82d6ee_0    conda-forge
async-timeout             3.0.1                   py_1000    conda-forge
attrs                     20.3.0             pyhd3deb0d_0    conda-forge
boto                      2.49.0                     py_0    conda-forge
boto3                     1.17.45            pyhd8ed1ab_0    conda-forge
botocore                  1.20.45            pyhd8ed1ab_0    conda-forge
brotlipy                  0.7.0           py39hb82d6ee_1001    conda-forge
bz2file                   0.98                       py_0    conda-forge
ca-certificates           2020.12.5            h5b45459_0    conda-forge
cachetools                4.2.1              pyhd8ed1ab_0    conda-forge
catalogue                 2.0.1            py39hcbf5309_2    conda-forge
certifi                   2020.12.5        py39hcbf5309_1    conda-forge
cffi                      1.14.5           py39h0878f49_0    conda-forge
chardet                   4.0.0            py39hcbf5309_1    conda-forge
click                     7.1.2              pyh9f0ad1d_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
cryptography              3.4.7            py39hd8d06c1_0    conda-forge
cymem                     2.0.5            py39h415ef7b_1    conda-forge
cython-blis               0.7.4            py39h5d4886f_0    conda-forge
dataclasses               0.8                pyhc8e2a94_1    conda-forge
en-core-web-sm            3.0.0                    pypi_0    pypi
ftfy                      5.9                      pypi_0    pypi
google-api-core           1.26.2             pyhd8ed1ab_0    conda-forge
google-auth               1.28.0             pyh44b312d_0    conda-forge
google-cloud-core         1.5.0              pyhd3deb0d_0    conda-forge
google-cloud-storage      1.19.0                     py_0    conda-forge
google-crc32c             1.1.2            py39h3fc79e4_0    conda-forge
google-resumable-media    1.2.0              pyhd3deb0d_0    conda-forge
googleapis-common-protos  1.53.0           py39h35db3c3_0    conda-forge
grpcio                    1.36.1           py39hb76b349_0    conda-forge
idna                      2.10               pyh9f0ad1d_0    conda-forge
intel-openmp              2021.2.0           h57928b3_616    conda-forge 
---> also tried downgrade but didn't work (conda install -n spacy intel-openmp=2020.2)

jinja2                    2.11.3             pyh44b312d_0    conda-forge
jmespath                  0.10.0             pyh9f0ad1d_0    conda-forge
joblib                    1.0.1                    pypi_0    pypi
libblas                   3.9.0                     8_mkl    conda-forge
libcblas                  3.9.0                     8_mkl    conda-forge
libcrc32c                 1.1.1                h0e60522_2    conda-forge
liblapack                 3.9.0                     8_mkl    conda-forge
libprotobuf               3.15.7               h7755175_0    conda-forge
markupsafe                1.1.1            py39hb82d6ee_3    conda-forge
mkl                       2020.4             hb70f87d_311    conda-forge
multidict                 5.1.0            py39hb82d6ee_1    conda-forge
murmurhash                1.0.5            py39h415ef7b_0    conda-forge
nl-core-news-sm           3.0.0                    pypi_0    pypi
numpy                     1.20.2           py39h6635163_0    conda-forge
openssl                   1.1.1k               h8ffe710_0    conda-forge
packaging                 20.9               pyh44b312d_0    conda-forge
pathy                     0.4.0              pyhd8ed1ab_0    conda-forge
pip                       21.0.1             pyhd8ed1ab_0    conda-forge
preshed                   3.0.5            py39h415ef7b_0    conda-forge
protobuf                  3.15.7           py39h415ef7b_0    conda-forge
pyasn1                    0.4.8                      py_0    conda-forge
pyasn1-modules            0.2.7                      py_0    conda-forge
pycparser                 2.20               pyh9f0ad1d_2    conda-forge
pydantic                  1.7.3            py39hb82d6ee_1    conda-forge
pyopenssl                 20.0.1             pyhd8ed1ab_0    conda-forge
pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
pysocks                   1.7.1            py39hcbf5309_3    conda-forge
python                    3.9.2           h7840368_0_cpython    conda-forge
python-dateutil           2.8.1                      py_0    conda-forge
python_abi                3.9                      1_cp39    conda-forge
pytz                      2021.1             pyhd8ed1ab_0    conda-forge
regex                     2021.4.4                 pypi_0    pypi
requests                  2.25.1             pyhd3deb0d_0    conda-forge
rsa                       4.7.2              pyh44b312d_0    conda-forge
s3transfer                0.3.6              pyhd8ed1ab_0    conda-forge
sacremoses                0.0.44                   pypi_0    pypi
setuptools                49.6.0           py39hcbf5309_3    conda-forge
shellingham               1.4.0              pyh44b312d_0    conda-forge
six                       1.15.0             pyh9f0ad1d_0    conda-forge
smart_open                2.2.1              pyh9f0ad1d_0    conda-forge
spacy                     3.0.5            py39hefe7e4c_0    conda-forge
spacy-alignments          0.7.2                    pypi_0    pypi
spacy-legacy              3.0.2              pyhd8ed1ab_0    conda-forge
spacy-transformers        1.0.1                    pypi_0    pypi
sqlite                    3.35.4               h8ffe710_0    conda-forge
srsly                     2.4.0            py39h415ef7b_2    conda-forge
thinc                     8.0.2            py39hefe7e4c_1    conda-forge
tokenizers                0.9.4                    pypi_0    pypi
torch                     1.8.1                    pypi_0    pypi
torchcontrib              0.0.2                    pypi_0    pypi
tqdm                      4.60.0             pyhd8ed1ab_0    conda-forge
transformers              4.2.2                    pypi_0    pypi
typer                     0.3.2              pyhd8ed1ab_0    conda-forge
typing-extensions         3.7.4.3                       0    conda-forge
typing_extensions         3.7.4.3                    py_0    conda-forge
tzdata                    2021a                he74cb21_0    conda-forge
urllib3                   1.26.4             pyhd8ed1ab_0    conda-forge
vc                        14.2                 hb210afc_4    conda-forge
vs2015_runtime            14.28.29325          h5e1d092_4    conda-forge
wasabi                    0.8.2              pyh44b312d_0    conda-forge
wcwidth                   0.2.5                    pypi_0    pypi
wheel                     0.36.2             pyhd3deb0d_0    conda-forge
win_inet_pton             1.1.0            py39hcbf5309_2    conda-forge
wincertstore              0.2             py39hcbf5309_1006    conda-forge
yarl                      1.6.3            py39hb82d6ee_1    conda-forge
zlib                      1.2.11            h62dcd97_1010    conda-forge
@polm polm added conda conda package manager windows Issues related to Windows labels Apr 7, 2021
@svlandeg
Copy link
Member

svlandeg commented Apr 7, 2021

Hm. I've used Anaconda & Windows with spaCy before, but I've never run into anything remotely similar :|

@rmitanch
Copy link

rmitanch commented Apr 23, 2021 •
edited by polm
Loading

Hi Edmond,
You can try

import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'

Hope this helps
Richard

@svlandeg svlandeg added the more-info-needed This issue needs more information label Apr 23, 2021
@EdmondStassen
Copy link
Author

This solved the issue!

import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'

Thank you @rmitanch

@no-response no-response bot removed the more-info-needed This issue needs more information label Apr 26, 2021
@azarkiel
Copy link

This solved the issue!

import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'

Thank you @rmitanch

I had same error but trying to execute EfficientPose github project, and this solved it.
I just added this 2 lines to .py file and now it is working.
Thanks a lot.

@polm polm mentioned this issue Jun 12, 2021
Closed
@Dhilen-Chin
Copy link

Hi Edmond,
You can try

import os
os.environ['KMP_DUPLICATE_LIB_OK']='True'

Hope this helps
Richard

Where are you meant to add these 2 lines to?

@polm
Copy link
Contributor

polm commented Jul 7, 2021

@Dhilen-Chin I answered you in #8366, please don't post the same question in multiple issues, it's disruptive.

I'm going to lock this issue, any comments on issues with this DLL can go in #8366.

@explosion explosion locked as resolved and limited conversation to collaborators Jul 7, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
conda conda package manager windows Issues related to Windows
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@polm @rmitanch @EdmondStassen @svlandeg @azarkiel @Dhilen-Chin