dc.get_resource("MSigDB", organism = "mouse") fails with errors in pypath

[Christian-Heyer]

Describe the bug
Decoupler fails to Import MSigDB resouce using when trying to get the "mouse" version of the genes.

I am not quite sure if this is a problem lies in deocupleR or in pypath. I've seen that similar issues have been raised in the past saezlab/pypath#218 but none of the workarounds solved the error.

I've run into the error on python 3.11 with the latest versions of pypath and decoupler-py installed from github.

To Reproduce

import decoupler as dc
msigdb = dc.get_resource('MSigDB', organism = "mouse")
msigdb

Traceback:

>>> msigdb = dc.get_resource('MSigDB',organism = "mouse")                                                                                                                                                                        
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/decoupler/omnip.py", line 229, in get_resource
    df = translate_net(
         ^^^^^^^^^^^^^^
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/decoupler/omnip.py", line 586, in translate_net
    hom_net = homology.translate_df(
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/homology.py", line 1898, in translate_df
    return manager.translate_df(**args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/homology.py", line 477, in translate_df
    table = self.which_table(
            ^^^^^^^^^^^^^^^^^
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/homology.py", line 178, in which_table
    self.load(key)
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/homology.py", line 209, in load
    self.tables[key] = self._load(key)
                       ^^^^^^^^^^^^^^^
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/homology.py", line 219, in _load
    return ProteinHomology(
           ^^^^^^^^^^^^^^^^
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/homology.py", line 712, in __init__
    self.load(source)
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/homology.py", line 728, in load
    self.load_homologene(source)
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/homology.py", line 1203, in load_homologene
    mapping.map_name(e, 'entrez', 'uniprot', self.target)
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/mapping.py", line 3564, in map_name
    return mapper.map_name(
           ^^^^^^^^^^^^^^^^
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/share/common.py", line 2772, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/mapping.py", line 2197, in map_name
    mapped_names = self.uniprot_cleanup(
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/mapping.py", line 2226, in uniprot_cleanup
    uniprots = self.primary_uniprot(uniprots)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/mapping.py", line 2838, in primary_uniprot
    primary = self.map_name(
              ^^^^^^^^^^^^^^
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/share/common.py", line 2772, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/mapping.py", line 1982, in map_name
    mapped_names = self._map_name(
                   ^^^^^^^^^^^^^^^
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/mapping.py", line 2514, in _map_name
    tbl = self.which_table(
          ^^^^^^^^^^^^^^^^^
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/mapping.py", line 1575, in which_table
    self.load_mapping(
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/mapping.py", line 3212, in load_mapping
    reader = MapReader(param = resource, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/mapping.py", line 257, in __init__
    self.load()
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/mapping.py", line 287, in load
    self.read()
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/mapping.py", line 449, in read
    getattr(self, method)()
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/utils/mapping.py", line 560, in read_mapping_file
    for i, line in enumerate(infile):
  File "/omics/groups/OE0436/internal/heyer/conda/envs/decoupler_test/lib/python3.11/site-packages/pypath/inputs/uniprot.py", line 479, in get_uniprot_sec
    enumerate(c.result)
TypeError: 'NoneType' object is not iterable

Expected behavior
Expect array to be loaded with MSigDB data from given organism.

System

• OS: CentOS 7 & MacOS 13.4.1
• Python version 3.11.4
• Versions of libraries involved
  • decoupler-py d768a29
  • pypath saezlab/pypath@d732a52
  • python 3.11.4 <- installed via conda

Let me know if you require any further information from me.

Additional context
Output Pip Freeze:

(decoupler_test)$ pip freeze
anndata==0.9.1
attrs==23.1.0
bcrypt==4.0.1
beautifulsoup4==4.12.2
boltons==23.0.0
certifi==2023.5.7
cffi==1.15.1
charset-normalizer==3.1.0
contourpy==1.1.0
cryptography==41.0.1
cycler==0.11.0
decoupler @ git+https://github.com/saezlab/decoupler-py@d768a29731377c7086c308d52004c4bf35014b43
dill==0.3.6
docrep==0.3.2
et-xmlfile==1.1.0
face==20.1.1
fonttools==4.40.0
future==0.18.3
glom==23.3.0
h5py==3.9.0
idna==3.4
inflect==6.0.4
kiwisolver==1.4.4
llvmlite==0.40.1
lxml==4.9.2
matplotlib==3.7.1
natsort==8.4.0
numba==0.57.1
numpy==1.24.4
omnipath==1.0.7
openpyxl==3.1.2
packaging==23.1
pandas==2.0.3
paramiko==3.2.0
Pillow==9.5.0
psutil==5.9.5
pycparser==2.21
pycurl==7.45.1
pydantic==1.10.9
PyNaCl==1.5.0
pyparsing==3.1.0
pypath-omnipath @ git+https://github.com/saezlab/pypath.git@d732a52f6a4ad30c0c56040d69d94c6583b06e42
pyreadr==0.4.7
pysftp==0.2.9
python-dateutil==2.8.2
pytz==2023.3
PyYAML==6.0
rdata==0.9
requests==2.31.0
scipy==1.11.1
six==1.16.0
soupsieve==2.4.1
sqlparse==0.4.4
tabulate==0.9.0
timeloop==1.0.2
toml==0.10.2
tqdm==4.65.0
typing_extensions==4.7.0
tzdata==2023.3
urllib3==2.0.3
wrapt==1.15.0
xarray==2023.6.0
xlrd==2.0.1
(decoupler_test) $ 

[PauBadiaM]

Hi @Christian-Heyer ,

Thanks for raising this issue. It looks like there might be an error with python 3.11 and pypath, @deeenes any idea what could be causing it?

[deeenes]

Because this happens in pypath.inputs.uniprot.get_uniprot_sec, most likely it is a download issue, a transient network error. It makes sense to try it again, or try on a different network. I've just tested it right now with empty cache and it worked.

However, even if this error is successfully addressed as I suggest above, we might expect another error due to the recent transition to the new UniProt API in pypath. In this case, I have to ask a little more patience, this is a major change in pypath and still a few things don't work, that we're trying to fix asap.

[PauBadiaM]

I see the same error happening in the builds of decoupler @deeenes. What is weird is that if I rerun the jobs, eventually it works, so it might be a transient network error. Is there anything that could be done to make it more stable?

[Christian-Heyer]

Thanks for the quick answer. RIght now I am also working behind a proxy server so that might also interfere with network connectivity. For now I'll just download the data with OmnipathR and ship it over to python (or just stick with R for now).