pip list --outdated fails on platforms where sem_open exists but is not implemented

[richboss]

This is not the same issue as #8161

Environment

• pip version: 20.2.4
• Python version: 3.8.6
• OS: Debian GNU/Linux bullseye/sid

Debian Chroot installed on a Synology DS214 NAS

Description

pip list --outdated fails, printing a stacktrace as follows:

ERROR: Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/cli/base_command.py", line 228, in _main
    status = self.run(options, args)
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/commands/list.py", line 165, in run
    packages = self.get_outdated(packages, options)
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/commands/list.py", line 174, in get_outdated
    return [
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/commands/list.py", line 174, in <listcomp>
    return [
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/commands/list.py", line 228, in iter_packages_latest_infos
    for dist in map_multithread(latest_info, packages):
  File "/usr/local/lib/python3.8/dist-packages/pip/_internal/utils/parallel.py", line 105, in _map_multithread
    with closing(ThreadPool(DEFAULT_POOLSIZE)) as pool:
  File "/usr/lib/python3.8/multiprocessing/dummy/__init__.py", line 124, in Pool
    return ThreadPool(processes, initializer, initargs)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 925, in __init__
    Pool.__init__(self, processes, initializer, initargs)
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 196, in __init__
    self._change_notifier = self._ctx.SimpleQueue()
  File "/usr/lib/python3.8/multiprocessing/context.py", line 113, in SimpleQueue
    return SimpleQueue(ctx=self.get_context())
  File "/usr/lib/python3.8/multiprocessing/queues.py", line 336, in __init__
    self._rlock = ctx.Lock()
  File "/usr/lib/python3.8/multiprocessing/context.py", line 68, in Lock
    return Lock(ctx=self.get_context())
  File "/usr/lib/python3.8/multiprocessing/synchronize.py", line 160, in __init__
    SemLock.__init__(self, SEMAPHORE, 1, 1, ctx=ctx)
  File "/usr/lib/python3.8/multiprocessing/synchronize.py", line 57, in __init__
    sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue, self._make_name(), unlink_now)
OSError: [Errno 38] Die angeforderte Funktion ist nicht implementiert

Analysis
On this system, multiprocessing.synchronize appears to be available (but it is not working). The check in file _internal/utils/parallel.py detects no problem since the statement import multiprocessing.synchronize does not fail. Hence multiprocessing or multithreading will be used in pip. But the necessessary functions are not implemented! Therefore an OSError is raised, as shown.

Proposed solution
An additional check should detect if multiprocessing is really working. For example by creating a ProcessPool or instantiating a SemLock. Checking for import failures only is not enough. This doesn't help at all if an error is only raised when an object is instantiated.

An alternative would be adding a configuration option to disable the use of multiprocessing, even if it seems available.

[McSinyx]

Hello there, thank you for reporting and suggesting the fixes. From pip's standpoint, I'm siding with

For example by creating a ProcessPool or instantiating a SemLock. Checking for import failures only is not enough.

Although it's not perfect, pip already has quite many options.

On the other hand, it seems that the CPython binary you're using is not properly built. IIUC, if sem_open is not implemented on the platform, CPython should be compiled without HAVE_SEM_OPEN.

[richboss]

On the other hand, it seems that the CPython binary you're using is not properly built. IIUC, if sem_open is not implemented on the platform, CPython should be compiled [without HAVE_SEM_OPEN]

I believe you are right. The package seems to be compiled with HAVE_SEM_OPEN=1 (see below), but the implementation is missing.

Python 3.8.6 (default, Sep 25 2020, 09:36:53)
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import _multiprocessing as mp
>>> mp.flags
{'HAVE_SEM_OPEN': 1, 'HAVE_SEM_TIMEDWAIT': 1}

I believe the following might be the package that is at fault (dpkg -l libpython3.8-stdlib):

||/ Name                      Version      Architektur  Beschreibung
+++-=========================-============-============-===============================================================================
ii  libpython3.8-stdlib:armhf 3.8.6-1      armhf        Interactive high-level object-oriented language (standard library, version 3.8)

I'll try to file a bug report at bugs.debian.org. But I am unsure if this will amount to anything. Alas, my attempts at finding a solution on my own were not successful. As a workaround for the time being I wrote a simple patch script for local use which I will run each time when pip gets updated.

[pradyunsg]

It seems that the issue is with how their Python binaries for ARM are compiled, so I'd also suggest filing a report with Debian. Please do link to the report from here, if and when you file it. :)

[richboss]

Please do link to the report from here, if and when you file it. :)

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=973460

[ichard26]

Closing this out as A) it's a really bug with how Python binaries for ARM are compiled, and B) pip list hasn't been parallelized for a good while (although it'd be nice to reintroduce it at some point...).