Bug: cyclic dependency with tomli==1.2.1 and flit_core==3.2.0

[lukasoyen]

I can not download tomli because there is a cyclic dependency with flit_core.

Reproducer:

python3.9 -m venv test
./test/bin/pip download --no-binary=":all:"  tomli  # fails
./test/bin/pip download --no-binary=":all:" flit_core # fails

Example output:

Collecting tomli
  File was already downloaded XXX/tomli-1.2.1.tar.gz
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 2:
   command: XXX/test/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-91g152mw/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core>=3.2.0,<4'
       cwd: None
  Complete output (31 lines):
  Collecting flit_core<4,>=3.2.0
    Downloading flit_core-3.4.0.tar.gz (27 kB)
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: finished with status 'done'
  Collecting tomli
    Downloading tomli-1.2.1.tar.gz (14 kB)
  ERROR: Exception:
  Traceback (most recent call last):
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/cli/base_command.py", line 186, in _main
      status = self.run(options, args)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/commands/install.py", line 357, in run
      resolver.resolve(requirement_set)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/legacy_resolve.py", line 177, in resolve
      discovered_reqs.extend(self._resolve_one(requirement_set, req))
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/legacy_resolve.py", line 333, in _resolve_one
      abstract_dist = self._get_abstract_dist_for(req_to_install)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/legacy_resolve.py", line 282, in _get_abstract_dist_for
      abstract_dist = self.preparer.prepare_linked_requirement(req)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 515, in prepare_linked_requirement
      abstract_dist = _get_prepared_distribution(
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/operations/prepare.py", line 94, in _get_prepared_distribution
      with req_tracker.track(req):
    File "/usr/lib/python3.9/contextlib.py", line 117, in __enter__
      return next(self.gen)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/req/req_tracker.py", line 148, in track
      self.add(req)
    File "XXX/test/lib/python3.9/site-packages/pip/_internal/req/req_tracker.py", line 115, in add
      raise LookupError(message)
  LookupError: https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442 (from https://pypi.org/simple/tomli/) (requires-python:>=3.6) is already being built: tomli from https://files.pythonhosted.org/packages/75/50/973397c5ba854445bcc396b593b5db1958da6ab8d665b27397daa1497018/tomli-1.2.1.tar.gz#sha256=a5b75cb6f3968abb47af1b40c1819dc519ea82bcc065776a866e8d74c5ca9442
  ----------------------------------------
ERROR: Command errored out with exit status 2: XXX/test/bin/python3.9 XXX/test/lib/python3.9/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-91g152mw/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.org/simple -- 'flit_core>=3.2.0,<4' Check the logs for full command output.

[lukasoyen]

Companion ticket: pypa/flit#451

[hukkin]

I don't think this is a Tomli issue. You may want to look into bootstraping the foundational build/installer packages https://flit.readthedocs.io/en/latest/bootstrap.html

[lukasoyen]

I agree, to some extend this is a problem with pip download installing build-dependencies even though I only requested to download a source-package (without building it).

But even installing flit_core does not help. The following still reproduces the problem. Only --no-build-isolation helps.

~ ➤ python3.9 -m venv test                                                                                                                                                                    
~ ➤ ./test/bin/pip install flit_core                                                                                                                                                          
~ ➤ ./test/bin/pip download --no-binary=":all:" tomli # fails
~ ➤ ./test/bin/pip download --no-binary=":all:" tomli --no-build-isolation # works

But I am fine with having this closed. Although some part of me wishes for stricter dependency requirements so that problems like this don't happen again.

[hukkin]

Hmm, yeah maybe pip's issue tracker is the right place for this?

[takluyver]

The thing with --no-build-isolation makes sense - by default, pip creates a new environment to build packages, and installs build dependencies into that. --no-build-isolation tells it to use the current environment instead. Why pip download tries to do any kind of building, I'm not sure, but given that it does, that's what I'd expect.

My best guess is that it wants to get the list of dependencies even if you tell it not to fetch them (--no-deps). Getting package metadata is similar to building.

# This works - flit_core has no build dependencies
pip download --no-binary :all: flit_core --no-deps

# This doesn't
pip download --no-binary :all: tomli --no-deps

[pradyunsg]

Why pip download tries to do any kind of building, I'm not sure, but given that it does, that's what I'd expect.

pypa/pip#1884 :)