Pip doesn’t allow you to self-depend

[flying-sheep]

Description

When specifying dependencies, you sometimes want to depend on other extras:

[project]
name = 'my_pkg'

[project.optional-dependencies]
test-utils = ['pytest', 'requests-mock', '...']
test = ['my_pkg[test-utils]']

Expected behavior

@uranusjr said it should work:

Circular dependency is a feature that Python packaging is explicitly designed to allow, so it works and should continue to work. […]

pip version

21.2.4

Python version

3.9.6

OS

Arch Linux

How to Reproduce

1. Create a new package with something like the above setup
2. Install it with the extra depending on itself: pip install .[test]
3. It can’t find the package because it doesn’t merge . with my_pkg

Specifying test = ['.[test-utils]'] doesn’t work.

Output

@henryiii said:

It does not pull the extra from the current package, but rather from the existing cached packages. So if you add a new extra […] and then depend on it, it starts spewing out a long list of package checking:

Collecting my_pkg[test-utils]
  Using cached my_pkg-1.11.1-py3-none-any.whl (1.5 MB)
WARNING: my_pkg 1.11.1 does not provide the extra 'test-utils'
  Using cached my_pkg-1.11.0-py3-none-any.whl (1.5 MB)
WARNING: my_pkg 1.11.0 does not provide the extra 'test-utils'
  Using cached my_pkg-1.10.0-py3-none-any.whl (1.5 MB)
WARNING: my_pkg 1.10.0 does not provide the extra 'test-utils'

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[uranusjr]

Hmm, this actually works for me.

$ pip --version
pip 21.2.4 from C:\Users\uranusjr\Documents\programming\pip\src\pip (python 3.8)
$ pip install './p[test]' --use-feature=in-tree-build
Processing c:\users\uranusjr\documents\programming\pip\p
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting requests-mock
  Using cached requests_mock-1.9.3-py2.py3-none-any.whl (27 kB)
Collecting pytest
  Using cached pytest-6.2.5-py3-none-any.whl (280 kB)
Collecting packaging
  Using cached packaging-21.0-py3-none-any.whl (40 kB)
Collecting pluggy<2.0,>=0.12
  Using cached pluggy-1.0.0-py2.py3-none-any.whl (13 kB)
Collecting iniconfig
  Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Collecting py>=1.8.2
  Using cached py-1.10.0-py2.py3-none-any.whl (97 kB)
Collecting atomicwrites>=1.0
  Using cached atomicwrites-1.4.0-py2.py3-none-any.whl (6.8 kB)
Collecting colorama
  Using cached colorama-0.4.4-py2.py3-none-any.whl (16 kB)
Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
Collecting attrs>=19.2.0
  Using cached attrs-21.2.0-py2.py3-none-any.whl (53 kB)
Collecting pyparsing>=2.0.2
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting six
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Collecting requests<3,>=2.3
  Using cached requests-2.26.0-py2.py3-none-any.whl (62 kB)
Collecting certifi>=2017.4.17
  Using cached certifi-2021.5.30-py2.py3-none-any.whl (145 kB)
Collecting charset-normalizer~=2.0.0
  Using cached charset_normalizer-2.0.6-py3-none-any.whl (37 kB)
Collecting idna<4,>=2.5
  Using cached idna-3.2-py3-none-any.whl (59 kB)
Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.7-py2.py3-none-any.whl (138 kB)
Building wheels for collected packages: my-pkg
  Building wheel for my-pkg (PEP 517) ... done
  Created wheel for my-pkg: filename=my_pkg-1-py2.py3-none-any.whl size=911 sha256=a61f0a53c2d17e65598e62f67eccd8e1f8bb7fa7819947fc6125c95918e18a31
  Stored in directory: C:\Users\uranusjr\AppData\Local\Temp\pip-ephem-wheel-cache-oxe9_p8l\wheels\4b\ef\d3\3fe69d05a77a120a507f1c02e5be870d176099b5187ddf28e6
Successfully built my-pkg
Installing collected packages: urllib3, pyparsing, idna, charset-normalizer, certifi, toml, six, requests, py, pluggy, packaging, iniconfig, colorama, attrs, atomicwrites, requests-mock, pytest, my-pkg
Successfully installed atomicwrites-1.4.0 attrs-21.2.0 certifi-2021.5.30 charset-normalizer-2.0.6 colorama-0.4.4 idna-3.2 iniconfig-1.1.1 my-pkg-1 packaging-21.0 pluggy-1.0.0 py-1.10.0 pyparsing-2.4.7 pytest-6.2.5 requests-2.26.0 requests-mock-1.9.3 six-1.16.0 toml-0.10.2 urllib3-1.26.7

Can you confirm your setup?

$ ls .
build  my_pkg.py  pyproject.toml
$ cat my_pkg.py
$ cat pyproject.toml
[build-system]
requires = ['flit-core']
build-backend = "flit_core.buildapi"

[project]
name = 'my_pkg'
version = "1"
description = "..."

[project.optional-dependencies]
test-utils = ['pytest', 'requests-mock']
test = ['my_pkg[test-utils]']

[no-response]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[henryiii]

Trying this on cibuildwheel breaks CI as promised, see pypa/cibuildwheel#870. Not all jobs broke, and the ones that did might have a slightly older pip. The "classic" AppVeyor job simply ignored the extras and didn't install them - I expect pip is old there. All Travis jobs failed with a similar issue; the pip there is 20.2.3, 20.2.2, 20.2.1, and 18.1 - all failed in the same way. CircleCI with pip 21.1.1 failed with the might-as-well-be-infinite pre-package search (interestingly, though, all the extras are now available in at least one previous package). The log is too large to load in full, so I can't really tell too much about what happened there, other than the pip version and the pattern it was repeating:

WARNING: cibuildwheel 0.1.2 does not provide the extra 'test'
WARNING: cibuildwheel 0.1.0 does not provide the extra 'bin'
WARNING: cibuildwheel 0.1.0 does not provide the extra 'mypy'
WARNING: cibuildwheel 0.1.0 does not provide the extra 'test'
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cibuildwheel to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of cibuildwheel[dev] to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install cibuildwheel[bin,mypy,test]==0.1.0, cibuildwheel[bin,mypy,test]==0.1.2, cibuildwheel[bin,mypy,test]==0.1.3, cibuildwheel[bin,mypy,test]==0.10.0, cibuildwheel[bin,mypy,test]==0.10.1, cibuildwheel[bin,mypy,test]==0.10.2, cibuildwheel[bin,mypy,test]==0.11.0, cibuildwheel[bin,mypy,test]==0.11.1, cibuildwheel[bin,mypy,test]==0.12.0, cibuildwheel[bin,mypy,test]==0.2.0, cibuildwheel[bin,mypy,test]==0.2.1, cibuildwheel[bin,mypy,test]==0.3.0, cibuildwheel[bin,mypy,test]==0.4.0, cibuildwheel[bin,mypy,test]==0.4.1, cibuildwheel[bin,mypy,test]==0.5.0, cibuildwheel[bin,mypy,test]==0.5.1, cibuildwheel[bin,mypy,test]==0.6.0, cibuildwheel[bin,mypy,test]==0.7.0, cibuildwheel[bin,mypy,test]==0.7.1, cibuildwheel[bin,mypy,test]==0.7.2, cibuildwheel[bin,mypy,test]==0.8.0, cibuildwheel[bin,mypy,test]==0.9.0, cibuildwheel[bin,mypy,test]==0.9.1, cibuildwheel[bin,mypy,test]==0.9.2, cibuildwheel[bin,mypy,test]==0.9.3, cibuildwheel[bin,mypy,test]==0.9.4, cibuildwheel[bin,mypy,test]==1.0.0, cibuildwheel[bin,mypy,test]==1.1.0, cibuildwheel[bin,mypy,test]==1.10.0, cibuildwheel[bin,mypy,test]==1.11.0, cibuildwheel[bin,mypy,test]==1.11.1, cibuildwheel[bin,mypy,test]==1.11.1.post1, cibuildwheel[bin,mypy,test]==1.12.0, cibuildwheel[bin,mypy,test]==1.2.0, cibuildwheel[bin,mypy,test]==1.3.0, cibuildwheel[bin,mypy,test]==1.4.0, cibuildwheel[bin,mypy,test]==1.4.1, cibuildwheel[bin,mypy,test]==1.4.2, cibuildwheel[bin,mypy,test]==1.5.0, cibuildwheel[bin,mypy,test]==1.5.1, cibuildwheel[bin,mypy,test]==1.5.2, cibuildwheel[bin,mypy,test]==1.5.5, cibuildwheel[bin,mypy,test]==1.6.0, cibuildwheel[bin,mypy,test]==1.6.1, cibuildwheel[bin,mypy,test]==1.6.2, cibuildwheel[bin,mypy,test]==1.6.3, cibuildwheel[bin,mypy,test]==1.6.4, cibuildwheel[bin,mypy,test]==1.7.0, cibuildwheel[bin,mypy,test]==1.7.1, cibuildwheel[bin,mypy,test]==1.7.2, cibuildwheel[bin,mypy,test]==1.7.3, cibuildwheel[bin,mypy,test]==1.7.4, cibuildwheel[bin,mypy,test]==1.8.0, cibuildwheel[bin,mypy,test]==1.9.0, cibuildwheel[bin,mypy,test]==2.0.0, cibuildwheel[bin,mypy,test]==2.0.1, cibuildwheel[bin,mypy,test]==2.1.1, cibuildwheel[bin,mypy,test]==2.1.2, cibuildwheel[bin,mypy,test]==2.1.3 and cibuildwheel[dev]==2.2.0a1 because these package versions have conflicting dependencies.

The conflict is caused by:
    cibuildwheel[dev] 2.2.0a1 depends on cibuildwheel 2.2.0a1 (from /Users/distiller/project)
    cibuildwheel[bin,mypy,test] 2.1.3 depends on cibuildwheel 2.1.3 (from https://files.pythonhosted.org/packages/02/81/725cbfea54e8b18ca82112530cec9ca48d35016178ce4411d3dbb89db33a/cibuildwheel-2.1.3-py3-none-any.whl#sha256=13c0950819866aaf90ca75802e194698cf1e425b406ab87c4737e5c0ea7d7b96 (from https://pypi.org/simple/cibuildwheel/) (requires-python:>=3.6))
    cibuildwheel[dev] 2.2.0a1 depends on cibuildwheel 2.2.0a1 (from /Users/distiller/project)
    cibuildwheel[bin,mypy,test] 2.1.2 depends on cibuildwheel 2.1.2 (from https://files.pythonhosted.org/packages/12/18/71b0478d630ce5f15db00acf811b8adffd043945d1593ef5351b7d519dbd/cibuildwheel-2.1.2-py3-none-any.whl#sha256=9ce22debe509b4d83f75b8f6e375ca529e65e28207fc08acfbd876d1526f0556 (from https://pypi.org/simple/cibuildwheel/) (requires-python:>=3.6))
    cibuildwheel[dev] 2.2.0a1 depends on cibuildwheel 2.2.0a1 (from /Users/distiller/project)
    cibuildwheel[bin,mypy,test] 2.1.1 depends on cibuildwheel 2.1.1 (from https://files.pythonhosted.org/packages/0a/ad/be2f099a601652622a5baa07336bace8b220c827012eccc49532e89f892f/cibuildwheel-2.1.1-py3-none-any.whl#sha256=002f016fa67b921367e86c0fdcafce86910f830ec4dd7ec32735c3a9abc74957 (from https://pypi.org/simple/cibuildwheel/) (requires-python:>=3.6))

[henryiii]

This is very strange. I implemented the following test:

import nox

@nox.session
@nox.parametrize('pip', ['20.2.3', '21.1.1', '21.3.0'])
def tests(session: nox.Session, pip: str) -> None:
    session.install(f"pip=={pip}")
    session.run("pip", "--version")
    tmp = session.create_tmp()
    session.cd(tmp)

    with open("pyproject.toml", "w") as f:
        f.write("""\
[build-system]
requires = ['flit-core']
build-backend = "flit_core.buildapi"

[project]
name = 'cibuildwheel'
version = "1"
description = "..."

[project.optional-dependencies]
test-utils = ['pytest', 'requests-mock']
test-combined = ['cibuildwheel[test-utils]']
""")
    with open("cibuildwheel.py", "w") as f:
        f.write("__version__ = '1.0.0'")

    session.install(".[test-combined]")

And this is the result:

nox > * tests(pip='20.2.3'): success
nox > * tests(pip='21.1.1'): failed
nox > * tests(pip='21.3.0'): success

21.3 worked, and surprisingly 20.2 worked, but 21.1 failed with:

Collecting cibuildwheel[test-utils]
  Downloading cibuildwheel-2.1.3-py3-none-any.whl (1.5 MB)
WARNING: cibuildwheel 2.1.3 does not provide the extra 'test-utils'
  Using cached cibuildwheel-2.1.2-py3-none-any.whl (1.5 MB)
WARNING: cibuildwheel 2.1.2 does not provide the extra 'test-utils'
  Downloading cibuildwheel-2.1.1-py3-none-any.whl (1.5 MB)
WARNING: cibuildwheel 2.1.1 does not provide the extra 'test-utils'
...

Notice that I am intentionally picking the name of a known package and using an extra that that package does not have.

Edit: never mind about the surprisingly, I'm not checking to see if the extras were actually installed, so it probably is failing as expected.

[henryiii]

Okay, results:

• pip 20.2 and before simply ignore the extras (only testing 20.2, since I'm on macOS 11)
• pip 20.3 to 21.1 fail by looking at all older, online versions
• pip 21.2 & 21.3 work correctly.

import nox

@nox.session
@nox.parametrize('pip', ['20.2.*', '20.3.*', '21.0.*', '21.1.*', '21.2.*', '21.3.*'])
def tests(session: nox.Session, pip: str) -> None:
    session.install(f"pip=={pip}")
    session.run("pip", "--version")
    tmp = session.create_tmp()
    session.cd(tmp)

    with open("pyproject.toml", "w") as f:
        f.write("""\
[build-system]
requires = ['flit-core']
build-backend = "flit_core.buildapi"

[project]
name = 'cibuildwheel'
version = "1"
description = "..."

[project.optional-dependencies]
test-utils = ['pytest', 'requests-mock']
test-combined = ['cibuildwheel[test-utils]']
""")
    with open("cibuildwheel.py", "w") as f:
        f.write("__version__ = '1.0.0'")

    session.install(".[test-combined]")
    session.run("python", "-c", "import pytest")

nox > * tests(pip='20.2.*'): failed
nox > * tests(pip='20.3.*'): failed
nox > * tests(pip='21.0.*'): failed
nox > * tests(pip='21.1.*'): failed
nox > * tests(pip='21.2.*'): success
nox > * tests(pip='21.3.*'): success

[uranusjr]

20.3 was when the default resolver switch, so that behavioural change makes sense. I can't quite point out what exactly changed in 21.2 that makes this work (that release contains quite several resolver tweaks), but glad to hear its working now. I think this would be a very big step toward more declarative metadata definition since we no longer need to either write custom tooling to preprocess metadata, or manually constructing extras in setup.py to avoid duplication.