[BUG] DistributionNotFound error for ==x!y.z.* dependency in editable-installed package

[MetRonnie]

setuptools version

setuptools==67.0.0

Python version

3.7-3.10

OS

RHEL 7, Ubuntu 22, MacOS 12

Additional environment information

Description

setuptools 67 is having a problem with a version specifier ==1!3.0.* for a dependency of the package in setup.cfg, whereas 66.1.1 did not have this problem. This only happens when the package was installed in editable mode (pip install -e .), and only seems to affect dependencies specified with an epoch and * at the end.

Problem appears to be here: https://github.com/pypa/setuptools/blob/v67.0.0/pkg_resources/__init__.py#L840

Output while paused at breakpoint at that line:

req.key  # metomi-isodatetime

# setuptools 66.1.1:
dist in req  # True

# setuptools 67.0.0:
dist in req  # False

Expected behavior

==1!3.0.* should work

How to Reproduce

1. Create this minimal example with the file structure given in filetree.txt: https://gist.github.com/MetRonnie/f30efd9d2567c9e3091ceb4943d60227
2. Run pip install -e . in the same directory as setup.cfg
3. Run the command darmok

Output

Traceback (most recent call last):
  File "~/.conda/envs/cylc8.1/lib/python3.10/site-packages/pkg_resources/__init__.py", line 617, in _build_master
    ws.require(__requires__)
  File "~/.conda/envs/cylc8.1/lib/python3.10/site-packages/pkg_resources/__init__.py", line 956, in require
    needed = self.resolve(parse_requirements(requirements))
  File "~/.conda/envs/cylc8.1/lib/python3.10/site-packages/pkg_resources/__init__.py", line 815, in resolve
    dist = self._resolve_dist(
  File "~/.conda/envs/cylc8.1/lib/python3.10/site-packages/pkg_resources/__init__.py", line 861, in _resolve_dist
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (metomi-isodatetime 1!3.0.0 (~/github/isodatetime), Requirement.parse('metomi-isodatetime==1!3.0.*'), {'cylc-flow'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "~/.conda/envs/cylc8.1/bin/cylc", line 33, in <module>
    sys.exit(load_entry_point('cylc-flow', 'console_scripts', 'cylc')())
  File "~/.conda/envs/cylc8.1/bin/cylc", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "~/.conda/envs/cylc8.1/lib/python3.10/importlib/metadata/__init__.py", line 171, in load
    module = import_module(match.group('module'))
  File "~/.conda/envs/cylc8.1/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "~/github/cylc-flow-8.1/cylc/flow/scripts/cylc.py", line 26, in <module>
    import pkg_resources
  File "~/.conda/envs/cylc8.1/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3249, in <module>
    def _initialize_master_working_set():
  File "~/.conda/envs/cylc8.1/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3223, in _call_aside
    f(*args, **kwargs)
  File "~/.conda/envs/cylc8.1/lib/python3.10/site-packages/pkg_resources/__init__.py", line 3261, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "~/.conda/envs/cylc8.1/lib/python3.10/site-packages/pkg_resources/__init__.py", line 619, in _build_master
    return cls._build_from_requirements(__requires__)
  File "~/.conda/envs/cylc8.1/lib/python3.10/site-packages/pkg_resources/__init__.py", line 632, in _build_from_requirements
    dists = ws.resolve(reqs, Environment())
  File "~/.conda/envs/cylc8.1/lib/python3.10/site-packages/pkg_resources/__init__.py", line 815, in resolve
    dist = self._resolve_dist(
  File "~/.conda/envs/cylc8.1/lib/python3.10/site-packages/pkg_resources/__init__.py", line 856, in _resolve_dist
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'metomi-isodatetime==1!3.0.*' distribution was not found and is required by cylc-flow

Also viewable here: https://github.com/MetRonnie/cylc-flow/actions/runs/4044453546/jobs/6954609806#step:10:71

[abravalheri]

Hi @MetRonnie, thank you very much for reporting this.

I think that this is probably related to upstream changes in pypa/packaging...
Let's see how the investigation goes.

---

Is there any chance you could reduce your reproducer to something more minimal we can test with different versions of pypa/packaging?

[MetRonnie]

A minimally example might be creating a package with

# setup.cfg
[options]
install_requires =
    metomi-isodatetime==1!3.0.*

and

# __init__.py
import pkg_resources

haven't tried it yet so not sure if that's a reproduceable example however

[abravalheri]

Ummm... this reproducer does not result in error:

> docker run --rm -it python:3.7 /bin/bash

apt update
apt install -y libzmq3-dev build-essential

python3 -m venv /tmp/venv
mkdir /tmp/dummy-pkg
touch /tmp/dummy-pkg/pyproject.toml
cat <<EOF > /tmp/dummy-pkg/setup.cfg
[options]
install_requires =
    metomi-isodatetime==1!3.0.*
EOF
/tmp/venv/bin/python -m pip install 'setuptools==67' /tmp/dummy-pkg
/tmp/venv/bin/python -c 'import pkg_resources'

[MetRonnie]

I don't think python -c 'import pkg_resources' would do the trick, I think it would have to be running a package that does the import, as it is failing when iterating over the dependencies of the package.

[MetRonnie]

OK I have a minimal reproduceable example: https://gist.github.com/MetRonnie/f30efd9d2567c9e3091ceb4943d60227

The bug is only present when you've done an editable install (pip install -e .)

[agroszer]

installing yearfrac breaks with

error in yearfrac setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Expected end or semicolon (after version specifier)
numpy>=1.14.*

Looking at the PEP 508 version spec 1.14.* should be a valid version spec.

kmedian/yearfrac#24

Who is right then?

[pradyunsg]

The error reported here is different from the one you're seeing @agroszer.

See pypa/packaging#657 (comment)

[abravalheri]

@MetRonnie, we should be able to fix this once a new version of packaging is available:

> docker run --rm -it python:3.7 /bin/bash

apt update
apt install -y libzmq3-dev build-essential

python3 -m venv /tmp/venv

## ---- Reproducer ----
git clone https://gist.github.com/f30efd9d2567c9e3091ceb4943d60227.git /tmp/dummy-pkg
cd /tmp/dummy-pkg
mkdir tmp
mv __init__.py tmp

## ---- Install and patch setuptools ----
/tmp/venv/bin/python -m pip install 'setuptools==67'
# patch `packaging` with its development version
git clone https://github.com/pypa/packaging /tmp/packaging
rm -rf /tmp/venv/lib/python3.7/site-packages/pkg_resources/_vendor/packaging
cp -r /tmp/packaging/src/packaging/ /tmp/venv/lib/python3.7/site-packages/pkg_resources/_vendor

## ---- Install project and run command ----
/tmp/venv/bin/python -m pip install -e .
/tmp/venv/bin/darmok
# => If you see this, all is well

[MetRonnie]

I have tested out with packaging installed from the main branch (https://github.com/pypa/packaging/tree/5b34465682d40ddd42dd6cba88e158e1674de7b9) and cannot reproduce the bug

[MetRonnie]

Now that packaging 23.1 has been released, can this be updated in the vendored version for the next release of setuptools?