Lockfiles: evaluate PDM for lockfile generation

[Eric-Arellano]

See https://frostming.com/2021/03-26/pm-review-2021/ for the blog comparing PDB to Poetry. Like Poetry, PDM uses pyproject.toml for direct deps, and it uses the same general TOML lockfile format as Poetry:

[[package]]
name = "requests"
version = "2.26.0"
requires_python = ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*,!=3.5.*"
summary = "Python HTTP for Humans."
dependencies = [
    "certifi>=2017.4.17",
    "charset-normalizer~=2.0.0; python_version >= \"3\"",
    "idna<4,>=2.5; python_version >= \"3\"",
    "urllib3<1.27,>=1.21.1",
]

# ...

[metadata.files]
"certifi 2021.5.30" = [
    {file = "certifi-2021.5.30-py2.py3-none-any.whl", hash = "sha256:50b1e4f8446b06f41be7dd6338db18e0990601dce795c2b1686458aa7e8fa7d8"},
    {file = "certifi-2021.5.30.tar.gz", hash = "sha256:2bbf76fd432960138b3ef6dda3dde0544f27cbf8546c458e60baf371917ba9ee"},
]

This issue relates to our evaluation of Poetry at #12568.

Improvements over Poetry

Uses resolvelib

Meaning, it uses the same resolver as Pex/pip. No churn for users switching to a new resolver.

Solvable issues

Handling --platform

Same as Poetry, we would need to post-process pdm.lock to remove unused platforms and eagerly validate the supported platforms work. See #12557.

Unsolved issues

Exported lockfiles are not cross-platform compatible

Whenever you run pdm export, it gives this warning:

The exported requirements file is no longer cross-platform. Using it on other platforms may cause unexpected result.

pdm.lock does have all the information needed to make things cross-platform, but they reverted that feature in pdm-project/pdm#456. They didn't give a motivation for why they reverted the feature.

We could presumably add back the feature, but it may be non-trivial and is not clear they'll accept it.

Does not support multiple specifications for same dependency

This pyproject.toml:

dependencies = [
    "setuptools>=50; python_version>'3'",
    "setuptools==44; python_full_version=='2.7.*'",
]
requires-python = ">=2.7"

results in:

[[package]]
name = "setuptools"
version = "44.0.0"
requires_python = ">=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*,!=3.4.*"
summary = "Easily download, build, install, upgrade, and uninstall Python packages"

This is a common pattern in Pants to have multiple entries for the same dep, e.g. Pytest 4.6 if using Python 2, else Python 5. We need to support this.

Fails on direct deps that don't match Python constraints

If you set requires-python = ">=3.5", then include "setuptools==44; python_full_version=='2.7.*'" as a direct dep in pyproject.toml, you'll get

Unable to find a resolution that satisfies the following requirements:
  setuptools==44; python_full_version == "2.7.*" (from project)
Please make sure the package names are correct. If so, you can either loosen the version constraints of these dependencies, or set a narrower `requires-python` range in the pyproject.toml.
[ResolutionImpossible]: [RequirementInformation(requirement=NamedRequirement(name='setuptools', marker=<Marker('python_full_version == "2.7.*"')>, extras=(), specifier=<SpecifierSet('==44')>, editable=False), parent=None)]

This would break some of our tool lockfiles that include setuptools==44; python_full_version == "2.7.*" even though the ICs are 3.6+. But this might not actually be very important, we could update our default requirements.

--find-links not supported

Same as Poetry, you can add a new index: https://pdm.fming.dev/pyproject/tool-pdm/#specify-other-sources-for-finding-packages. But it looks like there is no equivalent to --find-links.

Interesting edge cases found during the investigation

• recursion limit exceeded in pip's pyparsing.py because of "marker resolution w/o duplication": Cannot add "pyobjc~=7.2; sys_platform == 'darwin'" pdm-project/pdm#449

[Eric-Arellano]

Several of these are blockers. I don't think PDM is very feasible for us, and as concluded in #12568, we'll be leaning into Pex generating lockfiles. If that does not work as we intend, we can re-evaluate investing in PDM.