Lockfiles: Should we use pip-compile or teach Pex to generate lockfiles?

[Eric-Arellano]

Status quo & issues

We're using pip-compile. While it has a good UX, we've encountered some issues:

1. It does not use the new pip resolver.
2. The --platform pip option does not do anything when passed via --pip-args. We likely need this flag when you're using --platform w/ Pants, e.g. the platforms field of pex_binary.
3. PEP 440 direct reference support was only added two weeks ago and doesn't look like it will be cherry-picked to pip-compile 5, which is the last version to support Python 2.
4. Post-resolves as proposed in Lockfiles: determine if we need to post-process a resolve #12463 do not (naively) seem feasible. The pip-tools cache only has JSON files, rather than preserving the downloaded wheels/sdists. (It's possible I'm missing a secondary cache)

❯ ls -a /Users/ericarellano/Library/Caches/pip-tools/
# pkgs is empty
depcache-cp3.8.json  depcache-cp3.9.json  pkgs

We've offered to fix the resolver in jazzband/pip-tools#1190 (comment), but haven't heard back.

It's also unclear how they will respond to PEP 665. While it hasn't been accepted yet, the format has many advantages over pip-compile's requirements.txt style (including formalizing what environments the lock is valid for), and pantsbuild would also like to support standardization in the Python ecosystem.

Proposal: Pex generating lockfiles

PEX is well-positioned to generate lockfiles. We already use pip's new resolver and have experience with the packaging ecosystem.

A unique benefit Pex could bring: robustly handle using multiple environments, #12200. Pex already has technology to resolve against multiple Python versions in parallel, e.g. 3.5-3.9. It can use this + possibly post-processing resolves to robustly generate a lockfile that works in multiple environments. This compatibility info will be set in metadata.tags and/or metadata.marker.

Originally, some Pants team members were concerned with bifurcating the Python ecosystem even more, but PEP 665 instead allows Pex to help build momentum around standardization in the community. Rather than us introducing Yet Another format, we'll be using a standardized format.

The major downside seems to be development time and maintenance burden. It's plausible pip may eventually gain the ability to generate PEP 665 lockfiles and we could leverage that, but in the meantime, the functionality would likely be added to Pex.

[jsirois]

I think mixing the format of the lock files (PEP 665) in with this is not a great idea. First and foremost we need to generate locks that work. They need to handle interpreter constraints and platforms. Once we know how to do that, then the data needed to support those locks flows. The format to hold that data then flows from there.

[Eric-Arellano]

I think mixing the format of the lock files (PEP 665) in with this is not a great idea.

Fair. I rewrote this ticket, PTAL. I think PEP 665 is still relevant, such as wondering if/when pip-compile will add support. But I see what you mean and it's deprioritized.

[Eric-Arellano]

The major downside seems to be development time and maintenance burden.

@jsirois any insights or opinions on how expensive lockfile generation would be for Pex to implement/maintain?

[Eric-Arellano]

Sounds like pip-compile will not plan to add PEP 665 support until the PEP is accepted, whereas we're interested in being an early adopter and helping to push the standardization movement forward. We also have not heard back on our offer to switch to the new resolver, despite activity on the thread.

For both these reasons, I think we should lean into Pex/Pants generating PEP 665-compatible lockfiles.

[Eric-Arellano]

Instead of pip-compile vs Slack, I think the solution is: Poetry. As mentioned in Slack, it looks like Poetry robustly handles our concerns with environment markers! (#12200)

Edge case 1: interpreter constraint env markers. Black's lockfile varies if it was generated w/ Py36 vs Py39, e.g. the typing-extensions dep

Poetry handles this, and specifically based on what you set the python value to. If you set it to ==3.9, the lockfile only works w/ 3.9. If you set to >=3.7,<3.10, it works with all those intepreters, even tho you only run Poetry w/ one interpreter.

Edge case 2: platform env markers. ipython depends on appnode, but only on macOS. Regardless of if you generate lockfile on macOS vs Linux, Poetry handles that

Poetry is battle-tested and ready out-of-the-box, whereas improving pip-compile or teaching Pex to generate lockfiles are both large, risky projects.

--

Two known downsides:

1. We'll be using a new resolver.
   • At least, this resolver is only relevant when generating lockfiles, which you can opt out of. You can also provide your own lockfile if you so desire, rather than having Pants generate it.
2. Pex and Poetry do not share caches.
   • If this becomes important, we may be able to teach Pex how to consume Poetry's cache, such as downloads from PyPI.