Lockfiles: determine if we need to post-process a resolve #12463
Comments
This is false. You can read the metadata straight out of an sdist or whl. Where did the confidence for this assertion come from? |
Why? They can happen in parallel to save time at the cost of trees if their results turn out to not be needed in the end after post resolve processing. |
Oh, that's great! Thank you for clarifying.
I wasn't very confident on it, which was why I used a Is this the --
True, although that would somewhat nullify this pro you had mentioned in review of #12389, right?
|
|
That second quote is of yourself above in the description. I never said that. |
|
You said:
|
|
In the mentioned review I said:
The point there is avoiding invalid failures / warnings. Say the range is covered by 3.7, 3.8 and 3.9. |
I see. I think I understand now: if we want, we can still pessimistically generate for every Python interpreter we discover. That need not change. What changes is crying wolf when we didn't need to, as right now the error in #12389 is overly eager. Thank you for clarifying all of these points. I've updated the issue description to more accurately reflect the pros and cons. PTAL. |
|
So, now this is simply a prioritization question, it seems. I'm now convinced that post-processing is both important for correctness and a better UX of not crying wolf. The question becomes how important are those things, e.g. important enough that we either contribute to pip-compile or give up on it in favor of PEX generating lockfiles (#12470 (comment)) |
|
The description edits lgtm, but there still an incorrect bit. Perhaps this was the bit that got things off the rails in the 1st place:
The post-processing is a way to know when we don't need to do any more resolves. I think you understand this now with the cry-wolf verbage you're using, which is correct. |
|
If you want to avoid confusing collisions, you might kill mention of |
|
Talked more about this w/ @jsirois in Slack. I've updated the issue again. tl;dr: "pessimistic generation" should handle correctly IC ranges, but with two downsides:
We need to decide how to prioritize our engineering resources vs. the better UX and correctness we'd get. If we stick with pip-compile, this is much harder to implement. If Pex generates lockfiles, it's more feasible. See #12470 for that discussion. |
|
Excellent. @asherf tried to use lockfiles for Toolchain and was confused why the lockfile generated on macOS failed in Linux CI. Apparently the dep So, yes, we technically could punt on post-processing, but we really shouldn't for us to call this project complete. |
|
Because Poetry already robustly handles environment markers (#12470 (comment)), I think post-processing is moot. |
…inor_versions()` (#12515) Because Poetry already robustly handles generating a single lockfile that works with all requested Python interpreters (#12470 (comment)), there is no need for the "pessimistic generation" proposed in #12463. [ci skip-rust] [ci skip-build-wheels]
…-compile (#12549) **Disclaimer**: This is not a formal commitment to Poetry, as we still need a more rigorous assessment it can handle everything we need. Instead, this is an incremental improvement in that Poetry handles things much better than pip-compile. It gets us closer to the final result we want, and makes it much more ergonomic to use the experimental feature—like `generate_all_lockfiles.sh` now not needing any manual editing. But we may decide to switch from Poetry to something like pdb or Pex. -- See #12470 for why we are not using pip-compile. One of the major motivations is that Poetry generates lockfiles compatible with all requested Python interpreter versions, along with Linux, macOS, and Windows. Meaning, you no longer need to generate the lockfile in each requested environment and manually merge like we used to. This solves #12200 and obviates the need for #12463. -- This PR adds only basic initial support. If we do decide to stick with Poetry, some of the remaining TODOs: - Handle PEP 440-style requirements. - Hook up to `[python-setup]` and `[python-repos]` options. - Hook up to caching. - Support `--platform` via post-processing `poetry.lock`: #12557 - Possibly remove un-used deps/hashes to reduce supply chain attack risk: #12458 -- Poetry is more rigorous than pip-compile in ensuring that all interpreter constraints are valid, which prompted needing to tweak a few of our tools' constraints.
Given that Python resolves can vary by environment (interpreter constraints, OS/arch), it's sometimes necessary to generate the lockfile in each environment that will be used: #12200
@jsirois proposed post-processing as a way to know when it is safe to skip multiple resolves:
If the resolve is not fully valid for the IC range and platforms, we can warn or error, such as instructing that you must generate on macOS and Linux and then merge.
--
Alternatively, we could pessimistically perform the resolve against every valid interpreter in the IC range (#12389), and merge into a single lockfile. For example, if the IC range is 3.6-3.8, then we know to generate w/ 3.6, 3.7, and 3.8; then we merge into a single lockfile by using env markers to handle conditional dependencies.
Pros and cons
Post-processing has two advantages over pessimistic generation:
platform_version.--platform, where it is possible to do cross-machine resolves by only using wheels instead of sdists. I'm talking about generating a lockfile when--platformis not used and accounting for any of the deps having platform-specific env vars.The main downside is the development and maintenance burden:
I first posited that there may be performance regressions, but @jsirois clarified this is not so. See the edit history + below discussion.
Possible actions
The text was updated successfully, but these errors were encountered: