[internal] Add internal ability to generate lockfiles with PEX #14278
Conversation
# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]
|
I'm happy to close this if it's too premature, @jsirois . I tried turning it on for our Reason 1 of landing the code is helpful to me, but there's other stuff I can do. |
|
If we don't want to expose this option to users, another approach is to add a boolean field to That will let me get the first benefit of incremental progress on the code, without exposing this to premature dogfooding. |
|
Yeah, not appropriate. The lock export for universal will probably never support universal export since it requires rather insane env marker fabrication gymnastics. Neither Pants, nor anyone else, can use Pex locks until Pex supports resolving its own locks. |
# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]
Eric-Arellano
changed the title
[python].experimental_generate_lockfiles_with_pex# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]
# Rust tests and lints will be skipped. Delete if not intended. [ci skip-rust] # Building wheels and fs_util will be skipped. Delete if not intended. [ci skip-build-wheels]
| lock_content = digest_contents[0].content.decode() | ||
| header = dedent( | ||
| """\ | ||
| # This lockfile was autogenerated by Pants. To regenerate, run: |
There was a problem hiding this comment.
Does Pants pre-strip this stuff before handing to the tool to resolve with later? With a json lock file it will have to. With any lock file really. It's probably actually an error to be adding an extension to the file since Pants should not even know what the contents of those files are - and thus can't know if the comment style is corrupting of the file or not.
There was a problem hiding this comment.
It does not currently -- we'll have to strip when hooking up Pex consumption, I suppose.
Would you be willing to consider changing PEX to use TOML? We recently changed Pants's proprietary JVM lock format to use TOML, in part to accommodate comments and in part because it's arguably easier to read (less nesting). I imagine a major turnoff would be needing to vendor toml or tomli.
There was a problem hiding this comment.
Yeah, Pants can't expect to strong arm the world here aside from Pex being willing to do this. I strongly suggest what Pants is doing today is not right - it should not tamper with a - supposedly - opaque file format, which lock files should be considered.
There was a problem hiding this comment.
See #14281. Feedback welcomed :)
We can't consume PEX lockfiles until pex-tool/pex#1583, but the generation is now stable. Part 1 of #13964.
Why land this? It lets us finish up the PEX generation code. For example, a follow-up PR can wire up
[python-repos]. We can frontload some PEX-related work without blocking on pex-tool/pex#1583.Note that we generate in PEX's JSON format and do not call
pex3 lock export, as the export is lossy and does not attempt to handle things like environment markers.[ci skip-rust]
[ci skip-build-wheels]