Support resolving using a Pex lock file.

[jsirois]

Instead of exporting the lock file to requirements.txt format and letting Pip do the resolve, record the Requires-Dist metadata in the lock so that Pex can calculate the resolve (subset) using the logic it already has for --pex-repository and then parallel-download all required dists. The backend to this can then be the current Pip build and install phase (BuildAndInstallRequest).

[jsirois]

N.B.: This is also required to support the final phase of #1556 which requires Pex to do the hashing and hash-checking of loose source project requirements (local and VCS projects).

[jsirois]

Perf is decent.

Cold:

$ hyperfine -w1 -r3 -p 'rm -rf ~/.pex' 'python3.7 -mpex jupyter-server -o jupyter-server.pex -c jupyter --venv --layout packed' 'python3.7 -mpex --lock lock.json jupyter-server -o jupyter-server.pex -c jupyter --venv --layout packed'
Benchmark 1: python3.7 -mpex jupyter-server -o jupyter-server.pex -c jupyter --venv --layout packed
  Time (mean ± σ):      9.887 s ±  0.139 s    [User: 21.607 s, System: 2.452 s]
  Range (min … max):    9.786 s … 10.046 s    3 runs
 
Benchmark 2: python3.7 -mpex --lock lock.json jupyter-server -o jupyter-server.pex -c jupyter --venv --layout packed
  Time (mean ± σ):      6.420 s ±  0.603 s    [User: 39.429 s, System: 5.474 s]
  Range (min … max):    5.763 s …  6.949 s    3 runs
 
Summary
  'python3.7 -mpex --lock lock.json jupyter-server -o jupyter-server.pex -c jupyter --venv --layout packed' ran
    1.54 ± 0.15 times faster than 'python3.7 -mpex jupyter-server -o jupyter-server.pex -c jupyter --venv --layout packed'

Where cold standard breaks down as:

pex: Building pex: 9465.6ms
pex:   Resolving distributions (jupyter-server): 9463.6ms
pex:     Resolving requirements.: 9462.6ms
pex:       Resolving for:
  /home/jsirois/.pyenv/versions/3.7.12/bin/python3.7: 7902.4ms
pex:       Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='jupyter-server', processed_text='jupyter-server', source='<string>', start_line=1, end_line=1), requirement=Requirement.parse('jupyter-server'), editable=False): 0.1ms
pex:       Installing 45 distributions: 1408.7ms
pex:       Checking install: 144.0ms

And cold lock breaks down as:

pex: Building pex: 4674.0ms
pex:   Resolving distributions (jupyter-server): 4671.8ms
pex:     Resolving requirements from lock file lock.json: 4670.8ms
pex:       Parsing lock lock.json: 125.6ms
pex:       Parsing requirements: 0.9ms
pex:       Resolving urls to fetch for 1 requirements from lock lock.json: 17.3ms
pex:       Downloading 45 distributions to satisfy 1 requirements: 1915.4ms
pex:       Categorizing 45 downloaded artifacts: 2.9ms
pex:       Building 0 artifacts and installing 45: 2604.5ms
pex:         Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='jupyter-server', processed_text='jupyter-server', source='<string>', start_line=1, end_line=1), requirement=Requirement.parse('jupyter-server'), editable=False): 0.1ms
pex:         Installing 45 distributions: 2456.6ms
pex:         Checking install: 146.6ms

Warm:

$ hyperfine -w1 -r5 'python3.7 -mpex jupyter-server -o jupyter-server.pex -c jupyter --venv --layout packed' 'python3.7 -mpex --lock lock.json jupyter-server -o jupyter-server.pex -c jupyter --venv --layout packed'
Benchmark 1: python3.7 -mpex jupyter-server -o jupyter-server.pex -c jupyter --venv --layout packed
  Time (mean ± σ):      4.375 s ±  0.078 s    [User: 2.802 s, System: 0.099 s]
  Range (min … max):    4.305 s …  4.505 s    5 runs
 
Benchmark 2: python3.7 -mpex --lock lock.json jupyter-server -o jupyter-server.pex -c jupyter --venv --layout packed
  Time (mean ± σ):     662.5 ms ±   7.3 ms    [User: 528.4 ms, System: 46.8 ms]
  Range (min … max):   656.0 ms … 671.7 ms    5 runs
 
Summary
  'python3.7 -mpex --lock lock.json jupyter-server -o jupyter-server.pex -c jupyter --venv --layout packed' ran
    6.60 ± 0.14 times faster than 'python3.7 -mpex jupyter-server -o jupyter-server.pex -c jupyter --venv --layout packed'

Where warm standard breaks down as:

pex: Building pex: 4070.7ms
pex:   Resolving distributions (jupyter-server): 4068.8ms
pex:     Resolving requirements.: 4067.9ms
pex:       Resolving for:
  /home/jsirois/.pyenv/versions/3.7.12/bin/python3.7: 3863.0ms
pex:       Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='jupyter-server', processed_text='jupyter-server', source='<string>', start_line=1, end_line=1), requirement=Requirement.parse('jupyter-server'), editable=False): 0.1ms
pex:       Installing 45 distributions: 63.7ms
pex:       Checking install: 133.9ms

And warm lock breaks down as:

pex: Building pex: 468.3ms
pex:   Resolving distributions (jupyter-server): 466.3ms
pex:     Resolving requirements from lock file lock.json: 465.4ms
pex:       Parsing lock lock.json: 126.9ms
pex:       Parsing requirements: 1.0ms
pex:       Resolving urls to fetch for 1 requirements from lock lock.json: 17.6ms
pex:       Downloading 45 distributions to satisfy 1 requirements: 111.1ms
pex:       Categorizing 45 downloaded artifacts: 2.7ms
pex:       Building 0 artifacts and installing 45: 201.9ms
pex:         Calculating project names for direct requirements:
  PyPIRequirement(line=LogicalLine(raw_text='jupyter-server', processed_text='jupyter-server', source='<string>', start_line=1, end_line=1), requirement=Requirement.parse('jupyter-server'), editable=False): 0.1ms
pex:         Installing 45 distributions: 65.3ms
pex:         Checking install: 135.4ms

The only surprising result is the cold lock case taking 1 second longer to do the wheel installs. That's likely a bug to suss out.

[stuhood]

Nice! What is the granularity of "cold" vs "warm" here? Is it on a distribution-by-distribution basis, such that a resolve might be partially warm?

[jsirois]

The cold is rm -rf ~/pex in the hyperfines; so fully cold and the warms are -w1; i.e after a 1 full warm up, so fully warm.

The code though has atomic_directory locks around all individual parallelized downloads; so those are only ever done once. Likewise, with the downloads in-hand, the existing parallelized / atomic_directoried machinery for normal Pex resolves build and install of those dists is used. That infra too only ever builds an sdist once per target platform and ditto for the install. So ... the spectrum from cold to hot is, as ever in Pex 2.x, fully granular at all layers down to the individual distribution unit.