Fix uv PEX builder to use pex3 lock export#23227
Fix uv PEX builder to use pex3 lock export#23227seungwoo-ji-03 wants to merge 4 commits intopantsbuild:mainfrom
Conversation
|
Chose
(Based on reading |
jsirois
left a comment
There was a problem hiding this comment.
@seungwoo-ji-03 this looks good to me, but a few notes - the 2nd - the existence of pex3 lock export-subset, may prove useful at some point.
| # Try to export the lockfile via `pex3 lock export` so we can pass pinned | ||
| # versions with --no-deps (reproducible). This uses Pex's stable CLI rather |
There was a problem hiding this comment.
By "pinned versions" you really mean "locked requirements". I think the language of "pined versions" - which is common parlance - led you astray on your earlier foray. In the world of Python ecosystem requirements, requirements you can pin (name + version) cover 1/4 of the landscape only. There's also VCS, source from non-sdist archives and source from local project directories. These latter 3 do not necessarily have a known version up front - you have to build them to find that out.
| if isinstance(uv_request.requirements, PexRequirements) and isinstance( | ||
| uv_request.requirements.from_superset, Resolve |
There was a problem hiding this comment.
I'm just reading words here - I don't know the Pants code - but "Pex requirements from superset Resolve" suggest you only need a subset of the locked resolve to be exported / venv'd / PEXed up. Pex supports this:
# Export the full lock:
:; pex3 lock create ansible -o lock.json
:; pex3 lock export --format pep-751 -o pylock.full.toml lock.json
:; tomlq -r '.packages[] | .name + "==" + .version' pylock.full.toml
ansible==13.5
ansible-core==2.20.4
cffi==2
cryptography==46.0.6
jinja2==3.1.6
markupsafe==3.0.3
packaging==26
pycparser==3
pyyaml==6.0.3
resolvelib==1.2.1
# Export just a subset:
:; pex3 lock export-subset --format pep-751 -o pylock.subset.toml --lock lock.json cryptography
:; tomlq -r '.packages[] | .name + "==" + .version' pylock.subset.toml
cffi==2
cryptography==46.0.6
pycparser==3
# ~5ms hit for performing the subset resolve on the full lock to narrow it down to the transitive subset that covers `cryptography`:
:; hyperfine \
-w2 \
-n full \
'pex3 lock export --format pep-751 -o pylock.full.toml lock.json' \
-n subset \
'pex3 lock export-subset --format pep-751 -o pylock.subset.toml --lock lock.json cryptography'
Benchmark 1: full
Time (mean ± σ): 321.1 ms ± 2.5 ms [User: 296.5 ms, System: 24.6 ms]
Range (min … max): 317.9 ms … 326.2 ms 10 runs
Benchmark 2: subset
Time (mean ± σ): 326.5 ms ± 3.2 ms [User: 302.6 ms, System: 23.8 ms]
Range (min … max): 318.4 ms … 330.7 ms 10 runs
Summary
full ran
1.02 ± 0.01 times faster than subsetSo you may want to factor this in going forward. The savings may be in the margin, but even uv is slowed down a little bit if the full venv includes torch's transitive dependencies. If the PEX you're building needs some other portion of the venv, but none of the torch stuff - you just did a lot of extra work for nothing and this may show up on someones perf radar. I leave the experimentation / analysis to you all though.
When the uv PEX builder extracts pinned requirements from a PEX-native lockfile, it previously parsed the internal JSON format directly. This broke VCS and direct URL requirements (formatted as name==version instead of name @ url), and the PEX maintainer has warned that this internal format is unsupported and may change without notice.
This PR replaces JSON parsing with pex3 lock export --format pep-751 via PexCliProcess. The exported pylock.toml is passed directly to uv pip install -r, handling VCS refs, direct URLs, extras, and platform filtering out of the box.
Follow-up to #23197 (reported by @benjyw, approach suggested by @jsirois).
Supersedes #23218.