Skip to content

Fix uv PEX builder to use pex3 lock export#23227

Open
seungwoo-ji-03 wants to merge 4 commits intopantsbuild:mainfrom
seungwoo-ji-03:fix/uv-pex-lock-export
Open

Fix uv PEX builder to use pex3 lock export#23227
seungwoo-ji-03 wants to merge 4 commits intopantsbuild:mainfrom
seungwoo-ji-03:fix/uv-pex-lock-export

Conversation

@seungwoo-ji-03
Copy link
Copy Markdown

When the uv PEX builder extracts pinned requirements from a PEX-native lockfile, it previously parsed the internal JSON format directly. This broke VCS and direct URL requirements (formatted as name==version instead of name @ url), and the PEX maintainer has warned that this internal format is unsupported and may change without notice.

This PR replaces JSON parsing with pex3 lock export --format pep-751 via PexCliProcess. The exported pylock.toml is passed directly to uv pip install -r, handling VCS refs, direct URLs, extras, and platform filtering out of the box.

Follow-up to #23197 (reported by @benjyw, approach suggested by @jsirois).
Supersedes #23218.

@seungwoo-ji-03
Copy link
Copy Markdown
Author

Chose pep-751 over pip-no-hashes for two reasons:

  1. Direct URL sdist preservation: pep-751 preserves direct URL sdist references via archive sections (built from the lockfile's input requirements), while pip-no-hashes drops them to name==version in export output.

  2. VCS commit pinning: pep-751 requires commit_id on VCS artifacts, which is populated when using --pip-version 24.2 (Pants default). Locks created with older pip/Pex may fail export when VCS artifacts are missing commit_id (e.g. 20.3.4-patched); in Pants, the code falls back to transitive uv resolution in that case.

(Based on reading lock.py and pep_751.py)

Copy link
Copy Markdown
Contributor

@jsirois jsirois left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@seungwoo-ji-03 this looks good to me, but a few notes - the 2nd - the existence of pex3 lock export-subset, may prove useful at some point.

Comment on lines +563 to +564
# Try to export the lockfile via `pex3 lock export` so we can pass pinned
# versions with --no-deps (reproducible). This uses Pex's stable CLI rather
Copy link
Copy Markdown
Contributor

@jsirois jsirois Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By "pinned versions" you really mean "locked requirements". I think the language of "pined versions" - which is common parlance - led you astray on your earlier foray. In the world of Python ecosystem requirements, requirements you can pin (name + version) cover 1/4 of the landscape only. There's also VCS, source from non-sdist archives and source from local project directories. These latter 3 do not necessarily have a known version up front - you have to build them to find that out.

Comment on lines 570 to 571
if isinstance(uv_request.requirements, PexRequirements) and isinstance(
uv_request.requirements.from_superset, Resolve
Copy link
Copy Markdown
Contributor

@jsirois jsirois Apr 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm just reading words here - I don't know the Pants code - but "Pex requirements from superset Resolve" suggest you only need a subset of the locked resolve to be exported / venv'd / PEXed up. Pex supports this:

# Export the full lock:
:; pex3 lock create ansible -o lock.json
:; pex3 lock export --format pep-751 -o pylock.full.toml lock.json 
:; tomlq -r '.packages[] | .name + "==" + .version' pylock.full.toml
ansible==13.5
ansible-core==2.20.4
cffi==2
cryptography==46.0.6
jinja2==3.1.6
markupsafe==3.0.3
packaging==26
pycparser==3
pyyaml==6.0.3
resolvelib==1.2.1

# Export just a subset:
:; pex3 lock export-subset --format pep-751 -o pylock.subset.toml --lock lock.json cryptography 
:; tomlq -r '.packages[] | .name + "==" + .version' pylock.subset.toml
cffi==2
cryptography==46.0.6
pycparser==3

# ~5ms hit for performing the subset resolve on the full lock to narrow it down to the transitive subset that covers `cryptography`:
:; hyperfine \
    -w2 \
    -n full \
    'pex3 lock export --format pep-751 -o pylock.full.toml lock.json' \
    -n subset \
    'pex3 lock export-subset --format pep-751 -o pylock.subset.toml --lock lock.json cryptography'
Benchmark 1: full
  Time (mean ± σ):     321.1 ms ±   2.5 ms    [User: 296.5 ms, System: 24.6 ms]
  Range (min … max):   317.9 ms … 326.2 ms    10 runs
 
Benchmark 2: subset
  Time (mean ± σ):     326.5 ms ±   3.2 ms    [User: 302.6 ms, System: 23.8 ms]
  Range (min … max):   318.4 ms … 330.7 ms    10 runs
 
Summary
  full ran
    1.02 ± 0.01 times faster than subset

So you may want to factor this in going forward. The savings may be in the margin, but even uv is slowed down a little bit if the full venv includes torch's transitive dependencies. If the PEX you're building needs some other portion of the venv, but none of the torch stuff - you just did a lot of extra work for nothing and this may show up on someones perf radar. I leave the experimentation / analysis to you all though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants