[Mikola Lysenko (mikolalysenko)]

LLM Description written by Claude Code:claude-opus-5-5

Refs #1606
Refs #1607

This PR removes most of the bun and yarn berry rescan regression, but not all of it. Both issues stay open for the remaining part (see Not fixed here), so they're linked with Refs, not Fixes.

Root cause

Callgrind of bun-isolated/rescan and yarn-berry/rescan, comparing main (aa69125) with the bench base (cc938aa6) on the same fixture, showed the same new cost in both scans. It sits in the scan path every ecosystem shares, and it comes from repeated per-pair purl work:

hotspot	base	main	this PR
Discovery::contest_across_locks (every ref × every lock entry from elsewhere × every wiring, each pair re-splitting purls)	4.8M	64M	14M
PurlKey::new / canonical_base_purl (about 31k calls: every crawled purl keyed in 5+ separate passes, each rebuilt with format!)	31M	89M	60M
scan::with_pypi_equivalents + adopt_pypi_equivalents (key every purl even when the project has no PyPI pin)	0	34M	~0
yarn berry only: berry_selector_routes_nothing (re-parses every live entry's locator for each root resolutions selector)	0	113M	~0

Total instructions: bun-isolated/rescan 525M (base) → 715M (main) → 630M. yarn-berry/rescan 1,299M → 1,340M → 1,168M.

Changes

• contest_across_locks indexes the elsewhere entries by exact purl and by versionless name. Every match ResolvedElsewhere::covers can make uses one of those two spellings, so each ref now checks only its own package's entries. Wiring is grouped by install tree. Candidates are visited in their original order, so the contester find picks is the same entry as before.
• canonical_base_purl returns a purl as is when it is already canonical: trimmed, no ?/#/%, a lowercase type that isn't pypi/nuget/composer. That covers nearly every crawled npm/cargo/gem purl. A new test checks the fast path against the full rebuild on edge-case inputs.
• PyPI equivalents: with_pypi_equivalents skips non-PyPI purls and builds its key set only when an equivalent spelling actually exists. adopt_pypi_equivalents returns early when no lockfile-only PyPI pin exists.
• BerryLock builds its selector index (resolved urls, names, (name, descriptor) pairs) once, on first use, through a OnceLock. berry_selector_routes_nothing answers from that index. Entries are never changed after berry_entries builds them, so the cache can't go stale.

Evidence

Scan --json output on the bun rescan fixture is byte-identical before and after (durationMs aside).

socket-patch-bench compare, main aa69125 (base) vs this PR (head), 20 interleaved pairs, 4 vCPU sandbox:

scenario	base wall	head wall	Δ wall [95% CI]	Δ CPU
yarn-berry/rescan	231.3 ms	207.8 ms	-10.8% [-12.6, -4.7]	-9.6%
yarn-berry/hosted	272.0 ms	249.5 ms	-8.5% [-14.3, -5.6]	-7.9%
bun-isolated/rescan	166.4 ms	154.9 ms	-9.5% [-12.8, -3.2]	-6.8%
bun-isolated/hosted	203.1 ms	188.8 ms	-7.2% [-12.1, -3.1]	-6.8%
bun/hosted	163.9 ms	153.2 ms	-7.3% [-11.1, -4.4]	-6.3%

Against the issues' original base cc938aa6 (25 pairs): yarn-berry/rescan +10.1% [+6.8, +14.1] (was +21.4%) and bun-isolated/rescan +14.4% [+10.0, +19.4] (was +21.3%).

New tests:

• utils::purl_key::tests::canonical_fast_path_matches_the_rebuild
• vendor::lock_inventory::yarn::selector_tests::one_lock_answers_every_selector_from_its_index
• commands::scan::tests::pypi_equivalents_add_spellings_only_for_lockfile_only_pypi_pins

These are behavior-pinning tests: the change is a performance change, so there is no red→green result. The bench numbers above are the regression evidence.

Local checks on ddf8f9d (sandbox runs as root, uid 0):

• cargo fmt --all -- --check: clean
• cargo clippy --workspace --all-features -- -D warnings: clean
• cargo test --workspace --all-features --no-fail-fast: 14,834 passed, 10 failed. All 10 are environmental:
  • 9 assert that a write to a 0o555 directory fails, which can't happen as root: copy_tree::relax_loop_must_not_traverse_symlinked_root, vlt_heal::an_unremovable_hidden_lock_keeps_every_store_entry, pypi_poetry::wire_write_failure_*, pypi_requirements::wire_failure_*, in_process_redirect x3, repair x2.
  • 1 (e2e_bun_lockb::binary_shared_bundled_record_hosted_pin_is_managed) downloads a blob from the real patches-api.socket.dev, which the sandbox proxy refuses.
  • None of them touch the changed code. CI runs them unprivileged.
• After the review fixes: core lib 6,681 passed (the same 4 root-only failures), CLI lib 931 passed, the 108 scan/vex/yarn/bun/redirect/rollback integration suites 1,652 passed (the same 4 environmental failures).
• The real-toolchain yarn berry and bun e2e legs are left to CI.

/code-review high reported 8 findings. I fixed 7: adopt_pypi_equivalents case, a shared folding-types list, index keys moved next to covers, PyPI-only spelling scan, an allocation-free berry index, a HashMap index, and the is_pypi_purl scheme case. I skipped 1: returning Cow from canonical_base_purl would save one copy per purl, but it widens a public API, and the fast path already removes the rebuild cost.

Not fixed here (why the issues stay open)

• bun: the .bun store liveness walk (live_bun_store_entries_sync / read_bun_store_node_modules_sync, about 95M instructions) is the With Bun's isolated linker, vex refuses every hosted patch as not_applied after the usual in-place bun install, because it checks orphaned node_modules/.bun registry entries that Bun never removes (regression from #496) #599 orphan-entry check, already tuned in Perf regression: bun/hosted wall +110% (1169ae68, #472) #578. Making it cheaper means reusing the crawl's own listings, which is a separate change to the crawler.
• both: a rescan now makes one more references request (127 → 128). The withdrawn-pin sweep (Restore withdrawn hosted pins upstream during scan #1592) and the hosted redirect each fetch references for the same pinned uuids. Sharing that answer needs a per-run reference cache in ApiClient (or the sweep passing its map to the redirect). That changes the client for every caller, so I left it for a follow-up.
• yarn berry: peak RSS is +10% vs cc938aa6, and wall is still above CPU growth. Not profiled further.

https://claude.ai/code/session_01RPAAUNEcDh9q9bfJSTLHFs

---

Note

Low Risk
Performance refactors on hot scan paths with behavior-pinning tests and byte-identical scan JSON; discovery contest semantics are intended to be unchanged.

Overview
Speeds up shared scan/discovery work that regressed bun-isolated and yarn-berry rescans by cutting repeated purl canonicalization and full lock-entry scans, without changing scan JSON output (aside from timing).

VEX contest_across_locks now indexes elsewhere lock entries by exact purl and versionless name, groups wiring purls per install tree, and only considers candidate entries for each ref’s package—replacing nested scans over every ref × entry × wiring pair.

canonical_base_purl adds a fast path that returns already-canonical npm/cargo-style purls unchanged instead of always rebuilding via format!.

Scan PyPI equivalent handling (with_pypi_equivalents, adopt_pypi_equivalents) skips non-PyPI purls, lazily builds dedup keys only when equivalents exist, and bails early when there are no lockfile-only PyPI pins; is_pypi_purl centralizes case-insensitive PyPI type detection.

Yarn Berry caches a per-lock SelectorIndex (OnceLock) so berry_selector_routes_nothing no longer re-walks live entries for every root resolutions selector.

New unit tests lock equivalence of the fast paths vs the old behavior.

^{Reviewed by Cursor Bugbot for commit ddf8f9d. Configure here.}

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

BugBot review

---

Generated by Claude Code

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit ddf8f9d. Configure here.}

[Mikola Lysenko (mikolalysenko)]

Labeled Ready for review (burn-down agent).

• Head: ddf8f9d
• CI: 86/86 settled green on this head (36 success / 50 skipped).
• Bugbot: reviewed ddf8f9d, no findings; no open review threads.
• For the reviewer: a performance-only change to the shared scan path (contest_across_locks indexing, canonical_base_purl fast path, PyPI-equivalent short-circuits, a cached BerryLock selector index). The PR body notes scan --json output is byte-identical on the bun rescan fixture. Perf regression: bun-isolated/rescan wall +21% (cc938aa6..17793173; #1279 candidate) #1606/Perf regression: yarn-berry/rescan wall +21% (a43cf327..17793173; #1321/#1323/#1397 candidates) #1607 stay open for the remaining regression.
• Slack announcement not sent this run (send tool unavailable).

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Labeled Ready for review (burn-down agent).

• Head: ddf8f9d
• CI: 86/86 settled green on this head (36 success / 50 skipped). No main-wide failures affect it.
• Bugbot: the Cursor Bugbot check passed on ddf8f9d; no review threads.
• Mergeable with main; no CHANGELOG.md change.
• For the reviewer: perf-only change to shared scan/discovery. Look at the elsewhere index in vex/discover/mod.rs (contest_across_locks must still pick the same contester, since candidates are visited in their original order) and the canonical_base_purl fast path in utils/purl_key.rs. Perf regression: bun-isolated/rescan wall +21% (cc938aa6..17793173; #1279 candidate) #1606/Perf regression: yarn-berry/rescan wall +21% (a43cf327..17793173; #1321/#1323/#1397 candidates) #1607 stay open for the residual regression listed in the description.
• Slack: not announced. This run has no Slack send tool, so the next run will try again.

---

Generated by Claude Code

[Mikola Lysenko (mikolalysenko)]

Final review brief

What it does. A performance-only change to the shared scan path, aimed at the bun and yarn-berry rescan regression (#1606, #1607). It indexes cross-lock contest candidates by purl and versionless name instead of a full ref × entry × wiring sweep, skips the canonical_base_purl rebuild for purls that are already canonical, skips PyPI-equivalent work when there are no PyPI pins, and caches berry's selector index once per lock. Bench numbers in the description: about 7–11% faster wall time on the affected scenarios. Both issues stay open for the rest of the regression.

Risk: low. No output change intended (scan --json is byte-identical on the bun fixture). I checked each rewrite against the code it replaces:

• every ResolvedElsewhere::covers match (exact purl, versionless entry, or same PyPI release) shares a key from lookup_keys, so the index can't drop a contester, and candidates are visited in their original order, so find picks the same entry
• the canonical fast path only accepts inputs the rebuild returns unchanged
• the berry selector index matches the old per-entry loop branch for branch, and BerryLock.entries is never mutated after berry_entries builds it

Look here

• contest_across_locks index + wiring map and index_keys/lookup_keys
• is_canonical_base fast path
• SelectorIndex / berry_selector_routes_nothing
• with_pypi_equivalents / is_pypi_purl / adopt_pypi_equivalents: one small behavior widening. adopt_pypi_equivalents used to match only pkg:pypi/ and now also matches other cases of the type, like pkg:PyPI/. That is consistent with how PurlKey folds type case.

Verified

• read the full diff (4 files, +296/−50). CHANGELOG.md is untouched.
• ran cargo test -p socket-patch-core --lib -- purl_key selector_tests contest: 60 passed
• CI is 86/86 settled green on ddf8f9d, the Cursor Bugbot check passed, and there are no review threads
• the new tests pin behavior and don't go red→green, which is expected for a perf change; the bench table is the evidence

Changes I made: none.
Open questions: none blocking.

Auto-merge is armed, so approving sends this straight to the merge queue.

---

Generated by Claude Code