[frankmcsherry]

The proxy join stops building an owned timestamp per staged row and per match. Bridges become columns, with times in a column type the backend chooses, and matches become blocks of row ranges. On DDIR this is LDBC SF1 −9.5% CPU (−12.1% instructions), SCC's first tick −9.3% and reachability's first tick −22.3%, with identical answers.

What changes

DD (int_proxy)

• TimeColumn<T>: len, clear, extend_cross_join(a, b, blocks), plus two scaffolding exits, owned_get/owned_push, to delete once nothing calls them. No default bodies; Vec<T> implements it.
• JoinBridge<C, R, K, V>: keys, values, times (C) and diffs as columns. ProxyBridge stays for the reduce and the half join.
• Matches are JoinBlocks, pairs of row ranges: block (r0, r1) is every row of r0 with every row of r1, at t0[i] ∨ t1[j] with diff d0[i] · d1[j]. JoinMatches and its per-match ids/times go.
• ProxyJoinBackend gains type Times. advance fills two JoinBridges, and cross receives both bridges and the blocks. The backend forms times and diffs from the blocks it keeps, after any filtering.
• The direct path (either side under 16 rows) pushes one block per key and touches no times or diffs.
• The long-history replay keeps its owned ValueHistory and appends each step's edit and buffer rows to the bridges, so its matches are blocks too. That's correct because the meet a buffered time was advanced by is at or below the edit's time.
• The as-of wrapper keeps its semantics, going through the owned exits past its early return.

DDIR (interactive)

• ColTimes implements TimeColumn: a max per coordinate, per block.
• Staging is on lanes (join/lanes.rs). Each key is appended straight into the bridges, with lower joined while copying. Each side is consolidated in place: a side already in (token, time) order is left alone, and one out of order sorts a compact record. A key with an empty side is truncated off both bridges.
• cross works from blocks:
  • it cuts blocks into containers of about TARGET_OUT rows;
  • it expands blocks under a colliding hash token into the 1×1 blocks whose real keys agree;
  • it builds values by repeating and tiling, and times and diffs with the block operations.

Commits

1. The interface and a first DDIR version on lanes.
2. Cheaper staging copies and output columns. This includes not reserving output lanes up front, which had raised reachability's transient peak by 31 MiB.
3. Each key staged straight into the bridges and settled in place. Sorting each whole block instead was measured and dropped: LDBC was 9 s slower, because almost all keys are already in order.
4. Matching scans each key's run, rather than binary-searching the rest of the block. Lookup joins, with one row per key on each side, had slowed: IS2 mine's second and third joins see about 12,900 such keys a tick, and mine went from 5.75 ms/tick on a6b7922a to 6.43. The scan costs at most one comparison per match, plus one, and brings mine to 6.06. The remaining 0.30 ms/tick is per-key work in cross and staging. On one replay each, the whole LDBC run goes from 1877.3 to 1869.4 G instructions, with identical answers.

Measurements

One worker, mini-02. Medians of three alternating trials against a6b7922a, on commits 1–3.

Workload	CPU	Instructions	Peak footprint
LDBC SF1 one-off, 808 ticks	119.12 → 107.76 s, −9.5%	−12.1%	−3.5%
SCC (LiveJournal 250k), first tick	3.207 → 2.909 s, −9.3%	−13.6%	+1.6% (whole run)
SCC, 50 churn ticks	−0.9%	−1.8%
Reachability, first tick	0.371 → 0.288 s, −22.3%	−27.3%	−4.1% (whole run)
Reachability, 50 churn ticks	−0.6%	−2.2%

Answers were identical in every trial: all 808 LDBC answer digests and physical digests, and the SCC and reachability results at the start and end of churn.

• Where it comes from, on LDBC: the join iterator went from 51.0 to 39.6 s of the replay. Matching is 9.8 → 2.3 s, cross 11.9 → 10.3 s and staging 28.9 → 27.0 s. Owned-time work in the join drops from 14.1 ms to 0.001 ms per tick.
• SCC's +1.6% peak: live memory after the tick is identical; the peak is a transient during the first tick. It survived every variant tried, and is not explained.

Scaffolding left

Callers of owned_get/owned_push, none in DDIR:

• the long-history replay (both sides of a key with 16 or more rows), which no key reached in LDBC, SCC or reachability; the new oracle test exercises it;
• the as-of wrapper past its early return;
• the bridge-order debug check.

Known rough edge

About 150 lines of join/lanes.rs repeat the row staging's token minting, because the half join still stages rows. They go away when the half join moves to lanes.

Tests

• cargo test -p differential-dataflow -p differential-dogs3 -p interactive: 459 passed, 0 failed.
• The default difftest, seeds 0–49, passed on the same tree.
• New oracle: join_matches_direct_cross_product. It runs the whole tactic over DDIR's backend against a direct cross product, covering the replay path, cancellation on load, times of one and four coordinates, primitive and compound value tokens, and cut output containers.

Implemented by Claude (Opus 5.5) agents in background sessions; squashed, checked and opened by Claude Code.

🤖 Generated with Claude Code