Repository navigation
Next release: timely 0.32, columnar and tactic work, interactive/ and ddir - #1076
Merged
Merged
Conversation
* Integer-proxy chunks: backend-agnostic join and reduce over (key_hash, value_id)
A boundary where only integers cross: a storage backend presents each
record as ((key_hash, value_id), time, diff) — integer proxies for data
it keeps in its own layout — the operators own all the lattice/time
logic over those integers, and the backend supplies value semantics via
callbacks. Any columnar (or otherwise opaque-to-DD) value store can
then reuse join and reduce without materializing values.
- trace/chunk/int_proxy: ProxyChunk, a cursor-less Chunk of proxy
columns, with from_unsorted (integer sort+consolidate with
representative provenance) as the presentation-building helper.
- operators/int_proxy: ProxyJoinTactic / ProxyReduceTactic for the
join_with_tactic and reduce_with_tactic seams (made pub here), and
the backend traits: present-as-proxies (read), value callback with
hash-minted output ids (write), materialize (egress). Reduce output
ids are content hashes, so an output arrangement re-presents with the
same ids downstream with no registry; pending interesting times are
keyed by the stable key_hash across retires. The module doc carries
the boundary contract and design notes (why value_id is not
order-preserving; collision risk); each tactic and the in-memory
reference backend (VecChunk arrangements, fnv hashes) sit in their
own file under the module.
- Tests: join and count/distinct/min reduces against the row operators
over multi-round retracting inputs, and a scripted Product-time
retire sequence exercising synthetic corrections and pending.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* int_proxy: state the minimal value_id contract — hashing is one scheme, not a requirement
value_id never outlives one computation (a join unit's presentations;
one reduce retire, whose materialize resolves ids to real data before
anything leaves). The actual contract is a per-computation bijection
with value equality, plus within-retire agreement between the output
presentation and minted ids. Content hashing discharges all of it
statelessly (the reference backend's choice); exact schemes — dense
ordinals from grouping, a per-retire value→id map — are equally valid
and collision-free. Only key_hash must be a stable pure function of the
key (cross-retire pending, changed-key filter), making the key side the
irreducible collision exposure. Persisting ids into an output
arrangement itself would force stable value ids; this design does not.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* int_proxy: delta-proportional presents, checked — not asymptotically worse than the cursor tactics
Two regressions found by asking exactly that question, both fixed:
- The reference reduce backend implemented the changed-key restriction
by scanning every batch and filtering — O(trace) per retire where the
cursor tactic seeks, O(delta·log). Presents now seek the changed keys
(novel keys resolve from this retire's delta-sized input batches;
pending keys from the retire that pended them, which is exactly what
the persistent hash→key map retains — pruned to the changed set each
retire, so it is bounded by the delta, and the per-retire value map
is cleared).
- The join interface had no restriction at all, forcing ANY backend to
present the entire accumulated side per fresh batch — O(trace·log)
per unit where the cursor join seeks. present0/present1 now take an
optional sorted key-hash filter; the tactic presents the fresh side
first and passes its key set for the accumulated side, the join
analogue of reduce's changed-key restriction.
The check is mechanical, not wall-clock: counting backend wrappers
measure presented records — with 20k arranged keys and five single-key
rounds, reduce presents 37 records and join 10, where the scanning
versions present Ω(rounds·N) (join: 100,005) and fail the gate. What
remains above the cursor tactics is a log-factor sort of delta-sized
presentations, and per-(key, wave) rescans of a changed key's
presented range where the row replayer consolidates progressively —
same worst-case order, different constants.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* int_proxy: the self-contained assessment — identity backend, grid-oracle fuzz
The minimal harness for the framework, closed over itself: the tactics
demand only BatchReader of the trace, and ProxyChunk is already a
Chunk, so a batch of proxy chunks is the minimal arrangement — the
proxy data IS the data, no separate chunk class needed. The identity
backend makes values u64s with the identity as the id function: no
hashing, no resolution machinery, no collision possibility, and
materialize emits the proxy records verbatim (incidentally exercising
ChunkBatch<ProxyChunk> as a real output batch).
What remains under test is exactly the framework's own contribution —
interesting-time discovery, desired-vs-current deltas, pending, held
routing — fuzzed over Product-time grids: 300 random inputs retired
through random diagonal frontiers (so synthetic joins arise inside and
across intervals and must pend), driven through an emulation of the
reduce driver protocol, and checked against a brute-force oracle at
every grid point: the accumulated output must equal the reduction of
the accumulated input, everywhere, for count/distinct/min-shaped
reducers.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* int_proxy: wall-clock bench; fix a capacity leak the bench found
An ignored benchmark (cargo test --release --test int_proxy -- --ignored
--nocapture) compares full stacks — the stock row operators against
chunk arrangements plus the proxy tactics over the reference backend —
for a bulk load and a steady-state incremental phase (warmed past the
post-load merge-amortization transient).
The bench caught what the counting gates could not: the reference
backend's hash→key map was pruned by retain (and the per-retire value
map by clear), both of which keep the backing table — so after a
million-key load, every retire walked a million-bucket table to visit
one entry, ~130µs/round of pure capacity. shrink_to_fit after the
prune and a fresh map per retire fix it.
Steady-state single-key rounds after the fix, proxy vs row: reduce
~1.4x at every scale (flat from 10k to 1M keys — the delta-
proportionality gates hold in wall clock too), join at parity below 1M
(~1.0x). Bulk load carries the presentation layer's constant factor
(per-record hashing, clones, by-hash sort): reduce 3-5x, join ~2x —
the costs a columnar backend's bulk primitives are meant to attack.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* int_proxy: batch the reduce value callback per wave (reduce_many)
The longer-term goal for the boundary is fewer crossings into the
backend's value logic (interpreted or columnar execution pays per-call
overhead): each crossing should carry a list of keys and a longer
bracketed list of value entries.
ProxyReduceBackend gains reduce_many(keys, ends, input) — group_offsets
-shaped brackets, one per key, each non-empty — returning concatenated
per-key outputs with their own bracket ends. A default implementation
loops the per-key reduce, so simple backends implement only that;
backends with bulk value logic override reduce_many.
The tactic now calls only reduce_many: retire's key-major loop becomes
two passes — derive each changed key's active times, group the work
into waves by time, then play the waves in ascending order (Ord extends
the partial order, so a key's earlier deltas always precede a later
time's reads) with at most one callback per wave, batching every key
active at that time.
The identity backend overrides reduce_many (asserting the bracket
protocol from the backend's seat), so the grid-oracle fuzz exercises
the batched path; the reference backend uses the default, so the
row-comparison tests cover the loop. All gates and benches unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* int_proxy: one value-callback crossing per retire, not per wave
The desired output at a (key, time) moment is a function of the key's
input accumulation at that time alone — no output-side state — so no
time ordering constrains the batch: every moment of the retire can
share one reduce_many call, a key contributing one bracket per active
time. The order-sensitive part (subtracting the current output, which
includes deltas emitted at earlier moments) is pure proxy-space
arithmetic and moves to a separate pass that plays the moments in
ascending time order.
The bracket, not the key, is reduce_many's unit: keys may repeat.
Contract docs updated accordingly.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* int_proxy: port the history-replay machinery — the scaling shapes go linear
The row suite's reduce_scaling/join_scaling shapes (one key, many
distinct times, one batch) exposed quadratic behavior in both proxy
tactics: reduce rescanned a key's full presented range per interesting
time (and its interesting-time closure joined all pairs), and join
cross-produced full matched histories pairwise. Measured: reduce 6.4s
at scale 10k, 4.7x per doubling; join 9.8s at 10k, >90s at 20k.
The robust versions are the cursor variants with integers in place of
keys and values, as intended:
- history.rs: IdHistory, the id-space ValueHistory — (value_id, time,
diff) edits replayed in ascending time order into a buffer repeatedly
advanced by the meet of the times still to come and consolidated.
The advancement is the collapse that keeps a key with many distinct
times linear: accumulations read the small buffer, never the raw
history. The presentations serve as the fused per-key load.
- reduce: discover_and_accumulate ports history_replay::compute —
lazy interesting-time discovery (novel and pending seed; synthetics
from joins with the advanced batch buffer and times_current) replaces
the eager join-closure, and per-moment accumulation reads the
advanced buffers. Phase B replays the output side per key over the
discovered moments with the same machinery (suffix meets; emitted
deltas advanced and consolidated). Still one batched reduce_many
crossing per retire.
- join: join_key ports JoinThinker::think — each side's edits replayed
against the other side's advanced buffer (identical emitted times:
t0 ∨ (t1 ∨ meet) = t0 ∨ t1), with the dead-simple cross product kept
for small histories.
proxy_reduce_scaling / proxy_join_scaling (scale 100k, the row tests'
shapes) now pin this; scale 10k dropped from seconds to milliseconds
and growth is ~4x per 4x scale. The grid-oracle fuzz over partially
ordered times, the scripted pending test, the row comparisons, the
delta-proportionality gates, and the steady-state bench all pass
unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* int_proxy: materialize in TARGET-sized chunks — the last scaling term
The row-vs-proxy comparison on the scaling shapes showed proxy reduce
still superlinear (9x per 4x scale; timeout at 4M where row takes
0.5s). Profiling pinned it outside the tactic: the reference backend's
materialize built ONE giant VecChunk and let the builder settle it,
and settle's split path peels TARGET-sized pieces off the front with
split_off, copying the remaining tail each iteration — O(m²/TARGET)
in the batch size. Feeding the builder TARGET-sized chunks directly
fixes it: 4M drops from >120s to 3.2s.
With that, both operators are in the row implementations' complexity
class on the scaling shapes (growth ~5x per 4x scale ≈ n·log n; the
hot frames are the presentation sort and fnv hashing — constants, not
structure): at scale 4M, join row 0.96s / proxy 3.6s, reduce row 0.48s
/ proxy 3.2s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* interactive: CorgiChunk arrangement backend (cursor-less Route B on the #778 Chunk split)
A columnar differential-dataflow backend for DDIR whose native representation
is corgi Value columns and whose scalar logic is corgi `eval_graph`, parallel
to `backend::vec`.
Arrangement:
- CorgiChunk : Chunk (NOT NavigableChunk) — corgi columns + Vec time/diff,
ordered (key,val) by corgi structural order then time. merge/extract/advance/
settle ported from the VecChunk reference; drives corgi's discrimination sort
(sort_perm) + batched compare_idx, never per-pair. Rides #778's cursor-less
Chunk path, so it gets the fueled/graded ChunkBatchMerger for free.
- Tactics (Route B): cursor-less reduce (incremental key selection over the
input delta); join via corgi `find` (find_ranges → equal-range merge-join,
multi-record). as_collection reads columns directly.
differential-dataflow proper: widen the JoinTactic/ReduceTactic/Fresh/
join_with_tactic/reduce_with_tactic seam from pub(crate) to pub, so an
out-of-crate tactic can be implemented at all. (Candidate for upstreaming as
the public extension point the #773 tactics were designed to be.)
State: all 6 canonical programs match `vec`; 33 lib tests pass. reach ~1.9×
slower than vec, compute-bound linear ~3× FASTER (columnar eval avoids the
row backend's ~22% Value::cmp pointer-chasing). The remaining reach gap is
entirely the reduce, still row-wise — the only operator not yet columnar.
Notes / follow-ups: corgi_arrange.rs (old CorgiBatch impl) is superseded by
corgi_chunk.rs and retained only as reference; scratch examples alongside the
corgi_perf/corgi_progs/corgi_prof harnesses. Depends on frankmcsherry/wip
corgi branch dd-arrange-api.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* interactive: operator scorecard — per-operator corgi-vs-vec perf triage
A standing harness that isolates each backend operator in a minimal DDIR
program, asserts corgi == vec, and reports the corgi/vec ratio against a
target. `PROG=<name> BACKEND=<corgi|vec>` loops one case for samply.
Baseline (n=100k linear / 20k arrange-y):
map8 0.30x ✓ (columnar compute, no row boundary)
filter 0.75x ✓
arrange 1.5x ✗ \
join 1.5x ✗ > every operator with a columns<->rows
reduce_distinct 2.0x ✗ / transcode boundary loses
reduce_count 2.0x ✗
reach 2.0x ✗
Corrects an earlier whole-program-profile conclusion ("only the reduce is
slow"): per-operator, arrange and join are independently ~1.5x — the shared
cost is the transcode boundary, not one operator. Beating vec means removing
row boundaries (chunk-native ingest, columnar operator emit).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* interactive: remove arrange/join transcode boundaries — both now beat vec
The columns↔rows transcodes at operator edges were self-inflicted, not
inherent. Removed all three:
- as_collection: read chunk columns straight into a CorgiContainer
(chunks_to_columns), no untranscode→from_updates round-trip.
- join emit: the projection already yields corgi columns; emit them via
give_container into a CapacityContainerBuilder<CorgiContainer> and drop the
JoinToCorgi unary — no untranscode + per-row give + re-transcode.
- arrange ingest: CorgiChunker (a ContainerBuilder) sort-consolidates each
input CorgiContainer's columns directly into CorgiChunks — replacing
ContainerChunker's drain-to-rows + VecMerger + row-based builder. It
ACCUMULATES to TARGET before consolidating, so it emits few large chunks,
not one tiny chunk per input container (else the columnar per-chunk set-up
dominates on many small batches).
Scorecard (corgi/vec): arrange 1.5x→0.9x (BEATS vec), join 1.5x→0.7x (BEATS
vec), reduce_distinct/count 2.0x→1.35x, reach 1.9x→1.45x. All 6 programs match
vec; 33 lib tests pass. Remaining reduce gap is the row-wise reducer (Rust over
Value rows) — next: columnar consolidate via corgi group→fold_add→filter.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* interactive: columnar consolidate building block (proven, unit-tested)
columnar_sum_by_key: the monoid Count/consolidate fold as pure corgi ops
(group → map(fold_add) → filter via parse_ml + eval_graph), no Value rows.
Signed diffs passed as raw two's-complement u64 bits; wrapping fold_add gives
the correct i64 sum, `ne 0` the sign-agnostic zero-drop. Unit test:
[(1,+1),(1,-1),(2,5),(3,9),(3,1)] -> [(2,5),(3,10)] (net-zero key dropped).
Groundwork for the wave-based columnar reduce; not yet wired into the live
reduce (which stays row-wise). Depends on corgi dd-arrange-api wrapping-Reduce.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* interactive: Count reduce fast-path — skip the unnecessary per-value consolidation
Count = Σ diffs ≤ t, and summing the consolidated per-value diffs equals summing
the raw diffs — so for Count the consolidate_vals (a Value sort + clones) is pure
waste. Sum diffs directly; no Value churn. reduce_count 1.39x→1.30x, all 6
programs match vec.
Distinct/Min still consolidate per value (they reason about individual values) —
that's the corgi two-level group→fold→filter path next; Collect stays row-wise.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* interactive: columnar Distinct consolidate building block (two-level, tested)
columnar_distinct_keys: the keys with a present value (net diff != 0, the
nonzero simplification) as pure corgi ops — group by the (key,val) composite,
fold_add raw diffs, drop net-zero, distinct keys of survivors. group-by-(key,val)
is just group-by a projected composite; the nonzero test keeps everything on the
raw-bit `ne 0` path (no signed encoding needed yet). Unit test:
(1,10):+1,-1 → absent; (2,20),(2,21),(3,30) → present → keys [2,3].
Both monoid consolidate blocks now proven (sum for Count, two-level for
Distinct). Remaining: the wave integration (feed many keys per wave, driven by
the Rust time logic; reuse the existing delta/emit loop).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* interactive: group_offsets — the integers-only Rust<->corgi boundary primitive
Group a key column by identity, returning only integers: `perm` (row indices in
group order) + `ends` (per-group exclusive end within perm). DD walks groups by
index — group g is perm[ends[g-1]..ends[g]] — and drives its time/diff logic
without ever seeing a key Value; keys and payloads stay columnar in corgi. This
is the boundary model (corgi presents abstract int ids tied to times/diffs DD
sees; DD hands back indices to include). Built on sort_perm + one batched
compare_idx. Tested.
Foundation for the path-2 columnar reduce (wave loop over group ranges). Note:
cross-retire `pending` needs a STABLE key id (a u64 hash), since group indices
are per-retire — converges with hashed-keys-as-u64.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SPIKE.md: 2026-07-02 checkpoint — Chunk backend, transcode removal, boundary model
Records: arrange & join now BEAT vec (transcode boundaries removed); multi-record
primitives exposed; reduce consolidate blocks + group_offsets proven; and the
design capture for a data-blind reduce tactic (framework presents as int-id/
time/diff, owns time navigation; backend supplies id-mapping + value callback;
output ids are hashes — mint = hash the new value).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* SPIKE.md: corgi-backend hand-off — int_proxy framework (PR #781) consumption plan
Branch rebased onto int-proxy: the framework the SPIKE's boundary model
called for now exists upstream (renamed ids -> int_proxy in review);
the old in-branch copy is dropped. The hand-off section documents the
current interfaces, the one-crossing-per-retire reduce_many contract,
the assessment harness to replicate, the reduce-first plan, the traps
already hit once (scan-presents, capacity leaks, giant-chunk settle),
and the suitability questions the corgi agent should answer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* corgi ProxyReduceBackend: columnar reduce over int proxies
Implement ProxyReduceBackend for corgi and swap CorgiReduceTactic ->
ProxyReduceTactic. The DD tactic owns all time/lattice logic over
(key_hash, value_id, time, diff); the corgi backend supplies only:
- hashing: key_hash/value_id via corgi::arrange::hash_rows over the corgi
key/val COLUMNS (columnar, content-addressed; DD never hashes).
- reduce_many: ONE crossing per retire over every (key,time) bracket.
Count/Distinct are integer-only (no value untranscode); Min/Collect
gather+untranscode the bracket values once, never per key. Output ids
minted by hashing the produced value column (agrees with present_output).
- present_input/present_output: read arrangement chunks, restrict to changed
keys by columnar semijoin (novel whole, history filtered); delta-proportional.
- materialize: resolve ids -> real (key,val) rows, seal a CorgiChunk batch.
corgi_progs: all 6 match vec (scc min/recursion + unnest collect included).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* corgi reduce: burn the transcode — fully columnar id resolution
Replace the DValue resolution maps (HashMap<u64,DValue>) with corgi COLUMN
pools + integer id->row-index maps. The real keys/values never leave corgi
columns:
- present_input/output register representative key/val columns into per-retire
pools (key_index/val_index -> offsets into concatenated blocks), no untranscode.
- reduce_many builds each reducer's output value COLUMN directly: Count -> u64
prim, Distinct -> Unit, Min -> gathered input rows (id reused from input),
Collect -> List. Ids minted by hash_rows over the built column. No transcode.
- materialize resolves proxy ids to columns by gather + columns_to_batch
(new corgi_chunk helper), column-native egress (no from_rows transcode).
The only residual untranscode is Min/Collect's value ORDERING (the reduction
contract is DDIR Ord, which is not corgi's structural order); the chosen rows
are still taken columnar.
Scorecard corgi/vec: reduce_distinct 1.01x -> 0.92-0.96x (BEATS vec),
reduce_count 1.03x -> 0.90-0.93x (BEATS vec), reach 1.10-1.22x -> 1.06-1.13x.
corgi_progs: all 6 match vec; interactive lib tests green (36).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* corgi reduce: Min/Collect order via corgi sort_blocks — zero transcode
Replace the last untranscode (Min/Collect value ordering) with one columnar
corgi sort_blocks per retire:
- Min: gather positive-diff candidates, segment by bracket, sort_blocks ->
each bracket's argmin = perm[block_start]; output row taken columnar,
reusing the input value_id.
- Collect: segment entries by bracket, sort_blocks -> sorted run per bracket,
expanded by diff into a List column.
Uses corgi STRUCTURAL order, which equals DDIR Ord for the non-negative
scalar/tuple values these reductions see (all 6 programs); diverges only for
negative ints (unsigned leaf compare) and list-valued compares (length-first),
neither of which arises here — documented as an order invariant.
Needs corgi::arrange::sort_blocks (exposed on fm/corgi dd-arrange-api 9b41cdc,
the segmented sort re-exported from ops::cmp::order). The reduce backend is now
entirely transcode-free. corgi_progs: all 6 match vec; lib tests green (36);
reduce still beats vec (distinct 0.92-0.95x, count 0.93-0.94x).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* corgi reduce: identity hasher for the u64 id-index maps
key_index/val_index keys are already well-distributed hash_rows u64s, so
siphash on every register/lookup was wasted (~7% of the reduce tactic in
profiling). Pass the id through unchanged. reduce_count 0.93x->0.88x;
distinct/reach within noise (their id-maps are small). Gate green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* int_proxy: MSD-bucket sort in ProxyChunk::from_unsorted
The proxy records sort by (key_hash, value_id, time); key_hash is a full 64-bit
content hash, so one MSD counting pass on its high byte splits into 256
near-uniform buckets, each finished by a comparison sort on the full key. One
linear pass replaces most of the n log n (falls back to the plain sort for
n<512 or clustered hashes). Profiling: from_unsorted 20% -> 8% of the reduce,
driftsort gone. Wall-clock ~flat (0.89->0.90x) — the reduce is allocation-bound
(malloc 33-38%), so this just uncovers that as the next lever. Framework
(int_proxy + reduce_reference) tests green; corgi_progs all 6 match vec.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* int_proxy reduce: reuse per-key scratch across the retire
retire allocated fresh Vecs + an IdHistory for EVERY changed key (58% of the
reduce's Vec-growth in profiling, IdHistory another 17%). Hoist the transient
buffers out of the per-key loops and clear/reload them (IdHistory::load already
clears+refills, retaining capacity): rep, raw_moments (drain), and phase-B
meets, out_replay, emitted. Pure capacity reuse, no semantic change.
Scorecard: reduce_distinct/count 0.90->0.85-0.86x, reach 1.06->1.00x (parity)
at n=4000. Allocation was the real lever (sort CPU wasn't). Framework tests
(int_proxy + reduce_reference) green; corgi_progs all 6 match vec.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* int_proxy reduce: reuse the per-moment delta map (BTreeMap -> reused IdMap)
Phase B allocated a fresh BTreeMap per (key, moment) for the desired-vs-current
delta. Replace with one HashMap keyed by an identity hasher (value_ids are
already u64 content hashes), hoisted above the loop and cleared per moment
(retains capacity, no per-moment alloc/free). Iteration order is irrelevant —
deltas are consolidated and the output batch is sorted by from_unsorted.
Scorecard: reduce_count 0.86->0.85x, reach 1.06->0.98x at n=4000 (below parity).
Framework tests (int_proxy + reduce_reference) green; corgi_progs all 6 == vec.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* corgi_perf: add end-to-end SCC case (corgi vs vec)
SCC is the sharp recursive case (nested fwd/bwd label propagation via min,
negation, 4 joins). Findings: corgi is ~1.9-2.5x slower than vec and the gap
GROWS with n (unlike reach, which reaches parity) — scc's nested-recursion /
negation / min structure scales worse for corgi. And corgi DIVERGES from vec on
larger random graphs (n>=1000: 1297 vs 1292) — a corgi-backend correctness bug
(the proxy reduce tactic + reference backend pass the scc oracles in
reduce_reference.rs, and it is NOT the min value-ordering — tested with a
DValue-order argmin). Harness flags the mismatch instead of panicking.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Isolate SCC bug to corgi (not the int_proxy tactic)
Two debugging tools:
- reduce_reference.rs: add reduce_int_proxy (int_proxy ProxyReduceTactic + Vec
value backend over VecChunk, NO corgi) and scc_int_proxy_* tests that run SCC
with only the reduce swapped to int_proxy, vs the cursor reduce. They PASS up
to 50 nodes/120 edges/20 seeds — so the int_proxy reduce TACTIC is correct at
SCC's nested-product-time depth. (reduce_reference's own scc_* compare cursor
vs the *stock* reference tactic, never int_proxy — this fills that gap.)
- interactive corgi_scc_min.rs: sweep+delta-debug an SCC corgi-vs-vec divergence
to a minimal graph. Finds nodes=4, 5 edges [(0,0),(0,2),(1,3),(2,1),(2,2)]:
corgi=3 vs vec=2.
Conclusion: the divergence is in the CORGI backend, not the int_proxy work. The
int_proxy tactic + Vec values reproduces SCC exactly; swapping in corgi is what
breaks it. Next: narrow within corgi (reduce value backend vs join vs negate).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Narrow corgi SCC bug to the outer recursion (not scopes/reduce structure)
Four minimizer/trace harnesses (interactive examples):
- corgi_scc_min: minimal SCC divergence = 4 nodes, [(0,0),(0,2),(1,3),(2,1),(2,2)].
- corgi_scc_trace: on that graph, export each SCC intermediate. Result:
fwd::labels OK, trim_fwd OK, bwd::labels DIFF (node 2 -> label 1 in corgi vs
2 in vec), trim_bwd/scc DIFF. So the divergence enters at the backward min-
reduce, structurally identical to the correct forward one.
- corgi_cc_min: single-level fwd min-label propagation NEVER diverges (to 80
nodes) -> not single recursion.
- corgi_bwd_min: two-pass fwd->trim->bwd with NO outer loop NEVER diverges ->
not the chained nested scopes either.
Conclusion: the bug REQUIRES SCC's outer recursion (the iterate wrapping fwd/bwd
-> 3-deep product times, plus negation and feedback
). Symptom is a failed retraction (stale label survives).
Suspects: corgi 3-level dynamic-time handling (leave_dynamic/results_in/enter_at)
or negation under recursion. Not int_proxy (proven), not the reduce structure.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Pin corgi SCC bug to the reduce via update-stream diff
corgi_scc_stream: emit the full (data,time,diff) stream of SCC intermediates via
inspect, per backend; diff consolidated by (data,time) to find the first moment
of divergence (DD is deterministic at all times).
Result on the minimal 4-node graph: fwd::labels and trim_fwd streams AGREE
exactly (the 16-vs-10 raw counts were +1/-1 that consolidate away). The bwd
min-reduce: its INPUT (proposals+nodes) diverges only at inner-times 258/513,
but its OUTPUT diverges at 257/512 — one tick earlier. So at time (0,[1,257])
the reduce input agrees while the output diverges (vec retracts node2->1, corgi
does not). By the inputs-agree-output-differs test, the bug is IN the corgi
min-reduce. The int_proxy tactic is proven on SCC (scc_int_proxy), so it's the
CorgiReduceBackend (present/materialize/changed-key), not the tactic. The
forward min-reduce (same code) is correct -> a data-specific edge case.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Pin corgi SCC bug: reduce-input present/arrange LOSES a record
Instrumented CorgiReduceBackend::present_input (env CORGI_DBG) to dump node-2's
presented input on the minimal repro, and compared to vec's true input stream:
for (node2, label1), vec has [0,257]+1 [0,258]+1 then [1,257]-1 [1,258]-1
(added round 0, retracted round 1 -> gone); corgi's present has [0,257]+1
[0,258]+1 [1,258]+1 -- the round-1 retractions are MISSING (plus a spurious +1).
And corgi's input STREAM (bwdin) agrees with vec at [1,257] (both -1), so the
retraction enters the arrange but present_input never returns it. => corgi loses
a reduce-input update when arranging/presenting under nested times; the min then
reads a non-retracted label1 and stays stale = 1. Not the min logic, not the
int_proxy tactic (proven) -- the reduce-input arrangement (CorgiChunk) or
present_input's changed-key/novel-history read.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Complete SCC diagnosis: compaction cancels a novel retraction, erasing an interesting time
Full per-retire dump (CORGI_DBG in present_input/present_output/materialize) on
the 4-node repro nails the mechanism:
1. corgi's reduce-input TRACE compacts the round-0 add (node2,label1)[0,257]+1
(present in the input stream) forward to [1,257]+1.
2. the novel retraction (node2,label1)[1,257]-1 then EXACTLY CANCELS it inside
present_input's consolidation (from_unsorted drops the net-0 record) -> the
(node2,label1) record at time [1,257] DISAPPEARS from the presentation.
3. the int_proxy tactic seeds interesting times from the presentation, so with
[1,257] erased it never re-evaluates node2's min there and never emits the
retraction of node2->1 (materialize shows label1 only ever emitted +1, never
-1) -> node2->1 is stuck -> wrong SCC.
The reference (VecChunk) passes SCC because it doesn't compact that record onto
[1,257] at that retire, so the novel retraction stays a distinct interesting
time. Root: trace compaction advancing a history record onto the same time as an
incoming novel retraction cancels it in present, erasing an interesting time the
reduce needed. Not the min logic, not the int_proxy tactic. Debug taps are
CORGI_DBG-gated. gate green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Adopt int_proxy seed fix (fm/int-proxy 960a6ec4) + implement present_novel in corgi
Framework (cherry-pick of the model-prescribed fix): ProxyReduceBackend::key_hashes
-> present_novel(novel) -> ProxyChunk. The tactic now seeds interesting times and
the synthetic-join closure from the BATCH's own support (present_novel = the novel
batches presented ALONE), matching Model.lean's `seedSet = b.support ∪ pending`,
instead of time-filtering the merged present_input. This fixes the SCC divergence
(scenario1_cancels): a legally-compacted stored record could cancel a novel
retraction out of the merged view, silently dropping a seed while a retraction was
still owed. Kept my churn-reduce edits (buffer reuse, IdMap, radix from_unsorted).
Corgi backend: CorgiReduceBackend implements present_novel (read novel chunks alone,
unfiltered -> from_unsorted), drops key_hashes; present_input simplified (novel keys
are all in `keys`, so one binary_search filter — dropped the seen<novel_len hack).
Removed the CORGI_DBG diagnostic taps.
Verification: corgi_scc_min sweep NO mismatch (was 4-node divergence); corgi_progs
all 6 == vec; DD int_proxy (11, incl. new seed/cancellation + compaction-adversary
fuzz) + reduce_reference (21) green. Scorecard: reduce ~parity (present_novel adds
a per-retire present pass, same cost the reference backend accepts); reach ~parity;
scc now CORRECT end-to-end (still ~2x slower — the sharp recursive case).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Adopt seed_times refinement (fm/int-proxy 34cc815) + implement it for corgi
The other agent's slimmer fix: seeds need only the batch's (key_hash, time)
support, not a value-present. Interesting-time over-derivation is sound (a
non-changing seed yields a zero delta), and the tactic reads only TIMES from the
seed source — so no value hashing, value_ids, sort-by-value, or consolidation.
Trait: present_novel(&mut self) -> ProxyChunk => seed_times(&self) -> Vec<(u64,T)>
(stateless — seeds never refer back to real values). history.rs gains TimeHistory
(times-only replay, same meet-advancement, keeps the scaling shapes linear). This
also sidesteps the value_id=0 trap (a ProxyChunk would re-consolidate distinct
values at one time into a dropped zero, reintroducing the cancellation bug).
Corgi: CorgiReduceBackend::seed_times = hash_rows(novel key columns) zipped with
the Rust-side times, sorted by key_hash — no from_unsorted, no value hashing on
the seed path. The expensive half of the per-retire double-present is gone; only
a delta-sized second key-hash remains (noise). This matters most on the common
SNAPSHOT-then-increment load, where present_novel had been ~doubling the present
work on the big first batch.
Verify: corgi_scc_min NO mismatch; corgi_progs all 6 == vec; DD int_proxy (11) +
reduce_reference (21) green. Scorecard reduce back to ~parity (distinct 0.99-1.01x,
count 0.98-1.00x — was 1.01-1.04x under present_novel), reach 0.97-1.08x.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* corgi_perf: add native DD SCC (three-way native vs DDIR-vec vs DDIR-corgi)
SCC is now correct in corgi, so the original question is answerable: how does the
hand-written native differential-dataflow strongly_connected compare to the same
SCC run as a DDIR program (vec interpreter vs corgi columnar backend), on the same
random graph, same from-scratch single-worker methodology. native_scc_once drives
algorithms::graphs::scc::strongly_connected. Result: vec-DDIR ~1.2-1.6x native
(the DDIR interpretation tax is small); corgi-DDIR ~2.9-3.8x native / ~1.9-2.7x
vec, gap GROWING with n (SCC is arrangement+recursion-bound — corgi's worst case,
the mirror of the compute-bound linear cases where corgi is ~3x FASTER than vec).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* corgi_scc_prof: loopable SCC profiling target + attribution
Profiled corgi SCC (n=500, samply+pollard): the slowness is CORGI's arrangement/
merge layer, not the framework. CorgiChunk::merge = 27% total; 57% of it is the
JOIN re-merging its accumulated trace per work-unit (flatten_batches, O(accumulated)
per round -> the superlinear term, gap grows with n), 26% arrange-ingest batcher,
17% spine maintenance. corgi::compare_idx (structural discrimination compare) drives
another 24%; malloc ~25-30% from corgi rebuilding columns per merge. Framework is
small: ProxyReduceTactic::retire is 24% total but only 3.7% self (rest is corgi
present/materialize); timely scheduling negligible; scalar eval a sliver (CmpOp::eval
3.1%). Levers: cursor/fueled join read instead of re-flatten; merge without whole-
column re-gather; cut per-merge allocation. Not the reduce value logic, not int_proxy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* corgi join: delta-proportional restriction of the accumulated side
The cursor-less corgi join re-flattened the whole accumulated trace per
work-unit (O(trace)/round, superlinear on recursive joins). Restrict the
accumulated side to the fresh side's keys via find_ranges per chunk and
gather only matches — delta-proportional, structural (no hashing), no
full-trace flatten. Gate on acc_len > 2*fresh_len so the non-recursive
one-shot join (comparable sides) keeps the cheaper plain flatten and does
not pay the probe+reconsolidate overhead.
SCC n=1000: 2.84x -> 2.24x vec; non-recursive join unchanged at 0.66x;
corgi_progs 6/6 match vec, SCC corgi==vec at all sizes.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Make corgi-chunk shareable: git-dep corgi + field report + large-N SCC harness
Swap the corgi path-dep (../../wip/corgi) for a git-dep pinned to
frankmcsherry/wip@9b41cdc (dd-arrange-api), so the branch builds standalone
without a local wip checkout. Add int-proxy-columnar-findings.md (the field
report tying #781 + this branch + the corgi kernel crate, with the SCC numbers
and the columnar-Time gap) and the large-N SCC / vec-profiling harnesses the
report's repro section references.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Collect dead generations: drop pre-Chunk arrange + pre-int_proxy reduce
Remove the two superseded modules (corgi_arrange.rs pre-Chunk arrangement,
corgi_reduce.rs pre-int_proxy reduce) — nothing in the live path referenced
them — plus the ~150-line dead stratum inside corgi_chunk.rs that only they
used (from_rows/to_rows/batch_to_rows, rows_to_batch, CorgiChunkBuilder,
group_offsets, semijoin_history) and the orphaned imports.
Also delete the examples that only exercised the dead modules (corgi_arrange_smoke,
corgi_join_mechanism/dataflow, corgi_reduce_dataflow/trace) and the SCC
bug-isolation harnesses (corgi_scc_min/trace/stream, corgi_cc_min, corgi_bwd_min)
for the interesting-times bug now fixed and pinned by tests + Lean.
Live path unchanged: build clean (no warnings), corgi_chunk unit tests pass,
corgi_progs matches vec. Net -2157 lines. Git history preserves the journey.
Also correct int-proxy-columnar-findings.md: the DD subtree is NOT byte-identical
to #781 — ~148 lines (MSD-bucket sort, reduce buffer-reuse, tests) sit ahead of
the PR and should be pushed back to int-proxy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Re-port corgi onto the merged #781 seam (windowed reduce + iterator join)
Reduce backend → the windowed ProxyReduceBackend: seed_times(+ReduceInstance),
begin (open tiled session), next_window (single window = all changed keys; build
input+output ProxyBridges via collect_present + consolidate_updates), and the new
reduce_corrections (reduce_brackets for desired, then difference vs. presented
output per value_id — the backend now owns the desired−current subtraction),
emit/finish (tiled column materialization). Content-hash value_ids + in_index
resolution as before; ProxyChunk/from_unsorted gone.
Join tactic → the #790 iterator contract: defer/work collapse to prep returning
Box<dyn Iterator<Item = CorgiContainer>>; run_unit now returns the container
instead of a session give_container. meet ignored (correctness-first).
CorgiContainer already impls Accountable; fixed the join_with_tactic container
(not builder) generic at the call site.
Gates green on merged master-next DD: corgi_progs 6/6 match vec, incl. scc.
Structural order (correctness first); hash-order + join hash-probe = Phase 2b.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Value-as-id for primitive keys/values in the reduce (skip hash_rows)
For a primitive column — a bare Prim(64) or a 1-field Prod([Prim(64)]) — the
value itself is already a collision-free id (i64 as u64 is a bijection), so pass
it through directly instead of content-hashing. Compound shapes still hash. The
id is used only as an identity for netting/dedup — its numeric order is never
relied on — so the raw two's-complement u64 is correct even for negatives; no
order-preserving swizzle needed.
Applied CONSISTENTLY at every id site (both value presentations AND the
reduce_brackets output values), so desired-current still nets by matching ids.
Gate: corgi_progs 6/6 (all programs match vec). Scorecard: reduce_distinct
1.18x->1.07x, reduce_count 1.17x->1.07x, reach 1.26x->1.20x. join unchanged
(the Phase 2a join is already hash-free). SCC flat within single-run noise
(allocation-bound, not hashing-bound) — confirming columnar times is the SCC lever.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Add ColTimes<T>: SoA per-tuple time column (T: Columnar) — foundation
Introduces columnar times for CorgiChunk without a DD change. Our
Time = Product<u64, PointStamp<u64>> already derives columnar::Columnar at every
layer (Product, PointStamp, u64), with the container's Ref deriving Ord/PartialOrd
(#[columnar(derive(Ord, PartialOrd))] on both Product and PointStamp). So the same
per-tuple times that were Vec<T> (one PointStamp SmallVec per row — the dominant
SCC allocation) live in <T as Columnar>::Container as SoA (offsets + values), one
allocation pair per column instead of n.
ColTimes exposes exactly what the CorgiChunk consumers need: push(&T) / push_ref
(Ref straight across, no T), get(i)->T (materialize only for Lattice ops + emit),
len, and cmp/cmp_cross (in-place ordering via the derived Ref: Ord, HRTB-bounded).
Not yet wired — Inner.times stays Vec<T> this commit. Containment rationale: the
Chunk trait boundary is whole-chunk + frontier-antichain only (CorgiChunk is Chunk
but NOT NavigableChunk, so no cursor TimeContainer/TimeGat), and the batcher/spine
move whole Rc<ChunkBatch> — so every O(data) per-tuple T is Inner.times, ours.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Wire columnar times: Inner.times Vec<T> -> ColTimes<T>
CorgiChunk's resident per-tuple times are now SoA (ColTimes over
<T as Columnar>::Container) instead of Vec<T> — killing the per-row PointStamp
SmallVec allocation that dominated the SCC profile (~40%). The hot maintenance
paths (merge/emit/extract/advance/settle/concat) build and read times columnar:
compares go through the derived Ref: Ord in place (no T materialized), range
copies push Refs straight across, and an owned T is reconstructed only where a
Lattice op is unavoidable (advance_by in compaction) or at the egress boundary
(SortedRun for the join, CorgiContainer for as_collection — both want owned T).
Bounds: a ColTime alias (Timestamp + Lattice + Columnar) bundles the constraint;
the viral HRTB `for<'a> Ref<'a,T>: Ord` is discharged once in ColTime's blanket
impl behind ColTime::cmp_refs, so downstream code needs only T: ColTime. Ingress
paths (sort_consolidate, CorgiChunker accumulator, flatten_restricted) stay Vec<T>
(they already hold owned T), converting at from_columns.
No DD change. Gate: corgi_progs 6/6; 29 lib tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Drop unused Timestamp/Lattice imports (subsumed by ColTime bound)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Adopt corgi survey (merge) + group_bounds (advance) — bump to f556868
CorgiChunk::merge now drives off corgi `survey(kv1, kv2)`: one bidirectional
gallop over the (key,val) columns yields Runs — maximal exclusive A/B ranges
(bulk-copied, no per-row compare) + single Both matched pairs (time-ordered /
consolidated here, since corgi owns no time). Replaces the per-pair two-pointer
compare_at (the 22%-and-growing merge-compare tax). CorgiChunk::advance uses
`group_bounds(ckv)` for the group-boundary scan (one pass) instead of per-row
compare_at. Bumped corgi pin 9b41cdc -> f556868 (survey/group_bounds + hash_order
reverted).
Note on times: survey's Both is a positional (key,val) match, so it can
transiently mis-order/under-consolidate the out-of-band times; advance re-sorts +
re-consolidates each (key,val) group before any consumer reads the chunk, so it's
correct end-to-end (gate 6/6 + 29 lib tests incl. merge/advance/is_graded oracle).
SCC: 1.94x -> 1.62x vec (n=100k 6.55s -> 5.43s, ~17% off corgi). Branch
corgi-survey-merge.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* collect_present: keep the scan (find_ranges seek regresses SCC)
Tried a delta-proportional find_ranges seek of the changed keys (value-as-id
makes them seekable in structural order). It REGRESSED SCC 1.62x->2.16x: SCC's
changed set is broad (label propagation touches most keys each retire), so the
scan already touches ~every row while the per-chunk gallop only adds overhead.
Reverted to the scan; simplified the signature to `changed: &[u64]` and recorded
the negative result in the doc so it isn't re-tried. Gate 6/6.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Hash unification: compound ids via native corgi::hash, not arrange::hash_rows
The `ids()` compound fallback now hashes via the canonical native `corgi::hash`
(the designed boundary-id fold — width-blind, consistent-with-equality) instead
of the branch-local `arrange::hash_rows` (width-seeded). Safe for DDIR: every leaf
transcodes to u64 so width-blindness is a no-op, and there is no cross-path hash
comparison (value-as-id for primitives and native hash for compounds are never
used for the same value — shape is uniform per column). DDIR no longer references
hash_rows, so the corgi side can retire it freely. Gate 6/6 (incl. compound-keyed
adt/binders/unnest), 29 lib tests.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Revert the survey merge — it bloats the arrangement at multi-chunk scale
The survey merge (17ff06d0) was a single-chunk MIRAGE. survey aligns only
(key,val) (corgi owns no time), so its positional Both under-consolidates
cross-side times; combined with consuming both chunks and no suffix push-back, it
leaves canceling rows in the arrangement. Single-chunk (n<=100k) it looked like a
win (SCC 1.94->1.62x, compare_idx 21.9->8.9%), but past the chunk boundary
(edges e=2n cross TARGET=262144 at n~131k) the un-consolidated arrangement BLOATS
every iteration and the reduce re-presents ever-growing input -> super-linear
(n=150k: 104.9s / 19.5x vec vs 9.9s / 2.08x with two-pointer). Profile smoking
gun: retire 96%, consolidate_updates 24.8% + quicksort 22.7% + collect_present
17% — the REDUCE drowning in bloated input, not the merge itself.
Restored the two-pointer merge (full (key,val,time) consolidation + survivor
push-back). Kept group_bounds in advance (perf-neutral but a clean one-pass scan;
doesn't touch consolidation). Back to ~1.96x vec, LINEAR to n=250k. A survey merge
needs a group-RANGE Both to compose with out-of-band times — flagged to corgi.
Gate 6/6. corgi pin stays f556868 (group_bounds + native corgi::hash).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Fair compiled SCC baseline: strongly_connected_at + four-way corgi_scc_big
Adds scc::strongly_connected_at(graph, logic) — the prioritized variant the
code comment always suggested — routing trim_edges through propagate_at so
label introduction is log-bucketed 256*(64-lz(logic(label))), byte-identical
to DDIR's enter_at($1[0]) delay in vec.rs/corgi.rs. strongly_connected is
unchanged (= strongly_connected_at(_, |_| 0)).
Oracle-tested: tests/scc.rs gains scc_at_{10_20_1000,100_200_10,100_2000_1}
running the prioritized variant against the sequential SCC oracle (incl. 1000
incremental rounds); all 6 pass.
corgi_scc_big gains the fair(enter_at) column. Measured (arm64, e=2n):
n=100k: native 4.16s, fair 0.62s (0.15x nat), vec 3.06s (4.95x fair), corgi 5.94s (9.61x fair, 1.94x vec)
n=150k: native 7.42s, fair 0.95s (0.13x nat), vec 4.83s (5.08x fair), corgi 9.90s (10.41x fair, 2.05x vec)
The compiled target was ~6.7x closer than the plain-native column suggested.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* B3 ident-join benchmark: pointer-doubling reps + term-resolution joins
E-graph-flavored identifier-lookup workload: parent forest (chains of CHAIN,
roots self-looped) resolved to roots by an iterative min-join fixpoint
(pointer doubling), then a term table (id,a,b) resolved by two integer joins
against reps. Closed-form oracle (root = x - x%CHAIN) checked against vec,
corgi asserted == vec, hand-written native twin oracle-checked too.
Measured (arm64, chains=64, terms=2n):
n=100k: native 0.59s, vec 2.34s (4.00x nat), corgi 3.49s (5.95x nat, 1.49x vec)
n=1m: native 6.55s, vec 26.99s (4.12x nat), corgi 46.28s (7.07x nat, 1.71x vec)
corgi-vs-vec gap GROWS with n (1.31x @1k -> 1.71x @1m) — profile candidate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* B2 scc-compound benchmark: SCC over (group,index) tuple nodes
Same control structure as B1 but every node is a nested compound value
(x -> (x>>10, x&1023), order-preserving): no value-as-id fast path — compound
hashing, nested compares, and nested-column gathers carry the load.
enter_at($1[0][0]) buckets by group; the fair native twin is
strongly_connected_at over (i64,i64) with logic |n| n.0 (identical delay).
Correctness: corgi == vec on the full SCC edge set; compound output mapped
back through the encoding == the plain-encoding program on the same graph.
Measured (arm64, e=2n): compound keys hurt everyone, corgi worst:
n=100k: fair 2.76s, vec 21.56s (7.80x), corgi 61.42s (22.22x fair, 2.85x vec)
n=150k: fair 5.22s, vec 41.48s (7.95x), corgi 132.77s (25.43x fair, 3.20x vec)
(B1 plain-int corgi is 1.94-2.05x vec — the compound-key path is corgi's
weakest spot so far, and the ratio grows with n.)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* B4 ast-compute benchmark: list/flatmap/variant/fold compute bookend
Per row (a,b < 2^15): 8-element arithmetic list, flatmap explosion, Fwd/Bwd
variant + case + 4-element fold per element, min per 16 buckets (min so the
compute can't be dead-coded; reduce stays tiny). Closed-form Rust oracle
checked against vec, corgi == vec, native twin oracle-checked.
Values constrained non-negative + non-overflowing: the first draft (values to
1e6, a-b negative, e*e overflowing) exposed corgi min DIVERGING from vec on
negative ints — corgi's structural order is unsigned at the integer leaf.
Real abstraction gap, logged in the read-out; out of contract for B4.
Measured (arm64): corgi is NOT columnar on this path — flatmap/case/fold run
the row-wise fallback (ir::eval + transcode), so the expected compute win is
absent:
n=1m: native 1.09s, vec 8.35s (7.69x nat), corgi 8.89s (8.19x nat, 1.07x vec)
n=4m: native 4.55s, vec 34.61s (7.60x nat), corgi 37.19s (8.17x nat, 1.07x vec)
Columnarizing flatmap/case/fold is a top lever candidate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* B2 profiling harness: corgi_compound_prof (BACKEND=corgi|vec)
Loops one backend of scc-compound at a fixed size for samply, built identically
for apples-to-apples profiles.
Iteration-1 profile findings (n=50k, corgi 28.0s vs vec 11.4s):
- retire subtree = 77% of corgi; vec's Value::cmp/partial_cmp/eq = 26% self.
- consolidate_updates_slice_slow 17.6% total. Two tactic-side fixes (permutation
sort; interned-time integer consolidation + stable run-merge) were both
measured FLAT and reverted: the hot consolidation is NOT the tactic's delta
emit — it is next_window's presentation build in corgi_reduce_backend
(collect_present clones an owned PointStamp per presented row, then
consolidate_updates comparison-sorts the AoS records; the caller frame was
inlined away). That backend site is the next lever.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* B4 profiling harness: corgi_ast_prof (BACKEND=corgi|vec)
B4 corgi profile (n=1m): ir::eval 19% (row-wise List/Case fallback), Value
churn ~15%, arrangement merge of the 8m exploded rows ~29%, reduce 14%.
Both backends run B4 row-wise — the columnar win is unrealized here.
Iteration-4 lever scoped in the read-out: columnarize List/FlatMap/Case in
the backend via existing corgi kernels (Iota/Gather/Flatten/Fold/Inject/
MapSum + gather_lanes interleave).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* B3 profiling harness: corgi_ident_prof (BACKEND=corgi|vec)
B3 corgi profile at n=1m surfaces two concentrated corgi-kernel buckets the
SCC profiles never showed: sort_leaf_blocks 17% self (discrimination-sort
leaf pass, under ingress sort_consolidate + materialize from_columns) and
the join's find_ranges probe 17.8% total (interpreted CmpOp::eval compare
per binary-search step). Both kernel-general levers; details in the read-out.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* corgi_scc_big: SEED env for instance-variance measurement
5 timing runs at n=100k (fixed graph): corgi 3.76-3.89s, vec 2.97-3.03s,
ratio 1.26-1.30x; n=250k: ratio 1.28-1.29x. 3 graph instances (SEED=1,
12345, 999983): ratio 1.24-1.29x, vec/fair 4.78-4.94x. Both axes ~ +/-2%;
the reported ratios are solid to about two figures.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* phase 3: framework-tax decomposition profile + mimalloc for the suite
New native_scc_prof (fair enter_at compiled twin). Triptych at n=100k, self-time
by module: allocator native 2.2% / vec 28.4% / corgi 27.2%; memmove native 0.7%
/ vec 2.8% / corgi 11.9%; timely progress tracking ~0.5% in BOTH backends. The
4.8x shared tax is BUFFER LIFECYCLE (allocation + copy churn), not progress
tracking (kills lever 1b as a priority), not operator count (rung B), not edge
time repr (rung A). vec's churn is per-row ir::Value boxes (cmp+clone+eq ~20%
self); corgi's is column rebuilds (fresh Arc<Vec> per gather/merge/present).
The suite ran on the SYSTEM allocator (only ddir_server/ddir_vec linked
mimalloc). Adding mimalloc to all suite + prof binaries (one binary per
benchmark: every column shares it; native columns move <5%, DDIR 12-36%):
B1 250k: corgi 8.67 -> 7.07s (0.96x vec, 4.29x fair; was 5.2x fair)
B2 150k: corgi 53.45 -> 42.94s (1.22x vec) 100k: 1.08x vec
B3 1m: corgi 27.24 -> 19.79s (0.82x vec, 3.17x fair)
B4 4m: corgi 20.09 -> 13.00s (0.45x vec, 2.96x fair)
Gate 6/6 incl. fusion oracle; [checked] rows green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* CI prep: bump corgi pin to dd-arrange-api 1301b28; drop unused import
The pin moves from f556868 to the dd-arrange-api tip, which carries the four
landed kernel additions (wip#7). Gate green; workspace passes -D warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* examples: one ddir_bench runner; benchmark programs to .ddp files
Ten purpose-built binaries (four benchmarks + six profiling twins) collapse
into one runner: interpreted programs live in examples/programs/*.ddp where
they belong; the irreducibly compiled parts — input generators and the native
twins, including the enter_at-fair SCC — register in the runner by bench name.
BACKEND=/ITERS= select profiling mode (loop one column, a samply target),
replacing the prof twins. Output formats, env protocols (N/SEED/CHECK/CHAIN),
and all correctness assertions are unchanged; all four benches verified
[checked] plus a prof-mode smoke; gate green; workspace passes -D warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* examples: prune the build-up scaffolding
Eleven milestone/experiment binaries (rungs 0-4, M1-M4, Q3 of the backend
build-up) whose questions are answered and whose coverage the survivors carry:
the gate (corgi_progs) runs reach/scc and richer programs continuously, the
scorecard owns per-operator triage, and ddir_bench owns end-to-end measurement
and profiling. SPIKE.md narrates the arc they documented.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* restore the gate's scc.ddp; the benchmark program is scc_bench.ddp
The .ddp extraction overwrote the pre-existing gate program with the
benchmark's embedded SCC (different export shape, comments lost). The gate
kept passing because the benchmark program is also a valid corgi==vec check —
which is how it slipped through. scc.ddp is restored byte-identical;
ddir_bench loads scc_bench.ddp.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* interactive: gather the corgi backend internals under a corgi module
The five crate-root corgi_*.rs files become src/corgi/{container,chunk,join,
reduce,logic}.rs. No code change beyond the module paths: container is the
CorgiContainer on dataflow edges (was corgi_backend.rs), chunk the columnar
Chunk + arrangement plumbing, logic the DDIR-term compiler, join/reduce the
tactic bindings. backend/corgi.rs (the Backend impl, parallel to backend/vec.rs)
and col_times.rs (a generic columnar-time utility that imports no corgi) stay
where they are. Gate 6/6 both tactics; workspace passes -D warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* interactive: gate to a test; remove benchmark tooling and working notes
Addresses review of the subsystem PR:
- corgi_progs (the corgi==vec gate) moves from examples/ to a tests/ integration
test (tests/corgi_backend.rs) — it is a gate, so its home is cargo test / CI,
not examples/. Runs the six canonical .ddp programs, one #[test] each.
- Remove the benchmark/measurement tooling that isn't a DDIR example: ddir_bench,
corgi_scorecard, and the five benchmark-only programs (ast/ident/scc_bench/
scc_compound/scc_plain .ddp — the latter three the duplicate-SCC sprawl). These
are dev tooling, not part of the library; they stay on the dev branch and can
land separately if the later perf PRs want reproducible in-tree numbers.
- Drop the working notes (SPIKE.md + two findings files, ~970 lines of markdown).
examples/ is back to DDIR programs plus the pre-existing server/dump binaries.
strongly_connected_at stays (a general algorithms/ addition, tested by tests/scc.rs).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* interactive: move col_times into the corgi module
col_times exists to serve CorgiChunk (its own doc says so) and is used only by
the corgi backend's modules — it belongs in src/corgi/, not at the crate root.
No code change beyond the module path (crate::col_times -> crate::corgi::col_times).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* interactive: trim the corgi dependency comment
Drop the local-dev path note and the reference to a findings file that no longer
exists; keep the git rev pin (explicit and reproducible) with a one-line purpose.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* backend/corgi.rs: refresh the stale module header
The header described an early milestone: Spine<Rc<CorgiBatch>> (the arrangement
is a ChunkSpine<CorgiChunk> now), reduce 'awaits' a retire design (it is
implemented via the tactics), and a 'this iteration ... = todo!()' note (nothing
is todo). Rewrite it to describe the current, complete substrate.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* backend/corgi.rs: hoist the type aliases above their first use
CC/Row/Upd/CTrace were defined ~110 lines below apply_ops, whose signature reads
in terms of CC. Move them to just after the imports so the shorthands are in
scope where the reader first meets them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* backend/corgi.rs: document CorgiBackend as an uninhabited type-level tag
Explain the empty enum: it is never a value, only a type carrying the Backend
impl (selected via render_tree::<CorgiBackend>); the empty enum makes it
unconstructable. Mirrors VecBackend.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Remove strongly_connected_at; revert scc to master-next
strongly_connected_at was added for the benchmarks' fair-compiled SCC baseline;
those benchmarks left this PR, orphaning it (tests/scc.rs was its only remaining
user). Its scc_at tests exposed a PRE-EXISTING nondeterminism in propagate_at/
propagate_core (the rotated reduce_abelian propagation): scc_at_10_20_1000 is
flaky under 3 workers (the plain scc test, which uses a local iterate+enter_at
reachability, is deterministic). Rather than carry an orphaned API whose tests
surface someone else's bug, revert scc.rs + tests/scc.rs to master-next. The API
can return with the benchmark tooling once propagate_core's determinism is fixed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* corgi join: port to the int_proxy proxy-join seam, delete the bespoke tactic
CorgiJoinBackend implements ProxyJoinBackend (#809): advance draws blocks of
the key intersection as ((key, coordinate), time, diff) bridges; cross redeems
matched coordinates directly against the instance's chunks (gather_lanes) and
cuts TARGET_OUT-sized containers. run_unit and its whole-unit staging (the
large-snapshot OOM) are gone; peak state is one block plus one container.
Value tokens are canonical coordinates: the per-key merge across chunks gives
equal values the coordinate of their least occurrence, so bridge consolidation
cancels cross-batch churn (tokens are the unit of cancellation), while values
themselves are never copied into the presentation — they are gathered once,
from chunk storage into the projection input, for matched rows only.
Group tokens are the key's own u64 when the key column flattens to a single
64-bit leaf lane (what DDIR Int keys and 1-tuples transcode to): chunk order
is u64 order, so blocks resume by seeking from. Two regimes, mirroring the
bespoke tactic's probe heuristic: a much-smaller side drives and the other is
probed at the driver's keys (batched find_ranges); comparable sides are pulled
and merged symmetrically (probing would cost n log n against the merge's n).
Multi-lane keys fall back to a structural single-block walk (ordinal tokens).
Gate green 6/6, debug (harness asserts live) and release. Bench vs the bespoke
tactic: scc 3.71s vs 3.77s; ident@1M 21.79s vs 22.76s (0.92x vec); ast 0.83-
0.92x vec; scc-compound (2-lane keys, the structural fallback) 50.4s vs 46.5s,
a known follow-up: the probe/merge split applies there too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Remove the code the join port orphaned
The bespoke tactic's presentation layer (SortedRun, flatten_batches,
flatten_restricted) lost its last caller when run_unit was deleted, and the
DrainContainer row path (containers drained to rows for a reused row-wise
MergeBatcher) was never live: the corgi pipeline's one arrangement path is
chunk-native (CorgiChunker/ChunkBatcher). sort_consolidate and concat_blocks
stay — from_columns and the chunker still use them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Review pass: per-chunk as_collection, dead types, narrative pruning, hot-path TODOs
as_collection emits one container per chunk — key/val columns clone by Arc
bump, times materialize (the owned-time egress), diffs memcpy — instead of
gather-concatenating every batch's chunks into one container; the concat
bought nothing downstream, and chunks_to_columns goes with it. The vestigial
row-update alias (Upd) leaves chunk.rs, taking the layer's last row-type
import with it: nothing under src/corgi/ mentions DValue except logic.rs's
transcode boundary.
Comment hygiene: history-of-the-work narration (superseded implementations,
reverted experiments, rung/phase references, stale type and document names)
gives way to present-tense constraints; the fallback comments now say
accurately that the gap is unwritten lowering in logic.rs, not expressiveness
in corgi (which models sums and lists).
Two TODOs mark the complete hot-path row-at-a-time residue: merge() awaits
survey_groups (group-range Both; newer corgi revs export it) for a batched
rewrite, and collect_present can memoize key hashes, batch its membership
test, and move kept ranges via push_range.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Mark the container/chunk split and the row-trip fallbacks as convergence debt
Cross-reference CorgiContainer and the chunk's Inner as one payload that
should become one type: the differences (row-mutable Vec<T> times on edges,
sorted/consolidated/shared in the trace) each name the artifact that removes
them, with a bulk-mutation time container dissolving the last. from_updates/
into_updates likewise declare their intended end-state — external ingest and
inspection edges only — and each in-dataflow fallback caller names what
retires it (the Cas…
…er guard) (#811) * corgi join: row-wise fallback for projections the lowering cannot compile Backend::join gated its projection nowhere: a program with Case/List/Unary/ Hash inside a join's projection panicked on corgi and worked on vec — the one place capability (not just speed) depended on the lowering's coverage. The join now takes the same shape as linear's gate: when the projection is not compilable, join with the identity projection (Var(0); (Var(1), Var(2)) — compilable by construction) and apply the original terms as a row-wise Project, rebased from the join env [/bin/zsh=key, =lv, =rv] onto the row env [/bin/zsh=key, =(lv, rv)]. The join_fallback gate program puts a unary negation reading both sides in a join projection (verified to drive the fallback path) and asserts agreement with the vec backend; the gate grows to seven programs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * corgi lowerings: Neg/Not/Len unaries; signed-correct ordered compares Neg is corgi's wrapping negate on the raw bits (exactly -as_int()); Not compares scalars against zero and constant-folds non-Int truthiness to 1 (the cross-shape Eq precedent); Len is the static shape arity for tuples and an acc+1 Fold along each row's list. compilable admits the three; the fallback list shrinks to Case/Inject/IsTag, List intro, and Hash — the last now documented as the one true kernel gap (splitmix64 parity needs lane-wise xor and integer rem corgi does not yet have). The new scalar_ops gate program tripped a pre-existing divergence: ordered compares ran unsigned over raw two's-complement bits, so any negative operand disagreed with vec's signed semantics. Lt/Le/Gt/Ge now route both operands through ToSigned (the order-preserving signed encoding); Eq/Ne stay raw (bit-equality is sign-safe). The non-negative-int assumption narrows to order-sensitive contexts (Min, structural sort order). scalar_ops runs verified fallback-free; join_fallback moves to hash (still unlowerable) so it keeps driving the join fallback. Gate: nine programs... eight — reach/scc/stable/unnest/adt/binders/join_fallback/scalar_ops. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * corgi lowerings: literal-tag Inject and IsTag Inject with a literal tag is Op::Inject (lane t of a t+1-lane sum, the below-t lanes uncommitted); a data-driven tag has no static lane count and stays row-wise. IsTag on a sum maps every committed lane to its constant answer and unwraps (uncommitted lanes need no arm); on any other shape it folds to the constant 0, matching eval's non-Variant-is-never-the-tag. infer_term_shape learns Inject's sum shape so downstream cross-shape Eq folds stay correct. NB Term::Inject's fields are (tag, payload) — the first draft had them swapped, which the shape-free gate demoted to a silent fallback rather than a miscompile; the instrumented fallback-free check is what caught it. The sum_ops gate program covers intro, committed- and uncommitted-lane tag tests, and istag-on-tuple, verified fallback-free. Gate: nine programs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * corgi lowerings: Case, via the shape-aware compile gate compile becomes Option-returning — the gate at the moment of full information. apply_ops attempts each projection/predicate with the container's shapes and falls back to rows only when the lowering declines; the syntactic compilable check remains solely the join-INLINE gate (a declined join projection defers to a linear stage, where the shape-aware gate lowers it — deferral composes, so nothing stays row-wise for gating reasons). Out-of-range env references decline rather than panic: closed bodies truncate the environment by design. Case lowers as CapSum (distribute the environment into each committed lane), MapSum (each arm a closed body over Prod([ctx, payload]) — arms see the outer env plus the payload as top binder; a default runs WITHOUT the binder, matching eval), and Unwrap (arm result shapes joined ⊥-tolerantly; genuine conflicts defer to rows). A local shape_join stands in until corgi exports its own. Fold bodies now carry aligned [init, elem] shapes, and infer_term_shape learns Bound, Case, and joined If shapes — If must OVER- approximate sum lanes lest a downstream Case leave runtime rows unmapped. One engine caveat: Select over sums that commit different lanes trips an offset bug in the pinned rev's lane merge, so sum-shaped If results defer to rows until the next corgi pin bump. The case_ops gate program covers the compiled arm+default path (verified), the shape-clash decline (verified, via a filter so the program stays vec-valid), and dead compile_term_single is removed. Gate: ten programs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * ddir: one driver, backend-switchable; corgi renders rows at the boundary render_tree_rows is the corgi substrate's row-boundary entry point: imports convert to corgi containers (ToCorgi), the tree renders columnar, exports convert back (FromCorgi) — signature-compatible with vec::render_tree, so a row-speaking driver switches backends by switching one call. evaluate now routes through it (the conversion dance lived there twice). ddir_vec becomes ddir with --backend=vec|corgi (default vec): the corgi backend is now runnable on every program the vec runtime runs — file inputs, synthesized rounds, --explain, --diag — not only the test gate. Smoke: scc 1k/2k, 100 update rounds on both backends. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * corgi backend: single-worker guard on arrange The corgi arrange is Pipeline (no key exchange), so multi-worker execution would mis-place keys — silently wrong, not slow; the vec backend's arrange_by_key exchanges and has no such constraint. Assert one worker until a columnar exchange (radix partition by key hash) exists — that work pairs naturally with the stored-hash-column direction. Verified to fire under -w2. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Gate fixtures move next to the gate; tour.ddp joins the examples The per-feature programs (join_fallback, scalar_ops, sum_ops, case_ops) are test fixtures, not examples — contrived shapes whose comments discuss fallback routing — and now live under tests/programs/ beside the gate that owns them. The algorithm programs stay in examples/, where they honestly double as demos. tour.ddp is the complement, not a replacement: one program composing the constructs — scalars, sums with case/istag, lists with fold/len and the flatmap/collect round trip, all four reducers, iteration with concat — as the grammar's worked example and the gate's integration check. The pinned- path fixtures localize failures and keep verifiable fallback counts (a mixed term tree rows atomically, so a tour alone would silently shift coverage to the row path); the tour covers what they deliberately avoid, construct INTERACTIONS. Gate: eleven programs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * corgi join: lane-tuple key path — tuple keys leave the structural walk Tuple keys ((group, index) and friends) previously took the structural last-resort: a symmetric compare_at walk of BOTH sides, every unit — per-row structural dispatch, cost tracking the trace rather than the delta, and the super-linear round times the 250k scc-compound measurement showed. But neither dumbness is forced by tuple-ness: only the resumable-block bookmark truly needs a single-u64 key. advance_lanes pulls every key lane as u64 buffers and walks them with lexicographic machine compares, with the single-lane path's two regimes: a much smaller side drives and the other is probed wholesale (find_ranges with a gather_lanes-built tuple needle column — shape-generic), or comparable sides merge symmetrically. Ordinal group tokens, one block (the bookmark question defers to a digest scheme; the walk, not the blocking, is what scales). The structural walk remains only for keys that do not flatten to integer lanes at all (sums/lists in the key) — and fires zero times across the gate. The pair_keys fixture (composite keys with overlap, fanout, and one-sided keys) pins the path — verified routed to advance_lanes. Gate: twelve. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* corgi reduce: seek-vs-scan presentation, decided per retire collect_present was the one non-delta-proportional path left: an O(trace) rescan (plus per-chunk key re-hashing) every retire, ~50% of an incremental round's profile. It now decides per retire, when the sizes are known: a narrow changed set over seekable keys (single-leaf: ids ARE key values, so the ascending changed set converts to a needle column) gallops each chunk once per changed key with find_ranges — no hashing at all; broad churn keeps the scan, whose flat membership test beats marginal seeking (a find_ranges probe is a structurally-dispatched binary search; SEEK_ADVANTAGE=16 measured as a load regression before widening to 64). Hashed ids of structural keys cannot be inverted into needles and always scan. Steady-state (1000 rounds x batch 100, 100k/200k): reach 4.51s -> 2.10s — past the vec backend's 2.51s (1.8x -> 0.84x); scc 794 -> 597ms/round, the residual being retires whose label cascades genuinely broaden the changed set — the windowed-presentation seam's territory, not a threshold's. Load-shaped scc unchanged (3.60s). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Reattach ids' doc comment (seek_needles had been inserted mid-block) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* corgi reduce: seek-vs-scan presentation, decided per retire
collect_present was the one non-delta-proportional path left: an O(trace)
rescan (plus per-chunk key re-hashing) every retire, ~50% of an incremental
round's profile. It now decides per retire, when the sizes are known: a
narrow changed set over seekable keys (single-leaf: ids ARE key values, so
the ascending changed set converts to a needle column) gallops each chunk
once per changed key with find_ranges — no hashing at all; broad churn keeps
the scan, whose flat membership test beats marginal seeking (a find_ranges
probe is a structurally-dispatched binary search; SEEK_ADVANTAGE=16 measured
as a load regression before widening to 64). Hashed ids of structural keys
cannot be inverted into needles and always scan.
Steady-state (1000 rounds x batch 100, 100k/200k): reach 4.51s -> 2.10s —
past the vec backend's 2.51s (1.8x -> 0.84x); scc 794 -> 597ms/round, the
residual being retires whose label cascades genuinely broaden the changed
set — the windowed-presentation seam's territory, not a threshold's.
Load-shaped scc unchanged (3.60s).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Reattach ids' doc comment (seek_needles had been inserted mid-block)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Remove dead infer_shape; one shape-inference function, not two
infer_shape (single-sample) had no callers but its own recursion — container.rs
imports only infer_shape_cols — and its doc's justification ('stays for the
single-sample callers that never carry variants') described callers that no
longer exist. It also carried a panic for the Variant case that infer_shape_cols
exists precisely to handle. Gate green, debug and release.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Remove backend/col.rs: superseded by the corgi substrate, not merely parked
col.rs was commented out of backend/mod.rs pending 'a Columnar/columnar-storage
story for Value'. That story arrived as corgi. The module cannot be revived as
written: its Row(pub Vec<i64>) and columnar::Vecs<Vec<i64>, Strides> are the FLAT
i64 row model DDIR abandoned for structured ir::Value (Int/Tuple/Variant/List),
so reviving it would mean rewriting it — which is what the corgi backend is.
259 lines plus a .disabled example, neither compiled. git history keeps them.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* reduce: ids() borrows the leaf via corgi::leaf_slice ids() had its own copy of 'is this a u64 leaf' (bare Prim, or a 1-field Prod of one) duplicating what corgi can answer, and reached it via clone().into_u64() — the clone bumps the Arc, so into_u64's try-unwrap always fails and copies regardless, even for a freshly-gathered column with a single holder. leaf_slice answers the shape question once, in the layer that owns shapes, and hands back a borrow. 10 lines to 5, one shape-test instead of two. NOTE measured flat (scc 100 rounds: 51.05s vs 50.87s) — the copies removed were real but small; this is a simplification, not a performance change. Requires a corgi pin bump. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Bump the corgi pin to master (c4626fc) DDIR pinned 1301b281, a revision that existed on no branch — it lived only on the side line where corgi's arrange API was developed. That API is now on corgi's master (wip#9), and the read-path fixes on top of it (wip#8), so the pin can name master. The bump also delivers find_ranges' native u64 path, measured end-to-end here: scc, 100 rounds x batch 100: 63.66s -> 51.19s (-20%) reach, 1000 rounds x batch 100: 2.09s -> 1.56s (-25%) scc load-shaped @100k: 3.53s -> 3.50s (unchanged) Gate green, debug and release. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * reduce: record that bounded windows were measured and rejected The comment said single-window presentation was 'a later refinement', which invites an experiment that does not pay. Measured: WINDOW = 1<<14 costs 33% time (scc, 100 rounds x batch 100: 84.4s vs 63.7s) and returns 4.4% memory (356MB -> 340MB peak RSS). The seek path removed windowing's asymptotic barrier but not its constant — per-window, per-chunk seek setup multiplies by the window count — and the presentation was never the memory peak; the trace is. Doc only; replaces an inviting TODO with the measurement. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Nemo Yu <zyu379@wisc.edu>
* Avoid non-termination that results from in-band progress tracking * Retarget count/propose/validate onto half_join The three shared `lookup_map`, which matched arranged updates at times less-or-equal in the *partial* order, and emitted at the update's own time. That is a correct lookup but not a correct delta join. Pairs at incomparable times never met, and a chained stage compared against the previous stage's time rather than the original. They now sit on half_join, which compares in a caller-supplied total order and carries the original time beside a payload time advanced by lattice join. `propose` and `validate` are wrappers over the safe entry point. `count` uses the unsafe one, to sum a scalar and annotate the prefix without joining times. `lookup_map` and `propose_distinct` are deleted; the latter proposed presence without maintaining an indicator to test it against. Differences are pinned to `isize` and the `Multiply` bounds dropped. BiGJoin (Ammar et al., VLDB 2018) is stated over sets: its extension indices are set-valued and its intersection step is an existence test. Above arity two an extension index is a projection, whose multiplicity counts completions rather than the record's own annotation, so carrying annotations would need indicator projections and a contributes-once rule. That is InsideOut, a different algorithm with different indices. The set expectation is documented rather than enforced, so callers whose data are already distinct pay nothing. `count` routes a prefix with a zero count rather than dropping it, because the sum spans an interval and zero there does not mean empty at every time within it. It sums absolute values, since magnitude carries the extension count and polarity only churn. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Test the WCOJ path over partially ordered times `wcoj_triangle.rs` cannot see the payload time do any work. For a totally ordered time the total order used by `comparison` coincides with the lattice order, so every match satisfies `lub(t2, payload) == payload` and all matches collapse onto one output time. This runs the same triangle query over `Product<usize, usize>` with unordered input, against an exhaustive-search oracle, at every input time and every pairwise join of them. It adds a constructed case where an extension is born at (0,0) and retracted at (0,2), both preceding a delta at (1,0) in the total order while (0,2) stays incomparable to (1,0) in the lattice. The extension is therefore live at (1,0) while its updates sum to zero, and a `count` that gates on a non-zero sum drops the prefix and loses the triangle. Randomized generation does not reach that configuration: `count` is per key, so every extension of a key must be born and die inside the window, which twelve seeds never produced. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * State delta-query strictness directly, rather than in the timestamp `count`, `propose` and `validate` take `strict: bool` in place of a comparison closure. Only two comparisons were ever valid, and which one follows from the query: a relation looking up in a later one is strict, and cannot see updates concurrent with the delta it is responding to; looking up in an earlier one is non-strict, and can. A relation never looks up in itself. The closure existed for monomorphization rather than for generality, so the operators branch on `strict` once while building the dataflow and instantiate `half_join` twice, leaving nothing to test at each timestamp. `half_join` keeps its closure: it is the externally relied-on surface, and it knows nothing about which relations are being joined. That subsumes `AltNeu`, whose `neu` flag was a stand-in for the same choice. A delta enters at `alt`, so an arrangement left at `alt` is visible at the delta's own time and one delayed to `neu` is not; with strictness stated directly the tag carries no information. The four tagged indexes in the worst-case-optimal example and its tests collapse to one per orientation, and the nested scope goes with them. `delta_query.rs` compared two delta implementations, agreeing only after concatenating all three fragments. It now compares the concatenated fragments against a conventional three-way join, which shares no machinery with them and so is an independent answer rather than a second opinion from the same method. That also removes the last use of `Arranged::enter_at` in the repository. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Delete altneu and calculus, which no longer have consumers Both modules are gone from the worst-case-optimal path, which now states strictness at each lookup rather than encoding it in the timestamp. `calculus` cannot outlive `altneu`. `Differentiate` is defined by emitting a change at `alt` and retracting it at `neu`, and `Integrate` recovers the collection by filtering on `!t.neu`, so the tag is its representation rather than an implementation detail. This removes two public modules from the crate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nothing calls `Arranged::enter_at` any longer. The dogs3 delta query encoded its alt/neu ordering by entering traces at data-dependent times; `half_join2` now takes a comparison closure instead, which leaves the wrapper without a consumer. The wrapper was also the only one to name cursor data types. Its logic closure took a key, a value, and a time, where `enter` and `frontier` reason only about times and forward keys and values verbatim. Removing it leaves the remaining wrappers cursor-oblivious in their semantics, so a non-cursor framework can implement them. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Move the enter wrapper's cursor to the cursor module The `enter` wrapper had two halves. One presents a wrapped trace and batch, rewriting descriptions and compaction frontiers through `Refines`; it names no keys and no values. The other navigates the wrapped batch with a cursor. This moves the second half to `trace/cursor/wrappers/enter.rs`, leaving `trace/wrappers/enter.rs` free of any cursor mention. A reader who declines cursors keeps the wrapper and supplies its own module beside this one. The cursor reaches the wrapped batch through a new `inner()` accessor. It sits in its own unbounded impl block because the `Cursor` impl does not know that the wrapped storage is a `BatchReader`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Move the frontier wrapper's cursor to the cursor module As with `enter`, the wrapper splits into a half that presents a wrapped trace and batch and a half that navigates it. The second half moves to `trace/cursor/wrappers/frontier.rs`. The time semantics stay with the batch, as `advance_time`, which advances a time by `since` and reports whether `until` suppresses it. Any reader of the wrapped batch applies that rule, whether or not it reads through a cursor. The cursor no longer keeps its own copies of the two frontiers, so it holds nothing but the cursor it forwards to. Its bound strengthens to `Storage: BatchReader<Time = C::Time>`, which is what lets it hand its owned time to the batch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Borrow the frontiers once per traversal, not once per time `advance_time` borrowed `since` from the batch on every time it advanced. The borrow is a slice construction and the comparisons are unchanged, but it put a call boundary inside the per-update closure, where the prior code hoisted both frontiers above it. The batch now lends its frontiers as an `Advance`, which a reader acquires once and applies to each time. The semantics still belong to the batch, so a reader that does not use cursors still gets them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Revert "Borrow the frontiers once per traversal, not once per time" This reverts commit 2960609. The borrowed frontiers are `Antichain<Time>`, never a time container, so borrowing one per time costs a slice construction from an inlined method rather than anything a container layout could make expensive. The `Advance` type was not worth its own name. `advance_time` keeps an `#[inline]`, matching the forwarders around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Retire seed_times; present the novel run apart from the history
`seed_times` existed because `next_window` merged the accumulated input and the
novel delta into one bridge, which loses which records were novel.
A novel record that nets to zero against a history record compaction has
advanced onto its time vanishes, and its interesting time with it, so the
seeds had to be derived from a second, separate read of the delta.
The window now carries the two runs apart, mirroring the three cursors of the
conventional reduce, which has always held them apart for exactly this reason.
The seeds are the novel run's own times, read where they already are.
The two runs combine only in the per-moment accumulation, never as
presentations.
`next_window` also takes `from: &mut Option<u64>` — a key hash rather than an
index into a harness-built list — on the same contract as the join tactic's
`advance`: strictly increase it or report the key space exhausted, and report a
key entirely within the window that first mentions it.
The window is caller-owned and refilled, so its three runs keep their
allocations.
Neither side now scans the whole key space.
The harness contributes the keys it knows must be revisited, and the backend
adds the ones its novel batches touch, which it discovers in the scan the
presentation needs anyway.
That change to `changed` is where the time went, and it was a real redundancy
rather than a constant factor.
`changed` was every pending key; it is now only the keys with an interesting
time DUE in this interval.
A key whose times all lie at or beyond `upper` was being presented and walked
through determination on every retire, only to produce no moments — and its
history and output rows were gathered into the presentation to do it.
The conventional tactic never did this: it filters to the due times and takes
the no-work branch when there are none.
The closure is not lost by skipping, because a join of two not-yet-due times is
itself not yet due, and is synthesized when they become due.
Carrying the times of unvisited keys is now explicit, since a key that is not
`changed` is never presented and would otherwise be forgotten.
The split against `upper` happens up front: due times drive determination,
carried times seed the next round's pending set directly, and a key with both
unions them.
Measured on the corgi backend, both binaries built in the same target directory:
scc n=500 e=1000 batch=100 rounds=60 4.07s -> 1.92s
scc n=1000 e=2000 batch=100 rounds=100 12.33s -> 4.95s
scc n=2000 e=4000 batch=100 rounds=60 18.83s -> 5.72s
reach n=20000 e=40000 batch=200 rounds=100 flat
The gap grows with n, as it should if the old cost tracked the whole pending
set and the new one tracks the due set.
Program output is byte-identical across scc, reach, tour, adt, binders and
unnest, on both the corgi and vec backends.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Restore the no-work early return, correctly guarded
Narrowing `changed` to the due keys made the old early return unsafe, so it was
dropped along with `seed_times`.
It read
if changed.is_empty() { self.pending.clear(); return (Vec::new(), Antichain::new()) }
which was sound only because `changed` then held every pending key, so an empty
`changed` implied an empty pending set.
It no longer does: times at or beyond `upper` remain when nothing is due, and
clearing them while returning an empty frontier releases the capabilities that
were holding them.
The guarded form also requires the novel batches to be empty, and returns the
frontier over the untouched pending set rather than an empty one.
Without it a retire with nothing to do still opened a tiled output session and
shipped empty batches.
Also measured and rejected: advancing the accumulated input and output
presentations by `instance.lower` in the corgi backend, as its join backend
already does in `stage_runs`.
The reasoning is sound — every moment is at or above `lower`, so advancing
preserves each record's downset while letting the consolidation net more — but
it costs more than it saves, because the driver already sets logical compaction
to `upper` on both traces each activation, so the times largely arrive advanced
and the pass is work for nothing.
On scc (n=1000, corgi) it was a stable 5.09-5.10s against 4.85-4.89s, about 4%
worse, and flat everywhere else.
`ReduceInstance::lower` therefore remains unread by the only backend.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Improve docs; remove unneeded logic
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…#825) A reference VecReduceBackend, with tests and a tactic-vs-tactic benchmark
* Fuse discovery and evaluation in the proxy reduce tactic
The tactic enumerated each key's interesting times and then walked them a second
time to evaluate. The conventional reduce does neither: it runs one ascending
pass and evaluates as it discovers. It can because discovery never looks
backwards — every synthesized time is `next_time.join(t)` for a `t` not at or
below `next_time`, so it is strictly greater, and new work only ever lands ahead
of the sweep.
`Sweep` is that pass, cut where the conventional operator calls user logic. Each
`next_crossing` runs to the next in-interval time needing evaluation and
suspends with the buffers positioned to read the accumulations; the caller
evaluates and returns the corrections through `commit`; the next call resumes.
The tactic runs every live key to its next suspension and crosses them together,
which is what a batched backend wants, and no key enumerates its times first.
Written as a state machine rather than one body: `frontier` finds the position,
`absorb` steps the sources and decides whether the time is reached, `close`
joins forward, `settle` recomputes the meet. One tick reports `Passed`,
`Pended`, `Crossing(t)` or `Done`. That puts each clause of
`RoundCoverage.lean`'s `round_coverage` in one place — the novel witness is the
whole of `absorb`'s return value, and it is also why `close` is split, going
against the novel times always but against the prior times only for a time that
already carries a witness.
`discover_times` and the per-key moment lists go with it; so does the second
ordering of every key's records.
Measured against the two-phase schedule, on the reference backend as fixed by
the commits below it:
churn 0.64x -> 0.58x of the cursor tactic
multimoment 1.22x -> 1.09x
propagate 1.20x -> 1.13x
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Delete the determination pass the sweep replaced, and operators::common
`discover_times` lost its last caller when the fused sweep landed.
`KeyView`, `DiscoverScratch` and `TimeHistory` existed only to serve it.
The containment test that checked the sweep against it goes too: it was scaffolding for building the sweep, and the ten `int_proxy` tests and the corgi gate check the same property end to end against the conventional reduce.
`common.rs` said "Types and methods generally useful for differential computation".
Every live item in it had exactly one caller, and all three callers were in `int_proxy`.
Naming it that way claimed a generality it did not have, hid the tactic's size from anyone reading the tactic, and is how four items died unnoticed: a module named for a role nobody audits accumulates things.
So `Sweep`, its `update_meet` helper and `tile_descriptions` move to the tail of `int_proxy/reduce.rs`, and `bilinear_wave` to `int_proxy/join.rs`, in each case beside their only caller.
None of them is `pub` any more; none was ever part of an interface.
The sweep and the tactic that drives it now sit in one file, which is where they belong: the boundary between them is meant to dissolve as more of the work fuses.
A warning for anyone who benchmarks this commit.
Under the workspace release profile it appears to cost 8% on churn and multimoment, and that is an artifact of `codegen-units`, which the profile leaves at the default 16 despite `lto = true`.
Rustc partitions codegen units by module, so folding two modules into one merges two partitions and the optimizer makes different choices; setting `codegen-units = 1` makes this commit and its module-per-item alternative indistinguishable, and both land at the faster number.
Layout was ruled out separately, by permuting function order within an untouched file and remeasuring.
The branch was +301 lines against master-next and is now -67.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Tidy two clippy warnings in the fused sweep
The key-slot loop tested its terminating condition with a `let ... else { break }` where a `while let` says the same thing, and the step-live-keys loop ranged over indices into `live` only to index `live`, shadowing the loop variable with the value it read.
`join_key`'s `too_many_arguments` is left alone: it predates this branch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A DDIR variant column becomes a corgi `Sum`: a tag column plus one lane per constructor. Nothing carries the declared universe, so each shape derivation infers the arity from what it sees -- `infer_shape_cols` from the batch's tags, `infer_term_shape` from `Inject(tag, _)` as `tag + 1`. A collection holding only `Rare(_)` therefore gets one lane and one holding only `Common(_)` gets two, for the same DDIR type. Concatenating them into one arrangement panicked corgi while vec rendered it fine. frankmcsherry/WIP#10 fixed that where it lives: `gather_lanes` took the output Sum's arity from `sums[0]` while gathering out tags from every source, dropping lanes the tags still named. This bumps the pin onto it and keeps the two programs that found it as gate coverage -- one on the row-wise fallback, one on the compiled path, since both derivations under-approximate. Earlier revisions of this PR reconciled arities DDIR-side in `chunk.rs` at the four sites that read two independently-inferred columns. That is all gone: the corgi fix subsumes it, and dropping the per-merge shape walk is worth ~4% on scc steady-state (4.54s vs 4.72s, median of 3) now that #824/#826 have taken the baseline down to where it shows. Gate 14/14 debug and release; both new programs panic without the pin bump. Claude-Session: https://claude.ai/code/session_0168NJWzHwrLGW2RXToGooYP Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Stream VecReduceBackend output into open chunks per tile The backend staged every retire's output as rows in tile_rows, building chunks only at finish — instrumented at a 336MB high-water against a 2.6MB windowed presentation at 4M keys, the backend's whole memory spike. Emits arrive per tile in disjoint ascending key ranges, each consolidated, so emit can push rows straight into an open chunk and seal at TARGET; finish just wraps the sealed chunks. Also records why merged_run's cross-batch merge must stay: it is load-bearing for ordinal id sharing (netting in the id-keyed accumulations), measured at ~25% of churn when removed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * tighten documentation --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Drive VecReduceBackend windows from a resumable merge-walk MergedWalk replaces merged_run: a resumable cursor over one logical batch list, whose first `novel` entries are the retire's novel input and whose rest are its prior history, holding a (chunk, offset) per batch, an index into `changed`, and the next active hash. Only the novel prefix and `changed` activate a key; prior batches are seeked to whatever key the active set names, so a window stays proportional to the work asked for rather than to the accumulated trace. Windows are now formed lazily. The active keys are the merge of the novel batch heads with `changed`, so the whole-trace prescan that built keys_cache is gone, along with keys_stale; keys_cache now holds only the current window's keys, which the output pass replays as its own active set. The budget is spent in records rather than keys, since records are what the presentations cost, and it is checked at key boundaries so a key is never split across windows. Prior records are advanced to the compaction frontier as they are drawn and the draw is consolidated per (value, role, time). Advancement without consolidation cannot reduce anything, which is why the frontier bought nothing before; together they remove 46% of the presented records on the benchmark. Identifiers are minted from one namespace shared by both input runs and only after consolidation, so a novel retraction cancels against its own history and a value whose records all cancel spends nothing. Times and diffs move into the bridges rather than being cloned into them, leaving one owned time per presented record. Seeks are searches, the chunk advance included, and a run's end is found by doubling rather than by bisecting the records behind it. Both carry a fast path for an already-positioned cursor: searching the dense path rather than comparing cost 1.71x on churn. Against the previous backend: multimoment 21.2ms -> 17.5ms, wide 31.3ms -> 26.4ms, churn 27.6ms -> 25.7ms, propagate 3.03ms -> 3.15ms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Document that a retire evaluates only within its interval The proxy backend advances the accumulated input and output to `lower` as it loads them, which is sound only because no time below `lower` is ever evaluated or used as a comparison target. That was true but stated nowhere, so the backend depended on an invariant a reader could not check. `ReduceTactic::retire` now says what the interval bounds: every time a tactic evaluates at lies in `[lower, upper)`, times at or beyond `upper` are deferred rather than evaluated, and times below `lower` were evaluated by an earlier call. The bound is one-sided in the sense that matters — a sweep's frontier does reach past `upper`, to defer what it finds there — so the clause speaks about evaluation rather than about every time visited. `ReduceInstance::lower` now carries the consequence for a backend: it may and should advance by the frontier, but only together with a consolidation, since advancement alone rewrites times and removes no records. The novel input is exempt, as its times are the interesting-time seeds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Settle hash collisions when the window is formed Whether a hash carries more than one real key was rediscovered inside `reduce_corrections`, which resolved every identifier of every bracket on every wave to find out. The walk already sees each value once, so the question is cheaper to answer there: the drain loop's "the value changed" branch fires once per distinct value, and testing the real keys for agreement costs one comparison there. A window now records `collisions`, the hashes carrying more than one real key, and `reps`, one representative input identifier per key. `reduce_corrections` consults the set and reads the key off the bracket's first identifier, which is sound given the set and would not be without it: neither identifier space is key-ordered across a bracket, so both can read `[A, B, A]`, whose ends agree while its interior does not. Corrections cannot invalidate the set, as they mint rows under a key the presentation already showed. `reps` exists because a collision can hide from either pass alone. A hash whose input has fully cancelled for one real key still carries that key's stale output, so an input pass seeing only `A` and an output pass seeing only `B` are each internally consistent. Reaching that state needs a correction deferred past a retire's upper bound, which no test here constructs; the check is insurance against a wrong answer rather than tested behaviour. Two tests cover the rest. `reduce_collision_multiwindow` buckets several real keys into each of two hashes and spans several windows, so the collision set is built and discarded more than once per retire. `reduce_collision_inside_iterate` does the same under `Product` times. Suppressing the collision flag fails all five collision tests. churn 25.7ms -> 25.5ms, multimoment 17.5ms -> 17.2ms, wide 26.4ms -> 26.1ms, propagate 3.15ms -> 3.10ms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Build release with one codegen unit Rustc partitions codegen units along module boundaries, and fat LTO does not undo the choices each unit made before it ran. Rewriting one module's contents therefore swings a benchmark by however much the new partitioning happens to differ, in whichever direction. Measured: replacing `merged_run` with a merge-walk in `vec_backend.rs` read as multimoment 21.2ms -> 17.2ms, wide 31.3ms -> 26.4ms and churn 27.6ms -> 25.5ms at `codegen-units = 4`. Rebuilt with one unit, every ratio fell inside 0.98-1.02 with fully overlapping run ranges: the old code alone got ~19% faster, the new code barely moved. The apparent win was the old arrangement partitioning badly. The other tactics in the same binary are not a control for this. The cursor and inherent reductions, untouched by that change, moved at most 4.5% across the two builds while the proxy tactic moved 19% — the damage is confined to the module that was rewritten, which is exactly how the partitioning works. Full workspace release builds in 3m07s with one unit against 3m10s with four, because fat LTO already dominates and the build parallelises across crates rather than within them. A single release test target built alone costs about 24s more. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * Size the propagate benchmark from the environment It was the only program in the file with its shape hardcoded, so it could not be scaled to ask whether an effect grows with the work. PROP_NODES, PROP_EDGES, PROP_CHURN, PROP_ROUNDS and PROP_WARMUP now read through `sized`, as the other programs already do. The churn retracts edge `r * PROP_CHURN + c`, which has to be one of the edges initially inserted, so an assertion states that. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * tighten up comments --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* B4 ast-compute: the row-wise-fallback measurement harness Ports the B4 benchmark and its samply target from fable/work, where they were written but never landed: list build with arithmetic, flatmap explosion, variant tag + case + fold per element, then a tiny min reduce (16 keys, so the reduce is not the subject). A closed-form oracle checks vec, corgi checks against vec, and a hand-written native twin checks against the oracle. The program is chosen to sit entirely on `apply_ops`' row-wise fallback path, which is what it measures. On master-next today (arm64, mimalloc): n=1m native 1.08s vec 6.56s (6.08x nat) corgi 5.96s (5.52x nat, 0.91x vec) n=4m native 4.45s vec 32.45s (7.29x nat) corgi 25.70s (5.77x nat, 0.79x vec) Profile at n=1m: apply_ops is 55.9% of the run, of which ir::eval 27.1%, from_updates 18.0%, into_updates 5.5% — the untranscode/eval/re-transcode round trip is 50.6% of total. Value::clone (12.8%) plus drop_in_place<Value> (10.9%) is 23.7% spent on DValue trees that exist only for the fallback. corgi::eval_graph draws zero samples: every map declines. `Term::List` has no lowering, and the decline propagates up through Fold/Inject/If/Case, so the general Case lowering from #811 never engages on this program. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS * Columnar list-literal intro: the keystone lowering A homogeneous `list(e0, .., ek)` becomes k columns and a length-k list per row through corgi's existing kernel matrix, with no per-row work and no new corgi op: `Enlist` each element (a length-1 lane per row), `Iota` a per-row [0..k) tag list, `Weave` interleaves the lanes in field order into List<Sum{X x k}>, and `MapList(Unwrap)` strips the now-homogeneous sum. Empty and heterogeneous literals decline (Weave needs a lane, Unwrap needs the lanes to join) and rows handle them. `infer_term_shape` gains the matching arm, which is what makes `fold` over a literal work: the Fold lowering reads the list's shape to find its element shape, so without it a `fold(list(..), ..)` declined even though both halves were already written. That is the keystone property in general -- a `list(..)` subterm anywhere made the WHOLE projection fall back to rows, which is why ast's case and fold lowerings (landed in #811) had never once engaged. B4 at n=1m: eval_graph goes from 0 samples to 59, and corgi 0.91x -> 0.85x vec. `compilable` still answers false for `List`, like `Case`: it is the shape-free gate for join-INLINE projections and cannot judge homogeneity. Its doc said these were unwritten lowerings; it now says what the gate is actually for. Gate 14/14, 29 lib tests, B4 checked against its oracle at every size. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS * Drop the sum-If gate: the pin it was written against moved in #817 The `If` lowering refused any sum-shaped result, on the grounds that merging sum columns committing different lanes tripped an offset bug in the pinned engine's lane merge. The comment said "revisit at the next corgi pin bump". The bump happened: #817 moved the pin from c4626fc to cb26fbd, whose sole commit IS the fix -- "gather_lanes: commit every lane any source does, not source 0's (#10)", with a regression test for the exact differing-arity case. `Op::Select` is implemented by `gather_lanes`, so the gate has been dead weight since Aug 11. It was not a cheap gate. `if(c, Fwd(x), Bwd(y))` is the shape of every conditional constructor, so a `case` over one fell back to rows -- which, together with the list-literal gap, is why B4's whole compute chain ran row-wise. B4 (n, corgi vs vec, checked against the oracle at each size): 100k 0.94x -> 0.54x 1m 0.91x -> 0.59x 4m 0.79x -> 0.54x Against the compiled-DD twin, 5.5-5.8x -> 3.6-3.9x. Profile at n=1m, share of total: apply_ops 55.9% -> 32.6%, ir::eval 27.1% -> 0.5%, from_updates 18.0% -> 11.8%, into_updates 5.5% -> 2.2%, eval_graph 0% -> 13.7%. The untranscode/eval/retranscode round trip falls from 50.6% to 14.5%, and what remains is flatmap, which is still row-wise. Gate 14/14 (adt, case_ops, sum_ops, sum_skew, sum_skew_compiled and tour all exercise sums), 29 lib tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS * Columnar flatmap: explode the list column structurally `compile_flatmap` lowers the list term over [key, val] to a corgi List column, declining when it will not lower or is not list-shaped (the explode needs real bounds, where `ir::eval` would accept any List value it happened to produce). The backend then explodes that column WITHOUT moving any element: the list's flat element storage already IS the new value column, and each row's span in the bounds yields both the within-row position DDIR pairs with it and a repeat map that carries key/time/diff across. No per-row eval, no transcode. The row-wise path moves to `apply_flatmap_rows` and stays the fallback. B4 (n, corgi vs vec, checked against the oracle at each size): 100k 0.54x -> 0.42x 1m 0.59x -> 0.48x 4m 0.54x -> 0.45x Against the compiled-DD twin, 3.6-3.9x -> 3.0-3.2x. That finishes the row-wise fallback on this program. Profile at n=1m, share of total, from where this branch started: apply_ops 55.9% -> 17.2% ir::eval 27.1% -> ~0% eval_graph 0% -> 16.0% from_updates 18.0% -> 1.2% into_updates 5.5% -> ~0% apply_ops is now essentially eval_graph alone, and the remaining `from_updates` is the `ToCorgi` import unary -- the I/O boundary, which is where transcoding belongs. `Value::clone` + `drop_in_place<Value>` fall from 23.7% to 2.0%. Gate 14/14, with unnest and tour verified to drive the columnar path rather than the fallback (instrumented run: zero fallbacks across the gate). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS * Columnar EnterAt: read the delay as a column, join it into times in place `enter_at`'s key and val columns are the identity -- only times change -- but the row-wise path untranscoded the WHOLE container to get at one integer per row. `compile_scalar` now lowers the delay field to a `U64` column (declining when it is not `Prim`-shaped, since the delay has no per-row reading then), and the times are adjusted in place. The delta never has to be built: joining `Product(0, PointStamp([0,..,delay]))` is coordinate-wise `max` at index `level-1` and identity elsewhere, because 0 is u64's minimum. `PointStamp::new` re-strips the trailing minimums the resize can add, so a zero delay leaves the representation canonical. `compile_flatmap`/`compile_scalar`/`compile_predicate` now share one `compile_over_kv` -- they differed only in the shape they demand of the result. As sized earlier this is coverage, not speed: enter_at's input collection changes once per epoch, so it is not per-iteration hot. Verified engaged rather than assumed -- an instrumented run shows the gate taking the columnar branch twice (scc.ddp's two enter_at sites) and the fallback zero times. scc agrees with vec beyond the gate's tiny inputs: 500/1500 -> 1319/-34 on both, and 5000/15000 -> 13281/-39 on both (corgi 348ms vs vec 410ms). Gate 14/14, 29 lib tests, 10 explain tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS * Explain on corgi: the rewrite's own dataflows, cross-checked columnar `LinearOp::LiftIter` is synthesized by the explanation rewrite -- every `$host:` export is a LiftIter Linear, and it panics if it appears in a user program -- so an explained program is the only thing that exercises it. tests/explain.rs ran on `vec` alone, leaving the entire explain surface unexercised on corgi. These tests render the SAME rewritten dataflow on both backends and require every export to agree, which is the coverage a columnar LiftIter needs to be safe to write. Seven agree: reach, tc, flatmap, collect, a depth-1 `enter_at` loop, a depth-1 `min` loop, and an SCC-shaped program (depth-2 nesting, min, three joins, filtered feedback). Three do not, and the divergence PREDATES this branch -- reproduced at 78d75b0 in a scratch worktree, so it is neither the list work nor the columnar enter_at. On an explained program whose iterative feedback is NEGATED, corgi reports a strict SUBSET of vec's `demand:input0` (2 rows of 10 on the small scc instance). Isolated to the negation alone: `corgi_agrees_on_explained_scc_one_scope` and `explained_scc_one_scope_negated` differ in exactly one line (`var trim = trim_fwd | filter(..)` vs `var trim = trim_fwd - edges`) and only the negated one fails. Ruled out: `enter_at` (the depth-1 delay loop agrees, and forcing the row-wise enter_at path changes nothing), `min`, and depth-2 nesting on its own. The three are `#[ignore]`d with that reason rather than deleted, and the one-line pair is kept as the minimal reproducer. Note the plain (unexplained) programs do NOT diverge -- the corgi gate's scc passes, and scc at 5000/15000 matches vec exactly -- so the bug needs both the negated feedback and the rewrite. `query_rows` is factored out of `demand_for_queries` for reuse. Explain suite: 17 passed, 7 ignored (3 new known-divergence + 4 pre-existing sweeps). Gate 14/14, 29 lib tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS * Columnar LiftIter: one integer field, read out of the time The inverse of EnterAt. `LiftIter` reads each row's iteration coordinate out of its time and appends it to the value, so keys, times and diffs are untouched and no term is compiled -- which makes this path TOTAL. There is no gate and no fallback, unlike every other columnar op here. The empty product is where the two representations part company, and it is not a corner case: DDIR unit IS `Tuple([])`, which `append_iter` extends to `Tuple([iter])`, but columnar it arrives as `CValue::Unit`, not an empty `Prod`. Emitting `Prod([Unit, iter])` would be a silent one-field-too-many divergence from `backend::vec`, so `Unit` maps to `Prod([iter])`. The explain suite hits that branch 38 times (and the `Prod` branch 128) at the previous commit's coverage, so this is checked rather than argued. `append_iter` goes with it -- corgi has no remaining row-wise caller, and `backend::vec` keeps its own. Explain 17 passed / 7 ignored, gate 14/14, 29 lib tests. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS * DDIR's hash IS corgi's hash; the gates that assumed otherwise go Nothing ever required the two backends' hash to be a PARTICULAR function -- only that they agree, since `hash` is an observable program value that lands in keys and gets exported. Rust makes no cross-run promise for `Hash` either. So DDIR's `hash` becomes corgi's structural hash, and `ir::eval` follows corgi rather than the reverse: `ir::structural_hash` is a row-at-a-time transcription of `corgi::hash`'s fold, salts and all, pinned by five tests that hash the same values through both paths (scalars, tuples, units, lists, variants, nesting). `ir::eval` no longer calls `hash_u64`, which goes back to being what it is -- the benchmark row generator's mixer, unrelated to the language. The lowering is `Op::Hash` over the arguments as one tuple, `Shr(1)` for the sign bit, `Rem` by the bound. The bound guard needs no `Select`: `Rem`'s total `x % 0 = x` gives `bound == 0` the identity, and a NEGATIVE bound reads as a u64 at or above 2^63 -- above the shifted hash -- so it is the identity too. Both match `ir::eval`'s `if bound > 0` exactly. `compilable` can admit `hash` shape-free, since `Op::Hash` folds whatever structure it is handed. REQUIRES corgi's `BinOp::Rem` (~/Projects/WIP/corgi, branch `ddir-rem`, commit 2221578, NOT pushed). The pin here is a local path so this branch is buildable and testable; it must become a rev before this merges. The two gates keyed to hash being unlowerable are gone, per Frank: a test that asserts a thing we intend to fix is the bug. * `join_fallback.ddp` used `hash` purely as an unlowerable term. It now uses a `case`, which drives the SAME path for a permanent reason: `compilable` is the shape-free gate, it runs before any container exists, and it will always decline shape-dependent terms. Verified still driving it (instrumented: one ROWFALL join, and no project fallback after, so the rebased projection compiles columnar). * `sum_skew.ddp` and `sum_skew_compiled.ddp` were the same program twice, one wrapped in `hash` to force the row-wise path. They now compile identically, so they are one program: the surviving `sum_skew.ddp` keeps the property (differing Sum arity reconciled through ⊥ lanes) and the accounting of both derivations. Gate 14 -> 13. Audit: instrumenting every fallback and running the whole suite leaves exactly two, both correct declines rather than gaps -- a `filter` whose `Case` arms have genuinely conflicting shapes (Prim vs Prod), and a `project` whose `Case` arm is a `Fold` whose step reads `Bound(2)`, past the closed body's env. No term now falls back for want of a lowering. B4 unmoved at 0.50x vec. 34 lib, 13 gate, 17 explain. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS * Pin corgi to the pushed `Rem` rev `frankmcsherry/wip` @ 2221578 ("Integer Rem, with a total zero divisor"), branched off the previously pinned cb26fbd. Replaces the local-path pin the previous commit used to be testable before the push. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS * corgi reduce: present values by NON-ZERO net, not positive net `Distinct` and `Min` selected the values they present with `d > 0`. DD's `reduce` presents every value whose accumulation is non-zero, negatives included, and `backend::vec` -- the correctness reference -- acts on exactly what DD hands it: its `Distinct` pushes `1` without looking at the diffs at all, and its `Min` takes `vals.iter().map(|(v, _)| v).min()` over all of them. So corgi silently dropped any key whose values all accumulated negative, and could pick a different minimum in a bracket that mixed signs. Only a negated collection produces a negative accumulation, and in the six canonical programs the reducers only ever see non-negative data -- which is why this survived until the explain rewrite, whose demand dataflow subtracts. `Count` already tested the SUM (`c > 0`, matching vec) and `Collect` already emitted per bracket with `d.max(0)` items; both were right and are untouched. Found by bisection on operator-level traces of the two backends: the wave front was `n165 = n164 | Distinct` in the explain scope, whose input agreed exactly on both backends at that time -- one value accumulating to -1 -- and whose output had the row on vec and not on corgi. Everything previously reported about this bug (the demand loop stalling at iteration 2, a join yielding multiplicity 1 instead of 3, the nested clone diverging while the forward clone did not) was downstream of that one row. The three `#[ignore]`d divergence tests from the explain-on-corgi commit now pass and are un-ignored, including the one-line minimal reproducer pair. Ruled out along the way and left as a pinned invariant: `ColTimes`'s columnar time order matches the owned `Ord` (property test over mixed-length PointStamps, `[3]` vs `[3, 1]` and friends) -- the chunk layer sorts and merges by the former while every other layer reasons with the latter. 35 lib, 13 gate, 20 explain, plus all 4 heavy sweeps (fuzz, scc-100, join-partner-time, demand-excess). B4 0.47x vec, scc 5000/15000 identical to vec (13281/-39) at 369ms vs 408ms. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS * The ast benchmark becomes a DDIR program, not two Rust binaries `interactive/examples` holds three Rust files, all generic drivers, plus DDIR programs. The B4 harness added two more binaries, each carrying its DDIR program as a string literal -- exactly what the convention exists to prevent. `corgi_ast_prof` was wholly redundant: `ddir` already loads a `.ddp`, takes `--backend=vec|corgi`, synthesizes rows and reports timings, so profiling is `samply record -- ddir --backend=corgi programs/ast.ddp ..` with no new binary. `corgi_ast_compute` duplicated that too; what was genuinely its own -- a closed-form oracle and a hand-written compiled-DD twin -- is a benchmarking concern, and compiled baselines already live in `diagnostics/examples`, not here. Both go; a benchmarking pass can reintroduce a baseline deliberately. The program itself is worth keeping and is now `examples/programs/ast.ddp`, where `ddir` can run it and the gate can cover it (14 programs). It is the only one exercising list intro, columnar flatmap, `case` over an `if`-selected constructor and `fold` together, and since a `list(..)` subterm anywhere makes the whole projection fall back to rows, a regression in one shows up as the others going row-wise too. Its input contract (non-negative, `$0[0] - $1[0] + 32768` non-negative) is written down in the file. The corgi-vs-vec claim stays reproducible from the tree without the harness: `ddir --backend=vec|corgi programs/ast.ddp 2 32768 200000 200000 1` gives vec 4.01s vs corgi 1.86s (0.46x), matching the 0.47-0.49x the deleted harness measured at n=1m. What is not reproducible without a baseline binary is the ratio to compiled DD. Branch diff drops from +856/-146 to +658/-146. 35 lib, 14 gate, 20 explain. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The novel data's two roles part ways. Its time support — the thing that seeds interesting times — rides the window as a compact `seeds: Vec<(key_hash, T)>` channel, recorded raw from the novel batches before anything else touches them. Its records, which were only ever accumulants and join partners, merge into one input presentation with the prior history, where they net against it and are advanced by the meet like every other record. The invariant that forced the runs apart (#824) was about seeding, not presentation: consolidating novel against compacted history can cancel a record and lose its interesting time. Consolidation only cancels equal-((key, id), time) pairs, and such a time is by construction in the seed channel. `reduce_seed_survives_cancellation` pins exactly this: a novel retraction that exactly cancels the compaction-advanced prior record presents an EMPTY merged input, and the seed alone retracts the key's stale output. Coverage-wise, `round_coverage`'s witness clause reads only times — the seed list carries the same support the novel run did — and its closure's novel partners are the seed-derived `reached` set, whose clause the deleted novel-buffer clause merely duplicated. The Sweep loses its novel ValueHistory: one fewer load+build+step per key, `close` loses a clause, the frontier polls four sources, and the warned (due) times merge with the novel support into one seed list — they always played the same role. Novel records become meet-advanceable (they could not while moonlighting as witnesses), so buffers compact strictly better. Both backends present once per window: the vec backend nets cancelling pairs inline at the bridge push (novel and prior share the id space, so the pair meets adjacently), and corgi's single consolidation now nets across the runs, which the split presentations could not. Interleaved runs (x3, medians), cursor mode flat within 1.5% as the control: churn 19.7 -> 18.7ms/round (-5.1%, the netting — a round's retraction cancels the prior round's insertion in the presentation itself), wide -2.4%, multimoment -2.0%, propagate noise; ddir adt on corgi -1.7%, scc flat. Honest ledger: +31 code lines net. The invariant moves from structural (two runs, raw one in hand) to contractual (the seeds obligation) plus a regression test; in exchange the tactic's reasoning core shrinks and the runtime shape improves. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
…r', when the input consolidates to nothing (#834)
* Reduce proxy pending-state rebuild overhead * Merge ordered Corgi reduce presentations * Add SCC backend comparison benchmark * Remove profiling and benchmark scaffolding * Assert proxy pending frontier invariant * Move due proxy timestamps without cloning * Tighten Corgi presentation merge contract * Make proxy reduce invariants explicit
* Arrangements present an integer key, and are sorted by it Corgi must hand the proxy seam an integer key, and the seam works properly only when the arrangement is already sorted by that integer. Until now corgi supplied the integer but not the order: `ids()` derived it per retire and threw it away, so for anything but a primitive-integer key the arrangement was in structural key order and the identifier was in some other order. Everything downstream paid for that mismatch — the identifier was re-hashed every retire, and `changed` could not be turned into needles, so the accumulated history was scanned in full. The rule is now applied once, at ingest (`CorgiChunker::flush`): a key that is already a primitive integer is used as it stands; any other key shape is hashed and the hash PREPENDED, making the key `Prod([hash, key])`. `from_columns` sorts lexicographically over lanes, so that is hash order with the real key as tie-break, and the identifier is lane 0 either way. The real key stays in the column, so colliding keys land adjacent and sub-sorted rather than indistinguishable, and reads recover it by dropping a lane (an `Arc` bump). The hash is computed once and thereafter moves as data: merge, advance and settle permute key columns with `gather_lanes`, so no transducer recomputes it. This is the distinction the earlier hash-ordering attempt missed — a coordinate is carried, a sidecar is recomputed. What this deletes: * `collect_present`'s scan branch, and with it `seek_needles`' `Option`, the seek-vs-scan heuristic and `SEEK_ADVANTAGE`. Every key shape now seeks, and seeks by `find_ranges` over a `u64` leaf — corgi's fast path. * `ids()` on keys. It reads lane 0. Egress strips the lane in the two places a key leaves an arrangement: the container built by `as_collection`, and the key handed to the join's projection, which is written against the key the program declared. Values are untouched — value order is load-bearing for Min/Collect, and value identifiers are a separate question. 53 tests, plus the 4 heavy release ones. `pair_keys` caught the join egress. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS * Keep the seek-vs-scan choice; the identifier lane makes both branches cheap Deleting the scan branch cost the primitive-integer programs: scc +1.2%, ident +3.4%, both of which this change should not have touched at all. The heuristic was earning its keep — broad-churn retires (loads, label cascades) present most of the key space, and a gallop per changed key loses to a flat membership test there. Restored, with both branches reading the identifier lane. That is strictly better than the version this branch started from: the seek is `find_ranges` over a `u64` leaf for EVERY key shape, and the scan BORROWS the lane rather than materializing it — a structural key used to be forced onto the scan (its on-the-fly hash could not be inverted into a needle) and forced to allocate a hash vector per chunk per retire to take it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS * The join seeks the identifier lane, so every key shape blocks and resumes `advance_leaf` was the only regime that blocks: it seeks `from`, ends at a key boundary, and writes back where to resume. It was reachable only by keys that were a single u64 lane. `advance_lanes` and `advance_structured` opened with `*from = None` and staged the whole intersection in one call, which is the join half of the windowing problem. Now that every arrangement leads with its identifier and is sorted by it, the leaf walk applies to every key shape — it seeks and resumes on that lane. The dispatch tries it first and the two whole-key walks become the fallback. Three sites moved off whole-key-shaped needles onto the lane: the block seek, the batched probe, and `LeafView`'s pull. `needle_like` goes with them — building a needle in the key's own shape was only ever a way to search a column that had no integer to search by. The one exposure is a hash collision, which would cross-product two distinct keys sharing an identifier. `one_key` checks per matched key, not per row: runs are contiguous and sub-sorted by the real key, so a run holds one key exactly when its first and last rows agree, and the same pass over both sides' runs confirms they matched on the key rather than the hash. On failure the call is redone by the whole-key walks, which compare the real key and cannot be fooled. That path cannot be provoked with a real hash, so it is mutation-tested both ways: forcing the fallback for every hashed key leaves all 53 tests passing (it computes the same answer), and panicking in the fallback also leaves them passing (it is never taken, so the leaf walk really is handling these keys). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS * Decide the block's extent before reading it, then read exactly that The block bound used to be discovered from what had already been read: pull a fixed number of rows per chunk, then take the least last-pulled identifier, since anything at or beyond it might have more rows outside somebody's buffer. Whatever sat past that bound was discarded and read again by the next block. The waste is structural rather than incidental — a chunk whose keys are sparse spans a wide identifier range with its rows and a dense one spans a narrow range, and the bound is the minimum, so the sparse chunks re-read most of what they touched, every block. A totally ordered identifier makes the other order possible. `block_horizon` reads ONE value per chunk — the identifier a budget's worth of rows in, bumped past its own run so a key is never split — and takes the least. `block_ends` then binary-searches that bound in each chunk. Each chunk is read once, over exactly the rows the block needs, and nothing is read twice. This is why it could not have been written this way before: choosing a bound before reading requires a single ordered value you can both index by position and seek by value, and until the arrangement led with its identifier, a non-scalar key had neither. Falling out of it: * `pull_horizon` and its extend loop are gone. The degenerate case they existed for — a run longer than the budget denying all progress — cannot arise, because the bound is taken past the run at the budget, so the chunk that set it always contributes at least a budget of rows. Progress is by construction, not by retry. * `LeafView` borrows its identifiers straight from the chunk's lane instead of gathering them into a `Vec`. The block reads keys; it no longer copies them. * Seeking the resume point is a `partition_point` on that slice rather than a column probe. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YH7iq9JoXmf7ATpq1gZaQS * Reduce proxy pending-state rebuild overhead * Merge ordered Corgi reduce presentations * Add SCC backend comparison benchmark * Remove profiling and benchmark scaffolding * Assert proxy pending frontier invariant * Move due proxy timestamps without cloning * Tighten Corgi presentation merge contract * Make proxy reduce invariants explicit * Filter hash collisions without restarting join blocks * Keep collision handling off join and reduce fast paths --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
) * Track timely master: messages stamped by sets of capabilities Points the timely dependency at timely master, which stamps each message with a multiset of timestamps rather than exactly one. The adaptations are mechanical: Distributor implementations receive the stamp in place of a time and reproduce it on each produced sub-message, and capture events carry a stamp. All stamps remain singletons; behavior is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BrUdeCb6dsunVdk4acCPmh * Un-shred arrange: batches ship whole, stamped by their capabilities The arrange operator retired capabilities one at a time, carving the batcher into per-capability tiles because each message could carry only one capability (its comment: 'Until timely dataflow supports multiple capabilities on messages, at least'). It now seals one batch per frontier advance and ships it under a CapabilitySet of the retiring capabilities; for totally ordered times the capability antichain has at most one element and behavior is unchanged. TraceReplayInstruction's capability hint becomes a Stamp (empty exactly for empty batches), and trace import replays batches under capability sets minted with delayed_stamp. Consumers accept multi-stamp batches: join retains the stamp and lower-bounds its unit's consolidation meet by the lattice meet of the stamp's elements (it must be the meet — any single element would be unsound for the history advance); reduce, count, threshold, and arrange's own input retain each stamp element. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BrUdeCb6dsunVdk4acCPmh * ReduceTactic retires one batch spanning the interval The tactic contract required output batches to tile [lower, upper), one per held capability time, because each message could carry only one capability. With stamped messages the tiling is unnecessary: retire now returns a single optional batch spanning the interval, which the driver ships stamped with the held times not in advance of the upper limit — precisely the times that justify its contents. Everything that existed per tile becomes single: the cursor tactics keep one output buffer and one builder in place of one per held time, and their thinkers append to a plain vector rather than routing each update through a reverse search for its covering capability; the proxy tactic accumulates one delta buffer per window rather than one per held time, and the tile_descriptions machinery (descriptions, held-time tags, and the tile_of routing map) is deleted. The ProxyReduceBackend lifecycle keeps its shape — begin, then windows of corrections and emissions, then finish — but begin now receives its one description owned rather than borrowing a list it must clone, emit loses its tile index, and finish yields the batch rather than a vector of them. On scc_bench with one worker (spreads under 1%): 100 nodes / 200 edges / 2,000 concurrently open rounds runs in 0.88s against 4.06s before (4.6x); 20 / 40 / 20,000 rounds in 14.2s against 20.6s (1.45x). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BrUdeCb6dsunVdk4acCPmh --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Preliminary HalfJoin tactic * Tighten up interfaces * reorganize and tighten * Generalize to allow a different mid container type
* Relocate batch merging traits into the fueled spine The Merger trait and the Batch::Merger associated type were consumed only by spine_fueled.rs, but lived on the common Batch trait, obliging every batch type to carry this one spine's merge opinion. Move them into the spine as a spine-local trait, SpineBatch: Batch, along with the Rc forwarding (RcMerger). The common Batch trait retains only `empty`, which has a consumer outside the spine: TraceWriter::seal mints empty batches to pad otherwise empty intervals of time. Implementors (OrdValBatch, OrdKeyBatch, ChunkBatch) split their impls accordingly. No behavior changes; call sites are path and name updates only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Collapse begin_merge into Merger::new `SpineBatch::begin_merge` and `Merger::new` were two entry points to the same construction, and the two implementors overrode `begin_merge` with the trait's own default body. Keep `Merger::new` and drop `begin_merge`; `SpineBatch` is now just the association of a batch type with its merger, and the documentation of what starting a merge means moves to `Merger` itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Batches as data: Batch<T, B> { desc, inner: Option<B> }
An exploratory replacement of the BatchReader and Batch traits with a
concrete struct: a batch is a Description plus an optional payload,
absent exactly when there are no updates. Payloads are unconstrained;
reading them is the Navigable capability, and everything else is the
business of whichever harness holds them.
Consequences, in rough order of significance:
* Empty batches are constructible by anyone (a description with no
payload): Batch::empty is a constructor, not a capability, and the
minting question (TraceWriter::seal, Spine::close) dissolves.
* Rc sharing is a payload type parameter, not a forwarding-impl family:
rc_blanket_impls shrinks to the Navigable/cursor forwarding, and
RcBuilder disappears (builders emit Rc payloads directly). External
Arc newtypes would need no batch impls at all.
* Description algebra hoists out of the payload mergers into the spine:
MergeState computes each merge result's description (adjacent lower/
upper, sinces joined with the compaction frontier), and mergers become
pure payload mergers handed a since. A merged payload that cancels
entirely becomes an absent payload.
* len() leaves the public batch surface: its only consumers were the
spine's accounting and logging, so it lives on SpinePayload (the
renamed SpineBatch), alongside the Merger opinion.
* TraceReader grows a Payload associated type; Tr::Batch becomes the
BatchOf<Tr> alias. The join/reduce/half-join tactics take payloads
(empty batches have nothing to join) with an explicit time parameter,
and the drivers keep descriptions for their frontier accounting.
The two kinds of nothing remain deliberately distinct in the spine:
MergeState::Single(None) is structural fuel bookkeeping with no
description at all; Batch { inner: None } is a recorded empty interval.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Builders produce payloads, not batches
A builder was handed a `Description` it only forwarded into the batch it
returned, and each builder decided independently to `Rc`-wrap its payload and
to report an absent payload when it had built nothing. That put three
responsibilities in each implementation, two of which are the caller's.
`Builder::Output` is now the payload, `done` and `seal` drop the description
parameter and return `Option<Self::Output>`, and callers assemble
`Batch::new(description, payload.map(Into::into))` under a
`Bu::Output: Into<Tr::Payload>` bound. `From<T>` for `Rc<T>` and `Arc<T>` are
both in std, so the sharing choice is a bound the caller satisfies rather than
a decision baked into each builder.
`Batch`, `Description`, and `Rc` all leave the batch implementation modules:
the payload types no longer name the description they travel with, nor how they
are shared.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Rename RcOrd*Builder to VecOrd*Builder
The `Rc` in these aliases stopped being true when builders began producing bare
payloads. The sharing opinion now lives in the spine alias next to them --
`OrdValSpine = Spine<Rc<OrdValBatch<..>>>` -- and reaches the builder only as
the caller's `Into<Tr::Payload>` bound. The prefix that is still load-bearing is
the layout the alias pins, so it names that instead.
The public re-exports (`ValBuilder`, `KeyBuilder`) are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Trim batch/payload commentary; drop serde on Batch
The struct signatures already say which type holds the description and which
holds the updates, so the prose restating it was noise. Removed from `Batch`,
`Builder`, `SpinePayload`, `MergeState::begin_merge`, `ChunkBatch`,
`OrdValBatch`, `TraceWriter::seal`, `JoinTactic::prep`, and the builder aliases.
The one comment kept in `begin_merge` records something the code does not show:
that a surviving lone payload is not advanced by the compaction frontier.
`Batch` also loses its serde derives. Nothing serializes a whole batch, and a
whole batch can be very large; easy to restore if a consumer appears.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* Mergers report absence, rather than the spine deriving it
`Merger::done` returned a payload, and `MergeVariant::work` then asked
`merged.len() > 0` to decide whether the merge had cancelled to nothing. The
merger already knew: `OrdValMerger::done` and `OrdKeyMerger::done` build their
result from `self.staging.total()`, and `ChunkBatchMerger::done` from a chunk
sequence that is empty exactly when nothing survived.
`done` now returns `Option<P>`, matching `Builder::done`. The rule both follow:
where a producer can see that it built nothing, it says so, rather than leaving
a consumer to measure the result. That keeps `SpinePayload::len` to the spine's
accounting, which is what it is for.
The three chunk merger tests now assert that absence tracks emptiness across
their randomized merges, which is the contract the trait just acquired.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Trace payload accessor; drop redundant Batch accessors; ChunkBatch is a payload
Three unrelated tidyings that the batch-as-struct change left behind.
Nine call sites read `batches_through(f).unwrap().into_iter().filter_map(|b|
b.inner).collect()`: fetch the batches, discard the descriptions, keep the
payloads, because that is what a cursor is built from. `TraceReader` now has
`payloads_through` with that body as its default. `batches_through` remains for
the three drivers that want the descriptions too.
`Batch::desc` is public, and `description()` and `since()` only read it.
`description()` had one caller, which wanted `lower()`/`upper()` anyway, and
`since()` had two, both in `begin_merge`. Removed; `lower()` and `upper()` stay,
with about thirty callers between them.
`ChunkBatch` holds a payload, not a batch: an ordered chunk sequence that goes
inside a `Batch`. Renamed to `ChunkPayload`, with `ChunkPayloadCursor`,
`ChunkPayloadMerger`, and `ChunkPayloadBuilder`. It keeps its own type rather
than collapsing to `Vec<C>`, because `new` enforces that every chunk is
non-empty. Also fixes a stale `SpineBatch::Merger` doc link.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Spine asks payload length, not payload presence
`Batch::is_empty` answers whether a payload is present. The spine wants to know
whether a batch has updates, and the two agree only for producers that report
absence when they built nothing -- which the in-tree builders do, but which a
foreign payload type need not.
The spine can measure instead, since `SpinePayload` gives it `len`, so
`batches_through` and `consider_merges`'s empty-fusion shortcut now go through
`batch_len`. A producer that hands over an empty payload costs an extra merge
rather than behaving differently from the absent case.
One of these sites is the straddle check in `batches_through`, which permits an
empty batch to span the requested cut. Nothing in the tree can reach it with a
present payload: wide empty intervals come only from `Batch::empty`, which has
no payload, and every built payload spans one tick of its operator's frontier,
inside which a cut is the error the panic reports. This is insurance for payload
types we do not own, not a fix for a reachable fault.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Rename the batch payload type to Updates
`Tr::Payload: Navigable` says nothing at the site where it is written, and it is
written where it matters: the payload name appears about 189 times against the
`Batch` struct's 48. The seven cursor aliases already disagreed with it, being
named `BatchCursor`, `BatchKey`, `BatchVal` and so on while being defined
through `Tr::Payload`.
`Batch` does not move onto it. In `begin_merge` a surviving payload is handed a
new description spanning the composed interval, so the same `Rc<OrdValBatch>` is
reachable from batches with different bounds. It has no interval and no identity
in time, which is what this branch separated; naming it `Batch` would put that
back. `batches_through`, `map_batches`, `TraceReplayInstruction::Batch` and
`logging::BatchEvent` also all mean the struct.
So it is named for what it is. `TraceReader::Payload` becomes `Updates`,
`payloads_through` becomes `updates_through`, `SpinePayload` becomes
`SpineUpdates`, and `ChunkPayload` becomes `ChunkUpdates` with its cursor,
merger and builder. `Tr::Updates: Navigable` reads as "a trace's updates can be
navigated", and `Bu::Output: Into<Tr::Updates>` as what it does.
This collides in spirit with `columnar::updates::Updates`, and with whatever a
`Collection` ends up calling the updates it carries. That collision is between
associated types in different traits, which costs little, and both would be
naming the same idea.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* The struct is a Span; the batch is what it carries
`Tr::Updates` was better than `Tr::Payload`, but the word the use sites want is
`Batch`, and the struct was holding it. A span is an interval of a trace's
history: a description, and the batch of updates within it, absent when the
interval brought none.
pub struct Span<T, B> { pub desc: Description<T>, pub inner: Option<B> }
`TraceReader::Batch` returns, so `Tr::Batch: Navigable` reads as it always did,
and the seven cursor aliases (`BatchCursor`, `BatchKey`, `BatchVal`, ...) are
once again defined through the thing they are named for. `Builder::Output` is a
batch again, which is what builders have always built.
The earlier argument for keeping `Batch` on the struct -- that `begin_merge`
hands a surviving payload a new description, so it has no interval -- was a
migration-cost argument in semantic clothes. Under this split it is simply the
shape of things: two spans may reference one `Rc<OrdValBatch>`, because the
batch is the data and the span is where it sits.
The trace's methods follow the nouns. `spans_through` returns spans;
`batches_through` returns the batches within them, and so the method most
callers want inherits the familiar name. `map_batches` becomes `map_spans`,
`BatchOf` becomes `SpanOf`, `TraceReplayInstruction::Batch` becomes `::Span`,
and `batch_len` becomes `span_len`.
`Span::is_empty` is gone. It meant "carries no updates", but on a span a reader
hears "empty interval", which is never true -- `insert` asserts
`lower() != upper()`. It is `has_updates()`, with the sense flipped.
`SpineBatch` and `ChunkBatch` (with its cursor, merger and builder) return to
the names they had before this branch, which is 96 fewer changed lines against
`master-next` and four fewer files touched.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Stop claiming Navigable is how a batch is read
`Span<T, B>` puts no bound on `B`, and `TraceReader::Batch` asks only for
`'static + Clone`. Saying that reading the batch "is the `Navigable`
capability" asserts a coupling neither type has, and the repo contradicts it:
`ProxyJoinBackend<T, B0, B1>` and `ProxyReduceBackend<T, B1, B2>` take batch
types with no `Navigable` bound, because they do not read through cursors.
The sentence is gone from `Span`, where the concept does not reach at all. On
`TraceReader::Batch` and on `Trace` it is replaced by what is true: the trace
has no opinion on how a batch is read, and the operators that want cursors add
the bound themselves.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Call spans spans where they are spans
`enter_batch` took a `Span` and returned a `Span`; it is `enter_span`. Its doc
had also survived the rename as "the updates is wrapped".
The same slip sat in every place the rename passed over a binding rather than a
type: the `Span` parameters of `Spine::insert` and `TraceWriter::insert`, the
`map_spans` closures in `TraceAgent` and `TraceFrontier`, and the `spans_through`
mapping in both scope wrappers were all still called `batch`.
`BatchEnter` and `BatchFrontier` keep their names, since after the split they do
wrap a batch. Their docs said "updates", and now say batch.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* Undo type-parameter and prose churn from the Payload detour
`Spine`, `RcMerger`, `MergeState` and `MergeVariant` were all parameterised by
`B` before this branch. They became `P` while the batch type was briefly called
`Payload`, and stayed `P` after it stopped being called that. Back to `B`, and
`Merger` keeps the `Output` parameter name it had. That is 64 fewer changed
lines in `spine_fueled.rs`, with no change to what the code does.
Some doc prose had drifted the same way and is restored or repaired:
`JoinTactic::prep` joins lists of batches, not "batch updates";
`ChunkBatch::new` assembles a batch; `ProxyReduceBackend::finish` yields the
batch it described.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
`Arrange` existed to let two overlapping impls coexist on `Collection`: one for `(K, V)` collections arranged by key, and one for `K` collections arranged by the whole record. As inherent methods they would collide, since `K` instantiated at a pair covers the first impl's shape. The second impl was sugar for `.map(|k| (k, ()))` before arranging, and it is worth deleting for its own sake. Which of the two impls a call site selected was decided by the builder type's `Input`, three type arguments away from the choice, so the same turbofish could arrange by key or by record depending on a parameter that named neither. Three call sites in `graspan1` and `spines` were reaching the record-arranging impl while spelling `ValBatcher`/`ValBuilder`/`ValSpine`. They now say `.map(|k| (k, ()))` and keep the spine they had; six others move to `arrange_by_self`. With one impl left there is no overlap, so `arrange` and `arrange_named` become inherent methods on `Collection<(K, V)>` with unchanged signatures. Call sites are untouched: inherent resolution matches what the trait provided. Only the `use ...::Arrange;` imports go away. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
`dogsdogsdogs::half_join` declared its own `Batcher` trait, because
`trace::Batcher` did not fit a consumer that stages updates in a form of
its own and releases them by a total order rather than an antichain. Two
traits with the same name and the same job, one of them half the size.
This adopts the smaller one, and deletes the duplicate.
pub trait Batcher<T, C0, C1> {
fn insert(&mut self, container: &mut C0);
fn extract<'a>(&'a mut self, upper: AntichainRef<'_, T>)
-> (Option<C1>, AntichainRef<'a, T>);
}
Four consequences, in rough order of how much they matter.
Input and output types are now distinct. `trace::Batcher` pinned them to
one associated type, which is why `arrange_core` carried a chunker whose
container had to equal `Ba::Output`. A batcher may now release whatever
its consumer means by a batch: half_join's is a sequence of chunks, and
arrange's is a chain it still hands to a `Builder`, which a following
commit narrows to the batch itself.
Construction leaves the trait. It has to: a trait meant to cover both a
merge batcher and, later, a builder-shaped one cannot mandate a
`new(logger, operator_id)` constructor, since only the arrange path has
a differential logger or an operator id to give. `arrange_core` takes the
constructor as an argument and `MergeBatcher::new` is inherent. Call
sites move the batcher from turbofish to argument position and are
otherwise unchanged.
The `Description` leaves too. `MergeBatcher::lower` existed only to
manufacture one, duplicating `arrange_core`'s `prev_frontier`; the
operator now builds it from the frontiers it already tracks.
`seal` and `frontier` collapse into one call, so the ordering invariant
between them ("the frontier after the most recent seal") is gone. The
retained frontier is reported by the extraction that determines it.
`insert` takes its container by reference, as `Builder::push` does, and
the implementor decides whether to claim the allocation or drain it. The
recycling decision belongs there rather than at the call site, which no
longer writes `std::mem::take`; `arrange_core` drops a `Default` bound as
a result. This also keeps the door open for a builder-shaped batcher:
`reduce` allocates one buffer outside its per-key loop and hands it over
by reference every key, which a by-value `insert` would cost it.
One behaviour change worth naming: the empty-batch branch used to call
`seal` and discard the result, to advance the lower bound that no longer
exists. Dropping that call also drops the full chain merge it performed
on every empty frontier advance. No held capability precedes the input
frontier there, so no update does either, and nothing was extractable.
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
* The batcher seals its own batches
`arrange_core` used to take both a batcher and a builder, pull a chain of
chunks out of the batcher, and call `Bu::seal` on it to get a batch. The
batcher named the batch type in its output, but did not produce one.
Now `MergeBatcher` carries the builder and seals internally, so `extract`
returns a batch. `arrange_core` drops its `Bu` parameter: the bounds go from
Ba: Batcher<Tr::Time, Chu::Container, Vec<Bu::Input>>,
Bu: Builder<Time=Tr::Time, Output: Into<Tr::Batch>>,
to
Ba: Batcher<Tr::Time, Chu::Container, B>,
B: Into<Tr::Batch>,
Arrange no longer knows that builders exist; it asks for updates and gets
something it can turn into a batch. Call sites stop naming a builder, which
is the bulk of the diff.
The builder is `PhantomData` on `MergeBatcher` because `seal` is an
associated function. Sealing still happens at extraction with the whole
chain in hand, so `Builder::with_capacity` still pre-sizes from the full
key, value, and update counts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* `Batcher`'s time and output become associated types
The input a batcher accepts stays a parameter, because a batcher may
reasonably accept several shapes of container. What it produces, and the
timestamps it carves by, are facts about the batcher, so they become
associated types. Neither impl in the tree implements `Batcher` twice for one
type, so the parameters were buying no polymorphism.
-pub trait Batcher<T, C0, C1> {
+pub trait Batcher<C0> {
+ type Time;
+ type Output;
Callers that want to pin the output still can, as half_join does with
`Output = Vec<CMid>`. Callers that only need it convertible say so directly:
- Ba: Batcher<Tr::Time, Chu::Container, B> + 'static,
- B: Into<Tr::Batch>,
+ Ba: Batcher<Chu::Container, Time = Tr::Time, Output: Into<Tr::Batch>> + 'static,
That drops a type parameter from `arrange_core`, `Collection::arrange`,
`arrange_named`, and `consolidate_named`, and with it the anonymous `_` that
every call site had to write in its turbofish.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
* The batcher chunks its own input
`arrange_core` used to hold a chunker, push each incoming container into it,
drain the chunks it emitted into the batcher, and drain it again before
sealing so a partial final chunk would not be stranded. That last part needed
a comment explaining a seam the operator had no business knowing about.
`MergeBatcher` now holds the chunker, so `insert` takes the container from the
wire and `extract` flushes before merging. `arrange_core` drops its `Chu`
parameter and ingest becomes `batcher.insert(data)`.
-pub fn arrange_core<'scope, P, C, Chu, Ba, Tr>(
+pub fn arrange_core<'scope, P, C, Ba, Tr>(
- Chu: ContainerBuilder + for<'a> PushInto<&'a mut C> + 'static,
- Ba: Batcher<Chu::Container, Time = Tr::Time, Output: Into<Tr::Batch>> + 'static,
+ Ba: Batcher<C, Time = Tr::Time, Output: Into<Tr::Batch>> + 'static,
The chunker is not a separate axis from the batcher: it adapts whatever
container is on the wire to the merger's preferred representation, so it is
part of what selects a batcher rather than a tax on everyone downstream. The
columnar batcher pairs with `TrieChunker` and with `ContainerChunker<ColChunk>`
depending on its input, and those are now two batchers rather than one batcher
and two chunkers. Corgi keeps its column-native ingest by naming its chunker
where it names its batcher.
One consequence: `MergeBatcher` implements `Batcher<C>` for every `C` its
chunker accepts, and `extract` does not mention `C`, so a caller holding a
concrete batcher must say which implementation it means. Generic callers pin
`C` in their bounds and are unaffected; the three sites that needed a turbofish
are tests driving a batcher by hand.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…-of time (#1026) A program reads every import as of `(e - 1, [])` and runs at times at or beyond it, so every time a half join's arrangement presents is at or beyond the as-of time. An update at exactly that time meets only arranged rows at it, and a strict half join meets none. An update keeps its own time along a chain of half joins, so once every path from a lifted stream reaches a strict half join, its updates at the as-of time produce nothing. In a delta join that is every path but the one that is non-strict throughout, which today carries the imports' whole history only to discard it. Install now drops those updates where the stream is lifted, with a filter on the time `lift` writes, so any backend runs it with the operators it has. A lift that other readers share, as both paths of a self-join do, keeps its rows, and the filter goes on the edges to the readers it is dead for. Only time-preserving stages may sit between the lift and the strict half join. The rewrite applies in nested scopes too, where the as-of time has zero coordinates. The shapes are inferred again after it, and install falls back to the program as written if they disagree. On a late-installed three-way delta join over imports with eight epochs of history, install plus first tick took 696-713 ms against 783-795 ms with dense keys and 61 ms against 124 ms with sparse ones; steady ticks are unchanged. Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
#1010) * LDBC: an SNB Interactive v2 driver for DDIR, with DuckDB as a baseline interactive_v2/driver.py reimplements the official Java driver's workload from its source: per-operation updates on the simulation clock, complex reads on their frequency grid with the paramgen's parameters, the short-read walk, TCR, and the per-type 95% on-time audit. systems.py runs it against DDIR (group-commit ticks) and DuckDB (concurrent connections). snb/operations.py turns the BI CSV inserts and deletes into single operations with indexed cascades; factors.py rebuilds the paramgen's inputs so the official paramgen can run. README.md lists what is reproduced, where it departs, and first SF1 results. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC Interactive v2: per-tick log and server/process CPU time for attribution Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC: IC9 ranks per friend before per request; IC14 iterates costs, then recovers the path; driver --batch-ms IC9's final top 20 is keyed by request, so every candidate went to one worker; each slow-path friend now keeps their own 20 first (4 workers: 223 -> 116 ms answer plus retract). IC14 runs Bellman-Ford on costs alone, keeps every least-cost predecessor, and walks the smallest next node from the source over the nodes on a cheapest path (121 -> 74 ms). Answers equal the previous plans on 62 SF1 bindings and the fixture. The driver can hold DDIR ticks so they are fuller; the client passes DDIR_DIAGNOSTICS. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC Interactive v2: replay and per-read cost tools; README results and attribution Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC Interactive v2: one-off complex reads (--oneoff), from split plans oneoff/split.py splits IC1, IC9, IC10, IC12, IC13 and IC14 into a standing program (their binding-independent statements, published) and a one-off program that each read installs, answers once and drops at the next tick instead of retracting. Experimental: a one-off installed late reads published traces without #982's consistent-frontier import. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC Interactive v2: one-off splits for all 14 complex reads; replay --oneoff and --only Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC Interactive v2: delete cascades in the graph program; the driver streams its inputs graph.py's program takes inserted rows and explicit delete targets and derives the live relations by snb.updates.State's rules, ahead of the unchanged graph program. Over the SF1 stream they equal the BI daily journal (three checkpoints); 20,000 operations answer identically either way; test_cascade.py checks the rules in CI. stream.py reads the snapshot in chunks and the operations a day at a time, and complex reads are generated lazily: the driver peaks at 181 MB at SF1 (was 8 GB); the server at 7.5 GB (was 4.7). --cascade client keeps the old path; DuckDB uses it. Also --sf 0.1 (SF1's mix on SF0.1 data) and factors.py reading plain CSV partitions. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC Interactive v2: sort each relation once when feeding the client-side snapshot Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u * LDBC Interactive v2 README: one-offs read imports as of their tick; rejoin the results table Since #1024 and #1025 a one-off reads every import as of the tick it starts, so the README no longer calls the one-off runs experimental. The 0.0025 run, rerun on master-next 7eb7da8, passes again (1,049 ops/s); 0.002 still fails. The one-off row was split from the results table by a blank line. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…te (#1028) The as-of rewrite (#1026) re-checks shapes after it runs, which cannot show that results match. `Server::set_as_of_drop` (default on) turns the rewrite off for programs installed afterwards, and a new `asof` difftest oracle uses it: each case's delta form is installed after its inputs have history, once with the rewrite and once without, and every epoch's exports must agree. Over 2,000 seeds the rewrite fired 2,938 times in place and 1,264 times on shared-lift edges, 1,892 of them in nested scopes. When a lift is shared, the rewrite put filters on the dead readers' edges only if the lift was its node's last op. The filter reads the time `lift` wrote into the row, and `filter` and `negate` leave the row as it is, so the edge filters now also go after a node whose ops past the lift are only those. Other tails (`project`, `flat_map`, `lift_iter`) may move or drop the lifted time and are left alone; handling them would need tracking the time through lambdas, or duplicating the tail on the dead edges at a cost every tick. Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o Co-authored-by: GPT-6 Sol (Codex) <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The authored plan built every path a-b-c in both directions and kept a < b < c after the join. Orienting each friendship a < b first builds each wedge once. Standing, over the 40 official SF1 bindings at 4 workers: 4.6 -> 3.3 ms per binding, 3.5 -> 2.2 ms per release, identical triangle counts. The fixture check (authored/check.py --fixture bi11) agrees with the generated plan. Claude-Session: https://claude.ai/code/session_01GVgz9LXpYfNWPmu352Yt1u Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Parse `set name = expr` as a feedback binding in a named scope. Keep `var` for optional declarations and inferred shape checks. Reject the old binding syntax with a migration message. Update DDP programs, generators, tests, and docs. Check duplicate and missing bindings while lowering, without changing the scope IR. Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o Co-authored-by: GPT-6 Sol (Codex) <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Parse paths of any depth. Resolve each path against the scope tree before lowering, then carry its converged value through child exports and ordinary imports. Reject invalid paths, scope name shadowing, and scope cycles with clear scope context. Keep plain outer names and existing shorthand. Add path and SCC answer tests. Document the rule in the parser and the short README scope reference section. Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o Co-authored-by: GPT-6 Sol (Codex) <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Collect type declarations before parsing statements. Resolve payload types in dependency order and report cycles. Make all types available to shapes and lambdas regardless of declaration position. Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o Co-authored-by: GPT-6 Sol (Codex) <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Render each text result row beside the feed parser. Keep the prior column dump under `as columns` and leave binary frames unchanged. Update the server tests, transaction benchmark, and protocol guide. Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o Co-authored-by: GPT-6 Sol (Codex) <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The bench fed binary frames but read peeks as corgi's Debug text, parsed by regex. Read them with `peek … as corgi` and the LDBC client's frame decoder instead: the timed rollback read measures the server's answer rather than its text rendering, and the parser no longer depends on corgi's Debug format. Answers are checked against the serial reference as before; cycle times are unchanged. Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Update DDIR readme * Respond to comments * README: the access example as a program and session under examples/server The README splits the program across sections and shows the session in pieces. Add the whole program and session, load the program from its path, and say how to run the session from interactive/. The session's output contains every line the README shows, apart from tick timings. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * README: state the access rules as the program computes them Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The projection pass inserts a projection before each concat input and after each reduce output, but did not insert labels. Diagnostics use labels only when they line up with the items, so every program the pass rewrote reported its time under an empty statement name. Build the labels alongside the items. Each inserted projection takes the label of the statement it serves. A test checks the labels after a rewrite that inserts both kinds. Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o Co-authored-by: GPT-6 Astra (Codex) <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Lowering ordered a scope's lets and child scopes by repeatedly scanning for the first item with no unlowered dependencies, then removing that item from every other item's dependency set. That is quadratic in the number of statements: 20,000 lets took 2 s to lower, and a 170,000-statement program did not finish. Use Kahn's algorithm, always taking the earliest ready item in source order. That is the same order as before, so plans are unchanged, and lowering 170,000 statements takes about 0.3 s. Anything left unlowered at the end is on or behind a cycle, and reported as before. Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* DDIR: share visible scope names during lowering A program with many sibling scopes cloned the full visible-name map for each child. Insert each scope's children before visiting its body, then remove those names on return. Siblings still see the same names and shadowing checks keep their order. At 64k sibling scopes, lowering falls from 185.18s to 0.54s. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: move linear operations when fusing chains Each fusion cloned the entire accumulated prefix and retained that copy in a dead item. Take the single-consumer input's operation vector and append to it, preserving its capacity, operation order, surviving item, and label. At 8k maps in a child scope, optimization falls from 0.78s to 0.0011s. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: batch scope reference redirects Collapsing arrangements and removing duplicate operators each walked every reference once per removed item. Record the surviving item indices and apply them in one walk per pass. Arrangement chains resolve earlier redirects in dependency order; deduplication still compares the original nodes before applying its redirects, preserving rewrite rounds and item order. At 64k, duplicate-map optimization falls from 31.98s to 0.064s; redundant arrangements fall from 5.04s to 0.0011s. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: index feedback variable names while lowering Checking each set for duplicates and each variable declaration for a set scanned all previously collected sets. Keep a name set for these membership checks while retaining the original vector for variable and bind order. At 64k declared feedback variables, lowering falls from 5.59s to 0.090s. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Duplicate removal keyed operators on their printed text, so two maps that differed only in variable names never merged, and child scopes were never compared at all. Walk a scope's items once, in dependency order. Rewrite each item's inputs through the merges found so far, then look it up by structure: its kind, its inputs, and each lambda's stored graph rather than its source text. A duplicated chain merges at every level in one walk. Child scopes, already optimized, are keyed by their bodies, imports, feedback, exports and shapes, and readers of a duplicate scope's exports move to the survivor. Rewrite binds and exports once at the end. Remove duplicates before fusing linear operators, so fusion cannot copy a shared prefix into two readers and hide it. The survivor keeps its text, names and labels. Check alpha-renamed maps, a 128-level duplicate chain in one walk, and nested recursive scopes whose answers match the unmerged plan through insertions and retractions. Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o Co-authored-by: GPT-6 Astra (Codex) <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Walk each scope backward from its exports, feedback bindings and Inspect operators. Keep only child exports that live readers use, propagate surviving imports to their parent sources, and compact unread items and imports. Keep root positional inputs so clients can still feed every declared input number. Run liveness first in the existing optimizer loop so dead readers do not inflate fusion use counts. Optimize children before the loop and cache their Inspect flags; revisit a child only when pruning removes some of its exports. There is no additional fixed-point loop. Add two liveness tests for the dead recursive program and a nested Inspect, including child-export remapping and feeding an unread root input. Leave the deduplication tests and difftest harness unchanged. Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o Co-authored-by: GPT-6 Astra (Codex) <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* DDIR: print program preparation stages without installing Add a read-only report shared by ddir-plan and the server plan command. Show shapes, statement origins, rewrite changes and renderer choices. Index diagnostic import readers and cache observer names per scope walk while retaining the base’s batched redirects and moved linear operations. Observe liveness removals while retaining the new base's child revisit rules and root inputs. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: print compiled lambdas in plan reports Print typed corgi graphs with input patterns and shared let bindings. Check scalar spellings at their input shapes and retain source display. Pin corgi to cb67fe4d and use its public node view in every build. Bound scalar spelling depth and release expression text after its last reader. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: emit whole plans and preparation stages as DDP Keep source bindings and sum names in the DDP printer. Emit parsed, lowered, optimized, projected and as-of programs. Compare exact final plans and emitted text when checking a round trip. Index authored DDP names and fresh-name suffixes, schedule authored aliases at their definitions, and discard fused provenance names in one walk. Drop source names for pruned values and remap surviving names with liveness. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: review preparation pairs as DDP and stage diffs Keep the provider-aware corpus and its structural checks. Snapshot whole final DDP programs and successive stage diffs. Retain regeneration, byte stability and idempotence checks in the default build. Check every view without diagnostic text snapshots. Retain suffix counters when assigning sanitized export names. Review the liveness-pruned corpus and its smaller repeated-projection count for explanations. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: desugar reductions and compose value projections Lower min, distinct, count and collect to reduce expressions. Keep non-unit distinct maps in the IR, compose reduce value projections and remove duplicate concat masks. Spell composed reducers from their exact graphs, without a mechanical _call0 wrapper when that spelling is faithful. Spell unit distinct as ((), _) -> ((), 1). These are the reductions DD runs. Review the changes in the DDP corpus. Consume ordered distinct sites in one walk when expanding reductions. Keep liveness-aware compaction available to reduction cleanup and regenerate its plan pairs. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: compose linear maps and remove dead bindings Add named scope passes for map composition, identity maps, double negation, and unused imports and aliases. Compose checked graphs and require a faithful DDP spelling. Run cleanup again after projection exposes new opportunities. Keep export contracts and input declarations. Record only live provider dependencies at install. Refresh plan pairs and check map rewrites on generated batches and long map runs. Compose supported map runs and redirect empty chains in batches, and index the live traces used to filter unresolved imports. Keep liveness before structural rewrites in each canonicalization round. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: gather program rewriting in passes Move structural optimization, projection, and as-of filtering under passes. Share one observed preparation schedule between planning and installation. Keep source discovery and configuration in the callers. Preserve plan bytes and update optimizer callers without a Program forwarding method. Precompute observer ordinals, propagate feedback shapes through reader edges, and index as-of filters and coverage while moving the fixed structural algorithms. Move liveness into the structural pass with the other rewrites and keep reporting its removals. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: make every pass optional, with projection off by default Replace the projection flag and optional as-of time with passes::Config, one switch per pass. Config::default() is all except projection; all() and the explicit word all still enable every pass. Comma-separated options start from the default, including an empty list. The server reads DDIR_PASSES once; ddir-plan takes --passes. Both use Config::default() when unset. Projection currently costs 0.5–1 ms per install and is being rewritten. Install becomes the only place a loaded program is optimized. Projection's cleanup uses the shared schedule. Switched-off stages remain in reports, unchanged, and identify the disabled pass. Keep corpus tests on all passes. The difftest optimize oracle installs the unoptimized program with every pass off; passes_are_optional checks 24 generated cases. The structural switch also controls the liveness cleanup from the new base. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: fix optional-pass plan round trips Remove unread child exports and remap their readers. Preserve the distinction between declared and inferred variable shapes when printing. Settle projection, reduce-map fusion, and enabled cleanup with bounded, shape-checked iteration. Intern identical import edges and stabilize synthesized child export aliases. Add runtime subset oracles and regressions, and update the reviewed plan pairs. Exact subset round trips use structural cleanup until lowering reuses local arrangements; the full subset check and plan oracle follow with that change. Leave linear-node segmentation with structural off open. Index imports by source and shape, and schedule DDP aliases at their definitions with cached export names. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: reuse local arrangements during lowering Reuse local Arrange and Reduce inputs for joins, half joins and reductions. Apply the same check while expanding distinct. Whole-record distinct still needs an arrangement after its output map. Keep explicit arranges and structural cleanup. Refresh the lowered plan pairs and test arrangement reuse without passes. Enable exact round-trip regressions for every pass subset and add the plan difftest oracle from the round-trip fixes. These checks require lowering to reuse arrangements when structural cleanup is off. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: share arrangements across scope boundaries Share explicit arrangements of unchanged parent bindings across scopes, and remove arranges of parent values that already carry arrangements. Keep lone child arrangements in place. Both rules are independently switchable passes; lowering and the runtime retain their existing behavior. Regenerate the plan-pairs corpus: five physical arrangements and eight trace-clone statements disappear. Add scope-boundary and answer regressions. Batch arrangement replacements by child and use the shared observer ordinals. Use the common removal display for liveness and cross-scope arrangement rewrites. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: move loop-invariant statements beside their sources Move lone arrangements and pure maps, filters, flatmaps, negations, concats, joins and reductions through parent imports to their defining scope. Reuse ancestor arrangements. Variables, time operators, half joins and inspections stop motion; exports and feedback bindings keep their scope. Place moved statements after their latest source and its existing derivatives. Keep free child names, use source-and-operation names for clashes, and remove unread child aliases. Retain hoist-arranges as an alias for hoist-invariant. Keep focused checks for motion barriers, names, dependency order, optional passes and nested feedback with retractions. Refresh the plan-pairs corpus. Leave unary inlining, ArrNF, unit binders and unit-check compilation for the printing parcel. Batch invariant hoisting and name allocation, index placement positions, and use the shared observer ordinals. Find the retained liveness-test scope by name after invariant statements move beside its sources. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: report per-stage and per-pass planning times on request Return wall times, call counts, rewrite observations and fixed-point rounds from the preparation schedule. Include optimizer calls made by projection, and time shape checks separately from rewrites and diagnostic snapshots. Count bulk canonicalization and binding/import cleanup without extra scope walks. Add --timings to ddir-plan and to the server's plan command. With --emit-ddp, the table goes to stderr. Default reports, stage diffs, emitted DDP and plan-pairs snapshots are unchanged. Include liveness item-removal observations in the structural timing counts. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: combine liveness and dead bindings in the dead-code pass Compute liveness in one backward walk per scope, then remove dead operators, child scopes, child exports, imports, and redundant source aliases in a separate transform. Keep root inputs, inspections, and feedback bindings in retained scopes. The transform never re-optimizes children. Run dead-code before structural in each optimizer round so dead readers do not inflate fusion use counts. Replace dead-bindings in Config, DDIR_PASSES, --passes, timing reports, and documentation. It stays on by default; with it off, structural performs no liveness pruning. Adapt the existing cleanup tests and add one switch regression. Keep unread imports left by hoisting when dead-code is off. Emit unused parent edges as aliases so pass-subset plans retain those imports on a round trip, without a second cleanup hidden in invariant motion. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: move the Corgi pin to master Pin Corgi at 2751e04. That brings typed `compile`, `analyze_demand` and `narrow`, a linear parser, and canonical compiled graphs. - Projection asks `analyze_demand` for its input request, in place of the removed `demand_transform`, and keeps its decisions. - Spelling checks and the lambda printer work from compiled graphs. A readable spelling now compiles to the same graph it spells, so composed reducers that fell back to `_call0` print directly, and shared constants and units are written inline. - `head` is recognized whatever its zero counts rows from. - `ListImm` carries "not equal" as a flag. `Op::Project` is gone, and so are its printer and normalizer cases. Seven plan pairs change. Four top-k reducers and `custom` print from their graphs instead of `_call0` wrappers. `minimum_reduce` and the transaction plans' `best` apply their map before `enlist`. `collected_reduce` no longer builds a list it discards. A chain of 170,000 maps plans in 3.7 s, down from 103 s. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: expect Corgi's canonical NaN from a computed add The pin move brings Corgi #128, which gives every computed NaN the positive quiet NaN's bits. The float-words test filtered out NaN operands but not NaN results: `exp(a) + b` can be infinity minus infinity, which x86 hardware returns as a negative NaN. The test failed on Linux CI and passed on arm64, whose default NaN is positive. Compare against Corgi's canonical NaN. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: validate submitted programs before guarded preparation Fix review finding 1: check root feedback, input declarations, trace names, export collisions, and shapes before optional passes can discard bindings. Catch pass panics before any installation side effects. Preparation remains on each worker, where current provider shapes and as-of are available; this uses the review's catch_unwind alternative to moving it to the session. Keep unread clock imports: importing publishes the clock in the catalog, advances its trace on ticks, and owns its lifetime. Check unknown unread imports under both default passes and none, and clock ownership in the same test. Remove the old test's expectation that an absent import is accepted. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: batch invariant arrangement groups in one rewrite Fix review finding 2: gather every group before rewriting, route all groups through one replacement tree, and compact each affected scope once. Reuse the name index across groups. Preserve dependency order with linked item indices instead of repeatedly shifting the whole vector for each arrange. Read operator references directly, and move invariant operators instead of cloning their scalar graphs. The reviewed corpus and all eleven existing scale shapes at 2,000 retain identical DDP. The task's inputs/gen.py adds rootjoins; debug default-pass planning and emission grows approximately twofold per doubling through 64,000 scopes. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: compare emitted corpus plans and reject changed scalar graphs Fix review finding 3: check graph_changed, compare replanned final graphs, and check emitted text stability at every corpus stage. Name the 21 parsed distinct cases whose untyped expansion differs from the original typed fast path. All lowered and later views compare without that allowance. Recognize repeated as-of insertion with structural cleanup off too: printing splits the fused linear node, so normalize that grouping on both sides of the comparison with as-of insertion disabled. Require an original as-of rewrite before granting this allowance. Both as_of cases now pass all 32 configurations: 1856 same, 32 known as-of, 32 expected rejections. Keep the sweep ignored: its isolated debug run takes 18.41 seconds. Make --emit-ddp fail on graph_changed without requiring --check-lambdas. The reviewed corpus remains byte-identical. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: plan at the server's as-of frontier Fix review finding 4: default standalone planning to as-of zero, add --as-of T, and use that same frontier when checking a round trip. Expose explicit-frontier report functions and use the server's current import as-of for its plan command. Report and emitted comments describe the actual frontier instead of a placeholder installation epoch. Exercise CLI emission on a fresh server: a strict half join must retain its real epoch-1 update. Also check the explicit --as-of 7 spelling. The corpus changes only in import-frontier comments and the two as-of examples' filter constants (1 to 0): 96 expected files, listed in the task's series/04-corpus-files.txt. No other emitted bytes change. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: correct projection guidance and guard server plan reports Fix review finding 5: document projection as off by default and warn once at startup when DDIR_PROJECTION is set, naming DDIR_PASSES=projection. Share source-shape lookup between install and the server plan command, including diagnostics sources. Catch print_with panics before streaming any report lines, and count the report's stages in the terminal reply. A two-worker protocol test plans an unpublished diagnostics source, checks the deprecated-variable warning and unchanged default projection setting, and checks the reply's stage count against the actual report. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: batch statement hoisting in one rewrite Fix the second review finding on hoisting: statement hoisting rebuilt the parent's name map, shifted its items, rewrote every reference and reordered it once for every child it moved something out of, which is quadratic in the number of child scopes. Collect the moves from every child first, build the name map once, and insert, renumber and reorder the parent once. Linked indices keep the previous placement, and unnamed bindings use the existing name index rather than rescanning it. Plans are unchanged: the corpus and every scale shape at 2,000 emit the same DDP. N child scopes that each move a join statement out (debug, default passes, planning plus emission) take 0.22, 0.88, 3.6 and 14.9 s at 1k, 4k, 16k and 64k scopes, 2.0x per doubling; the review measured 31 s at 4k. One test covers sibling name collisions and dependent moves. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: gather only what a reduce lambda reads When a reduce's backend is built, narrow its compiled lambda to its own demand: `corgi::narrow` runs `corgi::analyze_demand` over the lambda's input `(key, List<(val, diff)>)` and rebuilds the graph to take only what it reads. The runtime then gathers a key only if the lambda reads it, projects the value columns by the values' demand (`Demand::project`) before gathering them, and projects the argument to the narrowed graph's input. Every list keeps its length and order: a value the lambda does not read is still an entry. So `count` gathers no keys or values, only diffs; `min` and `distinct` gather values but no keys; and a lambda that reads one field of each value gathers that field. The analysis is per column: it cannot skip rows, which is what `min` would need (see the next commit). Tests: `ordered_reduce` checks lambdas that read nothing but list lengths, only diffs, a field of each value, and a key field with a field of each value, against their definitions over retractions and negative nets. `windows_partition_a_retire` inspects the presented values, so it now reduces by `distinct`, which reads them whole, where `count` reads none. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: give reduce a limit, and lower min to limit 1 A reduce may now carry a limit: `reduce(limit k, (key, vals) -> ...)` hands its lambda each key's first k values of non-zero net, in value order, whatever their nets. A value with many copies is one value, and a negative net counts like any other. The limit is part of what the operator means (`Node::Reduce { limit }`), so it is printed and parsed, and plans round-trip: the DDP emitter, the parsed view and the plan report write `limit k, ` before the lambda. `min` lowers to `reduce(limit 1, ...)` with the same lambda. At run time each key's consolidated values are cut to their first k before anything is gathered, so a key gathers at most k rows. Explain: a limited reduce depends only on the values it hands its lambda. The witness and forward clones compute those values beside each limited reduce (a limited `distinct` of its input, a site of its own), and the reverse rule demands, at each demanded key, the input rows holding every value presented at the demand's chain or before it, up to that chain. With those rows every round's first values are as in the reference, whatever the lambda does with them. This restores `min`'s narrowing, lost when it became a plain reduce: on the demand report in tests/explain.rs (SCC, `min` in loops) the explanations hold 160 and 815 rows, against 182 and 853 at the PR tip and 172 and 827 when `min` compared values with its output. Plan pairs: every change is a `min` gaining `limit 1, ` (42 lines in the final and parsed DDP of 15 cases, and their stage diffs), including `min`s with a projection composed into them. Tests: `ordered_reduce` checks `reduce(limit 2, ...)` and `min` against their definitions through negative nets, a value with three copies, values sharing their first field, a value cancelled within a batch, and retractions that hand the lambda a different first value. The colliding-key reduce test runs each sugar with its limit. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Co-authored-by: GPT-6 Astra (Codex) <noreply@openai.com>
`PointStamp::join` builds a new `SmallVec` and trims trailing zeros on every call. DDIR's join called it once per staged row (advancing to the unit's lower bound) and once per match: on the LDBC replay, about 770M calls and 8% of CPU, although at the inner depths DDIR mostly runs (0 and 1) nothing spills to the heap. The representation is unchanged; three changes build fewer times: - Staging builds each row's advanced time in one step from the lanes, `ColTimes::get_joined(row, lower)`, instead of building the stored time and then joining it. - The proxy join's matches use `join_assign` in place. - `PointStamp::join` takes fast paths for empty, single-coordinate and equal-length stamps, and `Product::join` is inlined so callers see them. Server CPU, three alternating trials against the same commit, identical answers: LDBC replay -5.6%, SCC initial -5.5%, reachability initial -10.2%; churn unchanged. Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o Co-authored-by: GPT-6 Astra (Codex) <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* DDIR: gather only the payload a join projection reads For each block of matches, a join's `cross` gathered the whole key and both values, then ran the projection, which often reads only a few fields. Narrow the projection with Corgi's `narrow`, as the reduce already does, and keep what it reads of each input: - an input it never reads is not gathered; - an input it reads in part is projected before gathering, including the columns rebuilt from integer tokens. Projecting shares the column's leaves, so it costs the shape's size, not the rows'; - an input it reads whole is borrowed, as before. The narrowed graph runs on the compacted argument. Collision checks still compare the original keys; output rows, times and multiplicities are unchanged. One test compares narrowed and unnarrowed joins, including keys that collide and differ only in a field the projection doesn't read. LDBC replay: -2.0% server CPU (three alternating trials, identical answers). The gain is in reads that join wide rows and keep a few fields. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: gather only the payload a half join projection reads The half join's backend gathered its projection's three inputs whole (the key, the update's value and the arranged value) before projecting each matched pair. It now compiles the projection once, narrows it to the fields it reads, and gathers only those, passing a unit column for an input it does not read. This is the regular join's narrowing, reusing its `Reads` and `demanded`. Times and multiplicities are unchanged. Implemented by a Claude Fable 5.1 agent; measured and reviewed by Claude. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o --------- Co-authored-by: GPT-6 Astra (Codex) <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* DD: PointStamp takes its inline capacity as a const parameter `PointStamp<T, const N: usize = 1>` stores up to `N` coordinates inline, in a `SmallVec<[T; N]>`. With the default, its type, layout and API are unchanged for existing users. smallvec's `const_generics` feature is now declared, and the workspace requires columnar 0.13.3, whose derive accepts defaulted generic parameters. Implemented by GPT-6 Astra (Codex); reviewed by Claude. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DD: enable smallvec's union layout With smallvec's `union` feature, a `SmallVec` records whether it has spilled in its capacity field instead of an enum tag, which saves a word: `SmallVec<[u64; 2]>` is 24 bytes instead of 32, and `SmallVec<[u64; 3]>` 32 instead of 40. Behavior and API are unchanged. Cargo unifies the feature across a build, so every smallvec user in it gets this layout; Materialize already enables it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o --------- Co-authored-by: GPT-6 Astra (Codex) <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…1053) `less_equal`, `join`, `join_assign`, `meet` and `meet_assign` each walk the two coordinate slices once. They no longer index coordinate by coordinate, clone a time to then join into it, or trim a result that is already minimal. `less_equal` compares lengths first: a minimal sequence longer than another is never less than or equal to it. `join` keeps its empty-input fast path and `#[inline(always)]`. That length check relies on every `PointStamp` being minimal, so deserialization now goes through `new`, as construction already does. Implemented by a Claude Fable 5.1 agent; reviewed by Claude. Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
DDIR's time was `Product<u64, PointStamp<u64>>` under a `u64` root scope, because timely's `new_input` needs a totally ordered time. It is now `PointStamp<u64, 2>`, with the epoch as coordinate 0 and loop depth `j` at coordinate `j`, and the program and control dataflows are rooted at it. Inputs are unordered inputs whose capabilities the server holds, downgrades at each tick and releases on drop; the client API keeps `u64` epochs. Collections enter a scope unchanged and leave through a generic `retime` operator that keeps the parent's coordinates. Capacity 2 holds the epoch and one loop coordinate inline, as `Product` did, in the same 32 bytes, so this changes the root and the shape of the time and nothing else. Implemented by GPT-6 Astra (Codex); reviewed by Claude. Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o Co-authored-by: GPT-6 Astra (Codex) <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* DDIR: retire enter_at; its uses become delay `enter_at(f)` raised a loop's time coordinate to `256 * bit_length(f(k, v))`, with the scaling hidden in the renderer, and at the root it wrote a loop coordinate without being in a loop. `delay` already says what it means: a row's time joined with the time a lambda gives. Inside SCC's two-deep loops, `enter_at(f)` is `delay((k, v) -> (0, 0, (256, f(k, v) call bit_length) mul))`; `0` is the bottom of every coordinate, so it leaves that coordinate alone. `bit_length` is a registered host word in DDIR's base scope, computing what the renderer computed: `64 - (n as u64).leading_zeros()`. `explain` now handles `join_function`, where it panicked before. `enter_at` is gone from the parser, IR, passes, printers, renderer and docs. SCC's answers are unchanged. Implemented by GPT-6 Astra (Codex); reviewed by Claude. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: review nits for retiring enter_at Rewrap two comments, keep both statements about which time coordinates leave a row's time unchanged, and name the explanation loop's forward clone. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o --------- Co-authored-by: GPT-6 Astra (Codex) <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* DDIR: programs with independent root time widths A program declares its root width `k` at install (`install_with_root`, default 1): its times are `k` root coordinates, then one per loop. The width is carried through shape inference, preparation and rendering, and recorded on exports and published traces. Importing a trace checks widths. A narrower trace is shared unchanged, since a trimmed `PointStamp` at the root already reads its missing coordinates as zero; narrowing is refused. Each input has its own capability, advanced on its own (`advance_input`); each export has its own probe; point reads wait only on the trace they read; compaction is per trace. A program advance is sugar over its inputs' advances and its outputs' waits, against the antichain of its inputs' points. The `u64` epoch API, `tick` and the wire protocol are unchanged for width-1 programs. Implemented by GPT-6 Astra (Codex), from a bitemporal spike by a Claude Fable 5.1 agent; reviewed by Claude. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: dedent bitemporal.ddp Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: tri-temporal SCC against an oracle, run in the release CI job An ignored test: SCC as a width-3 program, two nested loops with five-coordinate delays, a width-1 allow list widened in and joined inside both loops, and a count in the outer loop. Edges are fed and corrected at nearby points, inputs advance independently, and the export is compacted partway. Every closed point of the grid is checked against a Kosaraju oracle, at one, two and four workers. It takes about a second in release (about thirteen in debug), so the release CI job runs it with the bitemporal oracle tests. Implemented by GPT-6 Astra (Codex); reviewed by Claude. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: multitemporal review: point reads refuse, Level throughout Programs wider than one coordinate must name a point to read: `snapshot` and `peek` without one return an error pointing at `snapshot_as_of` and `peek_as_of`. Every read refuses a point that the program's input and import frontiers haven't closed, before gathering anything, so a read never waits for progress that only another command could make. `advance_to` still refuses to move an input that is already ahead of the target, and now says which input, where it is, and what was asked. `Level` is carried through shape inference and the optimizer's helpers. `ServerInput::new` takes the width, `run` and `infer` take the root width instead of having `_with_root` twins, and the program clock and export fields are documented. Implemented by GPT-6 Astra (Codex), from a review; reviewed by Claude. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o --------- Co-authored-by: GPT-6 Astra (Codex) <noreply@openai.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
…#1058) * DDIR: project join values before assigning tokens Consolidate values that agree on the fields the join reads before matching. Keep the existing merge for whole values and leading leaf projections; sort other projections within each real key. Unread values share one token. Collision checks retain the original keys. Adapt the half join's staging API without changing its demand yet. Implemented by GPT-6 Astra (Codex); reviewed by GPT-6.1 Sol (Codex) and Claude. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: project both half-join inputs before matching Use the join's projected-value staging for the arranged side. Consolidate released stream rows on their projected values within each block, preserving real keys and both update and data times. Original coordinates still serve output gathering and collision checks. Keep the whole-read stream path. Implemented by GPT-6 Astra (Codex); reviewed by GPT-6.1 Sol (Codex) and Claude. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: project reduce identifiers when weighted sums permit it Port the compiled-graph analysis for a diff-weighted map and total sum, requiring empty-group suppression independent of the key. Use projected columns in input presentation only with this fact and without a value limit. Keep output identity and raw novel time seeds. Other reducers retain whole values: in particular, signed projected extrema can change when values cancel. Implemented by GPT-6 Astra (Codex); reviewed by GPT-6.1 Sol (Codex) and Claude. Co-Authored-By: GPT-6 Astra (Codex) <noreply@openai.com> Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: review fixes for first-phase projection - The reduce analysis evaluates the empty case directly. Its argument has the compiled input's shape and corgi's ops are total, so the evaluation cannot fail on data; a panic would be a corgi bug, and is no longer caught. - Half-join needles come from the consolidated stream, so identifiers whose rows all cancel are neither probed nor staged. The collision walk over the block's rows skips rows whose identifier is not a needle. - The join's ungrouped path reuses its row-index scratch across keys. - The join module's token documentation is rewrapped. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o --------- Co-authored-by: GPT-6 Astra (Codex) <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…e by ranges (#1064) * DDIR: sort arrange ingest by (key, val, time) at once, consolidate by run labels The chunker's flush concatenated its input blocks through a per-row block number and offset, sorted by (key, val), and then walked every (key, val) run: it copied the run, sorted it by time with a Rust comparison sort, and compared neighbouring times to consolidate. Two changes remove those passes. - Concatenation copies each block a range at a time. Each block goes to corgi's `gather_lanes` as a one-row list holding the whole block, so corgi copies ranges at every level it can (leaves, products, lists, references into one arena) and falls back to row positions only for sums, as before. No per-row index is built. - The time is part of the one corgi sort. The time lanes become trailing integer fields: lanes constant within the flush are dropped, the rest are taken as offsets from their least value and packed into one u64 wherever their spreads fit (in practice always), each word offset by 2^63 so that corgi's signed order is the lanes' unsigned order. The sort's run labels then number (key, val, time) classes, and consolidation is one pass that sums diffs within runs of equal labels, with no comparisons, into vectors sized from the number of runs. Output is identical: the sort is stable, so each class's representative row and the order its diffs are summed in are unchanged. On captured flush inputs the stage goes from 21.5 to 14.5 ns per input row on SCC and from 39.8 to 34.3 on LDBC. End to end, one worker, three alternating trials: SCC initial computation -13.1% CPU, reachability initial -14.2%, churn ticks unchanged, LDBC SF1 one-off replay -4.0%; answers identical. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o * DDIR: concatenate chunks for advance and settle a range at a time `CorgiChunk::concat`, which `advance` and `settle` use to join a run of chunks, built a block number and an offset for every row and copied the key and value columns row by row through them. It now uses `concat_blocks`, which the ingest stage already uses: each chunk's columns are copied as one range. End to end, one worker, three alternating trials: instructions fall in every phase (SCC -1.6%, reachability churn -2.0%, LDBC -0.3%); CPU SCC churn -1.1%, reachability churn -1.6%, LDBC -0.3%; answers identical. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o --------- Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`slot_graph` compiles each slot at the shapes it reads, and the join, half-join and reduce backends then compiled its graph again at the same input before narrowing it. `compile_projection` and `compile_reduce` now return the `corgi::Compiled` itself, and those backends narrow it directly; the map, filter, flatmap and join-function paths take its graph as before. Compiling again at the same input changed nothing: a temporary check over the interactive test suite compared the two graphs at all 53,801 slot compiles and found them equal. Claude-Session: https://claude.ai/code/session_01H6wB4H99TLw4tqu4x48pXL Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The blanket `Hashable` implementation fed `T::hash` into FNV, which mixes one byte at a time with a multiply per byte, and fails the avalanche test at every key size (for some input bit, some output bit flips with probability 0 or 1). It now uses foldhash 0.2.0's quality hasher with a fixed seed, so every worker and process routes identically. The trait is unchanged. The new documentation says generic hash values are not a persistent format. `hashed()`: u64 1.37 -> 0.56 ns, 64-byte strings 30.9 -> 3.0 ns, 256-byte strings 193 -> 6.7 ns. Arranging 10M 64-byte string keys at 4 workers takes 8% less wall time and CPU; integer keys and one worker are unchanged. Hash-derived values such as `identifiers()` change once. Implemented and measured by GPT-6.1 Sol (Codex). Claude-Session: https://claude.ai/code/session_019XsakwFGwtESwZ1gSYwG4o Co-authored-by: GPT-6 Sol (Codex) <noreply@openai.com> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
* Extract Extract supertrait * Update test
Point the workspace at timely 0.32 on crates.io in place of the git revision. Relax the foldhash pin from `=0.2.0` to `0.2`. Fix intra-doc links that failed under `-D warnings`. Most were in the experimental columnar and int_proxy modules, where links named private items. Two links named paths that do not resolve. Drop the scc example's references to `interactive/` and `diagnostics/`, which do not ship with this crate. Co-authored-by: Claude Haiku 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merges
master-nextintomasteras a non-squash merge, so the 186 commits land individually. This is the second of two PRs; the first, #1075, is merged intomaster-next.On merge,
release-plzopens the release PR fordifferential-dataflowanddifferential-dogs3, both publishable.What ships in the DD crate
Chunk,JoinTactic,ReduceTactic,int_proxy. Thecolumnarmodule dates from 0.24.0 and is kept.formal/(Lean models) is not part of the crate.What does not ship
interactive/andddir/workspace members arepublish = false.ddir/was a leftover directory and is not on this branch.server/,tpchlike/,advent_of_code_2017/, anddoop/are removed, about 14k lines. They remain in git history.Known open items, not blocking
differential-dataflow/src. One is an open question aboutspans_throughintrace/implementations/spine_fueled.rs(line 244), where an incomparableuppercan silently drop a non-empty batch. Tracked for after the release.differential-dogs3releases alongside DD under the same version group.Verified on
master-nextat #1075:cargo test --workspacepasses against timely 0.32,cargo docpasses with warnings denied, andcargo publish --dry-runsucceeds fordifferential-dataflow.🤖 Generated with Claude Code