Skip to content

DDIR: compile join, half-join and reduce lambdas once - #1070

Merged
frankmcsherry merged 1 commit into
TimelyDataflow:master-nextfrom
frankmcsherry:ddir-compile-once
Nov 3, 2026
Merged

frankmcsherry merged 1 commit into
TimelyDataflow:master-nextfrom
frankmcsherry:ddir-compile-once

Conversation

@frankmcsherry

Copy link
Copy Markdown
Member

The join, half-join and reduce backends no longer compile their lambda a second time at the same input before narrowing it.
On the LDBC SF1 one-off replay this takes 70 µs off each IC read's install (757 to 687 µs), with identical answers.

Why

slot_graph compiles every slot at the shapes it reads.
The three backends then ran corgi::compile again on that graph, at the same input, before corgi::narrow.
A profile of one-off installs put the second compile plus narrowing at 140 of IS2's 462 µs install, and 177 of the average IC read's 757.

What changes

  • slot_graph returns the corgi::Compiled it makes.
  • compile_projection and compile_reduce return that Compiled, so the join, half-join and reduce backends narrow it directly.
  • The map, filter, flatmap and join-function paths take its graph, as before.
  • Tests that compared or built from the returned graphs now take .graph().

Compiling again at the same input changed nothing.
A temporary check, not committed, compiled every slot a second time during the interactive test suite and compared the graphs: all 53,801 were equal.

Measurements

One run, mini-02, one worker, diagnostics on, against a stored run of a6b7922a with the same uncommitted install timers on both sides.

before after
install per IC read (weighted) 756.7 µs 686.8 µs
render phase, IC1 / IC8 967.6 / 259.7 µs 797.9 / 212.9 µs
server CPU outside operators 14.3 ms/tick 13.8 ms/tick
whole run, instructions 2,130.0 G 2,121.7 G
whole run, server CPU 123.29 s 121.80 s

Answer and physical digests are identical on all 808 ticks.
Per IC family the saving ranges from 25 µs (IC13) to 171 µs (IC1).

Tests

cargo test -p interactive: 294 passed, 0 failed, 9 ignored; no warnings.

Implemented and measured by Claude Code.

🤖 Generated with Claude Code

https://claude.ai/code/session_01H6wB4H99TLw4tqu4x48pXL

`slot_graph` compiles each slot at the shapes it reads, and the join, half-join and reduce backends then compiled its graph again at the same input before narrowing it.
`compile_projection` and `compile_reduce` now return the `corgi::Compiled` itself, and those backends narrow it directly; the map, filter, flatmap and join-function paths take its graph as before.
Compiling again at the same input changed nothing: a temporary check over the interactive test suite compared the two graphs at all 53,801 slot compiles and found them equal.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01H6wB4H99TLw4tqu4x48pXL
@frankmcsherry
frankmcsherry merged commit 13bfe2b into TimelyDataflow:master-next Nov 3, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant