refactor: stage boundaries milestone 1 — Stages 0–2 and Stage 3's frozen interface (#1047, #1054) by zzylol · Pull Request #1108 · ProjectASAP/ASAPPlanner

zzylol · 2026-10-30T16:53:42Z

Milestone 1 of the #1047 / #1054 refactor: Stages 0–2 are restructured behind stage boundary types, and Stage 3's select_plan interface is frozen for the #1095 rewrite. Closes #1089.

Why

The planner's stages called into each other, and Stage 3 re-sized sketches, carried Stage 2's execution-scope copies and drove the pane-width rounds itself. #1047 and #1054 give each stage one input type and one output type, and let the facade orchestrate them. #1095 can then replace Stage 3 behind a fixed interface. Every step was reviewed and merged into the integration branch refactor/stage-boundaries. This PR merges that branch into main after the merge gates below.

What

Stage 0 / Stage 1. The boundary types live in asap_types::stages. Pass 1 has one entry, walks memo groups, and decides every sketch size (X2). Pass 1 prunes summaries that no reading query can use for its accuracy target. Stage 1's public API is narrowed to what other crates use.
Stage 2. Stage 2 emits no execution-scope copies and records pruned_candidates. Each item has one public path, and the reference pricer sits behind a feature.
Stage 3. The facade runs the stages and the pane-width rounds. Stage 3 takes only a deadline through the frozen select_plan interface and returns DagOfSelectedCandidates; it only checks sizes. The experimental [Feature] Stage 3: no execution-scope copies, exact candidate-level MILP, one cost formula #1095 redesign is removed (refactor(plan-selection): remove the #1095 Stage 3 redesign; the old engine is the only engine #1133); the old engine is the only engine.
Cost. Stage 3's cost formulas live in asap-cost. A materialized consumer's working-memory copy is held for its longest reader.
Renames. budget_share becomes accuracy_budget_parts, CandidateMemo becomes MemoCandidates, and --plan-selection-settings becomes --cost-inputs. "Producer" replaces "supplier".
Tests and docs. Tests are updated by area. The sketch plan snapshots are new. CI checks the crate graph. A final docs pass leaves no cargo doc warnings.

Included PRs

git log origin/main..origin/refactor/stage-boundaries, oldest first:

PR	Title
#1098	refactor: decide every sketch size in Pass 1; Stage 3 only checks (#1054 X2)
#1099	refactor: narrow types test-only fields and frontend visibility (#1054 step 8b, part 2)
#1100	refactor: say producer, not supplier, across crates (#1095 §5)
#1101	test: snapshot the selected plans of sketch-selecting workloads
#1102	refactor: move Stage 3's cost formulas into asap-cost (#1095 C2)
#1103	feat(plan-selection): fold Stage 2's execution-scope copies into classes (#1095 Steps 1-2)
#1104	refactor: Stage 1 and Stage 2 record pruned_candidates (#1047 P6)
#1105	refactor: delete the legacy whole-workload Pass 2 path and early query pruning (#1047 P6b)
#1106	refactor: Pass 1 walks memo groups; shared aggregates keep their rows (#1047 P3c, fixes #1089)
#1107	refactor: facade runs the stages; Stage 3's frozen select_plan interface (#1047 P7)
#1109	test: area A tests pass on the stage-boundaries branch (types → physical, executor)
#1110	test: area B tests pass on the stage-boundaries branch; fix decoded-output validation (plan-selection, planner)
#1111	test: area C tests on the stage-boundaries branch; reference enumeration applies Stage 2's raw-data rule
#1113	fix: quantile operand count survives memo sharing; Pass 1 deadline reports the planning budget
#1114	fix(frontend-metricsql): lower_metricsql and lower_metricsql_query stay public API
#1115	feat: inputs of Stage 3's class MILP: dominance, class data, owners, accuracy weights (#1095 C3)
#1116	feat: Pass 1 prunes summaries no reading query can use for its accuracy target
#1118	test(integration): plan summaries with no meeting guarantee by ignoring guarantees
#1119	refactor: Stage 2 emits no scope copies; the selection is DagOfSelectedCandidates (#1095)
#1125	feat: the exact candidate-level selection MILP with warm start (#1095 C4)
#1127	feat: decode, realize and check the selection MILP's plan, behind ASAP_STAGE3_SEARCH=milp (#1095 C5)
#1128	perf(plan-selection): share in the warm start and tighten the selection MILP (#1095)
#1129	fix(stage3): A1 accuracy rows match satisfies; Pass 1 prunes unusable summaries; parts record the sized shares (#1095)
#1130	fix(plan-selection): the selection MILP's plans are acyclic (#1095)
#1131	refactor(accuracy): sized_for_failure_probability is private (#1054)
#1132	test(integration): CI checks the workspace crate graph (#1054)
#1133	refactor(plan-selection): remove the #1095 Stage 3 redesign; the old engine is the only engine
#1134	fix(cost): hold a materialized consumer's working-memory copy for its longest reader (#1095)
#1135	refactor(physical): one public path per Stage 2 item; reference pricer behind a feature (#1054)
#1136	refactor(cost): narrow asap-cost's public API to what other crates use
#1137	refactor: #1047 remaining renames (budget_share → accuracy_budget_parts, CandidateMemo → MemoCandidates)
#1138	fix(memo): an unwidened scan no longer becomes a widened reader of its own group
#1139	refactor(logical): one Pass 1 entry; the target list is Pass 1's working form (#1047 P6)
#1140	refactor(planner): copy memos only when inspecting (#1047)
#1141	refactor: rename --plan-selection-settings to --cost-inputs and PlanningModels.plan_selection to cost_inputs (#1054)
#1143	test: follow-ups to the memo self-loop fix (#1138)
#1145	refactor(logical): narrow Stage 1's public API to what other crates use (#1054)
#1146	refactor: Stage 0 and Pass 1 boundary types in asap_types::stages (#1054)
#1147	refactor(devtools, dag-viewer): name reference_pricing keys as Stage 3's Selection (#1047 P8)
#1148	docs: final docs pass for the Stages 0–2 refactor; fix every cargo doc warning (#1047, #1054)
#1150	test(devtools): snapshot harness reads Stage 3's Selection; accept the facade's slower planning

How

Each step moved one boundary at a time. Stage 1 hands Stage 2 its candidates through the asap_types::stages types, Stage 2 hands Stage 3 the physical candidates, and no stage calls another stage: the facade (asap-planner) calls them in order. The constituent PRs describe each step in full.

Key code interfaces added / modified

The constituent PRs list each interface change with permalinks. The ones that callers outside the planner see:

asap_types::stages: the Stage 0, Pass 1, Stage 2 and Stage 3 boundary types (refactor: Stage 0 and Pass 1 boundary types in asap_types::stages (#1054) #1146, refactor: Stage 2 emits no scope copies; the selection is DagOfSelectedCandidates (#1095) #1119).
Stage 3: select_plan, which takes the candidates and a deadline and returns Selection / DagOfSelectedCandidates (refactor: facade runs the stages; Stage 3's frozen select_plan interface (#1047 P7) #1107, refactor: Stage 2 emits no scope copies; the selection is DagOfSelectedCandidates (#1095) #1119, refactor(devtools, dag-viewer): name reference_pricing keys as Stage 3's Selection (#1047 P8) #1147).
CLI: stage_pipeline --cost-inputs (was --plan-selection-settings), and PlanningModels.cost_inputs (was plan_selection) (refactor: rename --plan-selection-settings to --cost-inputs and PlanningModels.plan_selection to cost_inputs (#1054) #1141).
stage_pipeline JSON: Stage 3's output shows the selected plan as Stage 2 candidate P1, with cost, accepted_accuracy_per_query and search_report. planning_report shows the facade's rounds (refactor: Stage 2 emits no scope copies; the selection is DagOfSelectedCandidates (#1095) #1119, refactor(devtools, dag-viewer): name reference_pricing keys as Stage 3's Selection (#1047 P8) #1147).

Major algorithm changes

Sketch sizing (refactor: decide every sketch size in Pass 1; Stage 3 only checks (#1054 X2) #1098). Pass 1 sizes every sketch for its accuracy target; Stage 3 only checks the sizes. See refactor: decide every sketch size in Pass 1; Stage 3 only checks (#1054 X2) #1098 and the X2 sizing note below.
Pane-width rounds (refactor: facade runs the stages; Stage 3's frozen select_plan interface (#1047 P7) #1107). The facade runs every pane-width round up to Complete under the deadline, and the per-query seed (seed_separate_plans) is gone. The plans are unchanged on the snapshot workloads, but planning takes longer (see the waived planning-time threshold) and some Stage 3 searches end worse (see the accepted losses).
Memo self-loop (fix(memo): an unwidened scan no longer becomes a widened reader of its own group #1138) and longest-reader working memory (fix(cost): hold a materialized consumer's working-memory copy for its longest reader (#1095) #1134). See those PRs.

Merge gates

The gates compared main (b822553, same planner as the merge base) with the integration branch (7e268d4, before #1148, which is docs only).

Plan diffs (22 snapshot workloads: 12 corpus queries, 10 sketch workloads)

No plan or cost changes in Stages 0–2. All 22 selected plans have the same cost; fuzz-903 differs by +1.8e-10 relative. Rows whose plan display, cost or status changed:

Workload	Main cost/s	Integration cost/s	Change	Cause
sketch fuzz-issue903-raw-retained	1.86933125237e-9	1.86933125270e-9	+1.8e-10 relative. KLL shown as k=567, delta=1e-7 (was k=345, delta=1e-3); main resized it to 567 in Stage 3, so the size that runs is the same.	Size: #1098. Cost: #1134.
sketch quantile-single-mvp	6.37724245448e-7	same	KLL shown as k=86, delta=3e-5 (was k=52). Status optimal becomes deadline_reached.	#1098 (Pass 1 sizing; the MILP stalls numerically). #1107 names the status deadline_reached.
sketch shared-quantiles-1h-mvp	3.99010517691e-6	same	optimal becomes deadline_reached	#1098, and #1107 for the status name
corpus q02, q06, q22, q23, q26	0.4816 / 0.4846 / 0.4769 / 0.4769 / 0.9162	same	proven_optimal false becomes true. Planning goes from 7–11 s to 47–72 s.	#1107
corpus q05	0.955439633013	same	planning goes from 16 s to 240 s and hits the budget	#1107
corpus q19, q25	1.83727148353	same	planning goes from 21 s to 194 s; still not proven optimal	#1107

#1106, #1113, #1116 and #1138 change none of the 22 snapshot plans. The snapshot harness read Stage 3's old JSON keys after #1119. #1150 fixes it, regenerates the snapshots and accepts the three rows above that the harness used to reject (see "Waived planning-time threshold").

Retro-check (67 runs per mode, integration vs main, 30 s budget)

Mode	same	cheaper	more expensive	new plan (main had none)	both no plan	integration lost a plan
Guarantees ignored	20	34	5	6	2	0
Guarantees enforced (`--enforce-accuracy-guarantees`)	31	7	23	3	3	0

Guarantees ignored:

Cheaper: all 30 random-mixed-seed-862 rows are 75–99.99% cheaper (main's 30 s incumbents were poor), and ex-example3a is 45% cheaper.
New plan: custom-binary-two-sketch-queries-mvp, ex-example1, ex-example2, ex-memo-rangeheavy-t8-t5-1h, sharing-923-default and sharing-923-raw.
More expensive:
- custom-shared-sketch-quantiles-raw: +354% (refactor: facade runs the stages; Stage 3's frozen select_plan interface (#1047 P7) #1107, refactor: Stage 2 emits no scope copies; the selection is DagOfSelectedCandidates (#1095) #1119);
- custom-or-two-sketch-queries-mvp: +140% (refactor: decide every sketch size in Pass 1; Stage 3 only checks (#1054 X2) #1098);
- ex-memo-rangeheavy-t5-q09-15m-1h: +61.6% (refactor: facade runs the stages; Stage 3's frozen select_plan interface (#1047 P7) #1107);
- ex-memo-ratio-of-quantiles: +0.53% (refactor: decide every sketch size in Pass 1; Stage 3 only checks (#1054 X2) #1098);
- custom-or-topk-quantile-raw: +0.0054% (refactor: decide every sketch size in Pass 1; Stage 3 only checks (#1054 X2) #1098).

Guarantees enforced:

Cheaper: custom-shared-sketch-quantiles-raw -39%, ex-example3a -37%, and 5 random-mixed rows about -0.01%.
New plan: custom-binary-two-sketch-queries-raw, custom-or-two-sketch-queries-raw and ex-example2.
More expensive:
- 19 random-mixed rows, +0.009% to +0.028% (fix(stage3): A1 accuracy rows match satisfies; Pass 1 prunes unusable summaries; parts record the sized shares (#1095) #1129; a 120 s budget gives a plan cheaper than main's);
- ex-memo-rangeheavy-t5-q09-15m-1h: +65.8% (refactor: facade runs the stages; Stage 3's frozen select_plan interface (#1047 P7) #1107);
- sharing-923-default and sharing-923-raw: +75.5% (fix(memo): an unwidened scan no longer becomes a widened reader of its own group #1138);
- ex-memo-rangeheavy-t8-t5-1h: +1.7e-7 relative (fix(cost): hold a materialized consumer's working-memory copy for its longest reader (#1095) #1134).

Accepted Stage 3 losses (until #1095)

In each case the cheaper plan's candidates are in Stage 2's output; Stage 3's search does not find them. No Stage 3 code changes before #1095.

Workload	Change	Issue
ex-memo-rangeheavy-t5-q09-15m-1h	+62% (guarantees ignored), +66% (enforced)	#1149
custom-shared-sketch-quantiles-raw, guarantees ignored	+354%	#1149
custom-or-two-sketch-queries-mvp, guarantees ignored	+140%	#1149
sharing-923-default/raw, guarantees enforced	+75.5%	#1142

#1149 records the root causes: #1107 removed the per-query seed; the joint price profile has no options for KLL candidates; seed weights are log-compressed when guarantees are ignored, which hides a 6× gap; and the joint MILP ends with lower bound 0 under the deadline. It marks these workloads as #1095 acceptance cases.

Waived planning-time threshold

#1107 runs every pane-width round up to Complete, so planning takes 5–15× longer on slow corpus queries: q02/q06/q22/q23 go from 9–11 s to 70–72 s, q19/q25 from 21 s to 194 s, q26 from 8 s to 47 s, and q05 from 16 s to the 240 s budget. The plans are unchanged, so this slowdown is accepted. In the snapshot gate (#1150):

q05 has a time_limit_reason, so reaching the time limit no longer fails it; its plan and cost are still compared.
quantile-single-mvp and shared-quantiles-1h-mvp have a deadline_reached_reason, so they may end deadline_reached; their plan, cost and status are still compared.

X2 sizing note

#1098 sizes KLL for its failure probability even when a query composes no guarantee, for example with guarantees ignored. KLL state for ex-memo-ratio-of-quantiles grows from 121.8 MB to 174.2 MB (+0.53% cost), and custom-or-topk-quantile-raw switches to DDSketch (+0.0054%). This stays as it is; #1054 records it.

Before this PR

On main, Stage 3 re-sized sketches, folded Stage 2's execution-scope copies, ran the pane-width rounds, and stopped after a 128-evaluation budget. Stages called each other across crate boundaries.

After this PR

Each stage has its own boundary types, and the facade orchestrates the stages. Pass 1 decides sketch sizes. Stage 3 is a replaceable select_plan that takes only a deadline. Selected plans are the same on the snapshot workloads; the Stage 3 losses above are tracked for #1095.

Verification

Each constituent PR was reviewed and merged into the integration branch with its changed-part tests, fmt and clippy.
The gate snapshots (22 workloads) and the retro-check (67 runs × 2 modes) are described above.
cargo test --release -p asap-devtools --test corpus_snapshots -- --ignored passes on the integration branch after test(devtools): snapshot harness reads Stage 3's Selection; accept the facade's slower planning #1150.
CI: test and format-and-lint pass on this PR's head.
Private corpus runs in the nightly after merge.

🤖 Generated with Claude Code

X2) (#1098) Pass 1 now sizes each memo candidate's approximate inputs for its readers' failure probability over the values their targets cover: delta / (n * V), n the candidate's accuracy-budget parts, V the values the scope covers (G from the workload bounds for the whole result), the smallest over the queries reaching the group; classic HLL precision is raised for its source's distinct bound. This replaces add_scoped_candidates (one sized candidate per split, in place) and Stage 3's resizing. Pass 2 inherits the sizes or takes the strictest shared reader's (size_for_all_readers). Stage 3 deletes sized_for_failure_probability / resize_for_failure_probability, the DAG recompilation after resizing, and RejectionCause::FailureProbabilityResizeFailed; a candidate whose sizes miss the composed target is rejected with AccuracyTargetMissed. template_price_is_a_floor is now always true. Docs updated. Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…step 8b, part 2) (#1099) asap-types: fields and methods that only tests read outside the crate become pub(crate). Tests reach them through accessors and constructors under `test-support` (SummaryCoverage, SelectionBox, ColumnIdentity, MaterializedView, ViewCoverage, MaterializedViewInputs, RawDataWindow, RawDataRetention, PhysicalASAPDAGEdge, FamilyCapability) or through `asap_types::test_support` wrappers for methods the crate itself uses (output_schema, validate_inputs, matches_node, timed, lookback_ms, is_materialized_view). `Schema::has_unique_key` and `compile_physical_asap_dag` are test-only and go under `test-support`. Frontends: the `error`, `histogram`, `promql`, `sql`, `frequency`, `resolve`, `schema_resolver` and `unresolved` modules become private (one public path per item). `SqlLowerer`, `SchemaResolver`, `SchemaCatalog`, `UsageDerivedCatalog` and `resolve_expr` become pub(crate). Single-query MetricsQL entry points and the SQL unresolved lowering move to new `test_support` modules behind `test-support`. Deleted: `HistogramCatalog::is_empty`, `UnresolvedOp::scalar_exprs`. The executor's and integration-tests' fixture code enable `asap-types/test-support` for the test-only DAG helpers. Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

* test: snapshot the selected plans of sketch-selecting workloads The corpus snapshots are single queries without an accuracy target, so their plans are exact and never exercise sketches, their sizes, sharing, materialization or pane chains. Add a second ignored test to the corpus_snapshots binary (so the nightly run picks it up) that pins the selected plans of 17 workloads with accuracy targets: shared DDSketch pane chains, nested windows, a ratio of two quantile sketches, KLL, DDSketch, KMV and a CMS heap, an `or` composition, and random-mixed subsets, under the default, raw-retained and MVP deployments. Each snapshot reuses the corpus snapshot of the selected plan (DAG with sketch parameters, per-node cost, accepted guarantees, materialized views, pane retention) plus every query's text and the search status. Every kept workload is proven optimal or stopped by an evaluation count, and gave the same plan in three runs. A failure names the workload and the first differing JSON path; UPDATE_SNAPSHOTS=1 regenerates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test: keep only optimal sketch plan snapshots; serialize snapshot tests The seed and pane-width rounds have 5 s wall-clock budgets, so a search that stops at any budget can select another plan on a slower machine: pinned to one core, planner-layering-3a changed plan without a status change. Keep only the ten workloads whose search proves its plan optimal, and fail any workload whose status is not optimal instead of recording it. Drop the single-quantile default and raw-retained, planner-layering 3a/3b, `or` and COUNT(DISTINCT) workloads. Hold one mutex for the whole of each planner snapshot test so the corpus and sketch plan tests in the corpus_snapshots binary never run at once under --include-ignored. Mention sketch_plan_snapshots/ in the planner snapshots doc and the PR template. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Stage 3 now computes no cost of its own: the per-node and per-plan costing that lived in plan-selection's pricing.rs moves to asap-cost, which depends only on asap-types. - pricing.rs -> asap-cost usage.rs, re-exported at the crate root: price_node -> node_usage, PricedNode -> NodeUsage (work_ms -> work_ms_per_execution), price_at_scale -> plan_cost, CandidateCost -> PlanCost; ProducerOutput, panes_of, trailing_window, node_map, kept_pane_chains, pane_retention_ms, query_reach and family_name move with them. price_each_node stays private. - execution_demand::schedule_groups(readers, horizon) returns the reader groups evaluated together (ScheduleGroup { readers, evaluations_per_second, schedule }); evaluation_rate is their sum, added in the same order as before, so every rate is unchanged. - plan-selection changes only imports and call sites; the test-only self-calibrated price helper moves into lib.rs tests, and checks.rs reads dag.scan_window(..).lookback_ms() directly. Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…ses (#1095 Steps 1-2) (#1103) Add `classes.rs`: given `PhysicalASAPMemoCandidates`, fold the candidates that are equal in operator node, timing, materialization, query-output and keep flags, reader constraints (other than ActivationRequires) and producer classes per input into one class, by partition refinement to a fixpoint. Each class records its copies (ascending), its capacity (the copy count), its producer classes per input, and the state it holds (#1095 Step 2), which makes it shareable; the others are stateless. ActivationRequires inside a class is absorbed by the capacity; any other one keeps both of its candidates in classes of their own. Only adds the module and its unit tests; the search does not use it yet. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

) One PrunedCandidate { group, candidate, reason } and one PruneReason enum move to asap_types::stages. Pass 1 fills its pruned_candidates with them. Stage 2 replaces pruning_metrics with pruned_candidates, keyed by PhysicalCandidateKey { operator_node, timing, materialization, scope_copy }, and records: PerEntityAggregateNotExecutable (new Stage 2 catalog rule), RawDataNotRetained, InputHasNoBuildableProducer, ReachesNoQueryOutput, TooFewQueryReaders, PreviousScopeCopyPruned. Stage 3 drops its raw-data and per-entity aggregate checks (and their RejectionCause variants); stage_pipeline and the viewer's Memo view show both stages' pruned_candidates. Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…y pruning (#1047 P6b) (#1105) * refactor(plan-selection): delete early query pruning (#1047 P6) Delete `prune_query_candidates` / `alone_violation` (`query_pruning.rs`), the route block that ran it before Pass 1's memo population, and the `early_query_pruning_ms` / `_counts` / `_examples` report fields. Stage 3 still checks every candidate; its pruning is replaced by #1095's region DP, and the per-query "no combination meets the target" check moves to Pass 1 (#1047). This removes the last caller of the legacy whole-workload Pass 2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw * refactor(logical): delete the legacy whole-workload Pass 2 path (#1047 P6b) Delete `pass2/workload_candidates.rs` (`stage1_candidates_of_choice`) and the whole-workload Pass 2 entry points the memo rules replace: - identical expression: `identical_expression_pass2_candidates`, `sharing_candidates`, `SharingCandidate`, `IdenticalExpressionPass2Candidate` (memo group identity); - summary capability: `summary_capability_pass2_candidates`, `capability_sharing_candidates`, `CapabilitySharingCandidate`, `SummaryCapabilityPass2Candidate`, `MAX_CAPABILITY_SHARING_MEMBERS` and their private helpers (`memo::rules::shared_producer`, `shared_configuration_readers`); - top-k reuse: `topk_reuse_pass2_candidates`, `TopKReusePass2Candidate` (`memo::rules::topk`). Also delete `explicit_assignment_plans` and `memo_covers_whole_workload_pass2.rs`, which compared the memo against this path, and the unit tests of the deleted functions. `planner_layering_example2`'s shared-UnivMon tests read the memo's shared candidates (`shared_pipeline`); the pairwise test is deleted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw * docs: describe the memo path where docs named the legacy Pass 2 path (#1047 P6b) `stage_pipeline`'s Stage 1 description, `planner-layering.md`, `asap-aware-mapping-architecture.md` (stage diagram and component table) and `extend-asap-aware-mapping.md` (adding a Pass 2 rule) now name the memo rules and memo enumeration instead of the whole-workload candidates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…#1047 P3c, fixes #1089) (#1106) Pass 1 now finds its targets by walking the memo groups of the DagOfLogicalMemoGroups instead of each query's frontend nodes: one target per (aggregate group, measure), however many queries read the group. - A key-only HAVING goes below an aggregate only when every reader of its group (parent groups and query roots, across queries) applies it. Before, this was decided per occurrence and memo population kept the first target per (group, measure), so query A's HAVING city = 'NYC' filtered the summary query B shares and B returned only NYC (#1089). - Fused computations (whole-expression top-k, nested sum) no longer require the inner aggregate to have a single reader. A fused computation is one more candidate of the outer group; the absorbed groups keep their own candidates for other readers, so private_absorbed is deleted. Pass 1 population skips a candidate whose structural id, input groups, fused groups and budget share equal one the group already has. - generate_per_computation_candidates takes &DagOfLogicalMemoGroups; generate_per_computation_candidates_with_memo, the per-(group, measure) choices cache and PerComputationCandidateError::MemoRootsChanged are deleted. - PerComputationCandidates drops Id; its roots are the DAG's roots over the groups' nodes. ComputationTarget.aggregate becomes aggregate_group, with PerComputationCandidates::aggregate for the node. - PerComputationCandidates::walked replaces DagOfLogicalMemoGroups::matches_root_dependencies; original_roots is deleted. Stage1ToStage3Run and CandidateSearchContext drop Id. - Regression test having_shared_aggregate (fails on the base branch: B's shared group returns only NYC). Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…ace (#1047 P7) (#1107) * refactor: the facade runs the stages; Stage 3 gets its frozen interface (#1047 P7) Stage 3 (`asap-plan-selection`) now has one public entry, `select_plan(physical, entries, data, &AccuracyCheckInputs, &CostInputs, &dyn AssignmentRealizer, deadline) -> Result<Selection, ErrorSelectingPlan>`, with today's engine behind it and no dependency on Stages 1 or 2. - Selection / SearchReport / SearchStatus follow the decided interface. Whole-plan rejections, rejected_by_cause, search_rejected, the Pareto front (and ClusterObjective::ParetoFront), per-plan cost/accuracy maps, plan id strings, runner-up, pane_widths_pending, exhaustive and keep_every_candidate are gone from Stage 3. - SelectedCandidates replaces SelectedCandidateGraph (no modeled_cost); AssignmentRealizer has one method, realize -> RealizedPlan {plan, pane_retention_ms, nodes_by_candidate}; Stage2AssignmentRealizer holds the planning context; pane retention moves to asap_cost and is computed by realize. - The engine searches the whole catalog once per call: no component split, no per-query seeding; its heuristics read only PhysicalASAPMemoCandidates. - asap-planner runs Stage 0, Pass 1, the pane-width rounds (Pass 2 -> Stage 2 -> select_plan), keeps the best round by asap_cost::PlanRanking, builds the plan with realize and applies the proven weights. The pass API, PlanningModels, PlanOutput (now with pane_retention_ms) and PlanningRun / PlanningReport live there. - The old whole-workload selector stays in plan-selection behind the reference-pricer feature; stage_pipeline --exhaustive is the devtools reference enumeration. - Docs and the DAG viewer's Stage 3 panel follow the new document. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw * refactor: ignore_accuracy_guarantees leaves CostInputs; the facade passes the accept-all model (#1047 P7) #1095 §1: `ignore_accuracy_guarantees` is no longer a `CostInputs` field. The facade's `PlanningModels::ignore_accuracy_guarantees` (`with_ignore_accuracy_guarantees`) makes `accuracy_check_inputs()` pass `asap_accuracy::IgnoreAccuracyGuarantees` to Stage 3, which no longer swaps models itself. The engine's accuracy heuristics read `AccuracyModel::may_accept_regardless_of_guarantee` instead of the field. - `IgnoreAccuracyGuarantees` moves from plan-selection to asap-accuracy. - `stage_pipeline` sets the models flag from `--enforce-accuracy-guarantees`; a `--plan-selection-settings` file with the field is refused. - Tests that set the field now set the models flag (facade) or the accept-all model (Stage 3). - `SearchStatus::DeadlineReached` documents that it also covers the engine's 128-evaluation stop until #1095; the reference pricer notes its calibration scale under `--price-enumerated`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…cal, executor) (#1109) * test(executor): set the semijoin edges' timing through data_state_mut #1099 rewrote `edge.data_state.timing = phase` as `edge.data_state().timing = phase`, which assigns to a copy: the edges kept QueryTime while the nodes moved to IngestionTime, and bind rejected the DAG. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw * test(logical): update Pass 1 expectations for memo-group targets (P3c) #1106 (P3c) walks memo groups: structurally equal aggregates and HAVING filters are one group, with one target per (group, measure) and HAVING placed per group; roots are named by workload index, and an inventory matches any DAG built from structurally equal roots. - having_on_one_consumer_of_a_shared_aggregate_stays_above: two equal HAVING filters are one group, so one entry. - memo_cached_rules_preserve_occurrence_specific_having: the two equal aggregates are one target; its HAVING stays above the output because an unfiltered consumer reads the group. - assigned_physical_annotation_asap_input_and_invalid_accuracy_are_rejected: InvalidAccuracy names the root by workload index ("0"); doc comment updated. - population_rejects_an_inventory_from_other_roots: the other roots now differ structurally (the same query text is the same DAG). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw * test(frontend-promql): move planner-facade helpers out of support.rs support.rs is also compiled by the private corpus, which does not depend on asap-planner. run_stage1_to_stage3 and selected_dag (asap-planner since #1107) move to planner_support.rs, used only by this crate's test binary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…utput validation (plan-selection, planner) (#1110) * fix(plan-selection): decoded selections assign each query output once P7 (#1107) made validate_selected_graph check the selection through to_selected_candidates, which sorts the roots by output index and drops the index. A root assigned to a nonexistent output (or two roots to one output) then passed validation. Check that the assigned output indices are exactly 0..n before converting, as the checker did before P7. decoded_graph_is_checked_independently_of_solver_values covers it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw * test(plan-selection): kept-pane tests' realizer retains query-time results P7 (#1107) moved the query-time-results-retained capability from the planning models into the Stage 2 realizer. The two kept-pane pricing tests still set it only on their deployment, so the test realizer refused MatAtQueryKeptAcrossEvaluations(Memory). Build the realizer with the deployment's flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw * test(planner): no-raw avg-of-p99 workloads expect no selected plan, not an infeasibility label Both tests asserted the infeasible_under_configured_model status and the early pruning's rejection text. P6b (#1105) deleted early query pruning, which proved these catalogs infeasible, and P7 (#1107) removed that status from Stage 3's report: today's engine now stops at its budget with no validated plan. The tests now assert that plan selection selected none (no valid candidate, or none by the deadline); the ignore-accuracy test's selected sketch plan is still checked end to end. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…ion applies Stage 2's raw-data rule (#1111) * fix(devtools): the reference enumeration drops plans that scan raw data a deployment does not keep #1047 P6 moved the raw-data rule from Stage 3's checks into Stage 2's catalog (PruneReason::RawDataNotRetained). The devtools reference path implements Stage 2 with optimize_physical_asap, which enumerates every assignment, so stage_pipeline --exhaustive, --price-enumerated and the integration tests' explicit fixtures priced query-time raw scans that the deployment cannot run, and selected them. memo_pricing::retain_deployable applies Stage 2's rule; physical_plans_of and run_explicit_candidates use it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw * test(devtools, dag-viewer): read Stage 3's Selection document #1047 P7 made stage3_plan_selection Stage 3's Selection (the plan P1 with its cost, accepted_accuracy_per_query and search_report) and moved the per-candidate table under --exhaustive to reference_pricing; the facade's PlanningReport is planning_report. The stage_pipeline, memo_pricing and viewer tests, and server.py's log line, read that shape; a Pass 1 failure's Stage 3 section is a Selection without a plan. Deleted: the Pareto-front check (ClusterObjective::ParetoFront is gone) and the component and candidates-checked report fields (the engine has no components). viewer_contract reads a JS array up to its own ']'. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw * test(integration): fixtures follow P3c's shared composition and P7's planning budget - Pass 1 now composes a candidate from memo groups (P3c), so queries that read an identical input share its node. The Example 1 and 3 fixtures build their independent variant with planner_layering_common::unshare, and materialization_e2e splits a node read at two timings (split_shared_by_timing) as Stage 2 does. - memo_enumeration: one target per quantile group (3 x 3 inventory assignments, P3c); a 90 s budget, since Stage 3 searches until the deadline when it validates no plan. - finer_aggregate_reuse::filtered_kll_states_cost_no_more_than_separate_plans is ignored until #1095: P7 removed the per-query seed that reached the separate plans' cost. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…ports the planning budget (#1113) * fix(planner): report a Pass 1 deadline as the planning budget running out With a zero planning budget, Pass 1's sizing hits its deadline and the facade reported it as "logical ASAP-aware optimization: candidate generation deadline exceeded". Map PerComputationLogicalMemoCandidatesError::Timeout to SelectionError::DeadlineReached, as before P7; other Pass 1 errors keep LogicalASAPCandidates. Fixes joint_search_equivalence::zero_budget_returns_no_unvalidated_plan and stage_pipeline::milp_time_limit_is_reported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw * fix(plan-selection): count quantile operands independently of node sharing The feasibility seed's DDSketch budget splits a query's relative error over its quantile inputs. It counted distinct quantile nodes by Rc identity. Since P3c (#1106) Pass 1 builds each query's root over the memo groups' nodes, so the two `quantile 0.5` operands of `avg by (job) (q0.5 / q0.5 + q0.9)` are one shared node: the count fell from 3 to 2, the seed admitted sketches sized for half the budget, whose outputs have no accuracy guarantee, and the search spent its whole budget cutting such plans one by one without reaching the exact plan. quantile_operands counts each quantile aggregate as often as the most operands that read it from one path (q / q reads q twice), and a diamond of different parents over one aggregate (an average's sum and count) once, so the count no longer depends on how equal nodes are shared. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…ay public API (#1114) tools/verify_metricsql_external_consumer.sh (CI) treats them as a stable external dependency; #1099 had moved them behind test-support. Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…accuracy weights (#1095 C3) (#1115) New `plan-selection/src/owners.rs` (pub(crate), not called by the search yet): `prepare_milp_inputs` computes, from the folded classes, - class dominance pruning (exact under cost model 1), with the pruned classes and the dominating class; - per-class data: approx, accuracy_budget_parts (smallest over a factored node's memo candidates), usage per execution, cost per execution, standing cost, held bytes and own latency, all from asap_cost::node_usage, with the context options (producer outputs, scan windows, pane retention) that change the usage recorded; - query owners and instance owners (m_s = min(cap, reaching queries)) with their regions, links and boundary reads; - accuracy weights w(q, k): 1 ordinary, 2 denominator, unusable for nested quotients and unclear operators; exact divisors scale and count aggregates read no value. asap-cost exposes `retained_panes` and `maintained_window_ms` (visibility only). Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…cy target (#1116) * feat(pass1): prune summaries no reading query can use for its accuracy target Pass 1 now removes a summary candidate (own or fused) when, for every query whose root reaches its aggregate group, the query's output composed with that summary (every other target exact) has no error bound that can meet the query's target at any size: guarantee propagation has no rule above it (e.g. a rank error through avg, a value HAVING over an estimate), its bound stays unknown once the data workload's sign declarations are bound (a relative error through avg over an undeclared gauge), or its metric is not one the target accepts. Other symbolic statistics are assumed bindable. A summary one reading query can use is kept; exact candidates are never pruned. The reason is the new PruneReason::NoErrorBoundForQueryTarget, carried in Pass 1's pruned_candidates. The facade prunes only when Stage 3's accuracy model and evidence cannot accept an output without its guarantee (not with ignore_accuracy_guarantees). Stage 3 reports a query root group left with no candidate as no valid candidate rather than a solver failure. The two no-raw avg-of-p99 runtime_install tests assert the reason again, on the facade's MemoSteps::pass1_pruned_candidates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw * feat(pass1): keep summaries Pass 2 may reuse; ask the accuracy model; check whole-result scope Follow-ups to the structural accuracy prune: - Pass 1 cannot see Pass 2's reuse of one group's summary by another (rollup, window rollup, finer-aggregate reuse, top-k reuse). A summary is now kept whenever another target, over another aggregate reading one of the same sources, offers the same family. This keeps more than those rules' preconditions need and relies on no dominance argument. - The metric check asks the deployment's accuracy model through a new default method, AccuracyModel::metric_can_meet. Its default is the built-in rule; the facade passes the model Stage 3 uses. - The guarantee is stated over the target's scope, as Stage 3 states it. A per-value guarantee that may fail cannot meet a whole-result failure probability over a number of values the data does not bound, so it is pruned with the same reason. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…ng guarantees (#1118) #1116 makes Pass 1 prune a summary no reading query can use for its accuracy target when guarantees are enforced. Seven integration tests read the reference enumeration under the built-in strict model to check the structure or execution of a summary whose output Stage 3 already rejected at that target before #1116: a heap-backed Top-K and a value filter or group key over an estimate have no guarantee, and a grouped KLL quantile's per-value guarantee meets no whole-result target over groups the data does not bound. Pass 1 now prunes those summaries, so the tests found no candidate. The tests now plan with every output accepted (logical_candidates_ignoring_guarantees, ignore_accuracy_guarantees), as they check rewrites and execution, not accuracy acceptance. Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…edCandidates (#1095) (#1119) * refactor: Stage 2 emits no scope copies; selections are a DAG of candidate runs (#1095) Stage 2 emits one PhysicalCandidate per (operator node, timing, materialization). ReaderConstraint::ActivationRequires, PhysicalCandidateKey::scope_copy and PruneReason::PreviousScopeCopyPruned are deleted, with catalog/scopes.rs. The selection is DagOfSelectedCandidates { candidate_of_node, edges, node_of_output_reference } with SelectedNodeId and SelectedEdge; its validate(physical) replaces validate_selected_connections and checks each reader constraint per node. SelectedCandidates and ProducerConsumerConnection leave the public API. The realizer builds one plan node per selected node (RealizedPlan::operator_node_of_selected_node); Selection carries selected_dag and nodes_with_proven_non_negative_weights. Today's Stage 3 engine makes the copies it needs itself (search/candidates/scopes.rs, moved from Stage 2) and decodes the selected copies into nodes; its activation constraint and connection type are private. Plans, costs and MILP sizes are unchanged on the memo examples. classes.rs is deleted: owners.rs works on PhysicalCandidateId, m_s is the number of queries that can reach s, and capacity is gone. stage_pipeline writes selected_dag and per-round Stage 2 producer links (PlanningRoundReport::physical_producer_links); the viewer drops scope_copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WviZ6EjGTm6ZuggMKX1mjr * docs: no scope copies in Stage 2; Stage 3 selects a DAG of candidate runs (#1095) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WviZ6EjGTm6ZuggMKX1mjr --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…C4) (#1125) * feat(types): Stage 2's operator nodes carry accuracy_budget_parts (#1095) Each OperatorNodeOfCandidateForMemoGroup now holds how many equal shares of a query's accuracy budget its sketch sizes meet, copied by Stage 2 from its memo candidate (the largest for a factored node). Stage 3's owners.rs reads it and drops its parts_per_memo_candidate parameter. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw * feat(plan-selection): the owner-scoped selection MILP and its cost model 1 terms (#1095 C4) Builds the §2.0 MILP from MilpInputs over Stage 2's candidates (no scope copies, #1119) and solves it once with HiGHS, the deadline as its time limit, returning the raw solution (outputs, each owner's candidates and links, active instances) for C5 to decode into a DagOfSelectedCandidates. - milp/model.rs: x, s, e, b, o/ob, reach a (exact from above where a reader constraint counts readers), latency L with the evaluator's per-node definition, accuracy A1 (one share per approximate instance, denominators need parts >= 2, unbounded positions 0), reader constraints, symmetry breaking. m_s is the number of queries that can reach s; there is no capacity row. - milp/cost_terms.rs: execution at the query's rate or the instance's rate (schedule-group levels for materialized instances, per-consumer rates for not-materialized ones), standing cost, held memory, raw retention levels, usage on producer links with AND rows, scan window levels. Every number comes from asap-cost; request fees are reported apart. Cost models 2/3 return UnsupportedCostModel. Terms plan_cost defines differently are counted as inexact by reason. - owners.rs: per-producer and per-window usage tables; MilpInputs' parts renamed to candidate names (candidate_data, CandidateData, pruned_candidates, DominatedCandidate, producer_candidates_per_input, output_candidates, CandidateUsage). - Tests: objective equals plan_cost after decode + Stage 2 realization (optimum, shared MatAtQuery held by readers of different latency, raw retention over two windows, a reader costed at its shared scan's window), latency bound, accuracy rows, symmetry breaking, cost models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WviZ6EjGTm6ZuggMKX1mjr * feat(plan-selection): the selection MILP finds plans: warm start, tighter rows, usage per producer output (#1095 C4) - Warm start (milp/warm_start.rs): each query planned alone with no sharing (cheapest derivation over the candidates it may use, its own instance of each shareable candidate), repaired for accuracy and latency; HiGHS completes the continuous variables from its binaries. A deadline now ends with a plan. If a query has no plan alone, the solve runs without a start and an infeasible model names that query (NoPlanForQuery). - Accuracy rows: N[q] as one variable and one big-M row per parts value, instead of a dense sum per instance (q05-q09: 1.89M -> 6k nonzeros). - Rates: a fixed rate * x[i] where all readers run an instance at one rate; otherwise rate components (schedule-group levels, consumers, and AND(z_m, y) for materialized consumers instead of the big-M P rows). Region execution is summed once per instance and charged per component (V), replacing Q per node; held time at a fixed rate needs no W. - Latency: an instance region's work is one variable u[i], charged per reaching query (u[q, i]), replacing w per node. - owners.rs: usage tabulated per producer output, transitively (each context's own output is recorded and readers key on it); a reader's term sits on AND(link, the producer node's terms giving that output), with equal outputs merged. "Depends on context" goes from 3,382 to 0 on #1036 workload-01; the remaining gaps are producers on a memo cycle. - Tests: the warm start is a feasible plan costing what plan_cost says; a short deadline yields a plan; no usage-table gap in the fixtures. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WviZ6EjGTm6ZuggMKX1mjr * fix(plan-selection): HiGHS runs on 4 threads; objective scaled by the warm start's cost (#1095 C4) - A thread per core gave HiGHS a malloc arena each: on a 56-core host #1036 workload-01 broke the 8 GB `ulimit -v` with std::bad_alloc at 4.8 GB resident. With `threads = 4` it plans within the limit. - The objective was scaled so its largest coefficient is 1,000, which left the costs actually compared near HiGHS's tolerances (workload-01's dual bound came out negative; rollupheavy-t11 stopped at 69.86 as "optimal" though 54.44 exists). It is now scaled so the warm start's cost on its binaries is about 1,000, the largest coefficient at most 10^7. - The HiGHS log switch read from the environment is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WviZ6EjGTm6ZuggMKX1mjr --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…P_STAGE3_SEARCH=milp (#1095 C5) (#1127) * feat(types): Stage 3 prune reasons DominatedBy and UsageNotCostable (#1095 C5) Stage 3 records what it removes before its selection MILP in Selection.pruned_candidates: a candidate another dominates (named), or one asap-cost could not cost, which the model leaves out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WviZ6EjGTm6ZuggMKX1mjr * feat(plan-selection): decode the selection MILP, realize once, check, behind ASAP_STAGE3_SEARCH=milp (#1095 C5) - milp/decode.rs: the raw solution as a DagOfSelectedCandidates. Each owner's use of a stateless candidate is one node, each active instance (s, j) one node; edges follow the selected e and b links (an instance owner's links into its own candidate are the instance node's inputs); outputs follow o / ob. Only what the outputs reach is kept. Replaces the test-only decode in milp/model/tests.rs. - select.rs: the new search. One MILP solve, decode, validate, one realization (a rejection is RealizerRejected), then exact checks: plan_cost minus request fees equals the objective within 1e-6 relative (else ObjectiveDiffersFromPlanCost, or SearchReport.inexact_objective when the model counted inexact terms); accuracy (satisfies) and latency must hold (else SelectedPlanFailedCheck). Fills Selection: count-min weight proofs per node, PlanCost, accepted accuracy, pruned candidates (dominated, uncostable), and the search report (status, dual bound plus request fees, MILP size, timings). No plan: NoValidCandidate with a RejectionCause::Infeasible (or LatencyBoundExceeded) rejection naming the query the warm start found no plan for. - select_plan runs it when ASAP_STAGE3_SEARCH=milp; the old engine stays the default until C6. A measured cost table needs a frozen calibration factor (UnsupportedCostModel otherwise), as do cost models 2 and 3. - The facade keeps the new error variants when it copies a round's error; the viewer lists the new rejection cause. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WviZ6EjGTm6ZuggMKX1mjr * test(devtools): the MILP search's objective equals plan_cost on example 1 and RangeHeavy T5 (#1095 C5) Runs stage_pipeline with ASAP_STAGE3_SEARCH=milp on the child process and documents the switch in the binary's header. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WviZ6EjGTm6ZuggMKX1mjr --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…on MILP (#1095) (#1128) - warm start: local search from the unshared, amortized and sequential plans (re-plan a query, move every reader off a candidate, merge two instances), every plan costed exactly by the MILP's own rows (new milp/completion.rs); HiGHS gets every variable's value. - cost rows: the -x[i] form of the y and V AND rows, a slowest-rate row on V, per-query big-M on H and W. - fewer AND rows: producer outputs with equal usage merge when no reader tells them apart; scan levels below a reader's own window count as its own level. - latency rows of a binding bound are divided by the bound. - a deadline the warm start used up ends with it, reported as the deadline; SelectionMilpSolution.warm_start_cost_per_second.

… summaries; parts record the sized shares (#1095) (#1129) The A1 accuracy rows now take each position from asap-accuracy's composition and Stage 3's guarantee_check, so the selection MILP admits exactly the single and pairwise placements satisfies accepts. A failing pair gets a[q,k1] + a[q,k2] <= 1, and AccuracyWeight::Two is gone. Pass 1 changes: - The reuse escape keeps a summary only for a reusing target whose own queries can use the family. - Classic HLL is pruned when the declared distinct bound is outside its contract at precision 18. accuracy_budget_parts now records the shares a sketch was sized for, for both shared builds and state-configuration rewrites. Fixes ratio-of-quantiles, rollupheavy-t11 and rangeheavy-t8-t5 under ASAP_STAGE3_SEARCH=milp with guarantees enforced.

…1130) Candidate links can close cycles (a widened scan's Project lists its own memo group, itself included, as producers), and rows 2-3 of the selection MILP admitted a self-supporting cycle with no source below it, so rangeheavy-t7-t4 failed DagOfSelectedCandidates::validate. Row 5: no class reads itself in an owner, and order ranks on the links inside each cyclic component of an owner's links and of the instances' reads. rangeheavy-t7-t4 now plans at 2.149e-7/s (optimal, equal to today's engine); other memo examples are unchanged.

…1131) No caller outside crates/accuracy/src/sizing.rs uses it; its two in-file callers stay public. split_shared_by_timing stays in asap-types: Stage 2's assigned_candidate calls it on the realizer path.

A test reads cargo metadata and fails on any [dependencies] edge between workspace crates that the target graph forbids (stage crates never depend on each other; asap-accuracy and asap-cost depend only on asap-types; the facade on stages and shared crates; devtools and integration-tests unrestricted), naming the edge and the rule. Exceptions: the optional logical -> frontend-promql edge behind test-support, and planner -> executor (#1016).

…engine is the only engine (#1133) refactor(plan-selection): remove the #1095 Stage 3 redesign; the old engine is the only engine The redesign (owner-scoped selection MILP, its decode, warm start and completion, owners and accuracy positions, the ASAP_STAGE3_SEARCH switch) is deleted so the integration branch can merge to main while Stage 3 is redesigned. It stays recoverable at the tag stage3-owner-milp-redesign. - plan-selection: delete milp/, owners.rs, owners/, select.rs, select/; realize and the count-min weight check move back into search/select.rs; remove InexactObjective, SearchReport.inexact_objective, ErrorSelectingPlan::{UnsupportedCostModel, ObjectiveDiffersFromPlanCost, SelectedPlanFailedCheck} and RejectionCause::Infeasible. - types: remove PruneReason::{DominatedBy, UsageNotCostable}. - planner: remove the matching clone_error arms. - cost: retained_panes and maintained_window_ms are private again. - devtools: remove the MILP search tests and the switch's header lines; dag-viewer drops the infeasible cause label.

… longest reader (#1095) (#1134) plan_cost charged a NotMaterialized node's query-time working memory for the rate-weighted mean latency of every query reaching it, times its computations per second. When a materialized node reads it, the node is computed once per execution of that materialized node and its working memory is held until that node's slowest reader finishes, as the MatAtQuery state itself is. plan_cost now charges each consumer's copy (asap_cost::execution_demand::consumers): a query's for its own latency at its rate, a materialized node's for its longest reader's latency at its rate. computations counts the same consumers. The default search's direct-cost model (selected_cost, encode, general_schedule::encode) certifies its objective against plan_cost, so it charges the same per-consumer term; the mean-lifetime B/Q and equality encodings are replaced by gated per-consumer lifetimes. Stages 0-2 are unchanged; only Stage 3 costs (and so plans) change.

…r behind a feature (#1054) (#1135) The catalog module is private: the planner's items (generate_physical_asap_memo_candidates, Stage2AssignmentRealizer, ErrorGeneratingPhysicalASAPMemoCandidates and its error types) are re-exported at the crate root, test-only catalog items move to test_support::catalog, and the re-exports of asap-types items (PhysicalWorkloadPlan, PhysicalCandidateError, RealizedPlan) are dropped. optimize_physical_asap compiles only under cfg(test) or the new reference-pricer feature, which devtools and integration-tests enable and test-support implies.

#1136) - operator_statistics is a private module; EdgeStatistics, which NodeUsage.output and ProducerOutput.output expose, is re-exported at the crate root. - analytical: ResourceEstimate, PhysicalOperator, the Promql* enums, HashJoinBuildSide and estimate_operator are pub(crate); only AnalyticalCostError stays public. - measured: aggregate_lookup is pub(crate).

…ts, CandidateMemo → MemoCandidates) (#1137) budget_share becomes accuracy_budget_parts in the stage_pipeline JSON (accuracy_budget_parts, accuracy_budget_parts_1), the DAG viewer and its fixtures, and the logical crate's internal names (compose_logical_candidate_with_accuracy_budget_parts, accuracy_budget_parts_per_measure, max_accuracy_budget_parts). CandidateMemo becomes MemoCandidates, the name #1047 gives it, with no alias. It stays pub in asap_types::stages: asap-types defines it as a field type of the stage-boundary outputs and the logical crate builds it, so it cannot be crate-private.

…s own group (#1138) Scan-label widening recomposes the guarantees of every root it returns, including roots it left unchanged. The rule compared each returned root with the stored original, which has no recomposed guarantee annotation, so a scan or time range that already had every label of its metric looked widened. Interning then mapped it back to its own group and wrapped it in an identity Project: a candidate reading the group it implements. Stage 2 wired that candidate to every candidate of its group, itself and its execution-scope copies included, so the physical candidate graph and the class folding of #1095 had cycles. The no-op readers also counted as a second identical build, which made single-reader shared state groups. The rule now compares with the recomposed original. The memo's topological order counts a group reading itself as a cycle, so Pass 2's existing acyclicity checks reject such a memo. Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

…ing form (#1047 P6) (#1139) generate_per_computation_logical_memo_candidates runs all of Pass 1 (targets, deployment and accuracy pruning, population, sizing); the facade calls it once and PlanningRun.per_computation_candidates holds its output. The target list (per_computation_candidates) is public only with test-support. Removed: stage_pipeline's pass1_pruned, MemoSteps.pass1/pass1_pruned_candidates, ComputationTarget.raw_data_samples_of, and from_roots' Id parameter. Plans, costs and Stage 0-2 counts unchanged on example1, rangeheavy t5-q05-q09 and t8-t5.

PlanSearchSettings gains inspect (default false). Without it run_planning copies no memo: no Stage 0 copy, no Pass 1 clone, no per-round unexpanded-memo clone, no MemoSteps; PlanningRun.per_computation_candidates and memo_steps are None. stage_pipeline and the reference enumeration set it. MemoSteps.stage0 is removed (Stage 0's groups are Pass 1's). Plans, costs and counts unchanged with inspect on or off.

…ingModels.plan_selection to cost_inputs (#1054) (#1141) The CostInputs type was already renamed; this renames the stage_pipeline flag, the PlanningModels / SelectionModels field, the with_plan_selection builders, and the prose that still said "plan selection settings". No alias for the old flag; behavior unchanged.

memo_enumeration::choices_whose_candidates_read_each_other_are_left_out looked for two groups whose candidates read each other. On its workload the only such pair was a scan reading its own group, the self-loop #1138 removed. Pass 2 now rejects any memo whose groups read each other, so the test checks that production's memo has a dependency order and no self-reads, that every enumerated choice composes, and that a cycle injected by hand is still reported as Cycle. sharing_issue_regressions::exact_coarse_sum_reuses_fine_sum is ignored with a pointer to #1142. Stage 2 still offers the cheap plan, but Stage 3 keeps a MatAtQuery build whose NotMaterialized sibling would avoid a second scan. Before #1138 the invalid self-reads made shared-build candidates that hid this. Stage 3 MILP work is paused. asap-cost's held_memory test calls optimize_physical_asap, so its asap-physical-asap-optimization dev-dependency now enables reference-pricer; cargo check -p asap-cost --all-targets builds.

…se (#1054) (#1145) pass1/pass2/deployment_capabilities become crate-private; the crate root re-exports the boundary items once each, and test_support re-exports the internals other crates' tests call. asap_aware_cse is crate-private. computation_candidates_for_intent and the unused pub items become crate-private or test-only. split_shared_by_timing moves from asap-types to Stage 2, its only production caller.

) (#1146) Move DagOfLogicalMemoGroups and PerComputationLogicalMemoCandidates, with the types Pass 1's output holds (ComputationTarget, FusedComputation, HavingEntry, HavingPlacement, HowToComputeAggregateMeasure), from asap-logical-asap-optimization into asap_types::stages, next to the other stage-boundary types. Callers import them from asap_types::stages; the logical crate does not re-export them. Hash-consing stays in the logical crate, so DagOfLogicalMemoGroups::from_roots becomes the free function generate_dag_of_logical_memo_groups; the type gets a constructor DagOfLogicalMemoGroups::new for it. No behavior change.

…3's Selection (#1047 P8) (#1147) stage_pipeline's reference_pricing block now uses Selection's field names: selected -> selected_candidate, costs -> cost_per_candidate, accepted -> accepted_accuracy_per_candidate, rejected -> pruned_candidates (entry id -> candidate), search_rejected -> pruned_candidates_not_enumerated, rejected_by_cause -> pruned_candidate_count_by_cause. The viewer's selection table takes the same names and still reads archived documents' older Stage 3 section; the dead runner-up and editor rejection reads are gone.

…c warning (#1047, #1054) (#1148) Crate headers (logical, physical, per_computation_candidates) and design docs (planner-layering, asap-aware-mapping architecture/contracts, joint-planning implementation plan) match the code after #1133-#1147. cargo doc --workspace --no-deps prints no warnings, with or without --all-features (69 + 2 fixed). Documentation only.

…e facade's slower planning (#1150) test(devtools): snapshot harness reads Stage 3's Selection; regenerate snapshots The corpus and sketch plan snapshot harness still read the old stage3_plan_selection keys (selected, costs, accepted, search.candidate_connections). Since the selection became DagOfSelectedCandidates, stage_pipeline writes the selected plan as Stage 2 candidate P1 with cost, accepted_accuracy_per_query and search_report, so a regeneration wrote every corpus snapshot with a null plan and every sketch workload failed with status null. - plan_snapshot reads P1, cost, accepted_accuracy_per_query. - proven_optimal: search_report.status is optimal and no pane width is pending (planning_report.pane_widths_pending). - hit_time_limit sums planning_report's Pass 1 and per-round times. Snapshot changes against the committed ones: - o11y_bench q02, q06, q22, q23, q26: proven_optimal false -> true. Same plan and cost; the facade now searches every pane-width round up to Complete, where the search proves the plan optimal. - fuzz-issue903-raw-retained: KLL shown at the size that runs (k 345 -> 567, failure probability 1e-3 -> 1e-7, from Pass 1 sizing); cost +1.8e-10 relative from charging working memory for the longest reader. - quantile-single-mvp: KLL shown at the size that runs (k 52 -> 86, failure probability 3e-5, from Pass 1 sizing); status optimal -> deadline_reached. Same cost. - shared-quantiles-1h-mvp: status optimal -> deadline_reached. Same plan and cost. The facade's slower planning is accepted (plans are unchanged): - A corpus query may carry a time_limit_reason; reaching the time limit then records and compares its plan instead of failing. o11y_bench q05 has one: it now reaches the 240 s limit (16 s before) with the same plan and cost. - A sketch workload may carry a deadline_reached_reason; it may then end deadline_reached (after every pane width), and its plan, cost and status are still compared. quantile-single-mvp and shared-quantiles-1h-mvp have one: their MILP stalls numerically about 8e-7 relative below the cost.

zzylol and others added 10 commits October 30, 2026 11:06

refactor: say producer, not supplier, across crates (#1095 §5) (#1100)

241aa60

Claude-Session: https://claude.ai/code/session_01K1M6bqhAk7WA4MZcfbAoMw Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

zzylol added optimizer Workload optimization and plan selection refactor labels Oct 30, 2026

zzylol mentioned this pull request Oct 30, 2026

test: area A tests pass on the stage-boundaries branch (types → physical, executor) #1109

Merged

zzylol mentioned this pull request Oct 30, 2026

test: area B tests pass on the stage-boundaries branch; fix decoded-output validation (plan-selection, planner) #1110

Merged

zzylol mentioned this pull request Oct 30, 2026

test: area C tests on the stage-boundaries branch; reference enumeration applies Stage 2's raw-data rule #1111

Merged

zzylol mentioned this pull request Oct 30, 2026

fix: quantile operand count survives memo sharing; Pass 1 deadline reports the planning budget #1113

Merged

zzylol and others added 4 commits October 30, 2026 14:52

zzylol and others added 2 commits October 30, 2026 16:55

zzylol and others added 23 commits October 30, 2026 19:32

refactor(accuracy): sized_for_failure_probability is private (#1054) (#…

a394883

…1131) No caller outside crates/accuracy/src/sizing.rs uses it; its two in-file callers stay public. split_shared_by_timing stays in asap-types: Stage 2's assigned_candidate calls it on the realizer path.

Merge origin/main into refactor/stage-boundaries

6e3552f

zzylol changed the title ~~refactor: stage boundaries milestone 1 — Stages 0–2 and Stage 3's frozen interface (#1047, #1054, #1095)~~ refactor: stage boundaries milestone 1 — Stages 0–2 and Stage 3's frozen interface (#1047, #1054) Oct 31, 2026

zzylol merged commit 91d874f into main Oct 31, 2026
4 checks passed

This was referenced Nov 2, 2026

devtools: print a stage_pipeline document as text #1158

Closed

test: run an out-of-tree MetricsQL corpus from $ASAP_EXTERNAL_CORPUS_DIR #1162

Merged

zzylol deleted the refactor/stage-boundaries branch November 3, 2026 02:16

Navigation Menu

Repository navigation

refactor: stage boundaries milestone 1 — Stages 0–2 and Stage 3's frozen interface (#1047, #1054) - #1108

refactor: stage boundaries milestone 1 — Stages 0–2 and Stage 3's frozen interface (#1047, #1054)#1108
zzylol merged 42 commits into
mainfrom
refactor/stage-boundaries

zzylol commented Oct 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

zzylol commented Oct 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Included PRs

How

Key code interfaces added / modified

Major algorithm changes

Merge gates

Plan diffs (22 snapshot workloads: 12 corpus queries, 10 sketch workloads)

Retro-check (67 runs per mode, integration vs main, 30 s budget)

Accepted Stage 3 losses (until #1095)

Waived planning-time threshold

X2 sizing note

Before this PR

After this PR

Verification

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

zzylol commented Oct 30, 2026 •

edited

Loading