Skip to content

refactor: one crate per stage (#1207 PR 0b) - #1229

Merged
zzylol merged 5 commits into
mainfrom
refactor/1207-crates-per-stage
Nov 10, 2026
Merged

zzylol merged 5 commits into
mainfrom
refactor/1207-crates-per-stage

Conversation

@zzylol

@zzylol zzylol commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Merge unit. PR 0b of #1207's merge unit (PRs 0–9, with 0b). The PRs merge into main together, after PR 9. This PR only moves code between crates (no logic changes) and deletes the tests that crossed stage crates (decided 2026-11-06); every other test passes or fails exactly as on PR 0.

Summary

  • Problem: [Refactor] Crate organization by stage: per-crate functionality, cross-stage code into asap-accuracy / asap-cost / asap-types, enforced crate boundaries, public vs private #1054's crate rules (stage crates never depend on each other; shared code in asap-types, asap-accuracy, asap-cost; each crate exposes only what others use; the facade calls the stages in order, checked by the crate-graph test) were not enforceable: Stage 0 and both Stage 1 passes were one crate, asap-logical-asap-optimization, kept apart only by a source-scanning test (pass_layering.rs).
  • Change: one crate per stage and pass (asap-stage0-memo, asap-stage1-pass1-per-computation-candidates, asap-stage1-pass2-reuse-candidates, and the renamed asap-stage2-physical-optimization, asap-stage3-plan-selection), moved with git mv; crate_graph.rs now also forbids stage-to-stage and stage-to-facade [dev-dependencies], so tests that crossed stage crates are deleted (79 on PR 0's head be6ea1d6a, plus pass_layering.rs's 2).
  • Effect: PRs 1–8 land in their final crates. Example: Pass 2 calling into Pass 1 was a source-scan failure; now it does not compile (no dependency edge), and crate_graph (6 tests) fails on any new edge. Test status: 1,062 passed / 191 failed / 9 ignored, every failure at a PR 0 todo!() except one stale fixture.

Review guide

  • Position: base main, after refactor: define every stage's interfaces from #1207 and delete the ones they replace (#1207 PR 0) #1226 (merged as 828a2aaae); refactor(types): properties and requirements from #1207; chains as time-indexed state (#1207 PR 1) #1227 (PR 1) builds on it. Head c715a25c8. Diff vs base: 95 files, +886 / −2,926.
  • Restack (2026-11-10): rebuilt on main after refactor: define every stage's interfaces from #1207 and delete the ones they replace (#1207 PR 0) #1226's file layout (R1–R3, L1–L6, E1–E3). The same four commits, now in the scope layout: Pass 1's entry (pass1_memo, Pass1Error) and Pass 2's (generate_logical_asap_memo_candidates and its two steps) are defined in their crate's lib.rs (L5, L6; no logical_memo_candidates.rs), and Pass 1's target code is computation_targets.rs with computation_targets_tests.rs. Tests: same results by name as the previous head a4290eeef (backup backup/1207-layout-crates-per-stage-a4290eeef).
  • Scope layout (2026-11-10 review, last commit c715a25c8): each stage crate's lib.rs holds the crate doc, the entry function and the re-exports; the other files are by scope (table under Interfaces, "Stage crate layout"). Pass 2's skeleton is PR 4's: the stubs of the old rules 2–7 that PR 4 replaces are deleted (see Decided). Moves and renames otherwise; no logic change.
  • Reading order: the root Cargo.toml members and each new crate's Cargo.toml; the three new crate roots (crates/stage0-memo/src/lib.rs, crates/stage1-pass1-per-computation-candidates/src/lib.rs, crates/stage1-pass2-reuse-candidates/src/lib.rs); integration-tests/tests/crate_graph.rs; planner/src/run.rs (paths only).
  • Mechanical: every file move (history follows with git log --follow); path rewrites inside moved files (crate::pass1::X → crate::X, crate::pass1::computation_targets → crate::computation_targets, crate::stage0::memo_of_queries → asap_stage0_memo::memo_of_queries in Pass 1 tests only; Pass 2's pass2.rs becomes the crate root, so memo::rules resolves from it); module visibility in the new roots (pub(crate) mod → mod); docs' crate names and paths.
  • Careful: the deleted tests (tables under Verification) and what coverage PR 3 must re-create; the crate_graph.rs rule; the dev-dependencies removed.

Behavior: before → after

No planning behavior changes: planning still reaches PR 0's first todo!() (Stage 0's memo_of_queries, filled in by PR 2). The design need is the crate boundary.

Before (PR 0) After (this PR)
Stage 0, Pass 1, Pass 2 modules of asap-logical-asap-optimization asap-stage0-memo, asap-stage1-pass1-per-computation-candidates, asap-stage1-pass2-reuse-candidates
Stage 2, Stage 3 asap-physical-asap-optimization, asap-plan-selection asap-stage2-physical-optimization, asap-stage3-plan-selection (content unchanged)
Pass 2 importing Pass 1 caught by a source-scanning test does not compile; crate_graph.rs forbids the edge
A stage crate's test calling another stage or the facade allowed ([dev-dependencies] not checked) forbidden, no exceptions
Pass 1 → asap-frontend-promql (optional, test-support) an allowed exception gone with lower_promql

Crate graph ([dependencies] edges between workspace crates; frontend helpers, devtools and integration-tests omitted). Before:

graph LR
  planner[asap-planner] --> logical[asap-logical-asap-optimization]
  planner --> physical[asap-physical-asap-optimization]
  planner --> ps[asap-plan-selection]
  planner --> fe[frontends]
  planner --> executor[asap-executor]
  logical --> types[asap-types]
  logical --> accuracy[asap-accuracy]
  logical -. optional, test-support .-> fepromql[asap-frontend-promql]
  physical --> types
  ps --> types
  ps --> accuracy
  ps --> cost[asap-cost]
  accuracy --> types
  cost --> types
  executor --> types
  fe --> types
Loading

After:

graph LR
  planner[asap-planner] --> s0[asap-stage0-memo]
  planner --> p1[asap-stage1-pass1-per-computation-candidates]
  planner --> p2[asap-stage1-pass2-reuse-candidates]
  planner --> s2[asap-stage2-physical-optimization]
  planner --> s3[asap-stage3-plan-selection]
  planner --> fe[frontends]
  planner --> executor[asap-executor]
  s0 --> types[asap-types]
  p1 --> types
  p1 --> accuracy[asap-accuracy]
  p2 --> types
  p2 --> accuracy
  s2 --> types
  s3 --> types
  s3 --> accuracy
  s3 --> cost[asap-cost]
  accuracy --> types
  cost --> types
  executor --> types
  fe --> types
Loading

Interfaces

Only crate paths change, and Pass 2's entry is PR 4's (below); no other function body, type or signature changes. Permalinks at the head c715a25c8.

logical-asap-optimization/src/lib.rs (base, deleted) → stage0-memo/src/lib.rs · stage1-pass1-per-computation-candidates/src/lib.rs · test_support.rs · stage1-pass2-reuse-candidates/src/lib.rs

 // crate asap-logical-asap-optimization (deleted)
-pub use stage0::memo_of_queries;
-pub use pass1::{pass1_memo, Pass1Error, PerComputationCandidateError};
-pub use pass2::{
-    generate_logical_asap_memo_candidates, pass2_after_local_window_composition,
-    pass2_local_window_composition,
-};
-pub use pass2::memo::rules::Pass2RuleError;
-#[cfg(any(test, feature = "test-support"))]
-pub mod test_support;
 // crate asap-stage0-memo
+/// Stage 0: the memo groups of the workload's query roots, no candidates (body `todo!()`, PR 2).
+pub fn memo_of_queries(
+    roots: &[QueryRoot],
+    accuracy_targets: &[Option<AccuracyTarget>],
+    data: &DataWorkload,
+) -> Result<Memo, SchemaDerivationError>;
 // crate asap-stage1-pass1-per-computation-candidates
+/// Why `pass1_memo` failed (moved from the old crate's `pass1.rs`, unchanged).
+pub enum Pass1Error {
+    PerComputation(PerComputationCandidateError),
+}
+/// Stage 1, Pass 1 (moved from `pass1.rs`; body `todo!()`, PR 3).
+pub fn pass1_memo(
+    memo: Memo,
+    supported_summaries: Option<&[SupportedSummary]>,
+) -> Result<Memo, Pass1Error>;
+/// A query's DAG Pass 1 cannot plan.
+pub use computation_targets::PerComputationCandidateError;
+/// Pass 1 internals other crates' tests call (feature `test-support`).
+#[cfg(any(test, feature = "test-support"))]
+pub mod test_support {
+    pub use crate::computation_targets::generate_per_computation_candidates;
+}
 // crate asap-stage1-pass2-reuse-candidates
+/// Pass 2's entry points (moved from the old crate's `pass2.rs`; signatures as in PR 0).
+pub fn generate_logical_asap_memo_candidates(
+    pass1: Memo,
+    entries: &[QueryWorkloadEntry],
+    evidence: &dyn AccuracyEvidence,
+) -> Result<LogicalASAPMemoCandidates, Pass2RuleError>;
+pub use rules::Pass2RuleError;
 // asap-planner
-pub use pass::PlanningModels;
+pub use models::PlanningModels;

Pass 2's steps, inside the crate (todo!() bodies until PR 4; pass2_local_window_composition and pass2_after_local_window_composition are not public any more, and the developer tools show one Pass 2 step):

-// pass2/memo/window_candidates.rs, rules 1–7 in order
-pub(crate) fn add_local_window_composition(memo: &mut Memo, entries: &[QueryWorkloadEntry]) -> Result<(), Pass2RuleError>;
-pub(crate) fn add_scan_label_candidates(memo: &mut Memo) -> Result<(), Pass2RuleError>;
-pub(crate) fn factor_summary_inputs(memo: &mut Memo) -> Result<(), Pass2RuleError>;
-pub(crate) fn add_topk_candidates(memo: &mut Memo) -> Result<(), Pass2RuleError>;
-pub(crate) fn add_finer_aggregate_candidates(memo: &mut Memo) -> Result<(), Pass2RuleError>;
-pub(crate) fn add_rollup_candidates(memo: &mut Memo) -> Result<(), Pass2RuleError>;
-pub(crate) fn add_window_rollup_candidates(memo: &mut Memo) -> Result<(), Pass2RuleError>;
+// rules/rollup.rs, step 1: rollup from the finest common group
+pub(crate) fn add_rollup_candidates(memo: &mut Memo) -> Result<(), Pass2RuleError>;
+// rules/topk.rs, step 2: top-k reuse
+pub(crate) fn add_topk_candidates(memo: &mut Memo) -> Result<(), Pass2RuleError>;
+// steps 3–4 (readout requirements, dead-candidate elimination): PR 4's requirements.rs, dead_candidates.rs
+// rules/chains.rs, step 5: chain groups for the surviving state groups
+pub(crate) fn add_chain_groups(
+    memo: &mut Memo,
+    entries: &[QueryWorkloadEntry],
+    pruned: &BTreeSet<CandidateForMemoGroupId>,
+) -> Result<(), Pass2RuleError>;

types/src/test_support.rs

 // asap_types::test_support (feature `test-support`)
+/// The maintained-population sub-DAG for executor tests (an IR fixture, no stage involved);
+/// was `asap_logical_asap_optimization::test_support::population_over_selector`.
+pub fn population_over_selector(
+    root: &Rc<OperatorNode>,
+    statistic: crate::ir::operator::maintained_population::PopulationStatistic,
+    grouping: &[&str],
+) -> Rc<OperatorNode>;

asap-stage2-physical-optimization and asap-stage3-plan-selection expose the same items as asap-physical-asap-optimization and asap-plan-selection. asap_planner::{DeploymentCapabilities, SupportedSummary} re-export from asap_types::deployment_capabilities (moved there by PR 0). The stage-document JSON keys (stage1_logical_asap_optimization, stage2_physical_asap_optimization, stage3_plan_selection) are a document format, not crate names, and are unchanged; so are the viewer and the snapshots.

The old crate's test_support.rs, split by caller:

Item Called by After
generate_per_computation_candidates (re-export of computation_targets) tests of executor, devtools, integration-tests asap_stage1_pass1_per_computation_candidates::test_support (feature test-support)
lower_promql only deleted tests (Stage 0's and Pass 1's tests that lowered PromQL) deleted; Pass 1 no longer depends on the PromQL frontend
share_common_sub_dags (re-export) — deleted in PR 0 with cse.rs; asap-stage0-memo has no test-support feature
population_over_selector executor tests asap_types::test_support
metric_scan, metric_scan_with_keys, scan, scan_from Pass 1's own tests only stage1-pass1-per-computation-candidates/src/test_fixtures.rs (#[cfg(test)], not exported)
cms_summary, weighted, estimate_over only the deleted cross-stage tests deleted (in PR 0, P11)
PerComputationCandidates (re-export) nobody outside the crate dropped from the re-exports
hll_classic_failure_statistic, aggregate, agg, without_agg, filter, dedup, time_range, entry, entry_with nobody (dead since PR 0) deleted

File → crate map (folder names are crate names without asap-):

Before After
logical-asap-optimization/Cargo.toml split into the three new Cargo.tomls, each with only its own dependencies (asap_sketchlib, serde, rmp-serde of the old crate were unused)
logical-asap-optimization/src/lib.rs deleted; each new root re-exports its entry points and holds its crate docs (the accuracy paragraph went to Pass 1)
src/stage0.rs stage0-memo/src/lib.rs (src/stage0/cse.rs deleted in PR 0)
src/pass1.rs (pass1_memo, Pass1Error); src/pass1/{computation_targets, computation_targets_tests, realization}.rs stage1-pass1-per-computation-candidates/src/lib.rs (the entry is defined there); computation_targets.rs, tests.rs, summary_realization.rs (only Pass 1 calls it)
src/pass2.rs (the entry functions); src/pass2/memo.rs, memo/rules.rs, memo/rules/{finer_aggregate,rollup,scan_labels,summary_inputs,topk}.rs, memo/window_candidates.rs stage1-pass2-reuse-candidates/src/lib.rs (the entry is defined there); rules.rs, rules/{rollup,topk}.rs, rules/chains.rs (was memo/window_candidates.rs); memo.rs and the finer_aggregate, scan_labels and summary_inputs stubs deleted
src/deployment_capabilities.rs already types/src/deployment_capabilities.rs on PR 0 (moved there 2026-11-07, same content)
tests/pass_layering.rs deleted: the crates now cannot import each other, and crate_graph.rs checks it
physical-asap-optimization/** stage2-physical-optimization/**; src/catalog/mod.rs → src/lib.rs (src/catalog/realize.rs deleted in PR 0)
plan-selection/** stage3-plan-selection/**; src/select.rs → src/lib.rs (src/models.rs deleted in PR 0)
planner/src/pass.rs (PlanningModels) planner/src/models.rs

Stage crate layout (2026-11-10)

Files a later PR creates are named with it; this PR only moves what exists.

Crate Files
stage0-memo lib.rs (memo_of_queries)
stage1-pass1-per-computation-candidates lib.rs (pass1_memo, Pass1Error), computation_targets.rs, candidates.rs (PR 3), summary_realization.rs (was realization.rs), tests.rs (was computation_targets_tests.rs), test_fixtures.rs, test_support.rs
stage1-pass2-reuse-candidates (no memo/ level) lib.rs (generate_logical_asap_memo_candidates, its steps in order), rules.rs (Pass2RuleError; PR 4's shared helpers), rules/rollup.rs, rules/topk.rs, rules/chains.rs (was memo/window_candidates.rs), requirements.rs and dead_candidates.rs (PR 4), tests (PR 4)
stage2-physical-optimization (no catalog/) lib.rs (generate_physical_asap_memo_candidates, was catalog/mod.rs), candidates.rs and pruning.rs (PR 5), tests.rs
stage3-plan-selection lib.rs (select_plan, was select.rs), milp/model.rs, milp/execution_contexts.rs and milp/objective.rs (PR 6–7), plan.rs (PR 6), report.rs, tests
planner lib.rs, run.rs, pass.rs (OptimizationPass, PassRegistry, optimize, check_contract), models.rs (PlanningModels, moved out of pass.rs), installable.rs

Usage, end to end (crates/planner/src/run.rs; the flow is unchanged, only the paths):

// `roots`: the workload's lowered query roots, one per query, from the frontends.
// `memo`: Stage 0's groups, read by Pass 1 (body `todo!()` until PR 2).
let memo = asap_stage0_memo::memo_of_queries(&roots, &targets, data)?;
// `pass1`: the memo with each group's initial candidates, read by Pass 2.
let pass1 = asap_stage1_pass1_per_computation_candidates::pass1_memo(memo, supported)?;
// `logical`: reuse rules and logical edges, which Stage 2 expands.
let logical = asap_stage1_pass2_reuse_candidates::generate_logical_asap_memo_candidates(pass1, &entries, evidence)?;
// Stage 2 and Stage 3 as before, under their new crate names; the user gets PlanOutput { plan }.

Algorithm changes

None.

Decisions and open questions

Needs reviewer decision: None.

Decided:

  • 2026-11-10 (review of this PR, comments on finer_aggregate.rs and scan_labels.rs): Pass 2's skeleton is PR 4's from this PR on. PR 4 replaces the old rules 2–7 with its steps: rollup from the finest common group (which covers finer-aggregate reuse, rollup and the pairwise scan-label matching), top-k reuse, readout requirements, dead-candidate elimination and chain groups. So the stubs of finer aggregate, scan widening, summary inputs and window rollup are deleted here, with pass2_local_window_composition and pass2_after_local_window_composition. Scan widening across label sets is not kept; it is tracked in Share one scan of a source across readers with different predicates, labels and time ranges (subsumption derivations) #951. Summary inputs need no rule: Stage 0 already makes every sub-expression its own group, and Pass 1 (PR 3) builds each summary over the aggregate's input group with the value expression in the build's SummaryUpdate, a pushed-down key-only filter becoming its own Filter group. OriginOfCandidateForMemoGroup::{SummaryInput, WidenedScans} stay in asap-types until PR 4 deletes them. add_chain_groups (step 5) carries #[allow(dead_code)] until PR 4 calls it after dead-candidate elimination.
  • 2026-11-10: the stage crates are organized by scope (table under Interfaces): each lib.rs holds the crate doc, the entry function and the re-exports; Stage 2 drops catalog/, Stage 3's select.rs becomes lib.rs, PlanningModels gets its own models.rs.
  • 2026-11-10 (refactor: define every stage's interfaces from #1207 and delete the ones they replace (#1207 PR 0) #1226 review, L5/L6): each pass's entry function lives in its crate's lib.rs; Pass 1's target code is computation_targets.rs. This PR is restacked onto main after refactor: define every stage's interfaces from #1207 and delete the ones they replace (#1207 PR 0) #1226's file layout; it changes no content beyond the crate split.
  • 2026-11-06 (Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 order): each stage and pass becomes its own crate before any stage logic is filled in, so PRs 1–8 land in their final crates.
  • 2026-11-06: tests that cross stage crates are deleted, not rewritten; PR 3 re-creates Pass 1 coverage against the new interfaces. Rejected: rewriting them now against interfaces whose bodies are todo!().
  • MetricsQL → PromQL (test-only) dev-dependency deleted, not kept: keeping it would need an exception or a new role for the frontends in crate_graph.rs. Its only users were external_corpus.rs (via PromQL's tests/support.rs, deleted for calling Pass 1) and vector_negation_lowers_as_in_promql.
  • summary_realization.rs (was realization.rs) stays in Pass 1 (only Pass 1 calls it, though it was documented as shared). If Pass 2's rules (PR 4) need it, it moves to asap-types (IR helpers) rather than Pass 2 depending on Pass 1.
  • One module added: stage1-pass1-per-computation-candidates/src/test_fixtures.rs (#[cfg(test)]), holding the fixtures only Pass 1's own tests call, so the crate does not export them. No new abstraction.

Verification

cargo test -p <crate> --all-features --no-fail-fast at the head; cargo fmt --check and cargo clippy --workspace --all-targets --all-features -- -D warnings clean; crate_graph: 6 passed. Newly failing vs base (PR 0), by test name: 0. Against PR 0 the only differences are the deleted cross-stage tests and the moved test paths (pass1::… → crate root); no test changed result. Every failure panics at a PR 0 todo!(), except the stale fixture (PR 9).

Scope Passed / failed / ignored Newly failing vs base Notes
asap-types 153 / 2 / 0 0 reject the invalid combination of section 2 (PR 1): 2
asap-accuracy 59 / 0 / 0 0
asap-cost 31 / 1 / 0 0 θ from the exported node (PR 6): 1
asap-frontend-common 49 / 0 / 0 0
asap-sql-function-catalog 10 / 0 / 0 0
metricsql_common 3 / 0 / 0 0
asap-frontend-promql 217 / 0 / 0 0 cross-stage tests deleted
asap-frontend-metricsql 15 / 0 / 0 0 cross-stage tests deleted
asap-frontend-sql 173 / 0 / 0 0 cross-stage test deleted
asap-executor 210 / 63 / 1 0 θ from the exported node (PR 6): 32; Hydra grid sizes are θ (PR 6): 13; retention_ms is $h_v$ (PR 6): 8; weighted-frequency θ (PR 6): 7 + 2; Stage 0 (PR 2): 1
asap-planner 8 / 18 / 1 0 Stage 0 (PR 2): 18
asap-devtools 37 / 56 / 3 0 Stage 0 (PR 2): 54; assertion failed: output.status.success(): 2
asap-integration-tests 83 / 51 / 4 0 Stage 0 (PR 2): 47; retention_ms (PR 6): 3; stale old-wire fixture (PR 9): 1; crate_graph 6 passed
asap-stage3-plan-selection 3 / 0 / 0 0
asap-stage0-memo 0 / 0 / 0 —
asap-stage2-physical-optimization 0 / 0 / 0 —
asap-stage1-pass2-reuse-candidates 0 / 0 / 0 —
asap-stage1-pass1-per-computation-candidates 11 / 0 / 0 0
Total 1062 / 191 / 9 0

Scope-layout check (head c715a25c8 vs cadf5d579): cargo test -p asap-stage1-pass1-per-computation-candidates -p asap-stage1-pass2-reuse-candidates -p asap-stage2-physical-optimization -p asap-stage3-plan-selection -p asap-planner -p asap-devtools --no-fail-fast: 59 passed / 74 failed / 4 ignored on both, the same tests by name; no test covered the deleted stubs (their bodies were todo!()), so none is deleted; cargo check --workspace --tests with no warnings, cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --check clean.

Restack check (2026-11-10, head cadf5d579 vs previous head a4290eeef): cargo test -p asap-types -p asap-accuracy -p asap-cost -p asap-stage0-memo -p asap-stage1-pass1-per-computation-candidates -p asap-stage1-pass2-reuse-candidates -p asap-stage2-physical-optimization -p asap-stage3-plan-selection --no-fail-fast: 258 passed / 3 failed / 0 ignored on both, the same tests by name (the workload doctests moved with workload/query.rs); crate_graph: 6 passed; cargo check --workspace --tests with no warnings, cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --check clean. The other crates were not rerun for the restack (their changes are import paths). The per-crate table below is from the previous head.

The move itself was checked test by test before the rebases: cargo test --workspace --no-fail-fast on PR 0 (dab3d064f) and on this PR's move commit (80b5b09b8) gave every test the same result by name, except pass_layering.rs's 2 deleted tests. On each deletion commit (dfc78f40d group 1, f6e2daa49 group 2, 4d5537310 group 3), cargo test was re-run for the crates it changes; each remaining test had its PR 0 result. Per crate then (PR 0 at dab3d064f → this PR): asap-types 200/2/0 → 201/2/0; asap-frontend-promql 221/6/1 → 218/0/0; asap-frontend-metricsql 18/0/0 → 15/0/0; asap-frontend-sql 175/0/0 → 174/0/0; asap-logical-asap-optimization 95/13/0 → (stage0-memo 24/0/0, Pass 1 12/1/0, Pass 2 0/0/0); asap-integration-tests 102/76/4 → 103/76/4; total 1259/221/10 → 1195/203/9; every other crate unchanged. The old crate's 95/13/0: 24/0/0 stayed in Stage 0 and 12/1/0 in Pass 1; 1/0/0 moved to asap-types (default_supports_everything_and_json_fills_the_rest); 56/12/0 deleted as cross-stage tests (groups 1 and 2); 2/0/0 deleted with pass_layering.rs. The frontends lost 7/5/1 (group 2) and 0/1/0 (group 3).

crate_graph.rs tests: optional_only_exception_rejects_the_edge_made_non_optional deleted (with the only_if_optional field, no optional edge left); checker_ignores_dev_dependencies_and_external_crates → checker_ignores_external_crates_and_non_stage_dev_dependencies; added checker_rejects_a_dev_dependency_between_stage_crates (a stage crate's test may not call another stage crate) and checker_rejects_a_frontend_dev_dependency_on_the_facade (nor the facade). The three new crates get the Stage role; the two renamed keep it.

Dev-dependencies removed: asap-cost on the logical and physical crates (no test used them since PR 0 deleted tests/held_memory.rs); asap-planner on the logical crate with test-support; after the deletions, Pass 1 on asap-stage0-memo and asap-frontend-promql, Stage 0 on Pass 1, asap-frontend-promql and asap-frontend-sql on asap-stage0-memo and Pass 1, asap-frontend-metricsql on asap-stage0-memo, Pass 1 and asap-frontend-promql; group 3: asap-frontend-promql and asap-frontend-sql on asap-planner (SQL's had no user), asap-frontend-promql on asap-accuracy. Pass 2 depends on asap-accuracy (PR 0 gave its entry the &dyn AccuracyEvidence). The metrics_observability test target is removed from frontend-promql/Cargo.toml (docs/develop_docs/metrics-observability-corpora.md says so; its fixtures stay).

Deleted tests

82 tests deleted as cross-stage (63 passed, 18 failed, 1 ignored on PR 0 as of 2026-11-06), plus pass_layering.rs's 2. On PR 0 7708f3668 they were 79 (17 passed, 61 failed, 1 ignored; PR 0 renamed asap_input_and_invalid_accuracy_are_rejected to asap_input_is_rejected and deleted count_min_estimate_over_raw_data_samples_is_gated_on_the_metric). Since 2026-11-09 PR 0 deletes Pass 1's whole-workload composition (P11), so 36 of the tests below are deleted there, not here: a_quantile_estimate_shares_its_raw_data_samples_sign_only_inside_the_unit_interval, a_quantile_of_merged_tumbling_windows_shares_its_raw_data_samples_sign, aggregate_over_topk_candidates_compose, aggregate_replacement_preserves_sql_field_identity, avg_states_out_of_order_are_refused, benchmark_corpora_are_total_and_keep_their_coverage, binding_is_total_over_the_entire_corpus, cms_count_updates_total_ten_for_zero_positive_and_negative_samples, composed_avg_divides_sum_and_count_states, composed_candidate_replaces_chosen_target_with_summary_evaluation, composed_filters_and_having, composed_guarantees, composed_mixed_exact_and_sketch_measures_keep_output_names, composed_multi_measure_aggregate_joins_its_measures, corpus_lowers_and_binds, count_ranked_topk_fuses_to_one_global_heap_sketch, frequency_count_candidates_use_unit_weights, grouped_avg_composition_preserves_the_whole_schema, grouped_count_offers_hydra_but_selection_keeps_per_group_state, heap_weights_over_raw_data_samples_record_an_open_requirement, kubernetes_mixin_aggregates_over_topk_compose, multidimensional_ranked_item_is_a_structured_tuple, nested_sum_merges_into_one_exact_accumulator, normalized_additive_errors_compose_as_absolute_errors, own_having_subquery_is_substituted, per_series_avg_with_series_identity_composes_from_sum_and_count, per_series_rate_keeps_labels_and_retypes_its_state, subpopulation_columns_are_not_part_of_the_ranked_item_tuple, temporal_average_divides_with_a_finite_division_guard, temporal_per_entity_topk_uses_series_identity_and_sample_value, topk_keeps_known_labels_beside_series_identity, uncalibrated_frequency_evaluations_do_not_bypass_accuracy_targets, values_estimating_raw_data_samples_name_their_sign_by_the_metric, whole_expression_topk_over_a_grouped_sum_of_range_sums_reads_the_raw_data_samples, whole_expression_topk_partitions_index_the_heap_input, whole_expression_topk_replaces_the_heap_over_the_exact_sum. The tables keep the original lists for the record (results measured on PR 0 as of 2026-11-06).

(1) Stage 0 ↔ Pass 1 (54 tests). Pass 1 takes Stage 0's Memo, so every Pass 1 test that built its input with memo_of_queries goes.

File Tests PR 0 result What they covered
stage0-memo/src/lib.rs (was stage0.rs), mod tests repeated_results_share_groups_but_quantile_levels_do_not, equal_keyless_deterministic_results_share_one_group, scalar_and_literal_roots_keep_workload_positions (3) 3 passed memo_of_queries over PromQL lowered through Pass 1's lower_promql: equal results share a group, different quantile levels do not; G2 / G2 is one group read twice; scalar and literal roots keep their positions. (cse.rs and its 24 tests were deleted in PR 0.)
stage1-pass1-per-computation-candidates/src/computation_targets_tests.rs target discovery and input checks (6): scalar_root_producers_are_discovered_once, example_1_exact_q1_keeps_exact_accumulators_only, asap_input_and_invalid_accuracy_are_rejected, multi_measure_aggregate_has_one_target_per_measure, aggregate_without_realization_is_not_a_target, filtered_measure_keeps_candidates_and_records_its_filter; HAVING placement (11): key_only_having_is_pushed_below_and_keeps_candidates, value_having_stays_above_output_and_keeps_candidates, having_on_one_consumer_of_a_shared_aggregate_stays_above, a_scalar_reference_consumer_keeps_a_key_only_having_above, only_the_adjacent_of_stacked_filters_is_having, global_aggregate_having_stays_above, scalar_reference_filter_is_not_having, promql_scalar_operand_is_not_having, having_shared_with_a_project_consumer_stays_above, without_and_per_entity_having_stays_above, memo_cached_rules_preserve_occurrence_specific_having; composing a chosen candidate (10): composed_candidate_replaces_chosen_target_with_summary_evaluation, composed_filters_and_having, composed_avg_divides_sum_and_count_states, composed_multi_measure_aggregate_joins_its_measures, composed_mixed_exact_and_sketch_measures_keep_output_names, composed_guarantees, own_having_subquery_is_substituted, per_series_avg_with_series_identity_composes_from_sum_and_count, per_series_rate_keeps_labels_and_retypes_its_state, grouped_avg_composition_preserves_the_whole_schema — 27 21 passed, 6 failed (composed_candidate_replaces_chosen_target_with_summary_evaluation, composed_filters_and_having, composed_guarantees, composed_mixed_exact_and_sketch_measures_keep_output_names, composed_multi_measure_aggregate_joins_its_measures, own_having_subquery_is_substituted) Frontend-to-Pass-1 acceptance: which computation targets a query has, where a HAVING goes, and the DAG a chosen candidate composes to. The file's 4 tests of computation_candidates_for_intent stay.
stage1-pass1-per-computation-candidates/src/computation_targets.rs, mod tests fused heaps and whole-expression top-k (8): whole_expression_topk_replaces_the_heap_over_the_exact_sum, whole_expression_topk_partitions_index_the_heap_input, whole_expression_topk_over_a_grouped_sum_of_range_sums_reads_the_raw_data_samples, heap_weights_over_raw_data_samples_record_an_open_requirement, grouped_range_sums_a_raw_data_samples_heap_cannot_rank_get_no_fused_heap, a_fused_computation_absorbs_what_its_absorbed_targets_merges_absorb, combinations_skip_choices_a_whole_expression_candidate_absorbs, count_ranked_topk_fuses_to_one_global_heap_sketch; nested-sum merges (4): nested_sum_merges_into_one_exact_accumulator, nested_sum_merge_needs_a_sum_of_sums, nested_sum_merge_never_groups_by_the_dropped_metric_name, nested_sum_merges_do_not_stack; (2) avg_states_out_of_order_are_refused, temporal_average_divides_with_a_finite_division_guard; ranked items (4): multidimensional_ranked_item_is_a_structured_tuple, subpopulation_columns_are_not_part_of_the_ranked_item_tuple, temporal_per_entity_topk_uses_series_identity_and_sample_value, topk_keeps_known_labels_beside_series_identity; raw-data-samples signs (3): a_quantile_estimate_shares_its_raw_data_samples_sign_only_inside_the_unit_interval, a_quantile_of_merged_tumbling_windows_shares_its_raw_data_samples_sign, values_estimating_raw_data_samples_name_their_sign_by_the_metric; (1) guarantee_composition::normalized_additive_errors_compose_as_absolute_errors; (2) stream_norms::a_norm_follows_the_update_weight, stream_norms::the_absolute_bound_names_the_weighted_norm — 24 19 passed, 5 failed (the two quantile-sign tests, normalized_additive_errors_compose_as_absolute_errors, both stream_norms tests) Pass 1's fused and whole-expression candidates, merges, ranked items, and the guarantees and signs of its estimates. Of the module's other 18 tests, 14 are deleted in group (2); 4 stay, with realization.rs's 5.

Helpers deleted with them: in computation_targets_tests.rs composed, scan, aggregate_over, aggregate, always, exact_promql, choices_of, assert_sql_count_choices, quantile, grouped_quantile, grouped_count, having_filter, equals, having_of, above_scalar, columns, without_output_aliases, summary_agg; in computation_targets.rs's tests inventory, inventory_root, inventory_at, open_inventory, heap_weight_domains, composed_with, heap_states, topk_target, enumerated_choices, nested_sum_target, fused_heap, heap_update, and the emptied mod stream_norms; in test_fixtures.rs cms_summary, weighted, estimate_over.

(2) Frontends → Stage 0 / Pass 1, Pass 1 → PromQL frontend, MetricsQL → PromQL (27 tests).

File Tests PR 0 result What they covered
frontend-promql/tests/count_planning.rs grouped_count_offers_hydra_but_selection_keeps_per_group_state, frequency_count_candidates_use_unit_weights, cms_count_updates_total_ten_for_zero_positive_and_negative_samples, logical_candidates_fails_on_a_candidate_that_does_not_compose, aggregate_over_topk_candidates_compose (5) 2 passed (logical_candidates_fails_on_a_candidate_that_does_not_compose, aggregate_over_topk_candidates_compose), 2 failed (frequency_count_candidates_use_unit_weights, cms_count_updates_total_ten_for_zero_positive_and_negative_samples), 1 ignored (grouped_count_offers_hydra_but_selection_keeps_per_group_state) Pass 1 candidates of PromQL count queries: Hydra for grouped counts, unit weights for frequency counts, CMS update totals, composition failures surfacing, aggregate-over-topk composing. The file's 4 tests that lower or plan through the facade stay.
frontend-promql/tests/univmon_candidates.rs (file deleted) uncalibrated_frequency_evaluations_do_not_bypass_accuracy_targets (1) failed UnivMon candidates are rejected by selection without calibration.
frontend-promql/tests/observability/promql_corpus.rs binding_is_total_over_the_entire_corpus (1) failed Pass 1 binding never errors over the Prometheus docs and testdata corpora. lowering_is_total_over_the_entire_corpus stays.
frontend-promql/tests/observability/metrics_observability.rs (file and test target deleted) benchmark_corpora_are_total_and_keep_their_coverage, kubernetes_mixin_aggregates_over_topk_compose (2) 1 failed, 1 passed Coverage ratchets of ASAP candidates per corpus; Kubernetes-mixin aggregates over topk compose through Pass 1.
frontend-sql/tests/frequency_l2.rs stage1_offers_exact_and_summary_l2_candidates (1) passed Pass 1 offers exact and summary candidates for the SQL L2 intent. The file's lowering tests stay.
frontend-metricsql/tests/external_corpus.rs (file deleted) corpus_lowers_and_binds, floor_line (2) 2 passed The MetricsQL external corpus lowers and binds through Pass 1 (skipped without the corpus directory); floor_line tested only that test's parsing helpers.
frontend-metricsql/tests/lowering_semantics.rs vector_negation_lowers_as_in_promql (1) passed MetricsQL vector negation lowers to the same IR as PromQL's (the only MetricsQL → PromQL use left).
stage1-pass1-per-computation-candidates/src/computation_targets.rs, mod tests aggregate_replacement_preserves_sql_field_identity, raw_data_samples_of_metric_ends_at_any_other_operator, classic_hll_confidence_needs_one_instance_over_a_sources_own_rows; guarantee_composition::{absolute_errors_compose_through_arithmetic, products_and_quotients_of_estimates_have_closed_or_symbolic_bounds, relative_and_rank_errors_compose_only_where_sound, clamp_and_abs_keep_the_largest_argument_error, comparisons_over_estimates_have_no_guarantee, a_scalar_subquery_inherits_its_guarantee, failure_probability_counts_once_per_distinct_estimate, join_beside_an_exact_input_on_exact_columns_keeps_the_guarantee, join_on_an_approximate_key_has_no_guarantee, case_on_exact_columns_keeps_the_branch_guarantee, projection_over_a_join_reads_each_columns_guarantee} (14) 13 passed, 1 failed (classic_hll_confidence_needs_one_instance_over_a_sources_own_rows) Pass 1's guarantee composition over PromQL-lowered estimates, raw-data-samples requirements, HLL confidence, SQL field identity through aggregate replacement. The emptied mod guarantee_composition (helpers with_bound, abs, rel, lit, arith, call, of, bound, delta, close, join, column, equals, case_over, project) is deleted.

Helpers deleted with them: lower_promql (Pass 1's test_fixtures.rs); node, node_over, scalar, approximate_and_exact (Pass 1's tests); logical_candidates, local_computation_candidates, LogicalCandidate, require_composed_operator_candidates, logical_plan (PromQL's tests/support.rs); reads_hydra (count_planning.rs); bind_tally, BindTally (promql_corpus.rs). MetricsQL's tests no longer include PromQL's support.rs.

(3) Frontends → the facade (1 test).

File Tests PR 0 result What they covered
frontend-promql/tests/count_planning.rs exact_counts_select_count_accumulators (1) failed (Pass 1's todo!()) count(up), count by(job)(up) and count_over_time(up[5m]) at an exact target select a count accumulator through asap_planner::run_planning. The file's 2 fixture tests of lowered count aggregates stay.

Helpers deleted with it: frontend-promql/tests/planner_support.rs (run_all_stages, selected_dag; file deleted); physical_asap_dag and declared_data in PromQL's tests/support.rs (no caller left). No SQL or MetricsQL test used asap-planner.

End-to-end evidence: not applicable (move-only; planning stops at PR 0's Stage 0 todo!()).

Self-review: over-abstraction none added; duplication none; dead code none (clippy -D warnings).

Follow-ups

@zzylol
zzylol force-pushed the refactor/1207-crates-per-stage branch 2 times, most recently from 84cf80d to 69fed95 Compare November 7, 2026 20:51
@zzylol
zzylol force-pushed the refactor/1207-crates-per-stage branch 3 times, most recently from b768e8d to 0e0f75a Compare November 9, 2026 03:33
@zzylol
zzylol force-pushed the refactor/1207-interfaces branch from e9d9f5a to 2ed6ec0 Compare November 9, 2026 04:25
@zzylol
zzylol force-pushed the refactor/1207-crates-per-stage branch from 0e0f75a to 058ea42 Compare November 9, 2026 04:31
@zzylol
zzylol force-pushed the refactor/1207-interfaces branch from 2ed6ec0 to 61541f7 Compare November 10, 2026 13:37
@zzylol
zzylol force-pushed the refactor/1207-crates-per-stage branch from 058ea42 to a4290ee Compare November 10, 2026 13:37
@zzylol
zzylol changed the base branch from refactor/1207-interfaces to main November 10, 2026 14:57
@zzylol
zzylol dismissed milindsrivastava1997’s stale review November 10, 2026 14:57

The base branch was changed.

Move-only. Split asap-logical-asap-optimization into asap-stage0-memo,
asap-stage1-pass1-per-computation-candidates and
asap-stage1-pass2-reuse-candidates; rename asap-physical-asap-optimization
to asap-stage2-physical-optimization and asap-plan-selection to
asap-stage3-plan-selection. The old test_support splits by caller. The
crate-graph test checks the new crates; no stage crate depends on another.
Pass 1's tests that build Stage 0's memo and Stage 0's tests that lower
PromQL through Pass 1 are deleted, with the dev-dependencies and helpers
only they used. lower_promql moves to Pass 1's test fixtures, so Pass 1
no longer depends on the PromQL frontend outside its own tests. The
crate-graph test now also checks dev-dependencies between stage crates.
…1207 PR 0b)

Frontend tests that compose Pass 1 candidates and Pass 1 tests that
lower PromQL through the frontend are deleted, with the helpers,
dev-dependencies and test targets only they used. MetricsQL's test that
compares with PromQL's lowering goes too, so no frontend tests depend on
another frontend. The crate-graph test checks dev-dependencies between
stage crates with no exceptions.
…R 0b)

The PromQL test that planned through asap-planner is deleted with its
planner_support helpers; the PromQL and SQL frontends drop their
asap-planner dev-dependency, and PromQL its unused asap-accuracy one.
The crate-graph test now also forbids a stage crate's dev-dependency on
the facade.
@zzylol
zzylol force-pushed the refactor/1207-crates-per-stage branch from a4290ee to cadf5d5 Compare November 10, 2026 15:01
@zzylol
zzylol marked this pull request as ready for review November 10, 2026 15:02
@zzylol zzylol self-assigned this Nov 10, 2026
#1207 PR 0b)

- Pass 1: realization.rs -> summary_realization.rs;
  computation_targets_tests.rs -> tests.rs.
- Pass 2: no memo/ level. rules.rs (Pass2RuleError), rules/rollup.rs,
  rules/topk.rs, rules/chains.rs (was memo/window_candidates.rs).
  generate_logical_asap_memo_candidates runs PR 4's steps in order:
  rollup from the finest common group, top-k reuse, then readout
  requirements, dead-candidate elimination, chain groups and the logical
  edges (todo!() until PR 4). The finer-aggregate, scan-label, summary-input
  and window-rollup stubs are deleted, and so are
  pass2_local_window_composition and pass2_after_local_window_composition;
  the developer tools show one Pass 2 step.
- Stage 2: catalog/mod.rs -> lib.rs.
- Stage 3: select.rs -> lib.rs.
- Planner: PlanningModels moves from pass.rs to models.rs.
@zzylol
zzylol merged commit 6bcf398 into main Nov 10, 2026
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Workload optimization and plan selection refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants