Skip to content

refactor(stage1-pass1): per-computation candidates over memo groups (#1207 PR 3) - #1232

Merged
zzylol merged 7 commits into
mainfrom
refactor/1207-pass1
Nov 11, 2026
Merged

zzylol merged 7 commits into
mainfrom
refactor/1207-pass1

Conversation

@zzylol

@zzylol zzylol commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: on main, Pass 1 stops at todo!(): target discovery walked operator-node trees, and candidates came from a whole-workload node-tree composition sized per budget share; neither fits the memo model (concepts doc §3: "Everything that was computed on operator-node trees is computed on groups").
  • Change: Pass 1 adds each group's initial candidates (exact, exact accumulator, summary with one candidate per θ, fused) to the memo, every intermediate result its own group and every candidate one operator over input groups, with no input requirement or accuracy floor. Its tests build their memo through the Memo methods, so no stage crate depends on another, even in tests.
  • Effect: for the concepts doc's running example (query A = p95 of latency, query B = p99 / p50), pass1_memo goes from todo!() to a memo of 14 groups: a KLL state group with 7 builds (k ∈ {26, 52, 132, 269, 547, 1404, 2863}), a DDSketch state group with 7 (α 0.1 … 0.001), and six readout groups. Planning now stops at Pass 2's first todo!() (PR 4).

Review guide

  • Position: PR 3 of the Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 stack (row "3. Pass 1" of Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207's Order). Base main (PRs 0, 0b, 1, 2 and 2b are merged); refactor(stage1-pass2): rollup from the finest common group, readout requirements, dead-candidate elimination, chain groups (#1207 PR 4) #1234 (PR 4) builds on it. Seven commits: the restacked PR 3 (321a1ae8d), the review fixes (343e0a863), build-pending-then-add (888512995), the round-3 review fixes (9aa66fe41), a test case (25e6f5267), a docs fix (1eb415cc5) and the C1–C11 review fixes (7bd4546d2, head).
  • Reading order (all in crates/stage1-pass1-per-computation-candidates/src/):
    1. lib.rs: pass1_memo, PerComputationCandidateError, the group accessors expression and schema.
    2. computation_targets.rs: HowToComputeAggregateMeasure, ComputationTarget, FusedComputation, computation_targets, computation_candidates_for_intent, then the fused targets add_whole_expression_targets, add_nested_distributive_targets, nested_distributive_input, whole_expression_input.
    3. candidates/mod.rs: add_pass1_candidates (split_having, key_only_having) → add_aggregate_candidates (start_way, single_measure_part) → add_measure_way / add_fused_candidate(memo, aggregate_group, result_group, ..); unless_unsupported.
    4. candidates/pending.rs: Pending (push, schema), commit_pending, add_state_group / add_group / add_candidate.
    5. candidates/measure_output.rs: measure_output, summary_readout (state, estimate, ranked rows; shared with the fused heap), accumulator_build, add_output_candidate, renaming_project, ranked_as_topk, sql_avg, join_measures.
    6. summary_realization.rs: summary_candidates, measure_input, counted_column, summary_update, summarised_input, ranked_value, realize_keyed_additive_summary_input, summed_per_series_samples.
    7. tests.rs, test_fixtures.rs.
    8. types: ir/schema/topk.rs (the top-k unique-key rule), stages/logical/group.rs (ranked_column_of_group, raw_data_samples_of_metric_of_group; tests in group/tests.rs), summary_type/statistic.rs (raw_data_samples_of_metric), stages/logical/memo/mod.rs (Memo::output_schema_of_pending_expression).
    9. accuracy: guarantee/sizing_oracle.rs (test-only; now also the HLL precision helpers), guarantee/hll.rs (the classic confidence model is test-only).
  • Mechanical: deletions (node-tree composition, Pass1Error, the test-support features of Pass 1 and asap-accuracy, HavingEntry/HavingPlacement, types' per_computation_logical_memo_candidates.rs); integration-test renames; the accuracy sizing oracle (a pure move); docs.
  • Careful: HAVING push-down per consumer Filter group; the plain exact candidate beside the accumulator; fused builds and their fused_groups; renamed readouts and open schemas (series identity); the top-k unique-key rule; the pending → check → commit order of every way, and the candidate's schema check before commit; the HAVING conjunct split into one Filter above, and which column-free conjuncts go below; nested_aggregates (a renaming Project and the __name__ rule).

Behavior: before → after

The concepts doc's running example (§1; test running_example_gives_one_build_per_theta_and_separate_readouts): query A = p95 of latency, query B = p99 / p50 of it. It serves the design need that all ways to compute a group are memo candidates Stage 3 can choose among.

Before (main): pass1_memo panics at todo!("#1207 PR 3: …"). After, the memo (printed by the test at 888512995):

G0: Scan
  [0] Scan inputs=[]
G1: TimeRange
  [0] TimeRange inputs=[G0]
G2: Aggregate[Quantile]
  [0] Aggregate[Quantile] inputs=[G1]
  [1] Project inputs=[G7]
  [2] Project inputs=[G9]
G3: Aggregate[Quantile]
  [0] Aggregate[Quantile] inputs=[G1]
  [1] Project inputs=[G10]
  [2] Project inputs=[G11]
G4: Aggregate[Quantile]
  [0] Aggregate[Quantile] inputs=[G1]
  [1] Project inputs=[G12]
  [2] Project inputs=[G13]
G5: BinaryOp
  [0] BinaryOp inputs=[G3, G4]
G6: SummaryAgg[Sketch(Kll)]
  [0] SummaryAgg[Sketch(Kll)] inputs=[G1] θ=Kll { k: 26 }
  [1] SummaryAgg[Sketch(Kll)] inputs=[G1] θ=Kll { k: 52 }
  [2] SummaryAgg[Sketch(Kll)] inputs=[G1] θ=Kll { k: 132 }
  [3] SummaryAgg[Sketch(Kll)] inputs=[G1] θ=Kll { k: 269 }
  [4] SummaryAgg[Sketch(Kll)] inputs=[G1] θ=Kll { k: 547 }
  [5] SummaryAgg[Sketch(Kll)] inputs=[G1] θ=Kll { k: 1404 }
  [6] SummaryAgg[Sketch(Kll)] inputs=[G1] θ=Kll { k: 2863 }
G7: SummaryEstimate[Quantile { q: 0.95 }]
  [0] SummaryEstimate[Quantile { q: 0.95 }] inputs=[G6]
G8: SummaryAgg[Sketch(DDSketch)]
  [0] SummaryAgg[Sketch(DDSketch)] inputs=[G1] θ=DDSketch { alpha: 0.1 }
  [1] SummaryAgg[Sketch(DDSketch)] inputs=[G1] θ=DDSketch { alpha: 0.05 }
  [2] SummaryAgg[Sketch(DDSketch)] inputs=[G1] θ=DDSketch { alpha: 0.02 }
  [3] SummaryAgg[Sketch(DDSketch)] inputs=[G1] θ=DDSketch { alpha: 0.01 }
  [4] SummaryAgg[Sketch(DDSketch)] inputs=[G1] θ=DDSketch { alpha: 0.005 }
  [5] SummaryAgg[Sketch(DDSketch)] inputs=[G1] θ=DDSketch { alpha: 0.002 }
  [6] SummaryAgg[Sketch(DDSketch)] inputs=[G1] θ=DDSketch { alpha: 0.001 }
G9: SummaryEstimate[Quantile { q: 0.95 }]
  [0] SummaryEstimate[Quantile { q: 0.95 }] inputs=[G8]
G10: SummaryEstimate[Quantile { q: 0.99 }]
  [0] SummaryEstimate[Quantile { q: 0.99 }] inputs=[G6]
G11: SummaryEstimate[Quantile { q: 0.99 }]
  [0] SummaryEstimate[Quantile { q: 0.99 }] inputs=[G8]
G12: SummaryEstimate[Quantile { q: 0.5 }]
  [0] SummaryEstimate[Quantile { q: 0.5 }] inputs=[G6]
G13: SummaryEstimate[Quantile { q: 0.5 }]
  [0] SummaryEstimate[Quantile { q: 0.5 }] inputs=[G8]

Mapping to the doc's table: G1 here is the doc's G1 (latency, via Scan G0 and TimeRange G1); G6 = doc G2 (KLL state, one build per θ of KLL's list; the doc's k = 200/345 are illustrative); G8 = doc G2′ (DDSketch); G2/G3/G4 = doc G3/G4/G5, each with its exact aggregate and a renaming Project over a readout group per algorithm (G7/G9, G10/G11, G12/G13); G5 = doc G6. The doc shows G4/G5 reading only G2′: that is the metric rule, which drops the KLL → ratio edges in PR 4.

What Pass 1 now does:

  • Target discovery over memo groups. computation_targets(memo, supported_summaries) reads each group's expression and group_properties and finds one ComputationTarget per measure of each Aggregate group with a realization besides plain exact execution, once however many queries read it.
  • Each way to compute a group becomes candidates through Memo::find_or_add_group and Memo::add_candidate: exact (every group's own expression, no θ); exact accumulator (a state group SummaryAgg[ExactAggregate(kind)] and a FinalizeExactAccumulator over it; avg finalizes its sum and count states and divides them, a PromQL BinaryOp, or for SQL a Join group and a Project); summary (one state group SummaryAgg[Sketch(algorithm)] per supported algorithm with one build per θ in parameter_list(algorithm), read through a separate readout group SummaryEstimate(state)); fused builds (the whole-expression top-k heap over the rows below the absorbed sums, and the nested distributive merge sum∘sum, min∘min, max∘max as one exact build in the outer aggregate's accumulator state group; builds list the absorbed groups in fused_groups); multi-measure aggregates (SQL: one single-measure Aggregate group per measure, joined back on the group keys); HAVING read from the memo (the aggregate's own having split by key_only_having; each parent Filter with a key-only predicate gets candidates that push it below the aggregate; a value HAVING stays a Filter above each readout).
  • Every intermediate result is its own group; each candidate is one operator over input groups. A readout whose column names differ from the group's is its own group, read by a renaming Project. A heap top-k's Float64 estimate is cast to the exact top-k's column type (a SQL count ranks Int64).
  • Series identity. A readout over per-series rows keeps the $promql_series_identity column, so its renaming Project outputs the group's exact (closed) schema. A group whose schema is open gets no renamed readout.
  • A way Pass 1 cannot build is not offered (Unsupported, e.g. a SQL count(*) with no non-null column, or a quantile outside [0, 1]); a candidate that breaks the node rules or the group's output is a bug and fails Pass 1.

Interfaces

Against main.

crates/stage1-pass1-per-computation-candidates/src/lib.rs · computation_targets.rs · Cargo.toml

-/// Why `pass1_memo` failed: a one-variant wrapper.
-pub enum Pass1Error {
-    PerComputation(#[from] PerComputationCandidateError),
-}
 /// Stage 1, Pass 1's entry: adds each group's initial candidates to Stage 0's memo (exact,
 /// exact accumulators, one summary build per listed θ with separate readouts, fused builds).
 /// The planner facade calls it after Stage 0. Was `todo!()`.
 pub fn pass1_memo(
-    memo: Memo,
+    mut memo: Memo,
     supported_summaries: Option<&[SupportedSummary]>,
-) -> Result<Memo, Pass1Error>;
+) -> Result<Memo, PerComputationCandidateError>;
 /// Why Pass 1 could not add a group's candidates. Stays; now defined in `lib.rs` (same public path).
 pub enum PerComputationCandidateError {
     /// Bad input: a query's DAG breaks the node rules (the planner facade's `validate_structure`).
     Structure(#[from] SchemaDerivationError),
     /// Bad input (Stage 0) or internal bug (a group Pass 1 built): not valid over its input groups.
     MemoGroup(#[from] MemoGroupError),
     /// Bad input: a Stage 0 group is already an ASAP operator.
     ContainsAsap,
-    /// A whole-workload choice index out of range.
-    InvalidChoice,
+    /// Internal bug: a candidate Pass 1 built breaks the node rules or does not output its group's result.
+    Candidate(#[from] CandidateError),
+    /// Internal bug: a candidate Pass 1 built has logical properties that cannot go together.
+    LogicalProperties(#[from] LogicalPropertiesError),
     /// A way Pass 1 cannot build; it is not offered (never returned by `pass1_memo`).
     Unsupported(&'static str),
 }
-#[cfg(any(test, feature = "test-support"))]
-pub mod test_support {
-    /// The whole-workload per-computation candidates of `memo`'s node trees.
-    pub fn generate_per_computation_candidates(memo: &Memo)
-        -> Result<PerComputationCandidates, PerComputationCandidateError>;
-}
-[features] test-support = []

Pass1Error: nothing to add over PerComputationCandidateError. InvalidChoice and test_support: there are no whole-workload choices any more.

crates/types/src/stages/mod.rs · stages/logical/group.rs · stages/logical/candidate.rs · stages/per_computation_logical_memo_candidates.rs (deleted)

-/// How one Pass 1 target is computed: exact or a summary. Moves, crate-private, into Pass 1.
-pub enum HowToComputeAggregateMeasure {
-    Exact,
-    ExactAccumulator(Vec<ExactKind>),
-    Summary(SketchAlgorithm),
-}
-/// Where a `HAVING` predicate goes.
-pub enum HavingPlacement {
-    BelowAggregate,
-    AboveOutput,
-}
-/// A recorded `HAVING` predicate of a target.
-pub struct HavingEntry {
-    pub filter: Option<Rc<OperatorNode>>,
-    pub predicate: Predicate,
-    pub placement: HavingPlacement,
-}
-/// One measure of an aggregate group with a realization besides exact execution.
-pub struct ComputationTarget {
-    pub aggregate_group: MemoGroupId,
-    pub measure_index: usize,
-    pub ways_to_compute: Vec<HowToComputeAggregateMeasure>,
-    pub measure_filter: Option<Predicate>,
-    pub having_filters: Vec<HavingEntry>,
-    pub fused_computation: Option<FusedComputation>,
-}
-impl ComputationTarget {
-    pub fn choice_count(&self) -> usize;
-    pub fn computation_for(&self, index: usize)
-        -> Option<(&HowToComputeAggregateMeasure, Option<&[usize]>)>;
-}
-/// A target's measure computed with the targets beneath it.
-pub struct FusedComputation {
-    pub absorbed_targets: Vec<usize>,
-    pub ways_to_compute: Vec<HowToComputeAggregateMeasure>,
-}
-pub use per_computation_logical_memo_candidates::{
-    ComputationTarget, FusedComputation, HavingEntry, HavingPlacement, HowToComputeAggregateMeasure,
-};
+/// The column a top-k reading memo group `group` ranks it by (`ir::schema::topk::ranked_column`
+/// over the group as an input). Pass 1's way to the ranked-column walk without exposing `MemoGroupAsInput`.
+pub fn ranked_column_of_group(memo: &Memo, group: MemoGroupId) -> Result<ColumnId, ResolveError>;
+/// `summary_type::raw_data_samples_of_metric` of memo group `group`: the metric whose raw data
+/// samples, or plain sums of them, are every value the group outputs. Pass 1 reads it for a keyed heap's weights.
+pub fn raw_data_samples_of_metric_of_group(memo: &Memo, group: MemoGroupId) -> Option<String>;
+impl Memo {
+    /// The output schema `expression` would have as a new group, without adding anything: the dry
+    /// run of `find_or_add_group`. `expression` reads memo groups or pending expressions: the input
+    /// `MemoGroupId(n + k)` (n = group count) is `pending[k]`, each with its schema from earlier
+    /// calls. Mirrors `validate_expression_over_input_groups`, minus the merge coverage check,
+    /// which runs when a merge is added. Pass 1 calls it to check a way before adding it.
+    /// Errors as `find_or_add_group` (a bug in the rule that built the expression).
+    pub fn output_schema_of_pending_expression(
+        &self,
+        pending: &[(Operator<MemoGroupId>, Schema)],
+        expression: &Operator<MemoGroupId>,
+    ) -> Result<Schema, MemoGroupError>;
+}
 /// One candidate of a memo group. Stays; only `fused_groups`' doc changes.
 pub struct CandidateForMemoGroup {
     /// $e_v$.
     pub expression: CandidateExpression,
     /// $p^{\log}_v$.
     pub logical_properties: LogicalProperties,
     /// $r^{\log}_{c,i}$, one per input group.
     pub input_requirements: Vec<LogicalRequirements>,
-    /// The groups a fused computation also implements, beyond the group whose candidate it is.
+    /// The groups of the operators a fused candidate replaces in one appearance (a top-k over an
+    /// aggregate as one heap; a sum over a sum as one aggregate); empty otherwise. Reuse across
+    /// appearances is Pass 2's, not fusion (concepts doc §3, "Fused groups").
     pub fused_groups: Vec<MemoGroupId>,
     /// Which rule added it.
     pub origin: OriginOfCandidateForMemoGroup,
 }

HavingEntry, HavingPlacement, measure_filter, having_filters: copies of memo data (the aggregate's having, its parent Filter groups, Aggregate.filters[measure_index]). ComputationTarget, FusedComputation and HowToComputeAggregateMeasure move, crate-private, into Pass 1's computation_targets.rs (absorbed_targets → absorbed_groups: Vec<MemoGroupId>, used as fused_groups; HowToComputeAggregateMeasure::Exact is deleted: plain exact execution is always the group's own expression, so a target lists only ExactAccumulator and Summary ways); only Pass 1 reads them (devtools' computation_label had no caller and is deleted). ranked_column_of_group and raw_data_samples_of_metric_of_group live in the group scope because crates/types/src/ir must not import stages.

crates/types/src/summary_type/statistic.rs · ir/schema/topk.rs · ir/operator/non_asap.rs · test_support/memo_fixtures.rs

+/// The metric whose raw data samples, or plain sums of them (time range, `offset`-only time
+/// shift, plain `sum`), are every value `input` outputs; `None` past any other operator.
+/// The one copy: accuracy's node-tree version is deleted. Re-exported as `asap_types::summary_type::raw_data_samples_of_metric`.
+/// It stays in `statistic.rs` beside `WeightDomain`, whose requirement it serves; PR 5 deletes it
+/// with the weight-domain code.
+pub fn raw_data_samples_of_metric<I: OperatorInput>(input: &I) -> Option<String>;
 /// The column a top-k ranks `child` by; now crate-private (no caller outside types).
-pub fn ranked_column<I: OperatorInput>(child: &I) -> Result<ColumnId, ResolveError>;
+pub(crate) fn ranked_column<I: OperatorInput>(child: &I) -> Result<ColumnId, ResolveError>;
 /// Whether any measure has its own filter; generic over the predicate's child type, so it reads a group's measures.
-pub fn any_measure_filtered(filters: &[Option<Predicate>]) -> bool;
+pub fn any_measure_filtered<C>(filters: &[Option<Predicate<C>>]) -> bool;
-/// (feature `test-support`) `Aggregate(reduction; measure)` over `child`, the measure over the rows `filter` keeps.
-pub fn aggregate_where(reduction: Reduction, measure: AggIntent, filter: Option<Predicate<MemoGroupId>>,
-    child: MemoGroupId) -> Operator<MemoGroupId>;
+/// (feature `test-support`) `Aggregate(reduction; measures)` over `child` with `having`: each measure
+/// unnamed, over the rows its filter keeps; no filter list when none is filtered, as the frontends
+/// write it. `aggregate` and Pass 1's `aggregate_group` build through it.
+pub fn aggregate_with(reduction: Reduction, measures: Vec<(AggIntent, Option<Predicate<MemoGroupId>>)>,
+    having: Option<Predicate<MemoGroupId>>, child: MemoGroupId) -> Operator<MemoGroupId>;

Crate-private additions in topk.rs (the schema derivation fix): item_identity_is_complete(input, item_columns), ranked_rows_unique_keys(columns, value), exact_topk_unique_keys(child, partitions, output); private RankedSource, ranked_source.

crates/accuracy/src/propagation.rs · accuracy/src/lib.rs · accuracy/src/test_support.rs (deleted) · crates/devtools/src/candidate_labels.rs

 pub mod propagation {
-    /// The metric whose raw data samples, or plain sums of them, are every value `node` outputs.
-    pub fn raw_data_samples_of_metric(node: &OperatorNode) -> Option<String>;
 }
-#[cfg(feature = "test-support")]
-pub mod test_support {
-    pub use crate::propagation_stats::PropagationStats;
-    pub fn composed_guarantee(node: &OperatorNode) -> Option<AccuracyGuarantee>;
-    pub fn scalar_guarantee(expr: &ScalarExpr, column: &dyn Fn(usize) -> Option<AccuracyGuarantee>)
-        -> Option<AccuracyGuarantee>;
-    pub fn stream_norm(node: &Rc<OperatorNode>) -> Option<StreamNorm>;
-    pub fn fold_stats(reduction: &Reduction, input: &Rc<OperatorNode>) -> Option<PropagationStats>;
-    pub fn estimates_raw_data_samples_of(node: &OperatorNode) -> Option<String>;
-    pub fn source_distinct_guarantee(/* ... */) -> Option<AccuracyGuarantee>;
-    pub fn cms_depth(delta: f64) -> u32;
-    pub fn kll_rank_error_99(k: u32) -> f64;
-    pub const QUOTIENT_OPERAND_DOMAINS: &str;
-    pub const STREAM_ROW_COUNT: &str;
-    pub fn source_norm_statistic(norm: &str, source: &str) -> String;
-    pub fn raw_data_samples_sign_statistic(metric: &str) -> String;
-    pub fn sign_statistic(stats: &PropagationStats) -> Option<&str>;
-    pub fn estimate_has_bound(family: &FieldDataType, update: &SummaryUpdate, query: &SketchStatistic) -> bool;
-    pub fn estimate_bound_over_raw_data_samples(family: &FieldDataType, update: &SummaryUpdate,
-        query: &SketchStatistic) -> Option<AccuracyGuarantee>;
-}
-[features] test-support = []
 pub mod guarantee::hll {
-    /// The smallest precision whose classic failure probability meets a target (no non-test caller).
-    pub fn min_precision(relative_error: f64, failure_probability: f64, max_distinct: u64, at_least: u8) -> u8;
-    pub fn classic_failure_probability(precision: u8, relative_error: f64, max_distinct: u64) -> f64;
+    /// Now test-only, with `ClassicHllConfidence` and `MAX_CONTRACT_DISTINCT` (only it reaches them).
+    #[cfg(test)]
+    pub(crate) fn classic_failure_probability(precision: u8, relative_error: f64, max_distinct: u64) -> f64;
 }
 // asap_devtools::candidate_labels
-/// A Pass 1 per-computation candidate's label, e.g. "exact", "Sum acc" or "Kll".
-pub fn computation_label(candidate: &HowToComputeAggregateMeasure) -> String;

min_precision moves into the test-only sizing_oracle (its one test with it), as do ClassicHllConfidence's two #[cfg(test)] methods, now free helpers there (classic_hll_precision; guarantee is replaced by classic_guarantee bound with classic_failure_probability) with the two hll tests that used them. classic_failure_probability has no non-test caller, so it and the model only it reaches are #[cfg(test)]; PR 8 uses the formula again and drops the attribute. asap-accuracy's test_support had no caller left (PR 4 deletes the file too); cms_depth and saturating_ceil are now #[cfg(test)], as their only remaining callers are tests. (source_distinct_guarantee's arguments are elided above.)

Usage, end to end (test running_example_gives_one_build_per_theta_and_separate_readouts, run at 888512995):

// What the user writes: query A = p95 of `latency`, query B = p99 / p50 of it.
// `memo`: Stage 0's memo, built as Stage 0 builds it (no source facts here).
let mut memo = Memo::new(&DataWorkload::default());
// `g1`: `latency` (instant selector) with its series identity realized (doc G1); every aggregate reads it.
let g1 = selector(&mut memo, TimeRangeKind::Instant, "latency", &[]);
// `g3`: query A's p95 aggregate group; `g4`, `g5`: B's p99 and p50 groups.
let g3 = aggregate_group(&mut memo, g1, Reduction::by(vec![]), vec![quantile(0.95)], None);
let g4 = aggregate_group(&mut memo, g1, Reduction::by(vec![]), vec![quantile(0.99)], None);
let g5 = aggregate_group(&mut memo, g1, Reduction::by(vec![]), vec![quantile(0.5)], None);
// `g6`: query B's ratio g4 ÷ g5.
let g6 = group(&mut memo, Operator::NonASAP(NonASAPOp::BinaryOp { lhs: g4, rhs: g5, /* ÷ */ .. }));
// The two queries: memo.add_query(MemoQuery { root, accuracy_target: None, number_of_values_per_query_evaluation: None }).
query(&mut memo, g3);
query(&mut memo, g6);
// `memo`: Pass 1's output, as the facade calls it after Stage 0 (`None` = every summary supported).
let memo = pass1_memo(memo, None)?;
// KLL's state group: 7 build candidates, θ = k ∈ {26, 52, 132, 269, 547, 1404, 2863};
// DDSketch's: 7, α ∈ {0.1 … 0.001}; G2/G3/G4: own aggregate + one renamed readout per algorithm;
// G5: its own BinaryOp. Pass 2 (PR 4) adds reuse candidates and the edges, where the
// metric rule drops KLL → B's ratio; Stage 3 picks one producer per input.

Algorithm changes

Pass 1 candidate generation

  • Problem: on main, Pass 1 stops at todo!(). Before PR 0, it built one whole-workload node tree per choice and sized each sketch for an equal budget share; the result was copied into private node chains.
  • Before: targets over node trees → compose_logical_candidate(choice) → node tree per choice → sizing per accuracy_budget_parts → private chains.
  • After:
    1. computation_targets walks memo.groups(). For each Aggregate expression it lists the ways per measure (computation_candidates_for_intent: the exact accumulator first when the intent has one, then the summaries; plain exact is not a way, it is the group's own expression; a measure with no way is not a target), prunes the exact accumulator of an avg over a nullable input and the summaries of a relational count (a target can end with no way of its own and keep its fused computation), detects fused computations over groups (add_whole_expression_targets, add_nested_distributive_targets, close_absorptions; neither applies where has_having), prunes a PromQL top-k's own heaps (the aggregate group's scanned_sources are all time series), and drops unsupported algorithms.
    2. add_pass1_candidates adds every Stage 0 group's own expression as a candidate; then, per aggregate group (its own having split by key_only_having) and per Stage 0 Filter group directly above it with a key-only predicate, add_aggregate_candidates builds each way bottom-up: add_state_group (θ list or one exact build), add_group for each intermediate (the operator is also its group's candidate), and add_output_candidate for the top operator (added to the result group, or, when its column names differ, as its own group read through renaming_project; an open group gets none; under the one non-key HAVING predicate, the candidate is a Filter over it).
    3. Every candidate goes through add_candidate (this crate) → LogicalProperties::new (θ in the list, none on exact operators) → vec![LogicalRequirements::default(); inputs] → Memo::add_candidate.
  • Correctness: every candidate passes the node rules and outputs its group's columns (Memo::add_candidate's exact check); pass1_candidates::{sql,promql}_candidates_output_the_query_columns run every SQL and PromQL query of the old composition test through Pass 1.
  • Complexity: linear in targets × ways; each summary way adds |parameter_list(algorithm)| build candidates (KLL/DDSketch 7, HLL/KMV 4, CMS/Count Sketch 28, UnivMon 40, heap variants 280) to one shared state group, instead of one node tree per whole-workload combination.
  • Assumptions and limits: no accuracy pruning (PR 4). Hydra algorithms have empty lists (Planner: error models and parameter lists for Hydra (and other unsized families), so Pass 1 can offer them #1228) and are not offered.

Build pending, check, then add

  • Problem: three checks need the schema of a group the way itself builds (a heap's top-k estimate must carry the result group's item columns; a per-series avg's sum must be Float64; a renamed output needs a closed group with as many columns). Before, they ran after the way's state and estimate groups were added, so an unsupported way left groups no query reaches.
  • After: each way starts a Pending with the key-only HAVING conjuncts as Filters over the aggregate's input, appends its expressions (Pending::push derives each one's schema once through Memo::output_schema_of_pending_expression; the id n + k names pending expression k), runs every check that can return Unsupported (ranked_as_topk, the per-series avg dtype check, renaming_project, counted_column), and only then adds them (commit_pending, which asserts no group was added meanwhile: a summary build as a state group with its θ candidates, anything else as its own group's candidate) and the candidate. add_output_candidate also pushes the final candidate, checks its schema against the result group's, and pops it before anything is committed, so a mismatch is Unsupported and leaves the memo unchanged. The fused heap now goes through the same summary_readout and add_output_candidate as every summary way (its builds' fused_groups passed through to commit_pending); the nested merge, every measure way, SQL avg and the multi-measure join are built this way too; the join's single-measure groups get their own ways once the join is added. unless_unsupported drops ways that failed before adding anything.
  • Correctness: a_heap_estimate_without_the_groups_columns_adds_nothing, a_readout_an_open_group_cannot_rename_adds_nothing (the memo's groups, candidates and hash-consing index are unchanged), an_unsupported_way_under_a_key_only_having_adds_no_group (SQL count(*) over all-nullable columns under HAVING k > 'a': no group beyond Stage 0's). The per-series avg check is not reachable from a test: a per-series sum of Int64 samples derives Float64, so the way is offered.
  • Complexity: each pending expression's schema is derived once when pushed (rebuilding its earlier pending inputs as temporary builds, a handful), then again when added.

Nested aggregates, HAVING split, multi-measure parts

  • Nested aggregates (computation_targets::nested_aggregates). One matcher for an outer grouped aggregate over the value of a one-measure aggregate beneath it, shared by the nested distributive merge and the keyed heap's summed per-series samples. It reads through a Project that only renames or reorders columns and keeps the key names, so SQL SELECT src, sum(s) FROM (SELECT src, dst, sum(bytes) AS s … GROUP BY src, dst) x GROUP BY src gets the merged sum (its build lists the Project and the inner aggregate in fused_groups); the keyed heap does not read through it. Over a per-series aggregate it never groups by __name__ (see Decided).
  • HAVING split (split_having). The key-only conjuncts of a HAVING (src > 'b' AND p > 3: src > 'b') go below the aggregate as a Filter of its input; the other conjuncts, of the aggregate's own having and of the consumer Filter together, stay one Filter above the output (one BoolAnd when there are several; was a chain of Filters when both had some). It reads ScalarExpr::conjuncts. Before, a mixed HAVING pushed nothing.
  • Which conjuncts go below (key_only_having). A conjunct that reads at least one key column and nothing else, or a column-free one built only of literals and comparison and boolean operators (HAVING 1 = 0 on a grouped aggregate). A column-free conjunct with a function call (random() < 0.5) may be volatile and stays above; the IR has no volatility flag. Was: any column-free conjunct.
  • Multi-measure parts (single_measure_part). Each measure's own aggregate is written as the frontends write one: its name, and a filter list only when it is filtered. SELECT src, count(*), sum(bytes) … GROUP BY src with SELECT src, count(*) … GROUP BY src now share the count(*) group (was a duplicate).
  • PromQL avg as sum and count. Grouped and global avg now check finite division as avg_over_time does (see Decided).

Schema derivation fix (types)

  • Problem: the exact top-k's group derived no unique key over per-series rows while the heap readout's Project carried (job, series identity) from topk_schema_of_build, so topk by (job) (10, sum_over_time(m[1m]))'s heap readout failed add_candidate's exact check.
  • After: one rule for a top-k's ranked rows, exact or a heap's readout: one row per (partition, item) when the item identity is complete (ranked_rows_unique_keys, item_identity_is_complete, exact_topk_unique_keys).
  • Correctness: ir::schema::topk::tests::exact_topk_and_heap_readout_derive_the_same_unique_keys (over operator nodes).

Decisions and open questions

Needs reviewer decision: None.

Decided:

# Item Decision Date
1 Accuracy requirements and floors Pass 1 sets none (θ is structure only; LogicalRequirements { required_accuracy } on readout inputs is Pass 2's). heap_size ≥ k is Pass 2's rule. 2026-11-06 / 11-08
2 Metric rule Per (producer, consumer input) edge: Pass 1 offers KLL and DDSketch readouts to every quantile group; edge building drops KLL → B's ratio (PR 4). 2026-11-06
3 Dropping θ whose error alone exceeds every reaching query's target Needs the queries reaching each edge; Pass 1 keeps the whole list (PR 4). 2026-11-06
4 fused_groups A fused candidate computes several operators of one appearance; fused_groups lists the replaced operators' groups. Rules: top-k heap, sum∘sum, min∘min, max∘max. 2026-11-06 / 11-07
5 Pass 1 reads groups directly expression(memo, g) / schema(memo, g) (crate-private, lib.rs); MemoGroupAsInput stays types-internal; Pass 1 reaches the generic walks through ranked_column_of_group and raw_data_samples_of_metric_of_group. 2026-11-06; 2026-11-11
6 add_candidate's output check Exact (==), as PR 2 defines it. per PR 2 / 2b
7 pass1_memo(memo, supported_summaries) No accuracy targets, model, evidence or data. 2026-11-06
8 Plain exact candidate beside the accumulator Every group offers its own expression; Stage 3 picks the cheaper. 2026-11-07
9 HAVING push-down A key-only consumer Filter's predicate goes below in the Filter group's own candidates. 2026-11-07
10 Nested distributive merge placement A candidate of the outer aggregate's accumulator state group, reading the raw rows. 2026-11-07
11 The SQL avg execution tests sql_avg_from_sum_and_count_executes_as_avg, sql_avg_counts_its_own_input, sql_avg_of_an_int64_column_keeps_the_exact_columns and their helper avg_as_sum_and_count realize a plan from the memo, which is PR 6's: they move to PR 6 (kept outside the repo until its restack). 2026-11-11
12 Review fixes (2026-11-11) ComputationChoices deleted (the ways are a Vec; a quantile outside [0, 1] gets no summary); PerComputationCandidateError in lib.rs; HowToComputeAggregateMeasure crate-private in Pass 1; one raw_data_samples_of_metric, generic over OperatorInput; asap-accuracy's test-support deleted; realize_keyed_additive_summary_input(memo, output_reduction, child) -> Option<(MemoGroupId, SummaryUpdate)> (its Count-Min branch never ran: the one caller asks with Count Sketch); the absorbed-target list in add_nested_distributive_targets deleted (targets follow the memo's group order, so an outer target is never one already absorbed; checked with an assertion over Pass 1, integration, planner and devtools tests before deleting); Reduction::group_keys in sql_avg and join_measures; key_only_having private in candidates.rs; Pass 1's fixtures reuse memo_fixtures (labels are non-nullable there; no Pass 1 test depends on it); integration tests renamed pass1_candidates.rs, pass1_candidates_sql.rs. 2026-11-11
14 Unsupported ways found after groups are added Option (c): derive the schema of each of a way's pending expressions over the memo (Memo::output_schema_of_pending_expression, reusing InputOrTemporaryBuild<MemoGroupAsInput>), check, then add. Rejected: building in a memo clone; a MemoCheckpoint rollback. 2026-11-11
15 __name__ in nested aggregates A grouped aggregate over per-series rows must not group by __name__: the per-series aggregate drops the metric name, the raw rows keep it, so the fused computation over the raw rows would split groups the query has as one. A key whose column cannot be read counts as __name__ (is_none_or, the stricter of the two old checks; the other, is_some_and, accepted it). For valid groups every key column exists, so no plan changes. 2026-11-11
16 Accuracy's test-only sizing Kept as one test oracle, guarantee::sizing_oracle (#[cfg(test)]): the closed-form sizing the predefined lists are checked against (parameter_list_is_the_sizing_over_the_error_grid) and the sizers' own tests; family tests that check guarantees import it. Pure move; 61 accuracy tests, same names. 2026-11-11
17 Finite division for PromQL avg checked_finite_division: true for grouped, global and per-series avg as sum ÷ count: Prometheus' avg aggregation, like avg_over_time, switches to an incremental mean when the running sum overflows, so sum / count matches it only for finite operands and quotient. 2026-11-11
18 Round-3 layout candidates.rs split into candidates/{mod, pending, measure_output}.rs; the summary-input helpers in summary_realization.rs; Memo::output_schema_of_pending_expression as the dry run of find_or_add_group; raw_data_samples_of_metric stays in statistic.rs beside WeightDomain (PR 5 deletes it with the weight-domain code). 2026-11-11
19 Review fixes C1–C11 (2026-11-11) HowToComputeAggregateMeasure::Exact deleted (C1); one non-key HAVING Filter above (C2); Pending holds (expression, schema) pairs (C3); the fused heap shares summary_readout and add_output_candidate, with the schema check before commit (C4); add_fused_candidate takes ids, the non-aggregate branch is unreachable! (C5); stale names in docs (C6); group-scope tests next to group.rs (C7); the Pass 2-dependent SQL test moves to PR 4 (C8); column-free conjuncts go below only without function calls (C9); stale Stage 0 names in docs (C10); HLL test-only code in the sizing oracle (C11). No Pass 1 memo of the review workloads changes. 2026-11-11
13 PerComputationCandidateError::Structure Kept: the planner facade returns it from QueryRoot::validate_structure before Stage 0 (crates/planner/src/run.rs). 2026-11-11

Deleted (replaced by the memo candidates): HowToComputeAggregateMeasure::Exact and prune_own (it only re-inserted Exact), the != Exact guards and unreachable! arms, add_output_candidate's Filter chain, ClassicHllConfidence::{guarantee, precision} (now sizing_oracle helpers), compose_logical_candidate, compose, compose_roots, rewrite, substitute, substitute_measure, with_names_of, whole_expression, nested_sum, Sizing, ChosenComputation, Chosen, PerComputationCandidates (with combinations, absorbed_choice_varies, without_summaries_no_query_can_use), output_can_meet_target, sources_read, absorbed_targets, generate_per_computation_candidates, logical_node_trees, ComputationChoices, PhysicalSummaryInput, summarised_column (inlined into summarised_input), candidates_by (inlined into summary_candidates), unless_unsupported's unit form, memo_fixtures::aggregate_where, Pass 1's metric_schema/scan_operator fixtures; the executor's dev-dependencies on Stage 0 and Pass 1.

Files

Scope File Lines Holds
Pass 1 crate lib.rs 102 pass1_memo, PerComputationCandidateError, expression, schema
computation_targets.rs 686 HowToComputeAggregateMeasure, ComputationTarget, FusedComputation, target discovery, fused targets, nested_aggregates
candidates/mod.rs 509 add_pass1_candidates, AggregateWays, SingleMeasure, add_aggregate_candidates, unless_unsupported, add_measure_way, add_fused_candidate
candidates/pending.rs 186 Pending, commit_pending, add_group, add_state_group, add_candidate
candidates/measure_output.rs 555 measure_output, summary_readout, accumulator_build, add_output_candidate, renaming_project, ranked_as_topk, last_as_float64, sql_avg, join_on_keys, join_measures, key_items
summary_realization.rs 473 which algorithms serve an intent, what a measure's state reads and each row's update, the keyed heap rule
tests.rs 1391 Pass 1's tests (incl. summary realization)
test_fixtures.rs 57 memo builders over memo_fixtures
integration tests/pass1_candidates.rs, tests/pass1_candidates_sql.rs 523, 208 frontends → Pass 1
accuracy (test-only) guarantee/sizing_oracle.rs 676 closed-form sizing the predefined lists are checked against; HLL precision helpers

Verification

cargo fmt --all --check, cargo clippy --workspace --all-targets -- -D warnings: clean. cargo doc: no warning in a changed file (Pass 1 has none). grep -rn "use crate::stages" crates/types/src/ir: empty. Workspace cargo test --workspace --no-fail-fast compared by test name with main (bd4fe8d9a):

Scope main This PR Newly failing Notes
asap-stage1-pass1-per-computation-candidates 11 / 0 / 0 37 / 0 / 0 0 all Pass 1 tests in tests.rs
asap-accuracy 60 / 0 / 0 61 / 0 / 0 0 +1 moved from Pass 1; 10 sizer tests and 3 HLL precision tests (2 from hll, 1 from guarantee) moved into guarantee::sizing_oracle
asap-types (lib) 142 / 0 / 0 148 / 0 / 0 0 +6: the top-k unique-key test, the pending-schema Memo method (2), ranked_column_of_group, raw_data_samples_of_metric(_of_group) (2)
asap-integration-tests 83 / 51 / 4 103 / 46 / 4 0 4 HAVING tests now pass; a_filter_or_group_key_over_an_estimate_has_no_guarantee (fails at Pass 2's todo!) moved to PR 4
every other crate unchanged unchanged 0
Workspace 1154 / 189 / 9 1207 / 184 / 9 0

New tests and what they check:

  • a_quantile_outside_the_unit_interval_offers_no_summary: a quantile outside [0, 1] lists no way (plain exact is its own expression).
  • whole_expression_topk_heap_candidates: the fused heap, through the shared readout path, gives the top-k group one ranked Project per heap over its estimate group, only the state builds list fused_groups, and no other group (the memo printout is identical to 1eb415cc5's).
  • key_only_having_pushes_keys_and_constants_but_no_function_call (IR-level: random() < 0.5 stays above, 1 = 0 and key conjuncts go below, nothing for a global aggregate) and a_constant_having_of_a_grouped_aggregate_is_pushed_below.
  • a_heap_estimate_without_the_groups_columns_adds_nothing, a_readout_an_open_group_cannot_rename_adds_nothing, an_unsupported_way_under_a_key_only_having_adds_no_group: an unsupported way leaves the memo unchanged.
  • a_multi_measure_aggregates_part_is_the_equal_stage0_group, sql_sum_of_a_subquerys_sum_is_one_sum_over_the_rows, mixed_having_pushes_its_key_conjunct_below (was mixed_key_and_value_having_stays_above), promql_avg_divides_sum_and_count_vectors (now also checks finite division for grouped, global and per-series avg): each fails at 888512995 and passes here.
  • an_algorithm_without_the_readout_is_not_offered: offered algorithms follow the real capability table (replaces the injected-table test; preference_order_names_each_algorithm_once deleted).

Pass 1 memos of the review workloads (temporary dump, deleted), groups / candidates, before (888512995) → after. At 7bd4546d2 every dump (every group and candidate) is identical to 1eb415cc5's, with 0 orphan groups and 0 candidate children whose group does not list the candidate's group as a parent:

Workload Before After
example1, example2, example3a, example3b 13/576, 19/70, 33/103, 7/21 same
promql_avg, promql_misc, promql_nested, promql_topk, promql_quantile 27/30, 62/879, 15/22, 22/1147, 9/23 same finite division only changes a flag
sql_multi 39/75 37/71 parts share Stage 0 groups
sql_dedup 12/16 11/14 the count(*) part is the Stage 0 group
sql_having 17/32 24/53 the mixed HAVING pushes src > 'b' below (one shared Filter group)
sql_nested 9/11 9/12 + the merged sum, fused_groups = [Project, inner aggregate]
sql_topk 9/570 same
sql_nullable_having 5/5, orphan G4 4/4, no orphan
  • running_example_gives_one_build_per_theta_and_separate_readouts: KLL and DDSketch state groups hold one build per θ; each quantile reads either through a readout group.
  • the target-discovery, composition, HAVING, fused-heap, ranked-item, nested-merge and intent tests in tests.rs, and pass1_candidates.rs / pass1_candidates_sql.rs (each builds its memo through Memo::find_or_add_group and Memo::add_query, or lowers SQL/PromQL through Stage 0).

Removed from this PR: to PR 6, the three SQL avg execution tests, which failed here at their todo! helper; to PR 4, pass1_candidates_sql::a_filter_or_group_key_over_an_estimate_has_no_guarantee and its helper reads_quantile_sketch, which run every stage and fail at Pass 2's rollup todo! (both kept outside the repo until those restacks). Screenshots and measurements: not applicable.

Follow-ups

@zzylol zzylol added refactor optimizer Workload optimization and plan selection labels Nov 6, 2026
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from 6686f66 to d4bde07 Compare November 6, 2026 23:55
@zzylol
zzylol changed the base branch from refactor/1207-memo to refactor/1207-lineage November 7, 2026 00:00
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from d4bde07 to 441f3a9 Compare November 7, 2026 04:14
@zzylol
zzylol force-pushed the refactor/1207-lineage branch from e2b5456 to 58549f2 Compare November 9, 2026 06:16
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from 2c6d993 to 61f585a Compare November 9, 2026 06:37
@zzylol
zzylol force-pushed the refactor/1207-lineage branch from 58549f2 to b3655e1 Compare November 9, 2026 16:50
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from 61f585a to a71e517 Compare November 9, 2026 16:50
@zzylol
zzylol force-pushed the refactor/1207-lineage branch from b3655e1 to 0479df3 Compare November 10, 2026 01:26
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from a71e517 to d77278a Compare November 10, 2026 01:26
@zzylol
zzylol force-pushed the refactor/1207-lineage branch from 0479df3 to 50686ae Compare November 10, 2026 13:37
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from d77278a to 0c11cd7 Compare November 10, 2026 13:37
@zzylol
zzylol force-pushed the refactor/1207-lineage branch 2 times, most recently from a6eca31 to 03163fd Compare November 10, 2026 22:08
…1207 PR 3)

Pass 1 fills Stage 0's memo with each group's initial candidates, every
intermediate result its own group and every candidate one operator over
input groups:

- computation_targets walks memo.groups(): one target per measure of each
  Aggregate group with a realization besides plain exact execution.
- add_pass1_candidates adds every group's own expression, exact
  accumulators (state group + FinalizeExactAccumulator; avg from sum and
  count), one SummaryAgg state group per sketch algorithm with one build
  candidate per configuration of parameter_list(algorithm) and separate
  SummaryEstimate readout groups, fused builds (whole-expression top-k
  heaps; nested sum/min/max merges) listing the replaced groups in
  fused_groups, multi-measure joins, and key-only HAVING pushed below in
  the consumer Filter group's candidates.
- Pass 1 sets no input requirement and no accuracy floor: every
  candidate's LogicalRequirements are empty; Pass 2 gives each readout its
  required accuracy (a top-k readout's heap_size >= k included).
- ComputationTarget and FusedComputation move into Pass 1 (crate-private),
  FusedComputation::absorbed_groups replaces target indices, HavingEntry,
  HavingPlacement and Pass1Error are deleted; pass1_memo returns
  PerComputationCandidateError.
- Layout: pass1_memo is defined in lib.rs; computation_targets.rs chooses
  the targets, candidates.rs adds their candidates (add_pass1_candidates
  and below).
- types: stages::ranked_column_of_group in the group scope
  (stages/logical/group.rs; ranked_column becomes pub(crate)), so ir does
  not import stages; one unique-key rule for a top-k's ranked rows, exact or a heap readout;
  any_measure_filtered generic over the predicate's input. The top-k
  unique-key test builds memo groups, so it lives in the memo tests.
- Tests build their memo through the Memo methods; the integration tests
  stage1_logical_asap_optimization and per_computation_candidates_sql run
  SQL and PromQL queries through Stage 0 and Pass 1.
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from 0c11cd7 to 321a1ae Compare November 11, 2026 03:35
@zzylol
zzylol changed the base branch from refactor/1207-lineage to main November 11, 2026 03:35
@zzylol
zzylol marked this pull request as ready for review November 11, 2026 13:40
…amples rule, avg plan tests leave

- Remove the three SQL avg execution tests and their todo! helper (they
  realize a plan from the memo, which Pass 1 does not do); rename the
  integration tests pass1_candidates.rs and pass1_candidates_sql.rs.
- HowToComputeAggregateMeasure moves crate-private into Pass 1
  (computation_targets.rs); types' per_computation_logical_memo_candidates.rs
  and devtools' unused computation_label are deleted.
- ComputationChoices is deleted: computation_candidates_for_intent returns
  the ways, a quantile outside [0, 1] gets no summary, prune_own returns
  nothing. PerComputationCandidateError, expression and schema move to
  lib.rs; key_only_having moves into candidates.rs; distributive is private.
- One raw_data_samples_of_metric, generic over OperatorInput
  (asap_types::summary_type), read for a memo group through
  stages::raw_data_samples_of_metric_of_group; accuracy's copy is deleted.
- realize_keyed_additive_summary_input takes no intent or summary type and
  returns (group, update); its Count-Min branch, never reached, is deleted.
- The absorbed-target list in add_nested_distributive_targets is deleted:
  targets follow the memo's group order, so it never matched.
- sql_avg and join_measures read Reduction::group_keys; the plain-exact
  arms of the candidate builders are unreachable!; the multi-measure output
  is not offered when unsupported, like every other way.
- asap-accuracy's test_support module and test-support feature are deleted
  (no callers); cms_depth and saturating_ceil are test-only.
- Pass 1's test fixtures build their expressions with
  memo_fixtures::{scan, aggregate_with}; summary_realization's tests move
  into tests.rs.
- Docs: Pass 1 rows of planner-layering.md, the deleted acceptance tests of
  Example 2, and the Pass 1 test paths.
An unsupported way to compute a group no longer leaves groups in the memo.
Each way's groups are built as pending expressions, checked against the
schemas the memo derives for them, and added only once every check that can
return Unsupported has passed.

- types: stages::output_schemas_of_pending_expressions(memo, pending) derives
  and validates the output schemas of expressions not yet in the memo, over
  memo groups or earlier pending expressions (InputOrTemporaryBuild over
  MemoGroupAsInput); the input MemoGroupId(n + k) names pending[k].
- Pass 1: measure_output, sql_avg, join_measures, last_as_float64 and
  join_on_keys append to a pending list instead of adding groups;
  ranked_as_topk and renaming_project take derived schemas;
  add_output_candidate checks, then commit_pending adds the way. The heap
  top-k (fused and own), the per-series avg and the multi-measure join are
  built this way; unless_unsupported now only drops ways checked before
  they add anything.
- Tests: a heap estimate without the result group's columns and a readout
  an open group cannot rename leave the memo unchanged (groups, candidates,
  hash-consing index).
…sted matcher, HAVING split, sizing oracle

Functional:
- Key-only HAVING filters are pushed pending with each way, so an
  unsupported way adds no Filter group (SQL count(*) over all-nullable
  columns under HAVING k > 'a' left an orphan group).
- A multi-measure aggregate's single-measure parts are written as the
  frontends write one (no filter list when unfiltered), so they share an
  equal Stage 0 group.
- The nested distributive merge reads through a column-renaming Project
  (SQL sum of a subquery's sum fuses); a mixed HAVING pushes its key-only
  conjuncts below and keeps the rest above; PromQL avg as sum/count checks
  finite division for grouped and global avg too.

Code quality and layout:
- Memo::output_schema_of_pending_expression is the dry run of
  find_or_add_group over InputOrTemporaryBuild<MemoGroupAsInput>; Pass 1's
  Pending stores each pending expression's schema once.
- candidates.rs is split into candidates/{mod, pending, measure_output}.rs;
  measure_input, counted_column, summary_update and
  with_raw_data_samples_requirement move to summary_realization.rs.
- One matcher, nested_aggregates, serves the nested merge and the summed
  per-series samples; a key that cannot be read counts as __name__.
- unless_unsupported returns the way's result; candidates_by is inlined;
  memo_fixtures::aggregate_where is deleted.
- asap-accuracy's test-only sizing family is one module,
  guarantee::sizing_oracle, with the tests that check the predefined lists
  against it.
- Tests next to the new types functions; the top-k unique-key test moves
  to ir/schema/topk.rs over operator nodes; stale Pass 1 pruning text and
  issue numbers removed from docs, comments and a devtools test.
Stage 0 is memo_of_queries producing a Memo through Memo::find_or_add_group;
Pass 1 builds one summary per parameter_list configuration with no
accuracy budget shares. Rewrite the example's Pass 1 step with the measured
groups and candidates, fix the code-interface rows, and drop
accuracy_budget_parts from the asap_planner memo output comment.
… filter, shared readout path

- HowToComputeAggregateMeasure::Exact deleted: plain exact execution is
  always the group's own expression, so ways list only exact accumulators
  and summaries; prune_own, the != Exact guards and unreachable! arms go.
- split_having joins all non-key conjuncts (the aggregate's own and its
  consumer Filter's) into one predicate: one Filter above the output.
- key_only_having pushes a column-free conjunct only when it is built of
  literals and comparison or boolean operators (no function call).
- Pending stores (expression, schema) pairs;
  Memo::output_schema_of_pending_expression takes them.
- The fused heap goes through summary_readout and add_output_candidate,
  which checks the candidate's schema on the pending expressions before
  anything is added and passes fused_groups to commit_pending.
- add_fused_candidate takes ids and starts its own Pending; the dead
  non-aggregate branch of add_aggregate_candidates is unreachable!.
- Group-scope tests move next to group.rs; HLL test-only helpers move to
  the sizing oracle; classic_failure_probability and its model are
  test-only; stale names and paths in docs fixed.
- pass1_candidates_sql::a_filter_or_group_key_over_an_estimate_has_no_guarantee
  leaves this PR (it needs Pass 2).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Workload optimization and plan selection refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant