Skip to content

refactor(stage1-pass2): rollup from the finest common group, readout requirements, dead-candidate elimination, chain groups (#1207 PR 4) - #1234

Merged
zzylol merged 35 commits into
mainfrom
refactor/1207-pass2
Nov 12, 2026
Merged

zzylol merged 35 commits into
mainfrom
refactor/1207-pass2

Conversation

@zzylol

@zzylol zzylol commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Closes #1281 (a coarser PromQL aggregate rolls up from a finer one although their scans list different labels).

Summary

  • Problem: on main Pass 2's rules and edges end in todo!(), so every planning run stops there; the old rules matched shapes pairwise by column name, and θ floors per input mixed ε and δ and could not express a multi-readout allocation.
  • Change: Pass 2 (generate_logical_asap_memo_candidates) runs the reuse rules over memo groups (rollup from the finest common group, top-k reuse), gives each readout its required accuracy, eliminates dead candidates with root causes, builds chain groups and the logical edges $(u,c,i)$ with $\alpha^{\log}$ (concepts doc §3 "Stage 1, Pass 2" and §4). The node-tree guarantee composition and propagation are replaced by per-operator error contracts plus readout requirements; Pass 2 also lists, per query, the groups it may read only exactly.
  • Effect: for quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m]) at relative 0.05 every 60 s, Pass 2 prunes both KLL readouts (MetricRule, rank error) and the DDSketch builds with α 0.1 and 0.05, gives each DDSketch readout five levels 0.02 … 0.001, and builds one 60 s DDSketch chain with five builds; planning now stops at Stage 2 (PR 5). The snapshot workload ratio-of-quantiles-1h-mvp planned at 7.481e-06/s with the full stack before this PR's review fixes (D1); it is re-measured when the stack above is restacked.

Review guide

  • Position: PR 4 of the Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 stack (row "4. Pass 2" of Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207's Order); part of the Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 merge unit. Base main (PR 3, refactor(stage1-pass1): per-computation candidates over memo groups (#1207 PR 3) #1232, is merged); refactor(stage2): physical expansion (decision-tree levels 3–4), physical edges, pruned alternatives; Count-Min weights via column_signs (#1207 PR 5) #1235 (PR 5) builds on it. 35 commits, head 9ca4f50a1.
  • Reading order:
    1. stage1-pass2-reuse-candidates/src/lib.rs (the six steps in order), edges.rs, readout_requirements.rs, dead_candidates.rs, evaluation_grids.rs.
    2. rules/rollup/{mod,member,finest_common_group,rewrite}.rs (add_rollup_candidates), rules/topk.rs, rules/chains.rs, rules/candidates.rs (how rules add candidates).
    3. accuracy/src/metric_rule/mod.rs (ErrorMetricsOfMemo: the contracts, allowed metrics and readout metrics of one memo state; exact_only_groups_of_query), logical_edge/mod.rs (edge_meets_logical_requirements, within_limit), error_contract/mod.rs (ErrorContract), readout_caps/mod.rs (the caps walk, readout_levels → ReadoutRequiredAccuracy), readout/mod.rs (readout_error_metric).
    4. types/src/stages/logical/{candidate,memo,group}/mod.rs (θ on pass-through candidates, Memo::producers, Memo::max_values_per_evaluation, same_candidate) and types/src/stages/pruning.rs.
  • Mechanical:
    • Deletion of asap_accuracy::{composition, propagation, propagation_stats, budget, function_rules, requirements, sizing}, guarantee/sizing_oracle.rs, the classic HLL and UnivMon distinct-count guarantees, the unused guarantee vocabulary in types/src/accuracy/guarantee.rs, and the never-set guarantee field of OperatorNode, FlatNode and PhysicalASAPDAGNode (with OperatorNode::with_guarantee; the executor and integration fixtures drop it).
    • File moves: main's accuracy/src/requirements.rs (the edge-check stub, target_limit, meets) is replaced by logical_edge/ (with within_limit); metric_rule/, readout/, readout_caps/ and error_contract/ are new code; binomial_upper_tail into guarantee/binomial.rs; MAX_ERROR_LEVELS is new in readout_caps/; types' group.rs, aggregate_coverage.rs and column_properties.rs into stages/logical/group/, stages::view_coverage into plan::view_coverage (no re-export from stages), the default-metric IR walk from accuracy/target.rs into accuracy/default_metric.rs, test_support.rs into test_support/{mod,forwarders}.rs; every touched source file's inline tests into a sibling tests.rs (x.rs → x/{mod,tests}.rs), hash-consing tests next to hash_consing.
    • Pass 1: tests.rs (51 KB) split by scope into tests/{running_example,series_identity,targets,candidates,having,fused,nested_merges,update_weights,intents,summary_realization,unsupported_ways}.rs; HAVING placement (having_filter_groups, has_having) into candidates/having.rs; computation_candidates_for_intent → ways_to_compute_intent, with_raw_data_samples_requirement → with_raw_data_samples_of; the unused asap-accuracy dependency is dropped.
    • Integration tests named by content: sharing_issue_regressions.rs + identical_exact_queries.rs → plan_sharing.rs with fixtures named by what they hold (plan_sharing_fixtures/), pass2_*_promql.rs → pass2_promql/{rollup,accuracy_and_grids,exact_only_groups}.rs plus pass2_sql.rs, pass1_candidates_sql.rs folded into pass1_candidates.rs, stored_outputs_on_main.json → stored_outputs_before_view_coverage.json, types/tests/physical_export.rs → physical_asap_dag.rs; one shared helper module tests/planning_common/ (fuzz_workload, with_queries, cost, promql_pass2, sql_pass2). PR and issue numbers are removed from code comments.
    • Pass 2's tests split into tests/{rollup/{filters,finest_common_group,measures},topk,readout_requirements,dead_candidates,chains,edges,evaluation_grids,exact_only_groups}.rs plus test_fixtures.rs; the accuracy-math tests sit in the accuracy crate next to their files; shared memo builders in asap_types::test_support::memo_builders (plus compare and arithmetic in memo_fixtures; candidates_of, groups_where and QuantileReadout in memo_builders); devtools drops the two removed origins from its labels.
  • Careful: the Filter, Project and reduction contracts that read column lineage (error_contract); the caps walk and the readout-level grid with D1's extension; the per-value δ split over the readout group's own values (max_values_per_evaluation); NoGuarantee; two-direction elimination and its traces; θ matching on edges (Memo::producers); the metric rule cutting edges, not candidates, and exact_only_groups_of_query (live candidates only, readout metrics per query, scalar roots); evaluation grids below subqueries, chain widths (gcd closure, @ T after every unanchored state) and the chain-edge divisibility check; the rollup pre-check before the compensating Filter and its frontend semantics.
  • Final layout:
    • crates/accuracy/src: lib.rs, evidence.rs, {error_contract,metric_rule,readout,readout_caps,logical_edge}/{mod,tests}.rs, guarantee/{mod,tests,binomial,ddsketch,hll}.rs, guarantee/{cardinality,cms,count_sketch,kll,univmon}/{mod,tests}.rs.
    • crates/stage1-pass2-reuse-candidates/src: lib.rs, edges.rs, evaluation_grids.rs, readout_requirements.rs, dead_candidates.rs, rules/{mod,candidates,topk,chains}.rs, rules/rollup/{mod,member,finest_common_group,rewrite}.rs, test_fixtures.rs, tests.rs, tests/{rollup,topk,readout_requirements,dead_candidates,chains,edges,evaluation_grids,exact_only_groups}.rs, tests/rollup/{filters,finest_common_group,measures}.rs.

Behavior: before → after

Concepts doc "ratio of quantiles" (test dead_candidates_are_pruned_and_listed_before_chain_groups): quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m]), relative error 0.05, every 60 s at phase 0. It serves the design need that Pass 2 enumerates only candidates some query can use, with a reason for every one it removes.

Before (main) After (this PR)
Pass 2 rules and edges todo!(); planning stops LogicalASAPMemoCandidates (memo, edges, pruned, evaluation grids, exact-only groups per query); planning stops at Stage 2 (todo! of PR 5)
KLL readouts — pruned: MetricRule { producer_metric: Rank, accepted_metrics: [Relative] }; every KLL build through its edge into a KLL readout
DDSketch builds — α 0.1 and 0.05 pruned BelowRequiredAccuracy; five survive
DDSketch readout groups — 5 readout candidates requiring relative 0.02, 0.01, 0.005, 0.002, 0.001 (δ 1)
Chains — one SummaryTumblingWindowChain[DDSketch, w = 60 s](Scan m) with the 5 surviving builds; SummaryMerge(TimeRange(5m, chain)) in the DDSketch state group
Exact-only groups — none for this query but the rows below the builds (the division carries relative error; the dead KLL readouts are skipped); … > 1 over the same ratio would list the ratio group and every group below it

Interfaces

Permalinks are at head 9ca4f50a1913666f68330e84b181753f8e8ab9f0.

crates/stage1-pass2-reuse-candidates/src/lib.rs

 /// Stage 1, Pass 2: steps 1–6; `entries[i]` is query i's workload entry (its evaluation grid).
 /// The planner facade calls it after Pass 1. Signature unchanged; now implemented.
 pub fn generate_logical_asap_memo_candidates(
     pass1: Memo,
     entries: &[QueryWorkloadEntry],
     evidence: &dyn AccuracyEvidence,
 ) -> Result<LogicalASAPMemoCandidates, Pass2RuleError>;
 /// Why Pass 2 failed on a memo: a rule bug or a cyclic memo. Stays; now defined in `lib.rs`.
 pub enum Pass2RuleError {
-    /// Never produced.
-    InvalidGroupReference,
-    /// Never produced.
-    TooManyGroups,
     /// A group, or one of its candidates, reads itself through other groups.
     CyclicDependencies,
+    /// Internal bug: a rule built a group expression the node rules reject.
+    Group(#[from] MemoGroupError),
+    /// Internal bug: a rule built a candidate expression the node rules reject, or gave it θ its
+    /// group's output cannot have.
+    Candidate(CandidateError),
 }

crates/types/src/stages/logical/candidate/mod.rs · memo/mod.rs · group/mod.rs · memo_candidates.rs · crates/types/src/stages/pruning.rs

+impl CandidateForMemoGroup {
+    /// The candidate computing `operator` with θ `parameters`, added by rule `origin`, fusing
+    /// `fused_groups`. Its inputs have no requirement yet: Pass 2 sets each readout's.
+    pub fn new(
+        operator: Operator<MemoGroupId>,
+        parameters: Option<SketchParams>,
+        origin: OriginOfCandidateForMemoGroup,
+        fused_groups: Vec<MemoGroupId>,
+    ) -> Self;
+}
+/// The θ of each build candidate of `build`: every listed θ of a sketch build, `[None]` for any
+/// other operator. A group gets one build candidate per entry.
+pub fn listed_build_parameters(build: &Operator<MemoGroupId>) -> Vec<Option<SketchParams>>;
 /// Which rule added a candidate. Stays; nothing in planning reads it.
 pub enum OriginOfCandidateForMemoGroup {
     /// Pass 1: exact, exact accumulator, or a summary type.
     Pass1,
-    /// Canonical value preprocessing feeding a summary build.
-    SummaryInput,
     /// A tumbling-window chain, or a merge reading one, of a summary state group.
     TumblingWindows,
     /// A coarse group rolled up from a finer one, the compensating `Filter` it reads, or a build of a finest common group.
     Rollup,
     /// A top-k read from another group's ranking.
     TopKReuse,
-    /// Its metric's scans widened to a shared label set (scan widening).
-    WidenedScans,
 }
 /// A candidate `Memo::add_candidate` rejects; the memo is unchanged. Stays.
 pub enum CandidateError {
     /// Internal bug: the expression fails the node rules.
     NotValidOverInputGroups(MemoGroupError),
     /// Internal bug: its output differs from the group's.
     OutputDiffersFromGroup { group: MemoGroupId },
+    /// Internal bug: its θ does not fit the group's output (a sketch state group's candidate needs
+    /// one from the algorithm's list, any other none).
+    LogicalProperties(LogicalPropertiesError),
 }
 /// Why Pass 1 failed. Stays (crates/stage1-pass1-per-computation-candidates/src/lib.rs).
 pub enum PerComputationCandidateError {
-    /// θ rejected while building a candidate (now checked once, in `Memo::add_candidate`).
-    LogicalProperties(LogicalPropertiesError),
     /* other variants unchanged */
 }
 /// θ of a candidate. A `SummaryMerge`, and a `Filter`, `TimeRange` or `TimeShift` over states,
 /// carry the θ of the states they read.
 impl LogicalProperties {
-    pub fn new(
-        expression: &CandidateExpression,
-        parameters: Option<SketchParams>,
-    ) -> Result<Self, LogicalPropertiesError>;
+    // Removed: `CandidateForMemoGroup::new` builds it; `Memo::add_candidate` checks θ against the group.
 }
 impl Memo {
+    /// The candidates `consumer` may read on input `input`: those of the input group, except that
+    /// a consumer with a θ reads only producers with the same θ or none.
+    pub fn producers(
+        &self,
+        consumer: CandidateForMemoGroupId,
+        input: usize,
+    ) -> Vec<CandidateForMemoGroupId>;
+    /// The most values one evaluation of `group` returns from the declared facts, the smaller of:
+    /// the series (groups of rows) of the sources scanned below it (`scanned_sources`), and, for a
+    /// readout of a state (or a state) grouped `by` keys, the groups the keys form (1 without keys,
+    /// the declared distinct combinations for keys copied from one source). A readout of
+    /// `quantile_over_time(0.9, m[5m])` over 1000 series: 1000; `SELECT count(DISTINCT dst)`: 1.
+    pub fn max_values_per_evaluation(&self, group: MemoGroupId) -> Option<u64>;
 }
 impl GroupProperties {
+    /// The sketch algorithm of the state the group outputs; `None` for rows, values and exact
+    /// states. The accuracy crate reads a readout's summary type through it.
+    pub fn sketch_algorithm_of_state(&self) -> Option<SketchAlgorithm>;
 }
 /// Stage 1, Pass 2's output. Stays.
 pub struct LogicalASAPMemoCandidates {
     pub memo: Memo,
     pub edges: Vec<LogicalEdge>,
     pub pruned: Vec<PrunedLogicalCandidate>,
     /// Per group: the grid its result is evaluated on (now: below a PromQL subquery `[10m:30s]`
     /// its 30 s steps; `None` below a subquery without a resolution).
     pub evaluation_grid_of_group: Vec<Option<EvaluationGrid>>,
+    /// Indexed by query: the groups it reaches through a live consumer with no error contract for
+    /// its metric, the readout groups whose error metric its target does not allow, plus, for a
+    /// query whose whole-result δ has no per-value one, each readout group another query gave a
+    /// level; closed downward through live candidates. A scalar root's operands are roots. In these groups the query may use only exact
+    /// candidates. Pass 2 only lists them: Stage 3 enforces them per query in a later PR.
+    pub exact_only_groups_of_query: Vec<BTreeSet<MemoGroupId>>,
 }
 /// One path from a pruned candidate, through candidates that are themselves eliminated, to the
 /// edge that fails. Stays.
 pub struct PruneCause {
-    /// The eliminated consumers on the path, nearest first.
-    pub through_pruned_consumers: Vec<CandidateForMemoGroupId>,
+    /// The eliminated candidates on the path, nearest first: consumers above, or, for a candidate
+    /// cut off from an input, producers below.
+    pub through_pruned_candidates: Vec<CandidateForMemoGroupId>,
     /// Why the edge at the end fails.
     pub root: RootPruneReason,
 }
 /// Why the edge at the end of a `PruneCause`'s path fails. Stays.
 pub enum RootPruneReason {
     /// Unsatisfiable: the producer's guarantee at the required δ exceeds the input's required error.
     BelowRequiredAccuracy {
         consumer: CandidateForMemoGroupId,
         input: usize,
         required: RequiredAccuracy,
     },
     /// Unsatisfiable: the producer is a readout whose metric the consumer does not accept on that
     /// input (on a readout's own input: at its output).
     MetricRule {
         consumer: CandidateForMemoGroupId,
         input: usize,
         producer_metric: AccuracyMetric,
-        consumer_carries: Vec<AccuracyMetric>,
+        /// The metrics accepted on that input (on a readout's own input: at its output).
+        accepted_metrics: Vec<AccuracyMetric>,
     },
+    /// Unsatisfiable: a top-k readout whose `k` exceeds the producer's heap size, whatever accuracy is required.
+    HeapSmallerThanTopK {
+        consumer: CandidateForMemoGroupId,
+        input: usize,
+        k: usize,
+    },
+    /// Unsatisfiable: under a query with an accuracy target, the readout has a metric but no
+    /// guarantee evaluable at the required δ (HLL's distinct count, failure probability unknown) or
+    /// no guarantee at all (UnivMon's distinct count): its error would reach the root unbounded.
+    NoGuarantee {
+        consumer: CandidateForMemoGroupId,
+        input: usize,
+    },
+    /// Unsatisfiable: query `query` sets a whole-result δ but its values per evaluation ($n_q$) are
+    /// unknown, so no per-value δ follows, and no other query reaching the readout gives it a level.
+    ValuesPerEvaluationUnknown {
+        consumer: CandidateForMemoGroupId,
+        query: usize,
+    },
+    /// Unsatisfiable: the producer is over a tumbling-window chain whose width does not divide the
+    /// consumer group's evaluation period (or the group has no known grid): the chain's windows do
+    /// not end at the consumer's evaluation times (a 15 s chain read every 10 s).
+    ChainWidthDoesNotDivideEvaluationGrid {
+        consumer: CandidateForMemoGroupId,
+        input: usize,
+        tumbling_window_width_ms: u64,
+        evaluation_period_ms: Option<u64>,
+    },
     /// The candidate never had a consumer and is not a query output.
     NoConsumer,
 }

add_candidate keeps its signature but now identifies a candidate by expression, θ and input requirements (same_candidate), and rejects a θ that does not fit the group.

crates/types/src/ir/operator/node.rs · ir/flat.rs · plan/physical_asap_dag.rs

 /// A node of the logical operator DAG. Stays.
 pub struct OperatorNode {
     pub operator: Operator,
     pub result_kind: OperatorResultKind,
     pub schema: Schema,
-    pub guarantee: Option<AccuracyGuarantee>,
 }
 impl OperatorNode {
-    pub fn with_guarantee(mut self, guarantee: Option<AccuracyGuarantee>) -> Self;
 }
 /// An `OperatorNode` whose children are `NodeId`s. Stays.
 pub struct FlatNode {
     pub operator: Operator<NodeId>,
     pub(crate) result_kind: OperatorResultKind,
     pub(crate) schema: Schema,
-    pub(crate) guarantee: Option<AccuracyGuarantee>,
 }
 /// A node of the installable physical plan. Stays; every other field unchanged.
 pub struct PhysicalASAPDAGNode {
     pub id: PhysicalASAPNodeId,
     pub payload: PhysicalASAPOperatorPayload,
     pub output_state: ExecutionDataState,
     pub lifetime: Lifetime,
     pub parameters: Option<SketchParams>,
     pub output_schema: Schema,
-    pub guarantee: Option<AccuracyGuarantee>,
     pub retention_ms: Option<u64>,
     pub selected_physical_candidate: Option<PhysicalCandidateId>,
     pub view_coverage: Option<ViewCoverage>,
 }

The only writer of these fields was the deleted asap_accuracy::propagation; every remaining site only copied the value, so it was always None. Wire format: a serialized plan node no longer has the guarantee key (PhysicalASAPDAGNode is deny_unknown_fields, so a plan written before this PR with "guarantee": null no longer parses); the fixtures plan_wire_version_10.json and stored_outputs_on_main.json drop it. INSTALLABLE_PLAN_WIRE_VERSION goes from 14 to 15 (decided in review), so an older plan file is refused by its version.

crates/types/src/ir/properties/selection.rs · stages/logical/group/aggregate_coverage.rs · accuracy/ · summary_type/

 /// `column = v`, `!=`, `<`, `<=`, `>`, `>=`, `[NOT] IN (...)` and `OR` of equalities on one column.
-pub(crate) fn constraint_of<C>(expr: &ScalarExpr<C>) -> Option<(ColumnId, Constraint)>;
+pub fn constraint_of<C>(expr: &ScalarExpr<C>) -> Option<(ColumnId, Constraint)>;
-#[cfg(any(test, feature = "test-support"))]
 impl AggregateCoverage {
+    /// The coverage definition; public outside tests (the rollup rule reads it).
     pub fn definition(&self) -> &AggregateDefinition;
+    /// Union of boxes over the rows that went in; public outside tests.
     pub fn selection(&self) -> &[SelectionBox];
 }
-#[cfg(any(test, feature = "test-support"))]
 impl SelectionBox {
+    #[cfg(any(test, feature = "test-support"))]
     pub fn new(columns: BTreeMap<SourceColumn, Constraint>) -> Self;
+    /// The per-column constraints; public outside tests.
     pub fn columns(&self) -> &BTreeMap<SourceColumn, Constraint>;
 }
 impl AccuracyMetric {
+    /// Every metric (`ABSOLUTE`, `ABSOLUTE_STREAM_L1`, `ABSOLUTE_STREAM_L2`, `Relative`, `Rank`).
+    pub const ALL: [Self; 5];
 }
 impl AccuracyTarget {
+    /// The metrics this target accepts an error in: its metric, every metric when it names none.
+    /// The caps walk and the exact-only walk seed each query root with it.
+    pub fn metrics(&self) -> Vec<AccuracyMetric>;
 }
 impl SketchStatistic {
+    /// The statistic a summary is read for to answer `intent` (was `asap_accuracy::sizing::sketch_statistic`).
+    pub fn for_intent(intent: &AggIntent) -> Option<Self>;
 }
-pub mod view_coverage; // in asap_types::stages
+pub mod view_coverage; // in asap_types::plan
-/// How a parent operator consumes its inputs' values.
-pub enum CompositionOperator { /* 10 variants */ }
-/// Why an accuracy check rejected a candidate.
-pub enum AccuracyError { /* 4 variants */ }
-/// The statistic an unbound number of values per result is left as.
-pub const VALUES_PER_RESULT: &str = "values_per_result";
 pub enum BoundExpr {
-    Zero,
     Constant { value: f64 },
-    Sum { terms: Vec<BoundExpr> },
-    Product { factors: Vec<BoundExpr> },
-    Scaled { factor: f64, inner: Box<BoundExpr> },
-    Max { terms: Vec<BoundExpr> },
-    Unknown { statistic: String },
 }
 impl BoundExpr {
-    #[cfg(any(test, feature = "test-support"))] // was `pub` on main
-    pub fn has_unknown(&self) -> bool;
-    pub fn is_zero(&self) -> bool;
     pub fn evaluate(&self) -> Option<f64>;
 }
 pub enum ProbabilityExpr {
     Zero,
     Constant { value: f64 },
-    UnionBound { terms: Vec<ProbabilityExpr> },
-    Scaled { count: BoundExpr, inner: Box<ProbabilityExpr> },
     /// HLL's failure probability.
     Unknown { statistic: String },
 }
 impl ProbabilityExpr {
-    pub fn is_zero(&self) -> bool;
     pub fn evaluate(&self) -> Option<f64>;
 }
 pub struct AccuracyGuarantee {
     pub spec: AccuracySpec<BoundExpr, ProbabilityExpr>,
-    pub provenance: Vec<GuaranteeSource>,
 }
 impl AccuracyGuarantee {
-    pub fn new(metric: AccuracyMetric, error: BoundExpr, failure_probability: ProbabilityExpr, provenance: Vec<GuaranteeSource>) -> Self;
+    pub fn new(metric: AccuracyMetric, error: BoundExpr, failure_probability: ProbabilityExpr) -> Self;
-    pub fn exact(reason: impl Into<String>) -> Self;
+    pub fn exact() -> Self;
-    pub fn bind(&self, value: &dyn Fn(&str) -> Option<f64>) -> Self;
-    pub fn is_exact(&self) -> bool;
-    #[cfg(any(test, feature = "test-support"))] // was `pub` on main
-    pub fn has_unknown(&self) -> bool;
-    pub fn over_scope(&self, scope: AccuracyScope, values_per_result: Option<u64>) -> Self;
-    #[cfg(any(test, feature = "test-support"))]
-    pub fn approximate_layer_count(&self) -> usize;
 }
-/// Provenance of a guarantee: no planning decision read it.
-pub enum GuaranteeSource { /* 8 variants */ }

Removed because nothing reads them: composition and budget types, AccuracyError, over_scope and approximate_layer_count, the symbolic bound and probability combinators, bind, is_exact, is_zero, has_unknown, and the provenance trail (GuaranteeSource; each edge serialized θ to JSON for it). The later stack branches use only Constant, ProbabilityExpr::{Zero, Constant, Unknown} and AccuracyGuarantee::new, apart from a pre-restack copy of the code this PR replaces. The /* N variants */ lines stand for whole enums removed with all their variants.

crates/accuracy/src/lib.rs · logical_edge/ · metric_rule/ · readout_caps/ · error_contract/ · guarantee/mod.rs

+/// The error metrics of one memo state, built once per memo state before an edge pass (and for
+/// step 3); Stage 3 reads the same contracts.
+pub struct ErrorMetricsOfMemo {
+    /// [group][candidate][metric]: the candidate's error contract; None = no contract.
+    error_contract: Vec<Vec<[Option<ErrorContract>; METRIC_COUNT]>>,
+    /// [group]: the error metrics its output may have; None = no query with a target reaches it.
+    error_metrics_allowed_at_output: Vec<Option<Vec<AccuracyMetric>>>,
+    /// [group][candidate]: the error metric of a sketch readout; None = not a sketch readout.
+    readout_error_metric: Vec<Vec<Option<AccuracyMetric>>>,
+}
+impl ErrorMetricsOfMemo {
+    /// `order` is `Memo::topological_order`.
+    pub fn of(memo: &Memo, order: &[MemoGroupId]) -> Self;
+    /// `candidate`'s error contract in `metric`; None when it has none.
+    pub fn error_contract(&self, candidate: CandidateForMemoGroupId, metric: AccuracyMetric) -> Option<&ErrorContract>;
+    /// α^log of edge (producer → consumer, input): the readout rule (the producer's own θ, heap ≥ k,
+    /// an evaluable guarantee, computed once per edge) or the metric rule; Err names the failing root reason.
+    pub fn edge_meets_logical_requirements(
+        &self,
+        memo: &Memo,
+        producer: CandidateForMemoGroupId,
+        consumer: CandidateForMemoGroupId,
+        input_index_of_consumer: usize,
+        evidence: &dyn AccuracyEvidence,
+    ) -> Result<(), RootPruneReason>;
+    /// Per query, the groups it may read only exactly (walking live candidates; readout metrics
+    /// per query; scalar roots' operands as roots), with the readout groups step 3 lists.
+    pub fn exact_only_groups_of_query(
+        &self,
+        memo: &Memo,
+        order: &[MemoGroupId],
+        dead: &BTreeSet<CandidateForMemoGroupId>,
+        exact_only_readout_groups_of_query: &BTreeMap<usize, BTreeSet<MemoGroupId>>,
+    ) -> Vec<BTreeSet<MemoGroupId>>;
+}
-/// α^log of one edge (main: a `todo!()` returning bool in `requirements.rs`).
-pub fn producer_satisfies_logical_requirements(
-    memo: &Memo,
-    producer: CandidateForMemoGroupId,
-    consumer: CandidateForMemoGroupId,
-    input_index_of_consumer: usize,
-    evidence: &dyn AccuracyEvidence,
-) -> bool;
-/// The largest evaluated error (or δ) that meets a required `limit`.
-pub fn target_limit(limit: f64) -> f64;
+/// Whether an evaluated error (or δ) meets a required `limit` (up to floating-point noise).
+pub fn within_limit(evaluated: f64, limit: f64) -> bool;
+/// The required-accuracy levels one sketch readout gets.
+pub struct ReadoutRequiredAccuracy {
+    pub levels: Vec<RequiredAccuracy>,
+    pub queries_without_per_value_failure_probability: Vec<usize>,
+}
+/// The levels of each sketch readout candidate without a requirement yet.
+pub fn readout_levels(
+    memo: &Memo,
+    error_metrics: &ErrorMetricsOfMemo,
+    order: &[MemoGroupId],
+) -> BTreeMap<CandidateForMemoGroupId, ReadoutRequiredAccuracy>;
-/// g_c's linear bound for errors in `metric` (main: a `todo!()`).
-pub fn error_propagation(consumer: &Operator<MemoGroupId>, metric: AccuracyMetric) -> Option<ErrorPropagation>;
-#[derive(Debug, Clone, PartialEq)]
-pub struct ErrorPropagation {
-    pub factor_per_input: Vec<f64>,
-    pub validity_cap_per_input: Vec<Option<f64>>,
-}
+/// g_c's linear bound in one metric: output error ≤ Σ β_i ε_i while each ε_i ≤ b_i. Built per
+/// candidate and metric by `ErrorMetricsOfMemo::of` (the builder `error_contract(memo, consumer,
+/// metric)` is crate-private; a `Filter`, `Project` or grouping reads its input's column lineage).
+#[derive(Debug, Clone)]
+#[cfg_attr(test, derive(PartialEq))]
+pub struct ErrorContract {
+    /// β per input.
+    pub error_factor_per_input: Vec<f64>,
+    /// b per input: the largest input error up to which the linear bound holds; None = any.
+    pub linear_bound_holds_up_to_per_input: Vec<Option<f64>>,
+}
 /// The error bound of θ for `statistic` at `failure_probability`; None if θ cannot fail that rarely
 /// or its δ is unknown. Signature unchanged; was `todo!()`.
 pub fn theoretical_guarantee(
     summary_type: &FieldDataType,
     parameters: &SketchParams,
     statistic: &SketchStatistic,
     failure_probability: f64,
 ) -> Option<AccuracyGuarantee>;
-/// The per-algorithm guarantees (`guarantee::hll`, `guarantee::univmon` were public).
-pub mod guarantee;
+mod guarantee;
-/// Node-tree guarantee composition.
-pub mod composition;
-/// Node-tree guarantee propagation through a DAG.
-pub mod propagation;
-/// The statistic a summary is read for.
-pub mod sizing;
-/// The target of one of `approximate_inputs` inputs composing into `target`.
-pub fn per_input_target(
-    target: &AccuracyTarget,
-    metric: AccuracyMetric,
-    approximate_inputs: usize,
-) -> Option<AccuracyTarget>;

Removed: node-tree composition and propagation (composition, propagation, propagation_stats, budget::per_input_target, private function_rules) are replaced by per-edge error contracts plus readout requirements. Also removed: the test-only guarantee::sizing_oracle, the classic HLL and UnivMon distinct-count guarantees (a later PR in this stack brings back certified distinct-count guarantees), the fixed-δ KLL and cardinality guarantee lookups (θ's own guarantee plus one δ check in theoretical_guarantee replace them), the test-only meets (its one caller checks the bound with within_limit), and PerComputationCandidateError::LogicalProperties (θ is now checked in one place, Memo::add_candidate). ErrorContract keeps PartialEq for tests only. guarantee and its per-algorithm modules are private. The readout metric (readout_error_metric), the metrics passed to inputs and the metrics allowed on a consumer's inputs are crate-private parts of ErrorMetricsOfMemo; Pass 2's step 3 returns a crate-private ReadoutRequirementsOutcome { readouts_without_per_value_failure_probability, exact_only_readout_groups_of_query }.

Usage, end to end (test dead_candidates_are_pruned_and_listed_before_chain_groups at 9ca4f50a1):

// What the user writes: quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m]),
// relative error 0.05, evaluated every 60 s at phase 0 (FixedIntervalAt).
// `memo`: Stage 0 + Pass 1 (built by the test as they build it): G2 = TimeRange(5m, Scan m);
// per quantile an Aggregate group with its exact candidate and readouts of a KLL state
// (7 builds) and a DDSketch state (7 builds); G6 = Divide(p99, p50), the query output.
// `target`: the query's accuracy target; Pass 2 derives each readout's required accuracy from it.
let target = AccuracyTarget::max_error(0.05).accuracy_metric(AccuracyMetric::Relative);
query(&mut memo, ratio, Some(target)); // MemoQuery { root: G6, accuracy_target, values_per_evaluation: 1 }
// `logical`: Pass 2's output, as the planner facade calls it after Pass 1. Inside, step 3 builds
// ErrorMetricsOfMemo::of(&memo, &order), whose contract for Divide in Relative is
// ErrorContract { error_factor_per_input: [1/(1-0.1); 2], linear_bound_holds_up_to_per_input:
// [None, Some(0.1)] }, and calls readout_levels(&memo, &error_metrics, &order); steps 4 and 5
// build one ErrorMetricsOfMemo per edge pass and call edge_meets_logical_requirements per edge;
// step 6 calls exact_only_groups_of_query over the survivors.
let logical = generate_logical_asap_memo_candidates(memo, &[entry(60_000)], &NoMeasuredGuarantees)?;
// logical.pruned: each KLL readout -> MetricRule { producer_metric: Rank, accepted_metrics: [Relative] };
//   every KLL build through it; DDSketch alpha 0.1 and 0.05 -> BelowRequiredAccuracy.
// Each DDSketch readout group: 5 readout candidates requiring relative 0.02, 0.01, 0.005, 0.002, 0.001 (δ 1).
// One chain group SummaryTumblingWindowChain[DDSketch, w = 60 s](Scan m) with the 5 surviving builds,
// and SummaryMerge(TimeRange(5m, chain)) in the DDSketch state group.
// logical.exact_only_groups_of_query[0]: the scan rows below the builds only (a build has no contract).
// logical.edges: only between survivors; Stage 2 (PR 5) expands them.

Algorithm changes

Pass 2's steps:

Step Code What it adds or decides
1. Rollup from the finest common group rules/rollup/ add_rollup_candidates Coarse aggregate group $G_c$ from finer $G_f$ of an equal definition (keys ⊆, selection ⊆ with compensation only on $G_f$'s keys by lineage, equal time window): SummaryMerge(group_by) for states, sum/min/max for finalized sums, counts, minima, maxima, each measure named as $G_c$'s column; one synthesized finest common group $G^*$ per cluster. The definition is compared through TimeRange, TimeShift and per-series layers (e.g. rate), and scans listing different labels match by column lineage
2. Top-k reuse rules/topk.rs A smaller exact top-k reads the largest top-k of the same input
3. Reach, caps, readout requirements readout_requirements.rs, accuracy::readout_caps One top-down walk: reach, accepted metrics and caps $\min(b_{c,i}, \max_q \varepsilon_q/\lambda)$ through the ErrorContracts of ErrorMetricsOfMemo; each sketch readout gets one candidate per requirement level (single-read query: the cap; composed query: the ε grid × δ grid not looser than the cap and per-value δ); per-value δ = δ / max($n_q$, Memo::max_values_per_evaluation of the readout group)
4. Dead-candidate elimination over the edges dead_candidates.rs, edges.rs Removes candidates no query can use, transitively; LogicalASAPMemoCandidates::pruned lists each with every root cause (one shortest path per failing edge)
5. Chain groups rules/chains.rs SummaryTumblingWindowChain[w](X) per width (gcd closure of $\gcd(g_q, W, o_j)$ over reaching queries with a known grid at the state, below a subquery its resolution), one build per surviving θ, re-wrapped TimeShift/TimeRange groups, SummaryMerge in the state group; @ T reuses the chains the unanchored states create (in any memo order); @ end()/@ start() none
6. Logical edges edges.rs, evaluation_grids.rs, lib.rs Every $(u,c,i)$ between surviving candidates with $\alpha^{\log}$ from ErrorMetricsOfMemo::edge_meets_logical_requirements; an edge from a chain into a reader whose evaluation grid (evaluation_grids: gcd of the reaching queries' grids) its width does not divide fails with ChainWidthDoesNotDivideEvaluationGrid
then ErrorMetricsOfMemo::exact_only_groups_of_query Per query, the groups it may read only exactly, over the live candidates

Rules add only through find_or_add_group/add_candidate and swallow only OutputDiffersFromGroup; add_candidate treats input requirements as part of a candidate's identity (two readouts with different requirements are different candidates); RELATIVE_OPERAND_VALIDITY_CAP = 0.1 is private; same_candidate compares input requirements first, and step 3 adds each readout copy with its requirement. theoretical_guarantee evaluates KLL (rank error scaled below δ = 0.01) and KMV (bisection on ε) at the required δ, and checks every other algorithm's own failure probability against it.

Rollup

  • Problem: three rules (scan widening, finer aggregate, rollup) each matched their own shapes by column name; no group finer than every member was ever built.
  • Before: rule 2 widened scans, rule 5 read a finer SQL result, rule 6 merged a finer state, each pairwise.
  • After: (1) cluster aggregate groups by coverage definition without keys, written out down to the source through TimeRange, TimeShift and per-series layers (Member::see_through_per_series_layers), so equal computations compare equal whatever their keys, label selections and the labels their scans list (scans match by lineage; an operator that names columns by position, such as a measure over a column, still needs equal column lists), and equal intervals; (2) per cluster, $G^*$ = union selection, keys ∪ narrowing columns, built over a member's rows; (3) every ordered pair gets Rollup(by keys)(Filter(compensation, G_f)) when keys ⊆ and selection ⊆ on keys by lineage. The rollup and its schema are checked over a pending Filter first, so a rollup the memo would reject leaves no compensating group behind.
  • Review fixes: a rolled-up Aggregate names each measure as $G_c$'s column, so count by (job) rolls up count by (job, instance) as a sum named count (a SQL global count over a grouped one is still rejected by the schema check: a sum over no group is NULL, the count 0). The compensating Filter is built from comparisons with the source frontend's semantics: a PromQL != matcher becomes Compare Ne with PromQL semantics, NOT IN a conjunction of Ne, IN a disjunction of Eq.
  • Correctness: a compensation reads only $G_f$'s keys, whose rows each combine all raw rows of one key. Tests: a_coarse_exact_sum_rolls_up_the_finer_sum, a_coarse_count_rolls_up_the_finer_count_as_a_sum, a_filtered_coarse_sketch_state_merges_the_finer_states_through_a_compensating_filter, a_finest_common_group_no_query_asks_for_serves_both, a_narrowing_on_a_column_that_is_no_key_gives_no_rollup, a_compensating_filter_over_a_sql_table_has_sql_semantics, a_not_equal_matcher_compensates_with_promql_semantics, a_rejected_rollup_leaves_no_compensating_filter, and through production PromQL lowering (crates/integration-tests/tests/pass2_promql/rollup.rs): an_instant_sum_rolls_up_from_the_finer_sum_over_a_scan_listing_more_labels, an_instant_count_rolls_up_from_the_finer_count, a_sum_of_rates_rolls_up_from_the_finer_sum_of_the_same_rates, a_label_matcher_becomes_a_compensating_filter_over_the_finer_sum, with the negative cases rates_over_different_ranges_do_not_roll_up, different_per_series_functions_do_not_roll_up, a_standard_deviation_does_not_roll_up, a_finer_sum_narrowed_by_a_matcher_serves_no_wider_sum.
  • Complexity: O(groups²) per cluster.
  • Assumptions and limits: see Q5. Not covered: without groupings, label filters between per-series layers, regex matchers; no new union Scan; equal intervals. The pre-check covers the schema only. The A filtered query (city = 'NYC') doesn't read its group from a grouped summary another query builds #902 case (city = 'NYC' over the per-city group) is not widened: Rollup rule: a filtered query (city = 'NYC') does not read the per-city group another query builds (#902 regression on the #1207 stack) #1252.

Readout requirements, the metric rule and dead candidates

  • Problem: θ floors per input (Pass 1 heap floor, Pass 2 cap floors) mixed ε and δ and could not express multi-readout allocation.
  • After:
    0. ErrorMetricsOfMemo::of: each candidate's ErrorContract per metric and each sketch readout's error metric, once per memo state (step 3, each edge pass); the walks below read them.
    1. readout_caps::caps: one walk down from the query roots in reverse topological order. A root with a target is reached with its target in each of its metrics (AccuracyTarget::metrics; a scalar root's operands are roots: scalar(v) and a scalar subquery with the target's metrics, any other scalar expression with none); every candidate of a reached group reaches its inputs and passes each metric its contract carries, with caps $\min(b_{c,i}, \text{cap}/\beta_{c,i})$, keeping the largest per query and metric; an input with $\beta = 0$ (a count's) gets the metric but no cap.
    2. readout_levels: per readout candidate without a requirement, its levels from its group's caps in its metric (single read: the cap at the per-value δ; composed: the ε grid MAX_ERROR_LEVELS × the δ grid 1e-2 … 1e-5 continued in powers of ten down to the first at or below a tenth of the per-value δ, D1).
    3. ErrorMetricsOfMemo::error_metrics_allowed_at_output = the metrics of the same walk per group (None: not reached); the metrics allowed on a consumer's inputs are its group's that its contract passes.
    4. $\alpha^{\log}$ per edge, then elimination to a fixpoint in two directions (a candidate whose consumers all died, or one with an input whose every edge is cut or reads a dead producer) and BFS traces.
    5. exact_only_groups_of_query: per query, over the live candidates, the metrics every path to a group passes (an intersection, where the metric rule's allowed metrics are a union over queries); a group where none is left, or that holds a sketch readout whose error metric is not among them, is exact-only for that query, with every group below it.
  • Review fixes:
    • A Project contract judges only the items reading a computed value: copied labels, the time and promql_drop_metric_name of the series identity keep any error; promql_abs/clamp/clamp_min/clamp_max and a negation keep an absolute error; a product with or quotient by a nonzero literal keeps a relative one. abs(count_over_time(m[1m])), -count_over_time(m[1m]) and quantile_over_time(...) / 2 now keep their sketch readouts (key_filters_and_value_preserving_projects_have_unit_contracts, a_function_of_the_value_keeps_the_readouts_below_it).
    • readout_metric no longer needs a finite δ (HLL's distinct count is relative); under a query with a target, an edge into a readout with no metric (UnivMon's distinct count) or whose guarantee's δ is unknown (HLL) fails with NoGuarantee (a_distinct_count_readout_without_an_evaluable_guarantee_serves_no_query_with_a_target).
    • The per-value δ splits over max($n_q$, the readout group's values per evaluation, Memo::max_values_per_evaluation): max(quantile_over_time(0.9, m[5m])) over 1000 series at δ = 0.01 requires 1e-5 per readout, not 1e-2 (a_max_over_many_readouts_splits_the_failure_probability_over_them); SELECT count(DISTINCT dst) FROM flows reads one value and keeps 0.01, so KMV k = 4323 survives (a_global_distinct_count_keeps_its_whole_failure_probability).
    • A factor-0 contract (count) passes no cap, so no readout gets an infinite requirement (a_count_over_readouts_gives_them_no_level).
    • exact_only_groups_of_query (a_query_comparing_a_shared_ratio_reads_it_only_exactly, a_readout_another_query_gives_a_level_survives_unknown_values_per_evaluation).
  • Error contracts: a reduction (count, min, max, avg) keyed by an estimate gets no contract: an error in a key can move a row to another group, so no bound on the output follows (a_reduction_keyed_by_an_estimate_has_no_contract).
  • Unknown values per evaluation: ValuesPerEvaluationUnknown prunes only a readout that no query gives a level; a readout another query reaches with a known $n_q$ keeps its levels, and the query without a per-value δ lists its group as exact-only (a_readout_another_query_gives_a_level_survives_unknown_values_per_evaluation).
  • Correctness: a pruned candidate has no edge to any live consumer, so no plan can use it; up to ten readouts each find a δ level whose sum stays within the per-value δ. Tests: dead_candidates_are_pruned_and_listed_before_chain_groups, a_single_readout_requires_the_query_target, and in the accuracy crate a_whole_result_failure_probability_is_split_per_value, a_ratio_reads_no_rank_error, a_ratio_whose_per_value_failure_probability_is_below_the_grid_splits_it_over_its_readouts, a_max_over_many_readouts_splits_the_failure_probability_over_them, a_count_over_readouts_gives_them_no_level.
  • Complexity: one ErrorMetricsOfMemo per memo state (contracts once per candidate and metric, readout metrics once per candidate); the caps walk reads the contract table; the guarantee once per edge (no JSON serialization); the evaluation periods once for steps 4 and 5; elimination to a fixpoint over edges.
  • Assumptions and limits: Q2–Q4 below; Stage 3 does not yet enforce exact_only_groups_of_query.
    • The exact-only list is per group, so it is conservative. avg by (job)(count_over_time(m[1m])) at absolute 0.05: avg has an absolute contract, the exact accumulator's sum/count candidate has none, so the group below and its readout are exact-only for that query even through avg. A finer list needs Stage 3 to enforce the metric rule per query on each edge. Likewise a sketch readout placed directly in a group with other candidates makes that whole group exact-only for a query that does not allow its metric (Pass 1 reads readouts through their own groups in production lowering).
    • PromqlSubquery has no error contract, so every sketch readout under a subquery serves no query with a target (sound). A contract for it must split δ over the subquery's inner steps (series × range / resolution), not over the series alone.
    • The per-value δ of a per-value-scope target over a per-series readout still splits over the readout's series (quantile_over_time at per-value scope and δ 0.01 over 1000 series: 1e-5): sound, stricter than needed.

Evaluation grids and chain groups

  • Problem: tumbling layouts were a property; widths came only from queries reaching a state through Stage 0 expressions; offsets were ignored; a chain could serve a reader whose evaluations it does not end on; below a PromQL subquery the outer grid was used.
  • After: evaluation grids per (query, group) walking down from each root (evaluation_grids.rs): below PromqlSubquery { resolution: Some(r) } the grid is r, with None it is unknown (no chain); evaluation_grid_of_group is the gcd over the reaching queries. Range-free chain groups after the reach; widths include offsets; built in two phases (all chain builds, then the TimeShift/TimeRange/merge groups) so a chain shared by state groups offers each every θ it has; a chain's edges into a reader whose own evaluation grid the chain's width does not divide fail with ChainWidthDoesNotDivideEvaluationGrid { consumer, input, tumbling_window_width_ms, evaluation_period_ms } (a 15 s chain never serves a reader every 10 s, which gave wrong answers before); dead-candidate elimination runs again after chain groups.
  • Correctness: tests a_chain_whose_width_does_not_divide_the_readers_grid_fails_its_edge, an_offset_enters_the_gcd_of_the_widths, one_range_free_chain_group_serves_every_range_and_offset, chain_groups_run_after_reach_so_a_rolled_up_query_adds_its_grid, an_absolute_anchor_reuses_only_existing_chains, no_chain_without_a_merge_an_anchor_or_a_grid, a_mergeable_state_reads_a_chain_with_one_build_per_surviving_parameters, a_groups_evaluation_grid_is_the_gcd_of_its_queries_grids, below_a_subquery_the_grid_is_its_resolution, below_a_subquery_without_a_resolution_there_is_no_grid, a_subquery_s_resolution_is_the_grid_of_the_groups_below_it.
  • Assumptions and limits: @ T reuses the chains the unanchored states create (added after them, so memo order does not matter: an_absolute_anchor_reuses_a_chain_a_later_group_creates); @ end()/@ start() get none.

Decisions and open questions

Decided (concepts doc updated):

  • Round-3 interface: ErrorMetricsOfMemo (contracts, allowed metrics and readout metrics of one memo state, built once per memo state) with edge_meets_logical_requirements; ErrorContract { error_factor_per_input, linear_bound_holds_up_to_per_input } (was ErrorPropagation { factor_per_input, validity_cap_per_input }); within_limit(evaluated, limit) -> bool (was target_limit); ReadoutRequiredAccuracy replaces the anonymous (levels, queries) tuple. Beside the agreed sketch, ErrorMetricsOfMemo has two more public methods: exact_only_groups_of_query (the per-query walk reads the contracts and readout metrics, which stay private) and error_contract (the read Stage 3 needs).
  • Round-3 fixes: a query reads a readout group only exactly when the readout's error metric is not one its target allows (quantile_over_time(0.9, m[5m]) at rank 0.01 and at relative 0.05: each reads the other's readout groups only exactly); the exact-only walk skips dead candidates; δ splits over at most the readout's own values; scalar roots' operands are roots; @ T chains independent of memo order.
  • Plan wire version: deleting the node guarantee field bumps INSTALLABLE_PLAN_WIRE_VERSION from 14 to 15.
  • A1 (Q1 = (b)): a merge carries θ. SummaryMerge, and a Filter, TimeRange or TimeShift over states, carry the θ of the states they read: one candidate per θ (add_candidates_per_input_parameters), input edges only to producers with the same θ (Memo::producers); Memo::add_candidate checks θ against the group. α^log of a readout reads its producer's own θ. Rejected (a): α^log(merge → readout) holds only when every θ the merge may carry meets the requirement (safe, but a chain with weak builds blocks tight levels). Example (a_merge_is_judged_by_the_parameters_it_carries): P = quantile_over_time(0.9, m[5m]) at rank 0.005, Q = quantile_over_time(0.9, m[10m]) at rank 0.01, one 60 s chain with KLL k ∈ {269, 547, …}: SummaryMerge[k=547] → P's readout holds; SummaryMerge[k=269] is pruned with BelowRequiredAccuracy { required: rank 0.005 }.
  • A2 (D4 / bug 6): the metric rule cuts edges, not candidates. The allowed metrics push down, per group, the metrics its output may be off in (root: the target's metrics; input i of c: the accepted metrics c's contract carries). An edge is cut only when its producer is a sketch readout whose metric the consumer does not accept on that input, or, on a readout's own input, when its metric is not accepted at its output. Every surviving readout a targeted query reaches has a requirement, or fails with NoGuarantee. The metric rule in dead-candidate elimination is a union over queries, so a query can still reach approximate producers through a consumer with no contract for its metric; Pass 2 lists those groups per query in exact_only_groups_of_query, which Stage 3 enforces per query. Example (the_metric_rule_cuts_the_readout_edges_and_keeps_the_exact_division): avg by (job)(q0.5 / q0.5 + q0.9) at absolute 0.05: the readout edges into the division are cut, the exact division is not pruned and its edge into + holds.
  • Two-direction elimination: a candidate is also dead when every edge on one of its inputs is cut or reads a dead producer; the side it died on is recorded and its trace follows it. PruneCause.through_pruned_consumers → through_pruned_candidates.
  • N10: RootPruneReason::ValuesPerEvaluationUnknown { consumer, query }: a whole-result δ with $n_q$ unknown no longer gives per-value δ 0; a readout no query gives a level is pruned with this reason (a_whole_result_failure_probability_without_values_per_evaluation_prunes_the_readout). Rejected: δ = 0 (no sketch meets it, with no stated reason).
  • D1: δ levels continue below the per-value δ. Before, a per-value δ below 1e-5 gave each readout that δ itself and two readouts summed past it, so (K) had no solution. Example ratio-of-quantiles-1h-mvp: quantile_over_time(0.9, data[1h]) / quantile_over_time(0.5, data[1h]), relative 0.05, δ = 0.01 over 10,000 series = 1e-6 per value: both readouts now get levels 1e-6 and 1e-7 (1e-7 + 1e-7 ≤ 1e-6), and the full stack planned it at 7.481e-06/s before this PR's review fixes (one DDSketch α = 0.02 chain of 60 s windows, refactor: tools and snapshots over the #1207 memo, edges and contexts (#1207 PR 9) #1239); re-measured on restack.
  • D6 (approved): Memo::producers, CandidateError::LogicalProperties, MetricRule.accepted_metrics (was consumer_carries). Example: for merge[k=547] in P's state group, memo.producers(merge, 0) = the k = 547 candidate of TimeRange(5m, G_w) only; adding a SummaryMerge over KLL states without θ fails with CandidateError::LogicalProperties(SketchBuildWithoutParameters) (a_merge_carries_the_parameters_of_the_states_it_reads).
  • P1 (N3): the error contract reads the memo, because the key-only Filter contract (E1(b)) reads the input group's column lineage. Example: over R of quantile by (job)(latency) (columns [job: SourceColumn(latency.job), value: Computed]), Filter(job = "api", R) gives Some(ErrorContract { error_factor_per_input: [1.0], .. }) in every metric; Filter(value > 0.5, R) gives None. Rejected: the doc signature (consumer, metric) with the caller deciding filters (two places decide contracts).
  • P2 (N4): edge_meets_logical_requirements returns the failing reason. Example: KLL k = 26 into the p99 readout requiring relative 0.02 gives Err(BelowRequiredAccuracy { consumer: G4#2, input: 0, required: relative 0.02 δ 1 }); LogicalEdge::satisfies_logical_requirement is .is_ok(). Rejected: a bool plus a second function (decides it twice).
  • P3 (N5): RootPruneReason::HeapSmallerThanTopK (the doc's α^log includes "a top-k's heap holds k", which fails also without a requirement). Example: topk(50, …) over CmsWithHeap: the edges from the 56 builds with heap 10 or 20 fail, so they are pruned with HeapSmallerThanTopK { k: 50 } (a_heap_smaller_than_k_fails_the_topk_readouts_edge).
  • P4 (N6, on condition it duplicates nothing): readout_error_metric (crate-private) reads the metric off the estimators' guarantee. Example: (Sketch(Kll), Quantile{0.5}) → Rank; (Sketch(DDSketch), Quantile{0.5}) → Relative; (Sketch(Hll), Cardinality) → Relative; (ExactAggregate(..), _) → None. Used by step 3 and the metric rule; a readout's summary type is read with GroupProperties::sketch_algorithm_of_state, the one helper for it.
  • P5 (N6): AccuracyMetric::ALL. P6 (N6): AggregateCoverage::{definition, selection} and SelectionBox::columns public (were test-support only); constraint_of public.
  • Q2 (R5): the metric rule applies to an edge only when a query with an accuracy target reaches the consumer; with accuracy not enforced every edge passes (without_targets_every_edge_is_admissible), matching "(K) has no rows".
  • Q3 (R5): a query is composed when a candidate it reaches has two or more inputs below which a readout group lies (a ratio); readouts of one value by different summary types are alternatives, not a composition.
  • Q4 (R5; the $n_q$-unknown case changed by N10): when no grid level is within the cap (exact target, cap 0) the readout gets one level at (cap, δ).
  • Q5 (R5): $G^$ is built over the rows of a member whose selection equals the union; no new Scan of the union of label sets, so {job="a"} with {job="b"} gets no $G^$ (scan widening, R7, not folded in). In production PromQL, sum by (job)(m) + sum by (instance)(m) get none either, as each scan lists only its own label: Rollup: no finest common group when PromQL scans of one metric list different labels #1285. Members grouped without keys or per entity are not rolled up; interval constraints must be equal within a cluster.
  • Q6 (R5): a chain group gets one build per θ of the state group's surviving builds (union over the state groups sharing it).
  • PR 1 Q6: test-only sizing replaced by the per-entry guarantee test (every_listed_parameters_has_a_guarantee_at_one_percent, a_guarantee_is_evaluated_at_the_required_failure_probability).
  • Cleanup review: one caps walk gives reach, allowed metrics and caps; readout_levels keys its result by readout candidate; MAX_ERROR_LEVELS lives with readout_levels; guarantee private; accuracy-math tests in the accuracy crate; shared memo builders in asap_types::test_support.

Verification

Test scopes run at head 9ca4f50a1 with cargo test -p <crate>; the workspace total is cargo test --workspace --no-fail-fast.

Scope Passed / failed / ignored Notes
asap-accuracy 29 / 0 / 0 error contracts, guarantee tables, readout metric, readout levels, metric rule, edge check
asap-types 148 unit, 77 integration, 8 doc / 0 / 0 unit tests: −6 symbolic-guarantee tests deleted with their code, +1 max_values_per_evaluation
asap-stage1-pass2-reuse-candidates 28 / 0 / 0 + an_absolute_anchor_reuses_a_chain_a_later_group_creates
asap-stage1-pass1-per-computation-candidates 37 / 0 / 0 split by scope
integration pass2_* 13 / 0 / 0 + 2 PromQL exact-only tests, + 1 SQL test
Workspace 1216 / 184 / 9 the same failing tests by name as at 312fa98b1 (1212 / 184 / 9); the only name differences are the integration tests renamed by content (sharing_issue_regressions::* and identical_exact_queries::* → plan_sharing::*)

Every workspace failure stops at a later-PR todo!() (Stage 2 entry stage2-physical-optimization/src/lib.rs, the executor's θ, devtools) or at materialized_view.rs:529 / materialized_views.rs:67 (fixture).

Round-3 workloads: the dump program (Pass 2's full output plus an integrity checker) ran the 91 round-3 workloads at this head: 0 integrity problems. Against 312fa98b1, only the exact-only lists and one live count change: quantile_over_time(0.9, m[5m]) at rank 0.01 and at relative 0.05 now lists the DDSketch groups as exact-only for the rank query and the KLL groups for the relative one (before: [0, 1] for both); count(quantile_over_time(0.9, m[5m])) lists [0, 1] (before also the quantile group, the DDSketch readout and its chain, through dead Count-Min and UnivMon builds), likewise 25 other workloads lose groups reached only through dead candidates; scalar roots now list their operands' groups per their contract (scalar(...) * 2: every group); SELECT count(DISTINCT dst) FROM flows at δ 0.01 requires 0.01 instead of 1e-5 and keeps two more KMV builds (live 17 → 19).

New tests:

  • Pass 2 crate: rollup (a_coarse_exact_sum_rolls_up_the_finer_sum, a_coarse_count_rolls_up_the_finer_count_as_a_sum, a_filtered_coarse_sketch_state_merges_the_finer_states_through_a_compensating_filter, a_finest_common_group_no_query_asks_for_serves_both, a_narrowing_on_a_column_that_is_no_key_gives_no_rollup, a_compensating_filter_over_a_sql_table_has_sql_semantics, a_not_equal_matcher_compensates_with_promql_semantics, a_rejected_rollup_leaves_no_compensating_filter); top-k (a_smaller_topk_reads_the_largest_topk_of_the_same_input); requirements (a_single_readout_requires_the_query_target, a_whole_result_failure_probability_without_values_per_evaluation_prunes_the_readout, a_readout_another_query_gives_a_level_survives_unknown_values_per_evaluation); dead candidates (dead_candidates_are_pruned_and_listed_before_chain_groups); chains (a_mergeable_state_reads_a_chain_with_one_build_per_surviving_parameters, a_chain_whose_width_does_not_divide_the_readers_grid_fails_its_edge, an_absolute_anchor_reuses_only_existing_chains, no_chain_without_a_merge_an_anchor_or_a_grid, chain_groups_run_after_reach_so_a_rolled_up_query_adds_its_grid, an_offset_enters_the_gcd_of_the_widths, one_range_free_chain_group_serves_every_range_and_offset); grids (a_groups_evaluation_grid_is_the_gcd_of_its_queries_grids, below_a_subquery_the_grid_is_its_resolution, below_a_subquery_without_a_resolution_there_is_no_grid); exact-only (a_query_comparing_a_shared_ratio_reads_it_only_exactly); edges (every_candidate_of_an_input_group_is_an_edge_of_matching_parameters, without_targets_every_edge_is_admissible, a_merge_is_judged_by_the_parameters_it_carries).
  • Round 3: each_query_reads_only_the_readouts_in_its_targets_metric, a_dead_candidate_makes_no_group_exact_only (pass2_promql/exact_only_groups.rs), a_global_distinct_count_keeps_its_whole_failure_probability (pass2_sql.rs), a_readout_of_one_value_keeps_the_whole_failure_probability (readout_caps), max_values_per_evaluation_bounds_a_readout_by_its_states_keys (types memo), a_scalar_roots_operands_are_roots (metric_rule), an_absolute_anchor_reuses_a_chain_a_later_group_creates (Pass 2 chains). The R1 and R2a cases reproduce the round-3 review's x15 and e05 workloads, whose dumps at 312fa98b1 show the old lists.
  • Accuracy crate: readout a_readouts_metric_is_its_algorithms (KLL quantile Rank, DDSketch and HLL Relative, exact None); readout_caps.rs a_whole_result_failure_probability_is_split_per_value, a_ratio_whose_per_value_failure_probability_is_below_the_grid_splits_it_over_its_readouts, a_max_over_many_readouts_splits_the_failure_probability_over_them, a_count_over_readouts_gives_them_no_level; metric_rule.rs a_ratio_reads_no_rank_error, the_metric_rule_cuts_the_readout_edges_and_keeps_the_exact_division, a_key_only_filter_keeps_the_ddsketch_readout_admissible, avg_of_readouts_under_a_relative_target_reads_exact_only; logical_edge within_limit_admits_only_floating_point_noise, a_distinct_count_readout_without_an_evaluable_guarantee_serves_no_query_with_a_target, a_heap_smaller_than_k_fails_the_topk_readouts_edge (56 builds with heap 10 or 20 fail HeapSmallerThanTopK { k: 50 }); error_contract a_reduction_keyed_by_an_estimate_has_no_contract, key_filters_and_value_preserving_projects_have_unit_contracts, error_contracts_of_a_ratio_a_count_and_a_sum; guarantee/mod.rs a_guarantee_is_evaluated_at_the_required_failure_probability, every_listed_parameters_has_a_guarantee_at_one_percent.
  • Integration: pass2_promql/rollup.rs (listed under Rollup); pass2_promql/accuracy_and_grids.rs a_function_of_the_value_keeps_the_readouts_below_it (abs, negation and / 2 over a sketch readout keep it), a_subquery_s_resolution_is_the_grid_of_the_groups_below_it.
  • Types: a_merge_carries_the_parameters_of_the_states_it_reads, kll_builds_with_k_269_and_k_547_share_one_group, set_candidate_input_requirements_replaces_only_the_named_candidate, add_candidate_checks_parameters_against_the_groups_output.

End-to-end evidence: none at this head, since planning stops at Stage 2's todo!. ratio-of-quantiles-1h-mvp planned at 7.481e-06/s on the #1239 snapshot before this PR's review fixes; it is re-measured on restack. Screenshots: not applicable.

Self-review: one add_candidate_if_output_matches helper in Pass 2, one readout_error_metric, one sketch-algorithm helper, one target-metric expansion (AccuracyTarget::metrics), contracts only in error_contract, one ErrorMetricsOfMemo per memo state, θ only in parameter_list (no strongest/weakest/floor tables); no #[allow(clippy::type_complexity)] (the two tuples are ReadoutRequiredAccuracy and ReadoutRequirementsOutcome); no #[allow(dead_code)] added.

Deleted fields and types

Item Where Why
GuaranteeSource, AccuracyGuarantee::provenance types/src/accuracy/guarantee no planning decision read it; serialized θ to JSON on every edge
BoundExpr::{Zero, Sum, Product, Scaled, Max, Unknown}, ProbabilityExpr::{UnionBound, Scaled} same only Constant (and ProbabilityExpr::{Zero, Unknown}) is built
AccuracyGuarantee::{bind, is_exact, has_unknown}, BoundExpr::{is_zero, has_unknown}, ProbabilityExpr::{is_zero, has_unknown, bind} same no non-test reader
LogicalProperties::new types/src/stages/logical/candidate one caller, CandidateForMemoGroup::new, builds it inline
Memo::series_count_scanned_below types/src/stages/logical/memo replaced by max_values_per_evaluation (from scanned_sources)
accepted_metrics, metrics_accepted_on_inputs, readout_metric_of_candidate, metrics_carried, producer_satisfies_logical_requirements, target_limit, ErrorPropagation, error_propagation (public) crates/accuracy folded into ErrorMetricsOfMemo / renamed (ErrorContract, within_limit)
the KLL and UnivMon contract-name constants, hll::generic_guarantee (now hll::guarantee) crates/accuracy/src/guarantee provenance only
Pass 2's exact_only_groups.rs, rules.rs Pass 2 the walk is ErrorMetricsOfMemo::exact_only_groups_of_query; helpers in rules/candidates.rs
test helpers range, shift, readout_of, groups_where, candidates_of (Pass 2), candidates, groups_where (Pass 1), summary_agg (types candidate tests) tests shared builders and fixtures

Follow-ups

@zzylol zzylol added optimizer Workload optimization and plan selection refactor labels Nov 7, 2026
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from d4bde07 to 441f3a9 Compare November 7, 2026 04:14
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from 72a4037 to 24dd00c Compare November 7, 2026 05:10
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from 2c6d993 to 61f585a Compare November 9, 2026 06:37
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from bdf0888 to bcae9e5 Compare November 9, 2026 07:11
@zzylol zzylol changed the title refactor(stage1-pass2): reuse rules over groups, logical edges, metric rule, validity and pruning caps (#1207 PR 4) refactor(stage1-pass2): rollup from the finest common group, readout requirements, dead-candidate elimination, chain groups (#1207 PR 4) Nov 9, 2026
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from 61f585a to a71e517 Compare November 9, 2026 16:50
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch 2 times, most recently from 8107319 to 5d3f6de Compare November 9, 2026 20:52
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from a71e517 to d77278a Compare November 10, 2026 01:26
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from 5d3f6de to dea088e Compare November 10, 2026 01:26
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from d77278a to 0c11cd7 Compare November 10, 2026 13:37
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from dea088e to e98e80b Compare November 10, 2026 13:37
@zzylol
zzylol force-pushed the refactor/1207-pass1 branch from 0c11cd7 to 321a1ae Compare November 11, 2026 03:35
…requirements, dead-candidate elimination, chain groups (#1207 PR 4)

Pass 2 runs, in order: the general rollup rule with one synthesized
finest common group per cluster of equal definitions (replaces scan
widening, finer-aggregate reuse and the rollup rules); top-k reuse; reach,
caps and readout requirements (one readout candidate per requirement level,
written with Memo::set_candidate_input_requirements); dead-candidate
elimination with every root cause (PrunedLogicalCandidate); chain groups
SummaryTumblingWindowChain[w](X) for the surviving state groups, widths
from the gcd closure of the reaching queries' grids, the window and the
offsets, then elimination again; and the logical edges between survivors.
A merge, and a Filter/TimeRange/TimeShift over states, carries the θ of
the states it reads (one candidate per θ, reading only producers with that
θ). The metric rule cuts edges from readouts whose metric the consumer's
input does not accept (accepted metrics pushed down from the targeted
queries), not candidates; a candidate cut off from an input is eliminated
too. A whole-result δ with n_q unknown prunes the readouts it reaches
(ValuesPerEvaluationUnknown).

- stage1-pass2: the entry generate_logical_asap_memo_candidates and the
  possible edges in lib.rs; rules.rs (Pass2RuleError, add_candidate,
  add_candidates_per_input_parameters, add_group_computing_itself),
  rules/{rollup,topk,chains}.rs, requirements.rs, dead_candidates.rs,
  tests.rs and tests/decided_examples.rs.
- types: AccuracyMetric::ALL; RootPruneReason::{HeapSmallerThanTopK,
  ValuesPerEvaluationUnknown}; MetricRule.accepted_metrics;
  PruneCause.through_pruned_candidates; Memo::producers; add_candidate
  checks θ against the group's state (CandidateError::LogicalProperties,
  candidate::parameters_fit); a candidate's input requirements are part of
  its identity (hash_consing::same_candidate); LogicalProperties::new
  accepts any θ on a candidate passing states on; aggregate coverage and
  selection accessors no longer test-only; origins SummaryInput and
  WidenedScans deleted.
- accuracy: error_propagation(memo, consumer, metric) with the constant
  contracts (key-only Filter by column lineage, abs / clamp / negation,
  min / max / avg / count, + - / *), b = 0.1 private;
  theoretical_guarantee at a failure probability (KLL and KMV evaluated at
  it, others checked against their own);
  producer_satisfies_logical_requirements returns the failing root reason;
  accepted_metrics and readout_metric. Node-tree composition, propagation
  (with its count-min estimate_has_bound test), budget split, function
  rules, PropagationStats and the test-only sizing (guarantee/sizing_oracle)
  deleted; the classic HLL precision tests move into guarantee/hll.rs.
- devtools: one pass2 stage; docs follow.
- integration-tests: pass1_candidates_sql re-adds
  a_filter_or_group_key_over_an_estimate_has_no_guarantee.
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from e98e80b to 5ac57fc Compare November 11, 2026 18:33
@zzylol
zzylol changed the base branch from refactor/1207-pass1 to main November 11, 2026 18:33
… guarantees no stage reads yet (#1207 PR 4)

classic_guarantee, ClassicHllConfidence, classic_failure_probability and
the classic statistic helpers, and UnivMon's distinct_guarantee with its
incomplete-top-layer statistic, had no caller outside their own tests.
PR 8 re-adds what its HLL and UnivMon distinct-count guarantees use.
… error contract (#1207 PR 4)

An Aggregate or exact accumulator build gave a count beta = 0 (and
min/max/avg beta = 1) without checking where its group-by keys come
from. Grouping by an estimate moves rows between groups, so the cap
passed down was infinite and no readout was pruned. The keys now have
to be copied from a source, as a Filter's predicate columns must.
…dout no query gives a level (#1207 PR 4)

A query whose whole-result failure probability has no per-value one
marked every readout copy it reached, including the levels other
queries wrote, so a readout another query could use was pruned with
ValuesPerEvaluationUnknown. Only a readout without any level is pruned
now.
…ation grid its width divides (#1207 PR 4)

Chain widths are the gcd closure over every query reaching a state
group, so a state read every 15 s and every 10 s got 5, 10 and 15 s
chains, and the 10 s reader could be given the 15 s chain. The edge
from a SummaryTumblingWindowChain into a group whose result depends on
the evaluation time now requires the width to divide that group's
evaluation grid; the merges over the other widths are pruned.
…er, with the source frontend's comparison semantics (#1207 PR 4)

rollup_from added the compensating Filter group before the memo checked
the rollup candidate, so a rejected rollup (COUNT(*) WHERE src = 'a'
from COUNT(*) GROUP BY src) left a Filter group no group reads. The
Filter now stays pending until the rollup over it derives the coarse
group's schema (Memo::output_schema_of_pending_expression).

The Filter also took PromQL comparison semantics for any group that is
not a relation, such as an exact sum state over a SQL table; it now
takes the semantics of the frontend that reads the column's source.
… and chain edges as the code does them (#1207 PR 4)

The TopKReuse row described a Limit(k_max) over Sort; the rule builds
Aggregate[TopK k] over the largest top-k group's ranked rows. A reader
without a declared phase has no known evaluation grid, so no chain is
built for it, rather than being taken to evaluate at multiples of its
interval. The TumblingWindows row now states the chain-width edge rule.
…valuation grid (#1207 PR 4)

A chain whose width does not divide the consumer group's evaluation period
no longer drops its edges silently: they fail with
RootPruneReason::ChainWidthDoesNotDivideEvaluationGrid, so the candidates
over that chain trace to it instead of an empty cause list.
…adouts without a guarantee (#1207 PR 4)

- A Project's contract judges only the items reading a computed value:
  copied labels, the time and promql_drop_metric_name of the series
  identity keep any error; promql_abs/clamp/clamp_min/clamp_max and a
  negation keep an absolute error; a product with or quotient by a nonzero
  literal keeps a relative one. abs(count_over_time(m[1m])),
  -count_over_time(m[1m]) and quantile_over_time(...) / 2 now keep their
  sketch readouts.
- readout_metric no longer needs a finite failure probability (HLL's
  distinct count is relative). Under a query with a target, an edge into a
  readout with no metric (UnivMon's distinct count) or whose guarantee's
  failure probability is unknown (HLL) fails with the new
  RootPruneReason::NoGuarantee { consumer, input }.
- sketch_guarantee is replaced by guarantee_of_parameters (θ's own
  guarantee) and the δ check in theoretical_guarantee.
…-in, per-query exact-only groups (#1207 PR 4)

- Evaluation grids are computed per (query, group) walking down from each
  root (evaluation_grids.rs): below PromqlSubquery { resolution: Some(r) }
  the grid is r, with None it is unknown (no chain). Chains and the chain
  edge check read it, and evaluation_grid_of_group is the corrected grid.
- A readout's per-value failure probability is δ / max(n_q, the readout
  group's values per evaluation), from the new
  Memo::series_count_scanned_below: max(quantile_over_time(0.9, m[5m]))
  over 1000 series at δ = 0.01 requires 1e-5 per readout.
- A factor-0 contract (count) passes no cap: no infinite requirement.
- New LogicalASAPMemoCandidates::exact_only_groups_of_query: per query,
  the groups it reaches through a consumer with no error contract for its
  metric, plus the readout groups another query gave a level while its own
  whole-result δ has no per-value one; closed downward. Stage 3 is to
  enforce it per query.
… with the frontend's comparison (#1207 PR 4)

- A rolled-up Aggregate names each measure as the coarse group's column,
  so count by (job) rolls up count by (job, instance) as a sum named
  count. The memo's schema check still rejects a SQL global count over a
  grouped one (sum over no group is NULL, the count 0).
- A compensating filter is built from comparisons with the source
  frontend's semantics: a PromQL != matcher becomes Compare Ne with PromQL
  semantics, NOT IN a conjunction of Ne, IN a disjunction of Eq.
…tracts, readout fan-in and exact-only groups (#1207 PR 4)
…ers, tests of floor helpers in the test module, updated file paths (#1207 PR 4)
…ted metrics and caps; readout metrics and accepted input metrics computed once per edge pass; readout.rs, binomial.rs, target_limit beside the edge check (#1207 PR 4)

- accepted_metrics derives from the readout caps' walk (a reached flag
  per group); AccuracyTarget::metrics is the one target-metric expansion.
- readout_levels keys its result by readout candidate, so Pass 2 no
  longer detects unrequired readouts a second time.
- readout_metric_of_candidate and metrics_accepted_on_inputs are computed
  once per edge pass / consumer instead of per edge; accepted metrics and
  evaluation grids once per memo state and passed in.
- A readout's summary type is read with
  GroupProperties::sketch_algorithm_of_state everywhere.
- MAX_ERROR_LEVELS moves next to readout_levels; guarantee, hll and
  univmon are private; kmv_guarantee and meets are inlined.
- Docs: a chain's edge fails with ChainWidthDoesNotDivideEvaluationGrid;
  max_distinct_per_evaluation has no reader yet.
…res in test_support; Pass 2 tests keep what the stage does with the accuracy results (#1207 PR 4)
…d failure probability, the delta grid, top-k and Hydra status, current Pass 2 and accuracy file layout (#1207 PR 4)
…d named readout results; exact-only per query reads readout metrics and live candidates (#1207 PR 4)

- ErrorMetricsOfMemo, built once per memo state before an edge pass,
  holds each candidate's error contract per metric, the error metrics
  each group's output may have and each sketch readout's error metric.
  Its methods check one logical edge (edge_meets_logical_requirements)
  and give the groups each query may read only exactly
  (exact_only_groups_of_query); accepted_metrics,
  metrics_accepted_on_inputs, readout_metric_of_candidate and
  metrics_carried are folded into it.
- ErrorPropagation { factor_per_input, validity_cap_per_input } is
  ErrorContract { error_factor_per_input,
  linear_bound_holds_up_to_per_input }, in error_contract.rs;
  target_limit is within_limit(evaluated, limit) -> bool.
- readout_levels returns ReadoutRequiredAccuracy per readout; step 3
  returns ReadoutRequirementsOutcome.
- A query whose target metric a sketch readout's error is not in reads
  that readout group only exactly; the exact-only walk skips dead
  candidates; a scalar root's operands are roots (scalar(v) and a
  scalar subquery keep the target's metrics, any other scalar
  expression reads its operands exactly).
- The per-value failure probability splits over at most the readout
  group's own values (Memo::max_values_per_evaluation: the series of the
  scanned sources, and the groups a state's keys form).
- A state under `@ T` reads the chains every unanchored state created,
  in any memo order.
- The guarantee is computed once per edge; the guarantee vocabulary
  keeps only what is read (BoundExpr::Constant, ProbabilityExpr::{Zero,
  Constant, Unknown}); provenance and GuaranteeSource are deleted.
- Readout copies are added with their requirement; same_candidate
  compares input requirements first.
- Rollup is split into rules/rollup/{member,finest_common_group,
  rewrite}.rs; the rule helpers move to rules/candidates.rs; chain_off_the_
  evaluation_grid is a function of evaluation_grids.rs; evaluation grids
  are computed once for steps 4 and 5.
…PR 4)

Architecture doc: current module paths for Pass 2, cost, physical
properties and ir::properties; crate dependency lines match the
manifests' used edges. post-asap-ir: note that the physical properties
assignment is gone. Accuracy proposal: status says what is implemented
and marks composition, the pipeline and the budget allocator as replaced
by per-operator error contracts.
… memo builders; HAVING placement in candidates; test_support forwarders and default metric files (#1207 PR 4)

- Pass 1's tests.rs is split into tests/{running_example,series_identity,
  targets,candidates,having,fused,nested_merges,update_weights,intents,
  summary_realization,unsupported_ways}.rs; candidates_of and
  groups_where move to types::test_support::memo_builders, and Pass 1's
  and Pass 2's copies are deleted.
- Pass 1 drops its unused asap-accuracy dependency.
- HAVING placement (having_filter_groups, has_having) moves to
  candidates/having.rs; computation_candidates_for_intent is
  ways_to_compute_intent; with_raw_data_samples_requirement is
  with_raw_data_samples_of.
- test_support.rs is test_support/{mod,forwarders}.rs; the default-metric
  IR walk moves from accuracy/target.rs to accuracy/default_metric.rs.
…res by behavior (#1207 PR 4)

Add planning_common (fixture loader, cost, PromQL Pass 2 runner); fold
identical_exact_queries into plan_sharing (was sharing_issue_regressions);
merge the two pass2 PromQL files into pass2_promql/ and pass1_candidates_sql
into pass1_candidates; name fixtures by what they hold, drop the unused one.
…limit names; per-value failure probability over the readout's own values; exact-only list per query, its per-group limit and subqueries without a contract; current Pass 2 and accuracy file layout (#1207 PR 4)
…e their files; tests for readout metrics per query, dead candidates in the exact-only list, one-value readouts, scalar roots and anchored chains in any memo order (#1207 PR 4)

- Every touched source file's inline tests move to a sibling tests.rs
  (x.rs becomes x/{mod,tests}.rs) in the accuracy crate and in types'
  accuracy and summary_type modules.
- New tests: each_query_reads_only_the_readouts_in_its_targets_metric
  and a_dead_candidate_makes_no_group_exact_only (PromQL through
  production lowering), a_global_distinct_count_keeps_its_whole_failure_
  probability (SQL), a_readout_of_one_value_keeps_the_whole_failure_
  probability, max_values_per_evaluation_bounds_a_readout_by_its_states_
  keys, a_scalar_roots_operands_are_roots,
  an_absolute_anchor_reuses_a_chain_a_later_group_creates.
- The integration helper takes each query's accuracy (promql_pass2) and
  runs SQL over a flows table (sql_pass2).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Workload optimization and plan selection refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Rollup across label sets is never offered for PromQL queries (#1239)

1 participant