Repository navigation
refactor(stage1-pass2): rollup from the finest common group, readout requirements, dead-candidate elimination, chain groups (#1207 PR 4) - #1234
Merged
Conversation
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 7, 2026 04:14
d4bde07 to
441f3a9
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 7, 2026 05:10
72a4037 to
24dd00c
Compare
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 9, 2026 06:37
2c6d993 to
61f585a
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 9, 2026 07:11
bdf0888 to
bcae9e5
Compare
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 9, 2026 16:50
61f585a to
a71e517
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
2 times, most recently
from
November 9, 2026 20:52
8107319 to
5d3f6de
Compare
This was referenced Nov 10, 2026
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 10, 2026 01:26
a71e517 to
d77278a
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 10, 2026 01:26
5d3f6de to
dea088e
Compare
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 10, 2026 13:37
d77278a to
0c11cd7
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 10, 2026 13:37
dea088e to
e98e80b
Compare
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 11, 2026 03:35
0c11cd7 to
321a1ae
Compare
…requirements, dead-candidate elimination, chain groups (#1207 PR 4) Pass 2 runs, in order: the general rollup rule with one synthesized finest common group per cluster of equal definitions (replaces scan widening, finer-aggregate reuse and the rollup rules); top-k reuse; reach, caps and readout requirements (one readout candidate per requirement level, written with Memo::set_candidate_input_requirements); dead-candidate elimination with every root cause (PrunedLogicalCandidate); chain groups SummaryTumblingWindowChain[w](X) for the surviving state groups, widths from the gcd closure of the reaching queries' grids, the window and the offsets, then elimination again; and the logical edges between survivors. A merge, and a Filter/TimeRange/TimeShift over states, carries the θ of the states it reads (one candidate per θ, reading only producers with that θ). The metric rule cuts edges from readouts whose metric the consumer's input does not accept (accepted metrics pushed down from the targeted queries), not candidates; a candidate cut off from an input is eliminated too. A whole-result δ with n_q unknown prunes the readouts it reaches (ValuesPerEvaluationUnknown). - stage1-pass2: the entry generate_logical_asap_memo_candidates and the possible edges in lib.rs; rules.rs (Pass2RuleError, add_candidate, add_candidates_per_input_parameters, add_group_computing_itself), rules/{rollup,topk,chains}.rs, requirements.rs, dead_candidates.rs, tests.rs and tests/decided_examples.rs. - types: AccuracyMetric::ALL; RootPruneReason::{HeapSmallerThanTopK, ValuesPerEvaluationUnknown}; MetricRule.accepted_metrics; PruneCause.through_pruned_candidates; Memo::producers; add_candidate checks θ against the group's state (CandidateError::LogicalProperties, candidate::parameters_fit); a candidate's input requirements are part of its identity (hash_consing::same_candidate); LogicalProperties::new accepts any θ on a candidate passing states on; aggregate coverage and selection accessors no longer test-only; origins SummaryInput and WidenedScans deleted. - accuracy: error_propagation(memo, consumer, metric) with the constant contracts (key-only Filter by column lineage, abs / clamp / negation, min / max / avg / count, + - / *), b = 0.1 private; theoretical_guarantee at a failure probability (KLL and KMV evaluated at it, others checked against their own); producer_satisfies_logical_requirements returns the failing root reason; accepted_metrics and readout_metric. Node-tree composition, propagation (with its count-min estimate_has_bound test), budget split, function rules, PropagationStats and the test-only sizing (guarantee/sizing_oracle) deleted; the classic HLL precision tests move into guarantee/hll.rs. - devtools: one pass2 stage; docs follow. - integration-tests: pass1_candidates_sql re-adds a_filter_or_group_key_over_an_estimate_has_no_guarantee.
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 11, 2026 18:33
e98e80b to
5ac57fc
Compare
… guarantees no stage reads yet (#1207 PR 4) classic_guarantee, ClassicHllConfidence, classic_failure_probability and the classic statistic helpers, and UnivMon's distinct_guarantee with its incomplete-top-layer statistic, had no caller outside their own tests. PR 8 re-adds what its HLL and UnivMon distinct-count guarantees use.
… error contract (#1207 PR 4) An Aggregate or exact accumulator build gave a count beta = 0 (and min/max/avg beta = 1) without checking where its group-by keys come from. Grouping by an estimate moves rows between groups, so the cap passed down was infinite and no readout was pruned. The keys now have to be copied from a source, as a Filter's predicate columns must.
…dout no query gives a level (#1207 PR 4) A query whose whole-result failure probability has no per-value one marked every readout copy it reached, including the levels other queries wrote, so a readout another query could use was pruned with ValuesPerEvaluationUnknown. Only a readout without any level is pruned now.
…ation grid its width divides (#1207 PR 4) Chain widths are the gcd closure over every query reaching a state group, so a state read every 15 s and every 10 s got 5, 10 and 15 s chains, and the 10 s reader could be given the 15 s chain. The edge from a SummaryTumblingWindowChain into a group whose result depends on the evaluation time now requires the width to divide that group's evaluation grid; the merges over the other widths are pruned.
…er, with the source frontend's comparison semantics (#1207 PR 4) rollup_from added the compensating Filter group before the memo checked the rollup candidate, so a rejected rollup (COUNT(*) WHERE src = 'a' from COUNT(*) GROUP BY src) left a Filter group no group reads. The Filter now stays pending until the rollup over it derives the coarse group's schema (Memo::output_schema_of_pending_expression). The Filter also took PromQL comparison semantics for any group that is not a relation, such as an exact sum state over a SQL table; it now takes the semantics of the frontend that reads the column's source.
… and chain edges as the code does them (#1207 PR 4) The TopKReuse row described a Limit(k_max) over Sort; the rule builds Aggregate[TopK k] over the largest top-k group's ranked rows. A reader without a declared phase has no known evaluation grid, so no chain is built for it, rather than being taken to evaluate at multiples of its interval. The TumblingWindows row now states the chain-width edge rule.
…valuation grid (#1207 PR 4) A chain whose width does not divide the consumer group's evaluation period no longer drops its edges silently: they fail with RootPruneReason::ChainWidthDoesNotDivideEvaluationGrid, so the candidates over that chain trace to it instead of an empty cause list.
…adouts without a guarantee (#1207 PR 4) - A Project's contract judges only the items reading a computed value: copied labels, the time and promql_drop_metric_name of the series identity keep any error; promql_abs/clamp/clamp_min/clamp_max and a negation keep an absolute error; a product with or quotient by a nonzero literal keeps a relative one. abs(count_over_time(m[1m])), -count_over_time(m[1m]) and quantile_over_time(...) / 2 now keep their sketch readouts. - readout_metric no longer needs a finite failure probability (HLL's distinct count is relative). Under a query with a target, an edge into a readout with no metric (UnivMon's distinct count) or whose guarantee's failure probability is unknown (HLL) fails with the new RootPruneReason::NoGuarantee { consumer, input }. - sketch_guarantee is replaced by guarantee_of_parameters (θ's own guarantee) and the δ check in theoretical_guarantee.
…-in, per-query exact-only groups (#1207 PR 4) - Evaluation grids are computed per (query, group) walking down from each root (evaluation_grids.rs): below PromqlSubquery { resolution: Some(r) } the grid is r, with None it is unknown (no chain). Chains and the chain edge check read it, and evaluation_grid_of_group is the corrected grid. - A readout's per-value failure probability is δ / max(n_q, the readout group's values per evaluation), from the new Memo::series_count_scanned_below: max(quantile_over_time(0.9, m[5m])) over 1000 series at δ = 0.01 requires 1e-5 per readout. - A factor-0 contract (count) passes no cap: no infinite requirement. - New LogicalASAPMemoCandidates::exact_only_groups_of_query: per query, the groups it reaches through a consumer with no error contract for its metric, plus the readout groups another query gave a level while its own whole-result δ has no per-value one; closed downward. Stage 3 is to enforce it per query.
… with the frontend's comparison (#1207 PR 4) - A rolled-up Aggregate names each measure as the coarse group's column, so count by (job) rolls up count by (job, instance) as a sum named count. The memo's schema check still rejects a SQL global count over a grouped one (sum over no group is NULL, the count 0). - A compensating filter is built from comparisons with the source frontend's semantics: a PromQL != matcher becomes Compare Ne with PromQL semantics, NOT IN a conjunction of Ne, IN a disjunction of Eq.
…tracts, readout fan-in and exact-only groups (#1207 PR 4)
…ing tests, view coverage in plan (#1207 PR 4)
…rator and plan nodes (#1207 PR 4)
…ers, tests of floor helpers in the test module, updated file paths (#1207 PR 4)
…ted metrics and caps; readout metrics and accepted input metrics computed once per edge pass; readout.rs, binomial.rs, target_limit beside the edge check (#1207 PR 4) - accepted_metrics derives from the readout caps' walk (a reached flag per group); AccuracyTarget::metrics is the one target-metric expansion. - readout_levels keys its result by readout candidate, so Pass 2 no longer detects unrequired readouts a second time. - readout_metric_of_candidate and metrics_accepted_on_inputs are computed once per edge pass / consumer instead of per edge; accepted metrics and evaluation grids once per memo state and passed in. - A readout's summary type is read with GroupProperties::sketch_algorithm_of_state everywhere. - MAX_ERROR_LEVELS moves next to readout_levels; guarantee, hll and univmon are private; kmv_guarantee and meets are inlined. - Docs: a chain's edge fails with ChainWidthDoesNotDivideEvaluationGrid; max_distinct_per_evaluation has no reader yet.
…res in test_support; Pass 2 tests keep what the stage does with the accuracy results (#1207 PR 4)
…d failure probability, the delta grid, top-k and Hydra status, current Pass 2 and accuracy file layout (#1207 PR 4)
…d named readout results; exact-only per query reads readout metrics and live candidates (#1207 PR 4) - ErrorMetricsOfMemo, built once per memo state before an edge pass, holds each candidate's error contract per metric, the error metrics each group's output may have and each sketch readout's error metric. Its methods check one logical edge (edge_meets_logical_requirements) and give the groups each query may read only exactly (exact_only_groups_of_query); accepted_metrics, metrics_accepted_on_inputs, readout_metric_of_candidate and metrics_carried are folded into it. - ErrorPropagation { factor_per_input, validity_cap_per_input } is ErrorContract { error_factor_per_input, linear_bound_holds_up_to_per_input }, in error_contract.rs; target_limit is within_limit(evaluated, limit) -> bool. - readout_levels returns ReadoutRequiredAccuracy per readout; step 3 returns ReadoutRequirementsOutcome. - A query whose target metric a sketch readout's error is not in reads that readout group only exactly; the exact-only walk skips dead candidates; a scalar root's operands are roots (scalar(v) and a scalar subquery keep the target's metrics, any other scalar expression reads its operands exactly). - The per-value failure probability splits over at most the readout group's own values (Memo::max_values_per_evaluation: the series of the scanned sources, and the groups a state's keys form). - A state under `@ T` reads the chains every unanchored state created, in any memo order. - The guarantee is computed once per edge; the guarantee vocabulary keeps only what is read (BoundExpr::Constant, ProbabilityExpr::{Zero, Constant, Unknown}); provenance and GuaranteeSource are deleted. - Readout copies are added with their requirement; same_candidate compares input requirements first. - Rollup is split into rules/rollup/{member,finest_common_group, rewrite}.rs; the rule helpers move to rules/candidates.rs; chain_off_the_ evaluation_grid is a function of evaluation_grids.rs; evaluation grids are computed once for steps 4 and 5.
…PR 4) Architecture doc: current module paths for Pass 2, cost, physical properties and ir::properties; crate dependency lines match the manifests' used edges. post-asap-ir: note that the physical properties assignment is gone. Accuracy proposal: status says what is implemented and marks composition, the pipeline and the budget allocator as replaced by per-operator error contracts.
… memo builders; HAVING placement in candidates; test_support forwarders and default metric files (#1207 PR 4) - Pass 1's tests.rs is split into tests/{running_example,series_identity, targets,candidates,having,fused,nested_merges,update_weights,intents, summary_realization,unsupported_ways}.rs; candidates_of and groups_where move to types::test_support::memo_builders, and Pass 1's and Pass 2's copies are deleted. - Pass 1 drops its unused asap-accuracy dependency. - HAVING placement (having_filter_groups, has_having) moves to candidates/having.rs; computation_candidates_for_intent is ways_to_compute_intent; with_raw_data_samples_requirement is with_raw_data_samples_of. - test_support.rs is test_support/{mod,forwarders}.rs; the default-metric IR walk moves from accuracy/target.rs to accuracy/default_metric.rs.
…res by behavior (#1207 PR 4) Add planning_common (fixture loader, cost, PromQL Pass 2 runner); fold identical_exact_queries into plan_sharing (was sharing_issue_regressions); merge the two pass2 PromQL files into pass2_promql/ and pass1_candidates_sql into pass1_candidates; name fixtures by what they hold, drop the unused one.
…d integration tests (#1207 PR 4)
…limit names; per-value failure probability over the readout's own values; exact-only list per query, its per-group limit and subqueries without a contract; current Pass 2 and accuracy file layout (#1207 PR 4)
…e their files; tests for readout metrics per query, dead candidates in the exact-only list, one-value readouts, scalar roots and anchored chains in any memo order (#1207 PR 4) - Every touched source file's inline tests move to a sibling tests.rs (x.rs becomes x/{mod,tests}.rs) in the accuracy crate and in types' accuracy and summary_type modules. - New tests: each_query_reads_only_the_readouts_in_its_targets_metric and a_dead_candidate_makes_no_group_exact_only (PromQL through production lowering), a_global_distinct_count_keeps_its_whole_failure_ probability (SQL), a_readout_of_one_value_keeps_the_whole_failure_ probability, max_values_per_evaluation_bounds_a_readout_by_its_states_ keys, a_scalar_roots_operands_are_roots, an_absolute_anchor_reuses_a_chain_a_later_group_creates. - The integration helper takes each query's accuracy (promql_pass2) and runs SQL over a flows table (sql_pass2).
zzylol
marked this pull request as ready for review
November 12, 2026 18:47
This was referenced Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1281 (a coarser PromQL aggregate rolls up from a finer one although their scans list different labels).
Summary
todo!(), so every planning run stops there; the old rules matched shapes pairwise by column name, and θ floors per input mixed ε and δ and could not express a multi-readout allocation.generate_logical_asap_memo_candidates) runs the reuse rules over memo groups (rollup from the finest common group, top-k reuse), gives each readout its required accuracy, eliminates dead candidates with root causes, builds chain groups and the logical edgesquantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m])at relative 0.05 every 60 s, Pass 2 prunes both KLL readouts (MetricRule, rank error) and the DDSketch builds with α 0.1 and 0.05, gives each DDSketch readout five levels 0.02 … 0.001, and builds one 60 s DDSketch chain with five builds; planning now stops at Stage 2 (PR 5). The snapshot workloadratio-of-quantiles-1h-mvpplanned at 7.481e-06/s with the full stack before this PR's review fixes (D1); it is re-measured when the stack above is restacked.Review guide
main(PR 3, refactor(stage1-pass1): per-computation candidates over memo groups (#1207 PR 3) #1232, is merged); refactor(stage2): physical expansion (decision-tree levels 3–4), physical edges, pruned alternatives; Count-Min weights via column_signs (#1207 PR 5) #1235 (PR 5) builds on it. 35 commits, head9ca4f50a1.stage1-pass2-reuse-candidates/src/lib.rs(the six steps in order),edges.rs,readout_requirements.rs,dead_candidates.rs,evaluation_grids.rs.rules/rollup/{mod,member,finest_common_group,rewrite}.rs(add_rollup_candidates),rules/topk.rs,rules/chains.rs,rules/candidates.rs(how rules add candidates).accuracy/src/metric_rule/mod.rs(ErrorMetricsOfMemo: the contracts, allowed metrics and readout metrics of one memo state;exact_only_groups_of_query),logical_edge/mod.rs(edge_meets_logical_requirements,within_limit),error_contract/mod.rs(ErrorContract),readout_caps/mod.rs(the caps walk,readout_levels→ReadoutRequiredAccuracy),readout/mod.rs(readout_error_metric).types/src/stages/logical/{candidate,memo,group}/mod.rs(θ on pass-through candidates,Memo::producers,Memo::max_values_per_evaluation,same_candidate) andtypes/src/stages/pruning.rs.asap_accuracy::{composition, propagation, propagation_stats, budget, function_rules, requirements, sizing},guarantee/sizing_oracle.rs, the classic HLL and UnivMon distinct-count guarantees, the unused guarantee vocabulary intypes/src/accuracy/guarantee.rs, and the never-setguaranteefield ofOperatorNode,FlatNodeandPhysicalASAPDAGNode(withOperatorNode::with_guarantee; the executor and integration fixtures drop it).accuracy/src/requirements.rs(the edge-check stub,target_limit,meets) is replaced bylogical_edge/(withwithin_limit);metric_rule/,readout/,readout_caps/anderror_contract/are new code;binomial_upper_tailintoguarantee/binomial.rs;MAX_ERROR_LEVELSis new inreadout_caps/; types'group.rs,aggregate_coverage.rsandcolumn_properties.rsintostages/logical/group/,stages::view_coverageintoplan::view_coverage(no re-export fromstages), the default-metric IR walk fromaccuracy/target.rsintoaccuracy/default_metric.rs,test_support.rsintotest_support/{mod,forwarders}.rs; every touched source file's inline tests into a siblingtests.rs(x.rs→x/{mod,tests}.rs), hash-consing tests next tohash_consing.tests.rs(51 KB) split by scope intotests/{running_example,series_identity,targets,candidates,having,fused,nested_merges,update_weights,intents,summary_realization,unsupported_ways}.rs; HAVING placement (having_filter_groups,has_having) intocandidates/having.rs;computation_candidates_for_intent→ways_to_compute_intent,with_raw_data_samples_requirement→with_raw_data_samples_of; the unusedasap-accuracydependency is dropped.sharing_issue_regressions.rs+identical_exact_queries.rs→plan_sharing.rswith fixtures named by what they hold (plan_sharing_fixtures/),pass2_*_promql.rs→pass2_promql/{rollup,accuracy_and_grids,exact_only_groups}.rspluspass2_sql.rs,pass1_candidates_sql.rsfolded intopass1_candidates.rs,stored_outputs_on_main.json→stored_outputs_before_view_coverage.json,types/tests/physical_export.rs→physical_asap_dag.rs; one shared helper moduletests/planning_common/(fuzz_workload,with_queries,cost,promql_pass2,sql_pass2). PR and issue numbers are removed from code comments.tests/{rollup/{filters,finest_common_group,measures},topk,readout_requirements,dead_candidates,chains,edges,evaluation_grids,exact_only_groups}.rsplustest_fixtures.rs; the accuracy-math tests sit in the accuracy crate next to their files; shared memo builders inasap_types::test_support::memo_builders(pluscompareandarithmeticinmemo_fixtures;candidates_of,groups_whereandQuantileReadoutinmemo_builders); devtools drops the two removed origins from its labels.Filter,Projectand reduction contracts that read column lineage (error_contract); the caps walk and the readout-level grid with D1's extension; the per-value δ split over the readout group's own values (max_values_per_evaluation);NoGuarantee; two-direction elimination and its traces; θ matching on edges (Memo::producers); the metric rule cutting edges, not candidates, andexact_only_groups_of_query(live candidates only, readout metrics per query, scalar roots); evaluation grids below subqueries, chain widths (gcd closure,@ Tafter every unanchored state) and the chain-edge divisibility check; the rollup pre-check before the compensatingFilterand its frontend semantics.crates/accuracy/src:lib.rs,evidence.rs,{error_contract,metric_rule,readout,readout_caps,logical_edge}/{mod,tests}.rs,guarantee/{mod,tests,binomial,ddsketch,hll}.rs,guarantee/{cardinality,cms,count_sketch,kll,univmon}/{mod,tests}.rs.crates/stage1-pass2-reuse-candidates/src:lib.rs,edges.rs,evaluation_grids.rs,readout_requirements.rs,dead_candidates.rs,rules/{mod,candidates,topk,chains}.rs,rules/rollup/{mod,member,finest_common_group,rewrite}.rs,test_fixtures.rs,tests.rs,tests/{rollup,topk,readout_requirements,dead_candidates,chains,edges,evaluation_grids,exact_only_groups}.rs,tests/rollup/{filters,finest_common_group,measures}.rs.Behavior: before → after
Concepts doc "ratio of quantiles" (test
dead_candidates_are_pruned_and_listed_before_chain_groups):quantile_over_time(0.99, m[5m]) / quantile_over_time(0.5, m[5m]), relative error 0.05, every 60 s at phase 0. It serves the design need that Pass 2 enumerates only candidates some query can use, with a reason for every one it removes.todo!(); planning stopsLogicalASAPMemoCandidates(memo, edges, pruned, evaluation grids, exact-only groups per query); planning stops at Stage 2 (todo!of PR 5)MetricRule { producer_metric: Rank, accepted_metrics: [Relative] }; every KLL build through its edge into a KLL readoutBelowRequiredAccuracy; five surviveSummaryTumblingWindowChain[DDSketch, w = 60 s](Scan m)with the 5 surviving builds;SummaryMerge(TimeRange(5m, chain))in the DDSketch state group… > 1over the same ratio would list the ratio group and every group below itInterfaces
Permalinks are at head
9ca4f50a1913666f68330e84b181753f8e8ab9f0.crates/stage1-pass2-reuse-candidates/src/lib.rs/// Stage 1, Pass 2: steps 1–6; `entries[i]` is query i's workload entry (its evaluation grid). /// The planner facade calls it after Pass 1. Signature unchanged; now implemented. pub fn generate_logical_asap_memo_candidates( pass1: Memo, entries: &[QueryWorkloadEntry], evidence: &dyn AccuracyEvidence, ) -> Result<LogicalASAPMemoCandidates, Pass2RuleError>; /// Why Pass 2 failed on a memo: a rule bug or a cyclic memo. Stays; now defined in `lib.rs`. pub enum Pass2RuleError { - /// Never produced. - InvalidGroupReference, - /// Never produced. - TooManyGroups, /// A group, or one of its candidates, reads itself through other groups. CyclicDependencies, + /// Internal bug: a rule built a group expression the node rules reject. + Group(#[from] MemoGroupError), + /// Internal bug: a rule built a candidate expression the node rules reject, or gave it θ its + /// group's output cannot have. + Candidate(CandidateError), }crates/types/src/stages/logical/candidate/mod.rs·memo/mod.rs·group/mod.rs·memo_candidates.rs·crates/types/src/stages/pruning.rsadd_candidatekeeps its signature but now identifies a candidate by expression, θ and input requirements (same_candidate), and rejects a θ that does not fit the group.crates/types/src/ir/operator/node.rs·ir/flat.rs·plan/physical_asap_dag.rs/// A node of the logical operator DAG. Stays. pub struct OperatorNode { pub operator: Operator, pub result_kind: OperatorResultKind, pub schema: Schema, - pub guarantee: Option<AccuracyGuarantee>, } impl OperatorNode { - pub fn with_guarantee(mut self, guarantee: Option<AccuracyGuarantee>) -> Self; } /// An `OperatorNode` whose children are `NodeId`s. Stays. pub struct FlatNode { pub operator: Operator<NodeId>, pub(crate) result_kind: OperatorResultKind, pub(crate) schema: Schema, - pub(crate) guarantee: Option<AccuracyGuarantee>, } /// A node of the installable physical plan. Stays; every other field unchanged. pub struct PhysicalASAPDAGNode { pub id: PhysicalASAPNodeId, pub payload: PhysicalASAPOperatorPayload, pub output_state: ExecutionDataState, pub lifetime: Lifetime, pub parameters: Option<SketchParams>, pub output_schema: Schema, - pub guarantee: Option<AccuracyGuarantee>, pub retention_ms: Option<u64>, pub selected_physical_candidate: Option<PhysicalCandidateId>, pub view_coverage: Option<ViewCoverage>, }The only writer of these fields was the deleted
asap_accuracy::propagation; every remaining site only copied the value, so it was alwaysNone. Wire format: a serialized plan node no longer has theguaranteekey (PhysicalASAPDAGNodeisdeny_unknown_fields, so a plan written before this PR with"guarantee": nullno longer parses); the fixturesplan_wire_version_10.jsonandstored_outputs_on_main.jsondrop it.INSTALLABLE_PLAN_WIRE_VERSIONgoes from 14 to 15 (decided in review), so an older plan file is refused by its version.crates/types/src/ir/properties/selection.rs·stages/logical/group/aggregate_coverage.rs·accuracy/·summary_type/Removed because nothing reads them: composition and budget types,
AccuracyError,over_scopeandapproximate_layer_count, the symbolic bound and probability combinators,bind,is_exact,is_zero,has_unknown, and the provenance trail (GuaranteeSource; each edge serialized θ to JSON for it). The later stack branches use onlyConstant,ProbabilityExpr::{Zero, Constant, Unknown}andAccuracyGuarantee::new, apart from a pre-restack copy of the code this PR replaces. The/* N variants */lines stand for whole enums removed with all their variants.crates/accuracy/src/lib.rs·logical_edge/·metric_rule/·readout_caps/·error_contract/·guarantee/mod.rsRemoved: node-tree composition and propagation (
composition,propagation,propagation_stats,budget::per_input_target, privatefunction_rules) are replaced by per-edge error contracts plus readout requirements. Also removed: the test-onlyguarantee::sizing_oracle, the classic HLL and UnivMon distinct-count guarantees (a later PR in this stack brings back certified distinct-count guarantees), the fixed-δ KLL and cardinality guarantee lookups (θ's own guarantee plus one δ check intheoretical_guaranteereplace them), the test-onlymeets(its one caller checks the bound withwithin_limit), andPerComputationCandidateError::LogicalProperties(θ is now checked in one place,Memo::add_candidate).ErrorContractkeepsPartialEqfor tests only.guaranteeand its per-algorithm modules are private. The readout metric (readout_error_metric), the metrics passed to inputs and the metrics allowed on a consumer's inputs are crate-private parts ofErrorMetricsOfMemo; Pass 2's step 3 returns a crate-privateReadoutRequirementsOutcome { readouts_without_per_value_failure_probability, exact_only_readout_groups_of_query }.Usage, end to end (test
dead_candidates_are_pruned_and_listed_before_chain_groupsat9ca4f50a1):Algorithm changes
Pass 2's steps:
rules/rollup/add_rollup_candidatesSummaryMerge(group_by)for states,sum/min/maxfor finalized sums, counts, minima, maxima, each measure named asTimeRange,TimeShiftand per-series layers (e.g.rate), and scans listing different labels match by column lineagerules/topk.rsreadout_requirements.rs,accuracy::readout_capsErrorContracts ofErrorMetricsOfMemo; each sketch readout gets one candidate per requirement level (single-read query: the cap; composed query: the ε grid × δ grid not looser than the cap and per-value δ); per-value δ = δ / max(Memo::max_values_per_evaluationof the readout group)dead_candidates.rs,edges.rsLogicalASAPMemoCandidates::prunedlists each with every root cause (one shortest path per failing edge)rules/chains.rsSummaryTumblingWindowChain[w](X)per width (gcd closure ofTimeShift/TimeRangegroups,SummaryMergein the state group;@ Treuses the chains the unanchored states create (in any memo order);@ end()/@ start()noneedges.rs,evaluation_grids.rs,lib.rsErrorMetricsOfMemo::edge_meets_logical_requirements; an edge from a chain into a reader whose evaluation grid (evaluation_grids: gcd of the reaching queries' grids) its width does not divide fails withChainWidthDoesNotDivideEvaluationGridErrorMetricsOfMemo::exact_only_groups_of_queryRules add only through
find_or_add_group/add_candidateand swallow onlyOutputDiffersFromGroup;add_candidatetreats input requirements as part of a candidate's identity (two readouts with different requirements are different candidates);RELATIVE_OPERAND_VALIDITY_CAP = 0.1is private;same_candidatecompares input requirements first, and step 3 adds each readout copy with its requirement.theoretical_guaranteeevaluates KLL (rank error scaled below δ = 0.01) and KMV (bisection on ε) at the required δ, and checks every other algorithm's own failure probability against it.Rollup
TimeRange,TimeShiftand per-series layers (Member::see_through_per_series_layers), so equal computations compare equal whatever their keys, label selections and the labels their scans list (scans match by lineage; an operator that names columns by position, such as a measure over a column, still needs equal column lists), and equal intervals; (2) per cluster,Rollup(by keys)(Filter(compensation, G_f))when keys ⊆ and selection ⊆ on keys by lineage. The rollup and its schema are checked over a pendingFilterfirst, so a rollup the memo would reject leaves no compensating group behind.Aggregatenames each measure ascount by (job)rolls upcount by (job, instance)as a sum namedcount(a SQL global count over a grouped one is still rejected by the schema check: a sum over no group is NULL, the count 0). The compensatingFilteris built from comparisons with the source frontend's semantics: a PromQL!=matcher becomesCompare Newith PromQL semantics,NOT INa conjunction ofNe,INa disjunction ofEq.a_coarse_exact_sum_rolls_up_the_finer_sum,a_coarse_count_rolls_up_the_finer_count_as_a_sum,a_filtered_coarse_sketch_state_merges_the_finer_states_through_a_compensating_filter,a_finest_common_group_no_query_asks_for_serves_both,a_narrowing_on_a_column_that_is_no_key_gives_no_rollup,a_compensating_filter_over_a_sql_table_has_sql_semantics,a_not_equal_matcher_compensates_with_promql_semantics,a_rejected_rollup_leaves_no_compensating_filter, and through production PromQL lowering (crates/integration-tests/tests/pass2_promql/rollup.rs):an_instant_sum_rolls_up_from_the_finer_sum_over_a_scan_listing_more_labels,an_instant_count_rolls_up_from_the_finer_count,a_sum_of_rates_rolls_up_from_the_finer_sum_of_the_same_rates,a_label_matcher_becomes_a_compensating_filter_over_the_finer_sum, with the negative casesrates_over_different_ranges_do_not_roll_up,different_per_series_functions_do_not_roll_up,a_standard_deviation_does_not_roll_up,a_finer_sum_narrowed_by_a_matcher_serves_no_wider_sum.withoutgroupings, label filters between per-series layers, regex matchers; no new unionScan; equal intervals. The pre-check covers the schema only. The A filtered query (city = 'NYC') doesn't read its group from a grouped summary another query builds #902 case (city = 'NYC'over the per-city group) is not widened: Rollup rule: a filtered query (city = 'NYC') does not read the per-city group another query builds (#902 regression on the #1207 stack) #1252.Readout requirements, the metric rule and dead candidates
0.
ErrorMetricsOfMemo::of: each candidate'sErrorContractper metric and each sketch readout's error metric, once per memo state (step 3, each edge pass); the walks below read them.readout_caps::caps: one walk down from the query roots in reverse topological order. A root with a target is reached with its target in each of its metrics (AccuracyTarget::metrics; a scalar root's operands are roots:scalar(v)and a scalar subquery with the target's metrics, any other scalar expression with none); every candidate of a reached group reaches its inputs and passes each metric its contract carries, with capsreadout_levels: per readout candidate without a requirement, its levels from its group's caps in its metric (single read: the cap at the per-value δ; composed: the ε gridMAX_ERROR_LEVELS× the δ grid 1e-2 … 1e-5 continued in powers of ten down to the first at or below a tenth of the per-value δ, D1).ErrorMetricsOfMemo::error_metrics_allowed_at_output= the metrics of the same walk per group (None: not reached); the metrics allowed on a consumer's inputs are its group's that its contract passes.exact_only_groups_of_query: per query, over the live candidates, the metrics every path to a group passes (an intersection, where the metric rule's allowed metrics are a union over queries); a group where none is left, or that holds a sketch readout whose error metric is not among them, is exact-only for that query, with every group below it.Projectcontract judges only the items reading a computed value: copied labels, the time andpromql_drop_metric_nameof the series identity keep any error;promql_abs/clamp/clamp_min/clamp_maxand a negation keep an absolute error; a product with or quotient by a nonzero literal keeps a relative one.abs(count_over_time(m[1m])),-count_over_time(m[1m])andquantile_over_time(...) / 2now keep their sketch readouts (key_filters_and_value_preserving_projects_have_unit_contracts,a_function_of_the_value_keeps_the_readouts_below_it).readout_metricno longer needs a finite δ (HLL's distinct count is relative); under a query with a target, an edge into a readout with no metric (UnivMon's distinct count) or whose guarantee's δ is unknown (HLL) fails withNoGuarantee(a_distinct_count_readout_without_an_evaluable_guarantee_serves_no_query_with_a_target).Memo::max_values_per_evaluation):max(quantile_over_time(0.9, m[5m]))over 1000 series at δ = 0.01 requires 1e-5 per readout, not 1e-2 (a_max_over_many_readouts_splits_the_failure_probability_over_them);SELECT count(DISTINCT dst) FROM flowsreads one value and keeps 0.01, so KMV k = 4323 survives (a_global_distinct_count_keeps_its_whole_failure_probability).a_count_over_readouts_gives_them_no_level).exact_only_groups_of_query(a_query_comparing_a_shared_ratio_reads_it_only_exactly,a_readout_another_query_gives_a_level_survives_unknown_values_per_evaluation).a_reduction_keyed_by_an_estimate_has_no_contract).ValuesPerEvaluationUnknownprunes only a readout that no query gives a level; a readout another query reaches with a knowna_readout_another_query_gives_a_level_survives_unknown_values_per_evaluation).dead_candidates_are_pruned_and_listed_before_chain_groups,a_single_readout_requires_the_query_target, and in the accuracy cratea_whole_result_failure_probability_is_split_per_value,a_ratio_reads_no_rank_error,a_ratio_whose_per_value_failure_probability_is_below_the_grid_splits_it_over_its_readouts,a_max_over_many_readouts_splits_the_failure_probability_over_them,a_count_over_readouts_gives_them_no_level.ErrorMetricsOfMemoper memo state (contracts once per candidate and metric, readout metrics once per candidate); the caps walk reads the contract table; the guarantee once per edge (no JSON serialization); the evaluation periods once for steps 4 and 5; elimination to a fixpoint over edges.exact_only_groups_of_query.avg by (job)(count_over_time(m[1m]))at absolute 0.05:avghas an absolute contract, the exact accumulator's sum/count candidate has none, so the group below and its readout are exact-only for that query even throughavg. A finer list needs Stage 3 to enforce the metric rule per query on each edge. Likewise a sketch readout placed directly in a group with other candidates makes that whole group exact-only for a query that does not allow its metric (Pass 1 reads readouts through their own groups in production lowering).PromqlSubqueryhas no error contract, so every sketch readout under a subquery serves no query with a target (sound). A contract for it must split δ over the subquery's inner steps (series × range / resolution), not over the series alone.quantile_over_timeat per-value scope and δ 0.01 over 1000 series: 1e-5): sound, stricter than needed.Evaluation grids and chain groups
evaluation_grids.rs): belowPromqlSubquery { resolution: Some(r) }the grid is r, withNoneit is unknown (no chain);evaluation_grid_of_groupis the gcd over the reaching queries. Range-free chain groups after the reach; widths include offsets; built in two phases (all chain builds, then theTimeShift/TimeRange/merge groups) so a chain shared by state groups offers each every θ it has; a chain's edges into a reader whose own evaluation grid the chain's width does not divide fail withChainWidthDoesNotDivideEvaluationGrid { consumer, input, tumbling_window_width_ms, evaluation_period_ms }(a 15 s chain never serves a reader every 10 s, which gave wrong answers before); dead-candidate elimination runs again after chain groups.a_chain_whose_width_does_not_divide_the_readers_grid_fails_its_edge,an_offset_enters_the_gcd_of_the_widths,one_range_free_chain_group_serves_every_range_and_offset,chain_groups_run_after_reach_so_a_rolled_up_query_adds_its_grid,an_absolute_anchor_reuses_only_existing_chains,no_chain_without_a_merge_an_anchor_or_a_grid,a_mergeable_state_reads_a_chain_with_one_build_per_surviving_parameters,a_groups_evaluation_grid_is_the_gcd_of_its_queries_grids,below_a_subquery_the_grid_is_its_resolution,below_a_subquery_without_a_resolution_there_is_no_grid,a_subquery_s_resolution_is_the_grid_of_the_groups_below_it.@ Treuses the chains the unanchored states create (added after them, so memo order does not matter:an_absolute_anchor_reuses_a_chain_a_later_group_creates);@ end()/@ start()get none.Decisions and open questions
Decided (concepts doc updated):
ErrorMetricsOfMemo(contracts, allowed metrics and readout metrics of one memo state, built once per memo state) withedge_meets_logical_requirements;ErrorContract { error_factor_per_input, linear_bound_holds_up_to_per_input }(wasErrorPropagation { factor_per_input, validity_cap_per_input });within_limit(evaluated, limit) -> bool(wastarget_limit);ReadoutRequiredAccuracyreplaces the anonymous(levels, queries)tuple. Beside the agreed sketch,ErrorMetricsOfMemohas two more public methods:exact_only_groups_of_query(the per-query walk reads the contracts and readout metrics, which stay private) anderror_contract(the read Stage 3 needs).quantile_over_time(0.9, m[5m])at rank 0.01 and at relative 0.05: each reads the other's readout groups only exactly); the exact-only walk skips dead candidates; δ splits over at most the readout's own values; scalar roots' operands are roots;@ Tchains independent of memo order.guaranteefield bumpsINSTALLABLE_PLAN_WIRE_VERSIONfrom 14 to 15.SummaryMerge, and aFilter,TimeRangeorTimeShiftover states, carry the θ of the states they read: one candidate per θ (add_candidates_per_input_parameters), input edges only to producers with the same θ (Memo::producers);Memo::add_candidatechecks θ against the group. α^log of a readout reads its producer's own θ. Rejected (a): α^log(merge → readout) holds only when every θ the merge may carry meets the requirement (safe, but a chain with weak builds blocks tight levels). Example (a_merge_is_judged_by_the_parameters_it_carries): P =quantile_over_time(0.9, m[5m])at rank 0.005, Q =quantile_over_time(0.9, m[10m])at rank 0.01, one 60 s chain with KLL k ∈ {269, 547, …}:SummaryMerge[k=547]→ P's readout holds;SummaryMerge[k=269]is pruned withBelowRequiredAccuracy { required: rank 0.005 }.NoGuarantee. The metric rule in dead-candidate elimination is a union over queries, so a query can still reach approximate producers through a consumer with no contract for its metric; Pass 2 lists those groups per query inexact_only_groups_of_query, which Stage 3 enforces per query. Example (the_metric_rule_cuts_the_readout_edges_and_keeps_the_exact_division):avg by (job)(q0.5 / q0.5 + q0.9)at absolute 0.05: the readout edges into the division are cut, the exact division is not pruned and its edge into+holds.PruneCause.through_pruned_consumers→through_pruned_candidates.RootPruneReason::ValuesPerEvaluationUnknown { consumer, query }: a whole-result δ witha_whole_result_failure_probability_without_values_per_evaluation_prunes_the_readout). Rejected: δ = 0 (no sketch meets it, with no stated reason).ratio-of-quantiles-1h-mvp:quantile_over_time(0.9, data[1h]) / quantile_over_time(0.5, data[1h]), relative 0.05, δ = 0.01 over 10,000 series = 1e-6 per value: both readouts now get levels 1e-6 and 1e-7 (1e-7 + 1e-7 ≤ 1e-6), and the full stack planned it at 7.481e-06/s before this PR's review fixes (one DDSketch α = 0.02 chain of 60 s windows, refactor: tools and snapshots over the #1207 memo, edges and contexts (#1207 PR 9) #1239); re-measured on restack.Memo::producers,CandidateError::LogicalProperties,MetricRule.accepted_metrics(wasconsumer_carries). Example: formerge[k=547]in P's state group,memo.producers(merge, 0)= the k = 547 candidate ofTimeRange(5m, G_w)only; adding aSummaryMergeover KLL states without θ fails withCandidateError::LogicalProperties(SketchBuildWithoutParameters)(a_merge_carries_the_parameters_of_the_states_it_reads).Filtercontract (E1(b)) reads the input group's column lineage. Example: over R ofquantile by (job)(latency)(columns[job: SourceColumn(latency.job), value: Computed]),Filter(job = "api", R)givesSome(ErrorContract { error_factor_per_input: [1.0], .. })in every metric;Filter(value > 0.5, R)givesNone. Rejected: the doc signature(consumer, metric)with the caller deciding filters (two places decide contracts).edge_meets_logical_requirementsreturns the failing reason. Example: KLL k = 26 into the p99 readout requiring relative 0.02 givesErr(BelowRequiredAccuracy { consumer: G4#2, input: 0, required: relative 0.02 δ 1 });LogicalEdge::satisfies_logical_requirementis.is_ok(). Rejected: a bool plus a second function (decides it twice).RootPruneReason::HeapSmallerThanTopK(the doc's α^log includes "a top-k's heap holds k", which fails also without a requirement). Example:topk(50, …)overCmsWithHeap: the edges from the 56 builds with heap 10 or 20 fail, so they are pruned withHeapSmallerThanTopK { k: 50 }(a_heap_smaller_than_k_fails_the_topk_readouts_edge).readout_error_metric(crate-private) reads the metric off the estimators' guarantee. Example:(Sketch(Kll), Quantile{0.5})→Rank;(Sketch(DDSketch), Quantile{0.5})→Relative;(Sketch(Hll), Cardinality)→Relative;(ExactAggregate(..), _)→None. Used by step 3 and the metric rule; a readout's summary type is read withGroupProperties::sketch_algorithm_of_state, the one helper for it.AccuracyMetric::ALL. P6 (N6):AggregateCoverage::{definition, selection}andSelectionBox::columnspublic (were test-support only);constraint_ofpublic.without_targets_every_edge_is_admissible), matching "(K) has no rows".Scanof the union of label sets, so{job="a"}with{job="b"}gets no $G^$ (scan widening, R7, not folded in). In production PromQL,sum by (job)(m)+sum by (instance)(m)get none either, as each scan lists only its own label: Rollup: no finest common group when PromQL scans of one metric list different labels #1285. Members groupedwithoutkeys or per entity are not rolled up; interval constraints must be equal within a cluster.every_listed_parameters_has_a_guarantee_at_one_percent,a_guarantee_is_evaluated_at_the_required_failure_probability).readout_levelskeys its result by readout candidate;MAX_ERROR_LEVELSlives withreadout_levels;guaranteeprivate; accuracy-math tests in the accuracy crate; shared memo builders inasap_types::test_support.Verification
Test scopes run at head
9ca4f50a1withcargo test -p <crate>; the workspace total iscargo test --workspace --no-fail-fast.max_values_per_evaluationan_absolute_anchor_reuses_a_chain_a_later_group_createspass2_*312fa98b1(1212 / 184 / 9); the only name differences are the integration tests renamed by content (sharing_issue_regressions::*andidentical_exact_queries::*→plan_sharing::*)Every workspace failure stops at a later-PR
todo!()(Stage 2 entrystage2-physical-optimization/src/lib.rs, the executor's θ, devtools) or atmaterialized_view.rs:529/materialized_views.rs:67(fixture).Round-3 workloads: the dump program (Pass 2's full output plus an integrity checker) ran the 91 round-3 workloads at this head: 0 integrity problems. Against
312fa98b1, only the exact-only lists and one live count change:quantile_over_time(0.9, m[5m])at rank 0.01 and at relative 0.05 now lists the DDSketch groups as exact-only for the rank query and the KLL groups for the relative one (before:[0, 1]for both);count(quantile_over_time(0.9, m[5m]))lists[0, 1](before also the quantile group, the DDSketch readout and its chain, through dead Count-Min and UnivMon builds), likewise 25 other workloads lose groups reached only through dead candidates; scalar roots now list their operands' groups per their contract (scalar(...) * 2: every group);SELECT count(DISTINCT dst) FROM flowsat δ 0.01 requires 0.01 instead of 1e-5 and keeps two more KMV builds (live 17 → 19).New tests:
a_coarse_exact_sum_rolls_up_the_finer_sum,a_coarse_count_rolls_up_the_finer_count_as_a_sum,a_filtered_coarse_sketch_state_merges_the_finer_states_through_a_compensating_filter,a_finest_common_group_no_query_asks_for_serves_both,a_narrowing_on_a_column_that_is_no_key_gives_no_rollup,a_compensating_filter_over_a_sql_table_has_sql_semantics,a_not_equal_matcher_compensates_with_promql_semantics,a_rejected_rollup_leaves_no_compensating_filter); top-k (a_smaller_topk_reads_the_largest_topk_of_the_same_input); requirements (a_single_readout_requires_the_query_target,a_whole_result_failure_probability_without_values_per_evaluation_prunes_the_readout,a_readout_another_query_gives_a_level_survives_unknown_values_per_evaluation); dead candidates (dead_candidates_are_pruned_and_listed_before_chain_groups); chains (a_mergeable_state_reads_a_chain_with_one_build_per_surviving_parameters,a_chain_whose_width_does_not_divide_the_readers_grid_fails_its_edge,an_absolute_anchor_reuses_only_existing_chains,no_chain_without_a_merge_an_anchor_or_a_grid,chain_groups_run_after_reach_so_a_rolled_up_query_adds_its_grid,an_offset_enters_the_gcd_of_the_widths,one_range_free_chain_group_serves_every_range_and_offset); grids (a_groups_evaluation_grid_is_the_gcd_of_its_queries_grids,below_a_subquery_the_grid_is_its_resolution,below_a_subquery_without_a_resolution_there_is_no_grid); exact-only (a_query_comparing_a_shared_ratio_reads_it_only_exactly); edges (every_candidate_of_an_input_group_is_an_edge_of_matching_parameters,without_targets_every_edge_is_admissible,a_merge_is_judged_by_the_parameters_it_carries).each_query_reads_only_the_readouts_in_its_targets_metric,a_dead_candidate_makes_no_group_exact_only(pass2_promql/exact_only_groups.rs),a_global_distinct_count_keeps_its_whole_failure_probability(pass2_sql.rs),a_readout_of_one_value_keeps_the_whole_failure_probability(readout_caps),max_values_per_evaluation_bounds_a_readout_by_its_states_keys(types memo),a_scalar_roots_operands_are_roots(metric_rule),an_absolute_anchor_reuses_a_chain_a_later_group_creates(Pass 2 chains). The R1 and R2a cases reproduce the round-3 review's x15 and e05 workloads, whose dumps at312fa98b1show the old lists.readouta_readouts_metric_is_its_algorithms(KLL quantile Rank, DDSketch and HLL Relative, exact None);readout_caps.rsa_whole_result_failure_probability_is_split_per_value,a_ratio_whose_per_value_failure_probability_is_below_the_grid_splits_it_over_its_readouts,a_max_over_many_readouts_splits_the_failure_probability_over_them,a_count_over_readouts_gives_them_no_level;metric_rule.rsa_ratio_reads_no_rank_error,the_metric_rule_cuts_the_readout_edges_and_keeps_the_exact_division,a_key_only_filter_keeps_the_ddsketch_readout_admissible,avg_of_readouts_under_a_relative_target_reads_exact_only;logical_edgewithin_limit_admits_only_floating_point_noise,a_distinct_count_readout_without_an_evaluable_guarantee_serves_no_query_with_a_target,a_heap_smaller_than_k_fails_the_topk_readouts_edge(56 builds with heap 10 or 20 failHeapSmallerThanTopK { k: 50 });error_contracta_reduction_keyed_by_an_estimate_has_no_contract,key_filters_and_value_preserving_projects_have_unit_contracts,error_contracts_of_a_ratio_a_count_and_a_sum;guarantee/mod.rsa_guarantee_is_evaluated_at_the_required_failure_probability,every_listed_parameters_has_a_guarantee_at_one_percent.pass2_promql/rollup.rs(listed under Rollup);pass2_promql/accuracy_and_grids.rsa_function_of_the_value_keeps_the_readouts_below_it(abs, negation and/ 2over a sketch readout keep it),a_subquery_s_resolution_is_the_grid_of_the_groups_below_it.a_merge_carries_the_parameters_of_the_states_it_reads,kll_builds_with_k_269_and_k_547_share_one_group,set_candidate_input_requirements_replaces_only_the_named_candidate,add_candidate_checks_parameters_against_the_groups_output.End-to-end evidence: none at this head, since planning stops at Stage 2's
todo!.ratio-of-quantiles-1h-mvpplanned at 7.481e-06/s on the #1239 snapshot before this PR's review fixes; it is re-measured on restack. Screenshots: not applicable.Self-review: one
add_candidate_if_output_matcheshelper in Pass 2, onereadout_error_metric, one sketch-algorithm helper, one target-metric expansion (AccuracyTarget::metrics), contracts only inerror_contract, oneErrorMetricsOfMemoper memo state, θ only inparameter_list(no strongest/weakest/floor tables); no#[allow(clippy::type_complexity)](the two tuples areReadoutRequiredAccuracyandReadoutRequirementsOutcome); no#[allow(dead_code)]added.Deleted fields and types
GuaranteeSource,AccuracyGuarantee::provenancetypes/src/accuracy/guaranteeBoundExpr::{Zero, Sum, Product, Scaled, Max, Unknown},ProbabilityExpr::{UnionBound, Scaled}Constant(andProbabilityExpr::{Zero, Unknown}) is builtAccuracyGuarantee::{bind, is_exact, has_unknown},BoundExpr::{is_zero, has_unknown},ProbabilityExpr::{is_zero, has_unknown, bind}LogicalProperties::newtypes/src/stages/logical/candidateCandidateForMemoGroup::new, builds it inlineMemo::series_count_scanned_belowtypes/src/stages/logical/memomax_values_per_evaluation(fromscanned_sources)accepted_metrics,metrics_accepted_on_inputs,readout_metric_of_candidate,metrics_carried,producer_satisfies_logical_requirements,target_limit,ErrorPropagation,error_propagation(public)crates/accuracyErrorMetricsOfMemo/ renamed (ErrorContract,within_limit)hll::generic_guarantee(nowhll::guarantee)crates/accuracy/src/guaranteeexact_only_groups.rs,rules.rsErrorMetricsOfMemo::exact_only_groups_of_query; helpers inrules/candidates.rsrange,shift,readout_of,groups_where,candidates_of(Pass 2),candidates,groups_where(Pass 1),summary_agg(types candidate tests)Follow-ups
Rollup: no finest common group when PromQL scans of one metric list different labels #1285: rollup with incomparable PromQL scan labels needs scan widening (
sum by (job)(m)+sum by (instance)(m)get no finest common group in production lowering).Stage 3 per-edge enforcement of the metric rule per query, for a finer exact-only list than per group.
An error contract for
PromqlSubquery(δ over the inner steps).Accuracy: Hydra states take the inner sketch's guarantee, but the design doc requires adding the shared-grid term #1283: Hydra shared-grid accuracy (the develop doc states the requirement; the code does not implement it yet).
Rollup rule: a filtered query (city = 'NYC') does not read the per-city group another query builds (#902 regression on the #1207 stack) #1252: A filtered query (city = 'NYC') doesn't read its group from a grouped summary another query builds #902 regression, rollup does not widen a narrower selection (
city = 'NYC'filter over the per-city group); among the remaining failures.Stage 3 enforces
exact_only_groups_of_queryper query: a later PR in this stack.Known rollup limits: no
withoutgroupings, no label filters between per-series layers, no regex matchers; the pre-check covers the schema only.Pruned chain consumers have an empty causes list in
PrunedLogicalCandidate.Certified distinct-count guarantees for classic HLL and UnivMon: a later PR in this stack.
Develop docs (
docs/develop_docs/asap-aware-mapping-*.md,end-to-end-accuracy-guarantees.md) and the concepts doc describe the error contracts, the θ lists with the guarantee at the required δ, and the current file layout;planner-layering.md's Pass 2 rows describe the current code. Its proposal sections still citememo::enumerationand "sized for the strictest" designs, which this PR does not rewrite.Scan widening across label sets (Q5, R7) is not folded into the rollup rule.
Stage 2 physical expansion: PR 5 (refactor(stage2): physical expansion (decision-tree levels 3–4), physical edges, pruned alternatives; Count-Min weights via column_signs (#1207 PR 5) #1235).