Repository navigation
refactor(stage1-pass1): per-computation candidates over memo groups (#1207 PR 3) - #1232
Merged
Merged
Conversation
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 6, 2026 23:55
6686f66 to
d4bde07
Compare
zzylol
changed the base branch from
refactor/1207-memo
to
refactor/1207-lineage
November 7, 2026 00:00
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 7, 2026 04:14
d4bde07 to
441f3a9
Compare
zzylol
force-pushed
the
refactor/1207-lineage
branch
from
November 9, 2026 06:16
e2b5456 to
58549f2
Compare
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 9, 2026 06:37
2c6d993 to
61f585a
Compare
zzylol
force-pushed
the
refactor/1207-lineage
branch
from
November 9, 2026 16:50
58549f2 to
b3655e1
Compare
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 9, 2026 16:50
61f585a to
a71e517
Compare
This was referenced Nov 10, 2026
zzylol
force-pushed
the
refactor/1207-lineage
branch
from
November 10, 2026 01:26
b3655e1 to
0479df3
Compare
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 10, 2026 01:26
a71e517 to
d77278a
Compare
zzylol
force-pushed
the
refactor/1207-lineage
branch
from
November 10, 2026 13:37
0479df3 to
50686ae
Compare
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 10, 2026 13:37
d77278a to
0c11cd7
Compare
zzylol
force-pushed
the
refactor/1207-lineage
branch
2 times, most recently
from
November 10, 2026 22:08
a6eca31 to
03163fd
Compare
…1207 PR 3) Pass 1 fills Stage 0's memo with each group's initial candidates, every intermediate result its own group and every candidate one operator over input groups: - computation_targets walks memo.groups(): one target per measure of each Aggregate group with a realization besides plain exact execution. - add_pass1_candidates adds every group's own expression, exact accumulators (state group + FinalizeExactAccumulator; avg from sum and count), one SummaryAgg state group per sketch algorithm with one build candidate per configuration of parameter_list(algorithm) and separate SummaryEstimate readout groups, fused builds (whole-expression top-k heaps; nested sum/min/max merges) listing the replaced groups in fused_groups, multi-measure joins, and key-only HAVING pushed below in the consumer Filter group's candidates. - Pass 1 sets no input requirement and no accuracy floor: every candidate's LogicalRequirements are empty; Pass 2 gives each readout its required accuracy (a top-k readout's heap_size >= k included). - ComputationTarget and FusedComputation move into Pass 1 (crate-private), FusedComputation::absorbed_groups replaces target indices, HavingEntry, HavingPlacement and Pass1Error are deleted; pass1_memo returns PerComputationCandidateError. - Layout: pass1_memo is defined in lib.rs; computation_targets.rs chooses the targets, candidates.rs adds their candidates (add_pass1_candidates and below). - types: stages::ranked_column_of_group in the group scope (stages/logical/group.rs; ranked_column becomes pub(crate)), so ir does not import stages; one unique-key rule for a top-k's ranked rows, exact or a heap readout; any_measure_filtered generic over the predicate's input. The top-k unique-key test builds memo groups, so it lives in the memo tests. - Tests build their memo through the Memo methods; the integration tests stage1_logical_asap_optimization and per_computation_candidates_sql run SQL and PromQL queries through Stage 0 and Pass 1.
zzylol
force-pushed
the
refactor/1207-pass1
branch
from
November 11, 2026 03:35
0c11cd7 to
321a1ae
Compare
zzylol
marked this pull request as ready for review
November 11, 2026 13:40
…amples rule, avg plan tests leave
- Remove the three SQL avg execution tests and their todo! helper (they
realize a plan from the memo, which Pass 1 does not do); rename the
integration tests pass1_candidates.rs and pass1_candidates_sql.rs.
- HowToComputeAggregateMeasure moves crate-private into Pass 1
(computation_targets.rs); types' per_computation_logical_memo_candidates.rs
and devtools' unused computation_label are deleted.
- ComputationChoices is deleted: computation_candidates_for_intent returns
the ways, a quantile outside [0, 1] gets no summary, prune_own returns
nothing. PerComputationCandidateError, expression and schema move to
lib.rs; key_only_having moves into candidates.rs; distributive is private.
- One raw_data_samples_of_metric, generic over OperatorInput
(asap_types::summary_type), read for a memo group through
stages::raw_data_samples_of_metric_of_group; accuracy's copy is deleted.
- realize_keyed_additive_summary_input takes no intent or summary type and
returns (group, update); its Count-Min branch, never reached, is deleted.
- The absorbed-target list in add_nested_distributive_targets is deleted:
targets follow the memo's group order, so it never matched.
- sql_avg and join_measures read Reduction::group_keys; the plain-exact
arms of the candidate builders are unreachable!; the multi-measure output
is not offered when unsupported, like every other way.
- asap-accuracy's test_support module and test-support feature are deleted
(no callers); cms_depth and saturating_ceil are test-only.
- Pass 1's test fixtures build their expressions with
memo_fixtures::{scan, aggregate_with}; summary_realization's tests move
into tests.rs.
- Docs: Pass 1 rows of planner-layering.md, the deleted acceptance tests of
Example 2, and the Pass 1 test paths.
An unsupported way to compute a group no longer leaves groups in the memo. Each way's groups are built as pending expressions, checked against the schemas the memo derives for them, and added only once every check that can return Unsupported has passed. - types: stages::output_schemas_of_pending_expressions(memo, pending) derives and validates the output schemas of expressions not yet in the memo, over memo groups or earlier pending expressions (InputOrTemporaryBuild over MemoGroupAsInput); the input MemoGroupId(n + k) names pending[k]. - Pass 1: measure_output, sql_avg, join_measures, last_as_float64 and join_on_keys append to a pending list instead of adding groups; ranked_as_topk and renaming_project take derived schemas; add_output_candidate checks, then commit_pending adds the way. The heap top-k (fused and own), the per-series avg and the multi-measure join are built this way; unless_unsupported now only drops ways checked before they add anything. - Tests: a heap estimate without the result group's columns and a readout an open group cannot rename leave the memo unchanged (groups, candidates, hash-consing index).
…sted matcher, HAVING split, sizing oracle
Functional:
- Key-only HAVING filters are pushed pending with each way, so an
unsupported way adds no Filter group (SQL count(*) over all-nullable
columns under HAVING k > 'a' left an orphan group).
- A multi-measure aggregate's single-measure parts are written as the
frontends write one (no filter list when unfiltered), so they share an
equal Stage 0 group.
- The nested distributive merge reads through a column-renaming Project
(SQL sum of a subquery's sum fuses); a mixed HAVING pushes its key-only
conjuncts below and keeps the rest above; PromQL avg as sum/count checks
finite division for grouped and global avg too.
Code quality and layout:
- Memo::output_schema_of_pending_expression is the dry run of
find_or_add_group over InputOrTemporaryBuild<MemoGroupAsInput>; Pass 1's
Pending stores each pending expression's schema once.
- candidates.rs is split into candidates/{mod, pending, measure_output}.rs;
measure_input, counted_column, summary_update and
with_raw_data_samples_requirement move to summary_realization.rs.
- One matcher, nested_aggregates, serves the nested merge and the summed
per-series samples; a key that cannot be read counts as __name__.
- unless_unsupported returns the way's result; candidates_by is inlined;
memo_fixtures::aggregate_where is deleted.
- asap-accuracy's test-only sizing family is one module,
guarantee::sizing_oracle, with the tests that check the predefined lists
against it.
- Tests next to the new types functions; the top-k unique-key test moves
to ir/schema/topk.rs over operator nodes; stale Pass 1 pruning text and
issue numbers removed from docs, comments and a devtools test.
Stage 0 is memo_of_queries producing a Memo through Memo::find_or_add_group; Pass 1 builds one summary per parameter_list configuration with no accuracy budget shares. Rewrite the example's Pass 1 step with the measured groups and candidates, fix the code-interface rows, and drop accuracy_budget_parts from the asap_planner memo output comment.
… filter, shared readout path - HowToComputeAggregateMeasure::Exact deleted: plain exact execution is always the group's own expression, so ways list only exact accumulators and summaries; prune_own, the != Exact guards and unreachable! arms go. - split_having joins all non-key conjuncts (the aggregate's own and its consumer Filter's) into one predicate: one Filter above the output. - key_only_having pushes a column-free conjunct only when it is built of literals and comparison or boolean operators (no function call). - Pending stores (expression, schema) pairs; Memo::output_schema_of_pending_expression takes them. - The fused heap goes through summary_readout and add_output_candidate, which checks the candidate's schema on the pending expressions before anything is added and passes fused_groups to commit_pending. - add_fused_candidate takes ids and starts its own Pending; the dead non-aggregate branch of add_aggregate_candidates is unreachable!. - Group-scope tests move next to group.rs; HLL test-only helpers move to the sizing oracle; classic_failure_probability and its model are test-only; stale names and paths in docs fixed. - pass1_candidates_sql::a_filter_or_group_key_over_an_estimate_has_no_guarantee leaves this PR (it needs Pass 2).
This was referenced Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
todo!(): target discovery walked operator-node trees, and candidates came from a whole-workload node-tree composition sized per budget share; neither fits the memo model (concepts doc §3: "Everything that was computed on operator-node trees is computed on groups").Memomethods, so no stage crate depends on another, even in tests.latency, query B = p99 / p50),pass1_memogoes fromtodo!()to a memo of 14 groups: a KLL state group with 7 builds (k ∈ {26, 52, 132, 269, 547, 1404, 2863}), a DDSketch state group with 7 (α 0.1 … 0.001), and six readout groups. Planning now stops at Pass 2's firsttodo!()(PR 4).Review guide
main(PRs 0, 0b, 1, 2 and 2b are merged); refactor(stage1-pass2): rollup from the finest common group, readout requirements, dead-candidate elimination, chain groups (#1207 PR 4) #1234 (PR 4) builds on it. Seven commits: the restacked PR 3 (321a1ae8d), the review fixes (343e0a863), build-pending-then-add (888512995), the round-3 review fixes (9aa66fe41), a test case (25e6f5267), a docs fix (1eb415cc5) and the C1–C11 review fixes (7bd4546d2, head).crates/stage1-pass1-per-computation-candidates/src/):lib.rs:pass1_memo,PerComputationCandidateError, the group accessorsexpressionandschema.computation_targets.rs:HowToComputeAggregateMeasure,ComputationTarget,FusedComputation,computation_targets,computation_candidates_for_intent, then the fused targetsadd_whole_expression_targets,add_nested_distributive_targets,nested_distributive_input,whole_expression_input.candidates/mod.rs:add_pass1_candidates(split_having,key_only_having) →add_aggregate_candidates(start_way,single_measure_part) →add_measure_way/add_fused_candidate(memo, aggregate_group, result_group, ..);unless_unsupported.candidates/pending.rs:Pending(push,schema),commit_pending,add_state_group/add_group/add_candidate.candidates/measure_output.rs:measure_output,summary_readout(state, estimate, ranked rows; shared with the fused heap),accumulator_build,add_output_candidate,renaming_project,ranked_as_topk,sql_avg,join_measures.summary_realization.rs:summary_candidates,measure_input,counted_column,summary_update,summarised_input,ranked_value,realize_keyed_additive_summary_input,summed_per_series_samples.tests.rs,test_fixtures.rs.ir/schema/topk.rs(the top-k unique-key rule),stages/logical/group.rs(ranked_column_of_group,raw_data_samples_of_metric_of_group; tests ingroup/tests.rs),summary_type/statistic.rs(raw_data_samples_of_metric),stages/logical/memo/mod.rs(Memo::output_schema_of_pending_expression).guarantee/sizing_oracle.rs(test-only; now also the HLL precision helpers),guarantee/hll.rs(the classic confidence model is test-only).Pass1Error, thetest-supportfeatures of Pass 1 andasap-accuracy,HavingEntry/HavingPlacement, types'per_computation_logical_memo_candidates.rs); integration-test renames; the accuracy sizing oracle (a pure move); docs.HAVINGpush-down per consumerFiltergroup; the plain exact candidate beside the accumulator; fused builds and theirfused_groups; renamed readouts and open schemas (series identity); the top-k unique-key rule; the pending → check → commit order of every way, and the candidate's schema check before commit; theHAVINGconjunct split into oneFilterabove, and which column-free conjuncts go below;nested_aggregates(a renamingProjectand the__name__rule).Behavior: before → after
The concepts doc's running example (§1; test
running_example_gives_one_build_per_theta_and_separate_readouts): query A = p95 oflatency, query B = p99 / p50 of it. It serves the design need that all ways to compute a group are memo candidates Stage 3 can choose among.Before (main):
pass1_memopanics attodo!("#1207 PR 3: …"). After, the memo (printed by the test at888512995):Mapping to the doc's table: G1 here is the doc's G1 (
latency, viaScanG0 andTimeRangeG1); G6 = doc G2 (KLL state, one build per θ of KLL's list; the doc'sk = 200/345are illustrative); G8 = doc G2′ (DDSketch); G2/G3/G4 = doc G3/G4/G5, each with its exact aggregate and a renamingProjectover a readout group per algorithm (G7/G9, G10/G11, G12/G13); G5 = doc G6. The doc shows G4/G5 reading only G2′: that is the metric rule, which drops the KLL → ratio edges in PR 4.What Pass 1 now does:
computation_targets(memo, supported_summaries)reads each group'sexpressionandgroup_propertiesand finds oneComputationTargetper measure of eachAggregategroup with a realization besides plain exact execution, once however many queries read it.Memo::find_or_add_groupandMemo::add_candidate: exact (every group's own expression, no θ); exact accumulator (a state groupSummaryAgg[ExactAggregate(kind)]and aFinalizeExactAccumulatorover it;avgfinalizes itssumandcountstates and divides them, a PromQLBinaryOp, or for SQL aJoingroup and aProject); summary (one state groupSummaryAgg[Sketch(algorithm)]per supported algorithm with one build per θ inparameter_list(algorithm), read through a separate readout groupSummaryEstimate(state)); fused builds (the whole-expression top-k heap over the rows below the absorbed sums, and the nested distributive mergesum∘sum,min∘min,max∘maxas one exact build in the outer aggregate's accumulator state group; builds list the absorbed groups infused_groups); multi-measure aggregates (SQL: one single-measureAggregategroup per measure, joined back on the group keys);HAVINGread from the memo (the aggregate's ownhavingsplit bykey_only_having; each parentFilterwith a key-only predicate gets candidates that push it below the aggregate; a valueHAVINGstays aFilterabove each readout).Project. A heap top-k's Float64 estimate is cast to the exact top-k's column type (a SQL count ranks Int64).$promql_series_identitycolumn, so its renamingProjectoutputs the group's exact (closed) schema. A group whose schema is open gets no renamed readout.Unsupported, e.g. a SQLcount(*)with no non-null column, or a quantile outside [0, 1]); a candidate that breaks the node rules or the group's output is a bug and fails Pass 1.Interfaces
Against
main.crates/stage1-pass1-per-computation-candidates/src/lib.rs·computation_targets.rs·Cargo.tomlPass1Error: nothing to add overPerComputationCandidateError.InvalidChoiceandtest_support: there are no whole-workload choices any more.crates/types/src/stages/mod.rs·stages/logical/group.rs·stages/logical/candidate.rs·stages/per_computation_logical_memo_candidates.rs(deleted)HavingEntry,HavingPlacement,measure_filter,having_filters: copies of memo data (the aggregate'shaving, its parentFiltergroups,Aggregate.filters[measure_index]).ComputationTarget,FusedComputationandHowToComputeAggregateMeasuremove, crate-private, into Pass 1'scomputation_targets.rs(absorbed_targets→absorbed_groups: Vec<MemoGroupId>, used asfused_groups;HowToComputeAggregateMeasure::Exactis deleted: plain exact execution is always the group's own expression, so a target lists onlyExactAccumulatorandSummaryways); only Pass 1 reads them (devtools'computation_labelhad no caller and is deleted).ranked_column_of_groupandraw_data_samples_of_metric_of_grouplive in the group scope becausecrates/types/src/irmust not importstages.crates/types/src/summary_type/statistic.rs·ir/schema/topk.rs·ir/operator/non_asap.rs·test_support/memo_fixtures.rsCrate-private additions in
topk.rs(the schema derivation fix):item_identity_is_complete(input, item_columns),ranked_rows_unique_keys(columns, value),exact_topk_unique_keys(child, partitions, output); privateRankedSource,ranked_source.crates/accuracy/src/propagation.rs·accuracy/src/lib.rs·accuracy/src/test_support.rs(deleted) ·crates/devtools/src/candidate_labels.rspub mod propagation { - /// The metric whose raw data samples, or plain sums of them, are every value `node` outputs. - pub fn raw_data_samples_of_metric(node: &OperatorNode) -> Option<String>; } -#[cfg(feature = "test-support")] -pub mod test_support { - pub use crate::propagation_stats::PropagationStats; - pub fn composed_guarantee(node: &OperatorNode) -> Option<AccuracyGuarantee>; - pub fn scalar_guarantee(expr: &ScalarExpr, column: &dyn Fn(usize) -> Option<AccuracyGuarantee>) - -> Option<AccuracyGuarantee>; - pub fn stream_norm(node: &Rc<OperatorNode>) -> Option<StreamNorm>; - pub fn fold_stats(reduction: &Reduction, input: &Rc<OperatorNode>) -> Option<PropagationStats>; - pub fn estimates_raw_data_samples_of(node: &OperatorNode) -> Option<String>; - pub fn source_distinct_guarantee(/* ... */) -> Option<AccuracyGuarantee>; - pub fn cms_depth(delta: f64) -> u32; - pub fn kll_rank_error_99(k: u32) -> f64; - pub const QUOTIENT_OPERAND_DOMAINS: &str; - pub const STREAM_ROW_COUNT: &str; - pub fn source_norm_statistic(norm: &str, source: &str) -> String; - pub fn raw_data_samples_sign_statistic(metric: &str) -> String; - pub fn sign_statistic(stats: &PropagationStats) -> Option<&str>; - pub fn estimate_has_bound(family: &FieldDataType, update: &SummaryUpdate, query: &SketchStatistic) -> bool; - pub fn estimate_bound_over_raw_data_samples(family: &FieldDataType, update: &SummaryUpdate, - query: &SketchStatistic) -> Option<AccuracyGuarantee>; -} -[features] test-support = [] pub mod guarantee::hll { - /// The smallest precision whose classic failure probability meets a target (no non-test caller). - pub fn min_precision(relative_error: f64, failure_probability: f64, max_distinct: u64, at_least: u8) -> u8; - pub fn classic_failure_probability(precision: u8, relative_error: f64, max_distinct: u64) -> f64; + /// Now test-only, with `ClassicHllConfidence` and `MAX_CONTRACT_DISTINCT` (only it reaches them). + #[cfg(test)] + pub(crate) fn classic_failure_probability(precision: u8, relative_error: f64, max_distinct: u64) -> f64; } // asap_devtools::candidate_labels -/// A Pass 1 per-computation candidate's label, e.g. "exact", "Sum acc" or "Kll". -pub fn computation_label(candidate: &HowToComputeAggregateMeasure) -> String;min_precisionmoves into the test-onlysizing_oracle(its one test with it), as doClassicHllConfidence's two#[cfg(test)]methods, now free helpers there (classic_hll_precision;guaranteeis replaced byclassic_guaranteebound withclassic_failure_probability) with the two hll tests that used them.classic_failure_probabilityhas no non-test caller, so it and the model only it reaches are#[cfg(test)]; PR 8 uses the formula again and drops the attribute.asap-accuracy'stest_supporthad no caller left (PR 4 deletes the file too);cms_depthandsaturating_ceilare now#[cfg(test)], as their only remaining callers are tests. (source_distinct_guarantee's arguments are elided above.)Usage, end to end (test
running_example_gives_one_build_per_theta_and_separate_readouts, run at888512995):Algorithm changes
Pass 1 candidate generation
todo!(). Before PR 0, it built one whole-workload node tree per choice and sized each sketch for an equal budget share; the result was copied into private node chains.compose_logical_candidate(choice)→ node tree per choice → sizing peraccuracy_budget_parts→ private chains.computation_targetswalksmemo.groups(). For eachAggregateexpression it lists the ways per measure (computation_candidates_for_intent: the exact accumulator first when the intent has one, then the summaries; plain exact is not a way, it is the group's own expression; a measure with no way is not a target), prunes the exact accumulator of anavgover a nullable input and the summaries of a relational count (a target can end with no way of its own and keep its fused computation), detects fused computations over groups (add_whole_expression_targets,add_nested_distributive_targets,close_absorptions; neither applies wherehas_having), prunes a PromQL top-k's own heaps (the aggregate group'sscanned_sourcesare all time series), and drops unsupported algorithms.add_pass1_candidatesadds every Stage 0 group's own expression as a candidate; then, per aggregate group (its ownhavingsplit bykey_only_having) and per Stage 0Filtergroup directly above it with a key-only predicate,add_aggregate_candidatesbuilds each way bottom-up:add_state_group(θ list or one exact build),add_groupfor each intermediate (the operator is also its group's candidate), andadd_output_candidatefor the top operator (added to the result group, or, when its column names differ, as its own group read throughrenaming_project; an open group gets none; under the one non-keyHAVINGpredicate, the candidate is aFilterover it).add_candidate(this crate) →LogicalProperties::new(θ in the list, none on exact operators) →vec![LogicalRequirements::default(); inputs]→Memo::add_candidate.Memo::add_candidate's exact check);pass1_candidates::{sql,promql}_candidates_output_the_query_columnsrun every SQL and PromQL query of the old composition test through Pass 1.|parameter_list(algorithm)|build candidates (KLL/DDSketch 7, HLL/KMV 4, CMS/Count Sketch 28, UnivMon 40, heap variants 280) to one shared state group, instead of one node tree per whole-workload combination.Build pending, check, then add
avg's sum must be Float64; a renamed output needs a closed group with as many columns). Before, they ran after the way's state and estimate groups were added, so an unsupported way left groups no query reaches.Pendingwith the key-onlyHAVINGconjuncts asFilters over the aggregate's input, appends its expressions (Pending::pushderives each one's schema once throughMemo::output_schema_of_pending_expression; the idn + knames pending expressionk), runs every check that can returnUnsupported(ranked_as_topk, the per-seriesavgdtype check,renaming_project,counted_column), and only then adds them (commit_pending, which asserts no group was added meanwhile: a summary build as a state group with its θ candidates, anything else as its own group's candidate) and the candidate.add_output_candidatealso pushes the final candidate, checks its schema against the result group's, and pops it before anything is committed, so a mismatch isUnsupportedand leaves the memo unchanged. The fused heap now goes through the samesummary_readoutandadd_output_candidateas every summary way (its builds'fused_groupspassed through tocommit_pending); the nested merge, every measure way, SQLavgand the multi-measure join are built this way too; the join's single-measure groups get their own ways once the join is added.unless_unsupporteddrops ways that failed before adding anything.a_heap_estimate_without_the_groups_columns_adds_nothing,a_readout_an_open_group_cannot_rename_adds_nothing(the memo's groups, candidates and hash-consing index are unchanged),an_unsupported_way_under_a_key_only_having_adds_no_group(SQLcount(*)over all-nullable columns underHAVING k > 'a': no group beyond Stage 0's). The per-seriesavgcheck is not reachable from a test: a per-series sum of Int64 samples derives Float64, so the way is offered.Nested aggregates,
HAVINGsplit, multi-measure partscomputation_targets::nested_aggregates). One matcher for an outer grouped aggregate over the value of a one-measure aggregate beneath it, shared by the nested distributive merge and the keyed heap's summed per-series samples. It reads through aProjectthat only renames or reorders columns and keeps the key names, so SQLSELECT src, sum(s) FROM (SELECT src, dst, sum(bytes) AS s … GROUP BY src, dst) x GROUP BY srcgets the merged sum (its build lists theProjectand the inner aggregate infused_groups); the keyed heap does not read through it. Over a per-series aggregate it never groups by__name__(see Decided).HAVINGsplit (split_having). The key-only conjuncts of aHAVING(src > 'b' AND p > 3:src > 'b') go below the aggregate as aFilterof its input; the other conjuncts, of the aggregate's ownhavingand of the consumerFiltertogether, stay oneFilterabove the output (oneBoolAndwhen there are several; was a chain ofFilters when both had some). It readsScalarExpr::conjuncts. Before, a mixedHAVINGpushed nothing.key_only_having). A conjunct that reads at least one key column and nothing else, or a column-free one built only of literals and comparison and boolean operators (HAVING 1 = 0on a grouped aggregate). A column-free conjunct with a function call (random() < 0.5) may be volatile and stays above; the IR has no volatility flag. Was: any column-free conjunct.single_measure_part). Each measure's own aggregate is written as the frontends write one: its name, and a filter list only when it is filtered.SELECT src, count(*), sum(bytes) … GROUP BY srcwithSELECT src, count(*) … GROUP BY srcnow share thecount(*)group (was a duplicate).avgas sum and count. Grouped and globalavgnow check finite division asavg_over_timedoes (see Decided).Schema derivation fix (types)
Projectcarried (job, series identity) fromtopk_schema_of_build, sotopk by (job) (10, sum_over_time(m[1m]))'s heap readout failedadd_candidate's exact check.ranked_rows_unique_keys,item_identity_is_complete,exact_topk_unique_keys).ir::schema::topk::tests::exact_topk_and_heap_readout_derive_the_same_unique_keys(over operator nodes).Decisions and open questions
Needs reviewer decision: None.
Decided:
LogicalRequirements { required_accuracy }on readout inputs is Pass 2's).heap_size ≥ kis Pass 2's rule.fused_groupsfused_groupslists the replaced operators' groups. Rules: top-k heap, sum∘sum, min∘min, max∘max.expression(memo, g)/schema(memo, g)(crate-private,lib.rs);MemoGroupAsInputstays types-internal; Pass 1 reaches the generic walks throughranked_column_of_groupandraw_data_samples_of_metric_of_group.add_candidate's output check==), as PR 2 defines it.pass1_memo(memo, supported_summaries)HAVINGpush-downFilter's predicate goes below in theFiltergroup's own candidates.avgexecution testssql_avg_from_sum_and_count_executes_as_avg,sql_avg_counts_its_own_input,sql_avg_of_an_int64_column_keeps_the_exact_columnsand their helperavg_as_sum_and_countrealize a plan from the memo, which is PR 6's: they move to PR 6 (kept outside the repo until its restack).ComputationChoicesdeleted (the ways are aVec; a quantile outside [0, 1] gets no summary);PerComputationCandidateErrorinlib.rs;HowToComputeAggregateMeasurecrate-private in Pass 1; oneraw_data_samples_of_metric, generic overOperatorInput;asap-accuracy'stest-supportdeleted;realize_keyed_additive_summary_input(memo, output_reduction, child) -> Option<(MemoGroupId, SummaryUpdate)>(its Count-Min branch never ran: the one caller asks with Count Sketch); the absorbed-target list inadd_nested_distributive_targetsdeleted (targets follow the memo's group order, so an outer target is never one already absorbed; checked with an assertion over Pass 1, integration, planner and devtools tests before deleting);Reduction::group_keysinsql_avgandjoin_measures;key_only_havingprivate incandidates.rs; Pass 1's fixtures reusememo_fixtures(labels are non-nullable there; no Pass 1 test depends on it); integration tests renamedpass1_candidates.rs,pass1_candidates_sql.rs.Memo::output_schema_of_pending_expression, reusingInputOrTemporaryBuild<MemoGroupAsInput>), check, then add. Rejected: building in a memo clone; aMemoCheckpointrollback.__name__in nested aggregates__name__: the per-series aggregate drops the metric name, the raw rows keep it, so the fused computation over the raw rows would split groups the query has as one. A key whose column cannot be read counts as__name__(is_none_or, the stricter of the two old checks; the other,is_some_and, accepted it). For valid groups every key column exists, so no plan changes.guarantee::sizing_oracle(#[cfg(test)]): the closed-form sizing the predefined lists are checked against (parameter_list_is_the_sizing_over_the_error_grid) and the sizers' own tests; family tests that check guarantees import it. Pure move; 61 accuracy tests, same names.avgchecked_finite_division: truefor grouped, global and per-seriesavgas sum ÷ count: Prometheus'avgaggregation, likeavg_over_time, switches to an incremental mean when the running sum overflows, sosum / countmatches it only for finite operands and quotient.candidates.rssplit intocandidates/{mod, pending, measure_output}.rs; the summary-input helpers insummary_realization.rs;Memo::output_schema_of_pending_expressionas the dry run offind_or_add_group;raw_data_samples_of_metricstays instatistic.rsbesideWeightDomain(PR 5 deletes it with the weight-domain code).HowToComputeAggregateMeasure::Exactdeleted (C1); one non-keyHAVINGFilterabove (C2);Pendingholds(expression, schema)pairs (C3); the fused heap sharessummary_readoutandadd_output_candidate, with the schema check before commit (C4);add_fused_candidatetakes ids, the non-aggregate branch isunreachable!(C5); stale names in docs (C6); group-scope tests next togroup.rs(C7); the Pass 2-dependent SQL test moves to PR 4 (C8); column-free conjuncts go below only without function calls (C9); stale Stage 0 names in docs (C10); HLL test-only code in the sizing oracle (C11). No Pass 1 memo of the review workloads changes.PerComputationCandidateError::StructureQueryRoot::validate_structurebefore Stage 0 (crates/planner/src/run.rs).Deleted (replaced by the memo candidates):
HowToComputeAggregateMeasure::Exactandprune_own(it only re-insertedExact), the!= Exactguards andunreachable!arms,add_output_candidate'sFilterchain,ClassicHllConfidence::{guarantee, precision}(nowsizing_oraclehelpers),compose_logical_candidate,compose,compose_roots,rewrite,substitute,substitute_measure,with_names_of,whole_expression,nested_sum,Sizing,ChosenComputation,Chosen,PerComputationCandidates(withcombinations,absorbed_choice_varies,without_summaries_no_query_can_use),output_can_meet_target,sources_read,absorbed_targets,generate_per_computation_candidates,logical_node_trees,ComputationChoices,PhysicalSummaryInput,summarised_column(inlined intosummarised_input),candidates_by(inlined intosummary_candidates),unless_unsupported's unit form,memo_fixtures::aggregate_where, Pass 1'smetric_schema/scan_operatorfixtures; the executor's dev-dependencies on Stage 0 and Pass 1.Files
lib.rspass1_memo,PerComputationCandidateError,expression,schemacomputation_targets.rsHowToComputeAggregateMeasure,ComputationTarget,FusedComputation, target discovery, fused targets,nested_aggregatescandidates/mod.rsadd_pass1_candidates,AggregateWays,SingleMeasure,add_aggregate_candidates,unless_unsupported,add_measure_way,add_fused_candidatecandidates/pending.rsPending,commit_pending,add_group,add_state_group,add_candidatecandidates/measure_output.rsmeasure_output,summary_readout,accumulator_build,add_output_candidate,renaming_project,ranked_as_topk,last_as_float64,sql_avg,join_on_keys,join_measures,key_itemssummary_realization.rstests.rstest_fixtures.rsmemo_fixturestests/pass1_candidates.rs,tests/pass1_candidates_sql.rsguarantee/sizing_oracle.rsVerification
cargo fmt --all --check,cargo clippy --workspace --all-targets -- -D warnings: clean.cargo doc: no warning in a changed file (Pass 1 has none).grep -rn "use crate::stages" crates/types/src/ir: empty. Workspacecargo test --workspace --no-fail-fastcompared by test name withmain(bd4fe8d9a):tests.rshll, 1 fromguarantee) moved intoguarantee::sizing_oracleMemomethod (2),ranked_column_of_group,raw_data_samples_of_metric(_of_group)(2)HAVINGtests now pass;a_filter_or_group_key_over_an_estimate_has_no_guarantee(fails at Pass 2'stodo!) moved to PR 4New tests and what they check:
a_quantile_outside_the_unit_interval_offers_no_summary: a quantile outside [0, 1] lists no way (plain exact is its own expression).whole_expression_topk_heap_candidates: the fused heap, through the shared readout path, gives the top-k group one rankedProjectper heap over its estimate group, only the state builds listfused_groups, and no other group (the memo printout is identical to1eb415cc5's).key_only_having_pushes_keys_and_constants_but_no_function_call(IR-level:random() < 0.5stays above,1 = 0and key conjuncts go below, nothing for a global aggregate) anda_constant_having_of_a_grouped_aggregate_is_pushed_below.a_heap_estimate_without_the_groups_columns_adds_nothing,a_readout_an_open_group_cannot_rename_adds_nothing,an_unsupported_way_under_a_key_only_having_adds_no_group: an unsupported way leaves the memo unchanged.a_multi_measure_aggregates_part_is_the_equal_stage0_group,sql_sum_of_a_subquerys_sum_is_one_sum_over_the_rows,mixed_having_pushes_its_key_conjunct_below(wasmixed_key_and_value_having_stays_above),promql_avg_divides_sum_and_count_vectors(now also checks finite division for grouped, global and per-seriesavg): each fails at888512995and passes here.an_algorithm_without_the_readout_is_not_offered: offered algorithms follow the real capability table (replaces the injected-table test;preference_order_names_each_algorithm_oncedeleted).Pass 1 memos of the review workloads (temporary dump, deleted), groups / candidates, before (
888512995) → after. At7bd4546d2every dump (every group and candidate) is identical to1eb415cc5's, with 0 orphan groups and 0 candidate children whose group does not list the candidate's group as a parent:count(*)part is the Stage 0 groupHAVINGpushessrc > 'b'below (one sharedFiltergroup)fused_groups= [Project, inner aggregate]running_example_gives_one_build_per_theta_and_separate_readouts: KLL and DDSketch state groups hold one build per θ; each quantile reads either through a readout group.HAVING, fused-heap, ranked-item, nested-merge and intent tests intests.rs, andpass1_candidates.rs/pass1_candidates_sql.rs(each builds its memo throughMemo::find_or_add_groupandMemo::add_query, or lowers SQL/PromQL through Stage 0).Removed from this PR: to PR 6, the three SQL
avgexecution tests, which failed here at theirtodo!helper; to PR 4,pass1_candidates_sql::a_filter_or_group_key_over_an_estimate_has_no_guaranteeand its helperreads_quantile_sketch, which run every stage and fail at Pass 2's rolluptodo!(both kept outside the repo until those restacks). Screenshots and measurements: not applicable.Follow-ups
a_filter_or_group_key_over_an_estimate_has_no_guarantee. It deletesaccuracy/src/propagation.rs(with the count-min test moved there) andbind_guarantees.raw_data_samples_of_metric(_of_group)then has no caller and goes with it.avgfrom sum and count; re-adds the three SQLavgexecution tests.