Repository navigation
fix(executor): global aggregates follow the input's result kind, not the identity column - #1350
Conversation
3433ca0 to
0c87736
Compare
…the identity column A PromQL aggregate with no grouping keys over another aggregate's output (`count(sum by (job) (m >= 1000))`) answered one row over an empty input, and `sum`, `min`, `max`, `avg` and `quantile` over such an output declared nullable outputs that failed to install against the Planner's non-null schema. The executor chose SQL or PromQL rules by whether the input still had a series identity column, which an aggregate's output never has. The lowering now reads the IR's result kind of the rows a node reads (`input_rows_kind`: its input's, or for summary states the rows they were built from). Over a `Relation`, an aggregate, per-group build or merge into no groups opts into `with_emit_empty_global`, which also makes SUM, AVG, MIN, MAX and percentile measures nullable; over a PromQL vector none does, and a quantile is PromQL's. The identity-column test is gone from the aggregate's nullability, its run and the per-group build.
…and NULL list The executor's `rows_kind` copied the IR's private `source_kind` and stopped at states built through `SummarySubtract`, `SummaryJoin`, `MaintainPopulation` or `Extension`, so a global SQL rollup of such states got PromQL's rule (no row over no input rows). The IR now exposes `evaluated_rows_kind` (a node's own kind, or for summary states the kind of the rows they were built from) and `input_rows_kind` calls it. `compile_internal` computes `over_relation` once, only for aggregates, summary builds and merges, and passes that one bool form through `compile_node`, `bind_operation` and `summary_agg_build`. `with_emit_empty_global` takes the flag (false leaves the operator as it is), replacing five `if … else Ok(op)` sites, and `Reduction::is_null_without_values` is the one list of measures that are NULL for a group with no value, read by it and by `with_measure_filters`.
Grouped stddev/stdvar still read the series identity column to choose between SQL and PromQL behaviour over no value. They now follow the same rule as the other global aggregates: the measure is nullable for a sample form or a nullable column, a global SQL aggregate gets its one NULL row from `with_emit_empty_global`, and a PromQL `stddev(...)`/`stdvar(...)` over an empty vector answers no sample. Tests: the stddev and stdvar cases of a_global_aggregate_follows_its_input_rows_kind (failed before: the derived output was nullable over a PromQL vector without an identity column), PromQL stddev/stdvar over an empty aggregate, SQL stddev_pop and var_samp over no row. The SQL variance and correlation tests build their global aggregate with `with_emit_empty_global`; the PromQL install test uses the shared `run` over a store.
Main deleted the read-counting CountingStore; the test reads the plan's raw-data tables directly.
0c87736 to
d78f984
Compare
|
Rebased onto Conflicts, all in Validation on the merged tree: |
Closes #1348.
Summary
count(sum by (job) (m >= 1000))answered{} 0over an empty vector, where Prometheus answers no sample.sum,min,max,avgandquantileover such an output declared a nullable column, so their plans failed to install withnative output type differs from Planner output.Relationor a PromQL vector), not from whether its input still has a series identity column. For summary states it asks the IR's ownevaluated_rows_kind(the kind of the rows the states were built from), so the executor has no copy of that walk. An aggregate, per-group build or merge into no groups over aRelationopts into SQL's one row over no input (with_emit_empty_global(true, …)). The identity-column test is deleted from the aggregate's nullability, its run and the per-group build.count(sum by (job) (m >= 1000))answers no sample;SELECT count(*) … WHERE falsestill answers one row of 0 andSELECT sum(x) … WHERE falseone NULL. Onmainevery corpus plan is unchanged. On the openwithoutPR's head (feat(executor): grouped stddev/variance, corr, group, count_values, exact topk and without grouping #1242), 13 corpus queries that failed to install now install (details under Verification).Review guide
mainat863153e3c. Head:0c877360b. Not stacked.crates/types/src/ir/operator/asap.rs:evaluated_rows_kind(new, public), next toselected_from; it returns the node's own kind, or for aStatenode the IR's existingsource_kind.crates/executor/src/physical_planner/mod.rs:input_rows_kind(its first input'sevaluated_rows_kind), the oneover_relationcomputed incompile_internal,compile_node(newover_relation: boolparameter), and its uses inbind_operation(Aggregate,SummaryAgg,SummaryMerge) andsummary_agg_build.crates/executor/src/operators/mod.rs:Operator::with_emit_empty_global(emit_empty_global, parameters), now coveringAggregateandPerGroupSummaryBuildas well asSummaryMerge.crates/executor/src/operators/aggregate/{mod,columnar}.rs: nullability (Reduction::is_null_without_values) and the empty global row.over_relationargument at every test call ofcompile_node(truefor the plain test tables,falsefor the PromQL binary tests); four executor tests that build a global SQL aggregate or build directly now call.with_emit_empty_global(true, None).Rebase onto main
863153e3c, and the #1242 coordination doneRebased from
59365ce80ontomain863153e3c(after #1242, #1309, #1339, #1330, #1324 and #1300). Earlier heads are kept asbackup/fix/1348-empty-global-promql-617615169andbackup/fix/1348-empty-global-promql-3433ca055. #1242 merged first, so this PR made the changes listed under "Coordination with #1242" in two new commits.crates/executor/src/physical_planner/mod.rscompile_node's doc: #1242's note on nodes that lower to several operators; this PR'sover_relationparagraphcrates/executor/src/operators/aggregate/mod.rswith_measure_filters(withStdDev/Variance); this PR'sis_null_without_values()is_null_without_values(), whose list now includesStdDevandVariancecrates/executor/tests/physical_dag.rsLifetime::Noneargument; this PR's.with_emit_empty_global(...)2175a8683fix(executor): global stddev and stdvar follow the input's result kind (test first).a_global_aggregate_follows_its_input_rows_kindnow also covers populationStdDevandVariance. Before the fix it failed withStdDev { col: Some(1), population: true }: invalid DAG: native output type differs from Planner output: the derived output was nullable over a PromQL vector without an identity column. The fix:!population || nullable;finish_measureloses itsNone if promql => NaNarm;with_emit_empty_global's doc names the standard deviation and variance measures.Also in this commit:
runtime_install::a_global_promql_count_over_an_empty_aggregate_answers_no_sampleaddsstddev(sum by (job) (m >= 1000))andstdvar(sum by (job) (m >= 1000)), which give no sample, andstdvar(count by (x) (m)), which gives 0.sql_table_scans::a_global_sql_aggregate_over_no_row_answers_one_rowaddsstddev_popandvar_sampover no row, which give NULL.sql_variance_and_stddev_follow_their_definitionsandcorrelation_follows_datafusionbuild their global SQL aggregate with.with_emit_empty_global(true, None).0c877360btest(planner): #1339 deletedCountingStore, so the PromQL empty-aggregate test runs overtime_series_tables(...)directly, through the sharedrunhelper.Checked on
0c877360b, by test name againstmain863153e3c:cargo check --workspace --all-targets,cargo fmt --all -- --checkandcargo clippy -p asap-executor -p asap-planner -p asap-types -p asap-cost --all-targets -- -D warningsare clean.cargo test --no-fail-fast -p asap-executor -p asap-planner -p asap-types: 645 passed, 1 failed, 2 ignored. The 1 (summary_sharing::identical_ungrouped_queries_share_their_producers) fails onmaintoo. Nothing newly fails.Checked against main
3e434c760After the Stage 3 execution-contexts refactor (#1237) merged, this head (
617615169) was rebased locally ontomain3e434c760with no conflicts and built there; it needs no new push.cargo check --workspace --all-targets,cargo fmt --all -- --checkandcargo clippy -p asap-executor -p asap-planner -p asap-types --all-targets -- -D warningsare clean.cargo teston those three crates: 567 passed, 2 failed, 2 ignored. The 2 (runtime_install::check_compiles_reports_the_executors_error,summary_sharing::identical_ungrouped_queries_share_their_producers) fail onmaintoo.Behavior: before → after
Exact,
plan_installable, run with the executor at T over gaugem(4 series, every value in (0, 1000)):count(sum by (job) (m >= 1000)){} 0count(count by (x) (m))with no sample{} 0count(sum by (job) (m >= 0)){} 2{} 2{} 2SELECT count(*) FROM http WHERE ts >= now() - INTERVAL '5 minutes' AND falseSELECT sum(latency) FROM http WHERE … AND falseExecutor, a global aggregate over
[job, v](the output ofmax by (job) (m)), jobs a = 1 and b = 3:Relationsum,avg,min,max,quantile(0.5)native output type differs)This serves correctness: a PromQL query's answer must not depend on whether an intermediate aggregate kept the series identity column.
Interfaces
crates/types/src/ir/operator/asap.rscrates/executor/src/physical_planner/mod.rscrates/executor/src/operators/mod.rs,crates/executor/src/operators/summary/mod.rsimpl Operator { - /// A merge into no groups that outputs one empty state when it has no - /// input state: the global SQL aggregate a rollup of a relation's - /// grouped states to no keys stands for, whose one row is NULL over no - /// input rows. ... - pub fn with_emit_empty_global(self, parameters: Option<SketchParams>) -> Result<Self, Error>; + /// An aggregate, per-group build or merge into no groups follows SQL's + /// rule for an empty input when `emit_empty_global`: its one global + /// group still has a row, as `SELECT count(*) FROM t WHERE false` + /// answers 0 and `SELECT sum(x) FROM t WHERE false` NULL. Otherwise (the + /// default; `false` leaves the operator as it is) an empty input has no + /// row, as a PromQL `count(v)` over an empty vector has no sample. + /// + /// An `Aggregate`'s SUM, AVG, MIN, MAX and percentile measures become + /// nullable (NULL over no row). A `PerGroupSummaryBuild` outputs an empty + /// state of its own θ, and a `SummaryMerge` one of `parameters`: the θ + /// of the states it merges (the exported build node's `parameters`), + /// `None` for an exact family and for the other two. + pub fn with_emit_empty_global( + self, + emit_empty_global: bool, + parameters: Option<SketchParams>, + ) -> Result<Self, Error>; }The private
Kind::AggregateandKind::PerGroupSummaryBuildgain anemit_empty_global: boolfield (the oneKind::SummaryMergealready had). It is serialized only whentrue, so a compiled operator without it reads back with the PromQL default. Saved compiled operators must be rebuilt: a SQL globalAggregateorPerGroupSummaryBuildcompiled and serialized before this PR reads back without the field and so gets the PromQL rule (no row over an empty input). Plans compiled at install, as today, are not affected.A private
Reduction::is_null_without_values()(executor, aggregate operator) holds the one list of measures that are NULL for a group with no (matching) value;with_emit_empty_globalandwith_measure_filtersboth read it.Usage (
a_global_promql_count_over_an_empty_aggregate_answers_no_sample,crates/planner/tests/runtime_install.rs):Algorithm changes
SQL-or-PromQL rule for a global aggregate.
count(sum by (job) (m >= 1000))answered{} 0;sum(min without (resource) (kube_resourcequota{…}))did not install.emit_empty_global = !input.has_promql_series_identity()forPerGroupSummaryBuild; the same test incolumnar::aggregateand inOperator::aggregate's SUM/AVG nullability; MIN/MAX/percentile were nullable for every global aggregate; quantile vs percentile was chosen by the same test. Only the merge used the IR (built_from_relation).compile_internalcomputesover_relationonce per node, and only for the operators that read it (a NonASAPAggregate,SummaryAgg,SummaryTumblingWindowChain,SummaryMerge):input_rows_kind(ir_node) == Relation, whereinput_rows_kindis the IR'sevaluated_rows_kindof the node's first input. For aStateinput the IR follows the states to the rows they were built from through every summary operator (SummaryAgg, merges,SummarySubtract,SummaryJoin,MaintainPopulation, state selections), the same walk it uses forSummaryEstimate's result kind andquantile_may_be_empty. Every other node passesfalse, which none of them reads.Relation, a globalAggregate, per-groupSummaryAggbuild (and chain build) or rollupSummaryMergecallswith_emit_empty_global(true, …); otherwise it passesfalse. A quantile isReduction::Percentileover aRelation,Reduction::Quantileotherwise.emit_empty_globalfield. An aggregate's measure is nullable when its input column is, or when it emits the empty global row (Reduction::is_null_without_values: SUM, AVG, MIN, MAX, percentile).finish_measureanswers NULL for a group with no value in both languages (a PromQL group always has one).a_global_aggregate_follows_its_input_rows_kindanda_global_merge_reads_the_rows_its_states_were_built_from(below); every corpus plan onmainis unchanged.source_kind; one memo group's states come from rows of one kind). Keyed and all-groups builds never emit an empty global state, as before. The per-measure filter check (with_measure_filtersrefusing filters over a series identity) is an input validation, not a semantics choice, and is kept.Design scope
evaluated_rows_kind(newpub fn,types::ir::operator::asap)selected_fromsource_kind) forSummaryEstimate's result kind andquantile_may_be_empty. Making it public deletes the executor's copy (rows_kind), which disagreed on states built throughSummarySubtract/SummaryJoin.Reduction::is_null_without_values(private method, executor aggregate operator)emit_empty_globalonKind::AggregateandKind::PerGroupSummaryBuildKind::SummaryMergealready had, with the same meaning.compile_node(…, over_relation: bool)compile_node,bind_operation,summary_agg_build); the plan format is unchanged.No new file, module or type.
input_rows_kindreplacesbuilt_from_relation(deleted) in the same file.Decisions and open questions
compile_noderather than adding a field toPhysicalASAPDAGNode. The plan format stays unchanged, and the executor already restores the IR. Rejected: storing the kind in the plan (a plan format change for a fact the IR already gives).over_relation) rather than the kind at every lowering function, so the== Relationcomparison happens once;with_emit_empty_globaltakes the flag as a builder setter, so the five call sites have noif … else Ok(op).Coordination with #1242
Whichever of this PR and #1242 merges second makes these changes (from the round-1 abstraction review, against #1242 at
586df8b7c):StdDev/Variancenullability (feat(executor): grouped stddev/variance, corr, group, count_values, exact topk and without grouping #1242aggregate/mod.rs:84-94): replace!input.has_promql_series_identity() && (!population || groups.is_empty() || nullable)with!population || nullable. PromQLstddev/stdvaralways lower withpopulation: true(frontend-promql/src/promql.rs:555-563), so a PromQL group is non-null without an identity check; SQL sample variance stays nullable.Reduction::StdDev { .. } | Reduction::Variance { .. }toReduction::is_null_without_values, which covers bothwith_emit_empty_globalandwith_measure_filters(feat(executor): grouped stddev/variance, corr, group, count_values, exact topk and without grouping #1242's ownwith_measure_filterslist already has them).finish_measure'sMomentsarm (feat(executor): grouped stddev/variance, corr, group, count_values, exact topk and without grouping #1242columnar.rs:619): deleteNone if promql => Value::Float64(f64::NAN)and keepNone => Value::Null. With population variance a PromQL group always has at least one value, soNonedoes not occur there;finish_measurehas nopromqlargument after this PR.aggregate/mod.rs:49(Sum/Avg),:66-70(Percentile) and:78(Min/Max) take this PR's versions.columnar.rs:494/506/519/530/653/719promqluses are replaced by the operator'semit_empty_global.summary/mod.rs:472andphysical_planner/mod.rs:1222/1234take this PR's versions.with_measure_filtersdestructure needs..for the new field, and its measure list becomesmeasures[index].is_null_without_values().compile_nodecalls, so the new argument causes no extra edits there.Verification
maincargo nextest run -p asap-executorcargo nextest run -p asap-plannermain(summary_sharing::identical_ungrouped_queries_share_their_producers,runtime_install::check_compiles_reports_the_executors_error)cargo nextest run -p asap-integration-testsmaincargo clippy -p asap-executor -p asap-planner -p asap-types --all-targets -- -D warnings,cargo fmt --all --checkNew tests (failing on
mainfirst):runtime_install::a_global_promql_count_over_an_empty_aggregate_answers_no_sample:count(sum by (job) (m >= 1000)),count(count by (x) (m))andcount(sum by (job) (m >= 0)), each planned exactly and run with no sample (empty) and with samples (empty, 1, 2). Onmain:count(sum by (job) (m >= 1000)) with no sample: left [0.0], right [].sql_table_scans::a_global_sql_aggregate_over_no_row_answers_one_row: SQL control.COUNT(*)andSUMover the last 5 minutesAND falseanswer one row each (0, NULL); withoutfalse, 300 and 314,850. Passes onmainand here.physical_semantics::a_global_aggregate_follows_its_input_rows_kind: a globalsum,avg,min,maxandquantile(0.5)over[job, v](no identity column). Over a PromQL vector: non-null, no row over empty input, 4/2/1/3/2. Over aRelation: refused against a non-null output, and with a nullable one it answers one NULL row (after a serialization round trip) and the same values. Onmainthis shape compiled only with SQL rules.physical_planner::tests::a_global_merge_reads_the_rows_its_states_were_built_from:SummaryMerge(group_by: [])overSummarySubtractand overSummaryJoinof aSummaryAggover a SQL table reads aRelation, and over a PromQL series anInstantVector. The executor does not lowerSummarySubtract/SummaryJoinyet (reserved ASAP operation has no native lowering), so the test checksinput_rows_kind, which decides the rule. On round 1's head (dd6117e7b, the executor's ownrows_kind) it failed withSummarySubtract: left: State, right: Relation: such a merge got the PromQL rule (no row over no input rows).sum(max by (job) (m > 1e9))could not be an end-to-end test here: onmainit fails earlier withsummary update column missing or ambiguous(#720), as dosum,max,min,avgandquantileover any… by (job)output. The executor test covers its rule.Corpus plan diff (
plan_installable, one query per workload, exact and 5 % targets, install outcome and DAG hash per entry). Corpora: the four PromQL corpora (promql_corpus_docs,promql_corpus_testdata,o11y_bench_promql,awesome_prometheus_alerts) and the four SQL corpora (data-quality, NetFlow, BGP analytics, BGP Jan-2024 workload).Round 2 (this head) vs
main(59365ce80) and vs round 1's head: 3,140 queries, 6,280 entries; 3,514 install on both. Every install outcome and installed DAG is the same as onmainand on round 1's head; the only text differences (19 vsmain, 20 vs round 1) are DataFusion's random "Did you mean …" suggestions in SQL lowering errors.Round 1:
mainvs this PR, plus the 253-querywithoutcorpus: PromQL 6,202 entries (3,298 install on both), SQL 584 entries (216 install on both). Every installed DAG is byte-identical and every error is identical. The only text differences are DataFusion's nondeterministic "Did you mean …" suggestions in SQL lowering errors.mainhas nonative output type differserror in these corpora: the nested shapes fail earlier there (A Sum over a grouped PromQL sum fails to execute: the update reads a sample-value column the grouped output doesn't have #720, andwithoutis not supported yet).Round 1: feat(executor): grouped stddev/variance, corr, group, count_values, exact topk and without grouping #1242's head
c462ccad5vs that head plus this fix: 15 queries (30 entries) change and all other DAGs are byte-identical. 13 distinct queries now install:sum(min without (resource) (kube_resourcequota{…}))and its 7 variants (with/ sum(kube_node_status_allocatable{…}) > 1.5);quantile(0.9, sum without (cpu) (rate(node_cpu_seconds_total{mode="system"}[5m])));sum(sum without (instance)(node_cpu) / ignoring (mode) group_left sum without (instance, mode)(node_cpu));sum(sum by (mode, job)(node_cpu) / on (job) group_left sum by (job)(node_cpu));sum(avg(mongodb_… - mongodb_…)) - sum(avg(… group_right …)) <= 0(alerts);quantile without(point)(NaN, data), which failed witha percentile requires Float64 input and a fraction in [0, 1]because the quantile was chosen as SQL's.sum without (device) (max without (fstype, mountpoint) (…)) / scalar(sum(max without …))moves to the next gap (native Planner binding currently requires a unary operation). So 10 of the 11without-corpusnative output type differsqueries now install.Review status
Round 1 (head
dd6117e7b): two independent reviews, abstraction + layering and functional + efficiency. Both approve; no blocker. Addressed in this head:rows_kindcopied the IR's privatesource_kindand disagreed on states built throughSummarySubtract,SummaryJoin,MaintainPopulation,Extensionrows_kinddeleted; the IR'sevaluated_rows_kindis public andinput_rows_kindcalls it. Testa_global_merge_reads_the_rows_its_states_were_built_fromfailed ondd6117e7b(State, the PromQL rule).Reduction::is_null_without_values, used bywith_emit_empty_globalandwith_measure_filtersif global && over_relation { with_emit_empty_global } else { Ok(op) }siteswith_emit_empty_global(emit_empty_global, parameters)takes the flag;falseleaves the operator as it isover_relation: bool, fromcompile_internalthroughcompile_node,bind_operationandsummary_agg_buildAggregate,SummaryAgg,SummaryTumblingWindowChainandSummaryMergenodes (FinalizeExactAccumulator's binding does not read it)with_emit_empty_global's doc covered only the aggregate in its first sentenceMin/Maxnullable regardless of result kind, while the executor now produces non-null for PromQLrows_kind; coordination with #1242Checked clean in round 1:
irdoes not importstages; no cross-stage calls;built_from_relationand the merge-onlywith_emit_empty_globaldeleted, not left as shims; everyemit_empty_globalfield is read; the remaininghas_promql_series_identityuses in the executor are structural or input validation, not a SQL/PromQL choice; compile cost about +2% (a few tenths of a millisecond per compile), run time unchanged.Follow-ups
sum/max/min/avg/quantileover a… by (…)output fails withsummary update column missing or ambiguousonmain.Min/Maxnullability should follow the input's rows kind, as the executor now does.scalar(…)in a binary operator overwithoutaggregates (the remaining corpus query).