Repository navigation
feat(executor): grouped stddev/variance, corr, group, count_values, exact topk and without grouping - #1242
Merged
Conversation
This was referenced Nov 7, 2026
This was referenced Nov 15, 2026
zzylol
force-pushed
the
exec/x2-reduce-aggregates
branch
from
November 15, 2026 05:25
33293b7 to
aef6562
Compare
This was referenced Nov 15, 2026
Merged
…s, exact topk and without grouping
Reduce aggregates the frontends emit that the executor refused:
Reduction::StdDev/Variance (Welford, population and sample, SQL NULL
rules), Reduction::PearsonCorr (DataFusion corr), Reduction::Group,
count_values as a value-text column plus a count, an exact
Aggregate{Reduce, [TopK]} as per-partition Sort + Limit + ranked-row
Project, and GroupKeys::without resolved to the kept columns (series
identity less the excluded labels and __name__) for Aggregate,
SummaryAgg, Sort and Limit. SQL sample STDDEV/VAR_SAMP are typed
nullable.
The exact top-k lowering is the only one now that Stage 2 no longer
lowers TopK, so the executor reads the ranked-row helpers in
types::ir::schema::topk, which become pub. The planner tests run the
installable plan's DAG directly and declare the SQL table at rest; an
ingesting whole-table aggregate hits a planner install bug (#1320).
- count_values ... without (...) installs and matches Prometheus: the IR adds the value-label column under without, and the without projection drops a same-named label from the series identity before grouping. - The IR marks a series identity kept by without as computed (OutputColumnSource::AddedLabel), with the rule documented. - Sort gains fetch: Option<u64>, a bounded per-group heap keeping the first n rows of each group; exact top-k and the current-series top-k lower to it instead of Sort then Limit. - The without identity rewrite parses borrowed labels and encodes once. - CoMoments is six flat fields (bit-identical, tested); the variance and correlation kernels read the Float64 array directly. - lower_exact_topk / lower_count_values / top_k_sort extracted from compile_internal; WITHOUT_PROJECTION_HELPER names the fixed helper index; one grouping bounds check. - Test helpers shared through physical_common and time_series; duplicate integration top-k and without cases dropped. - physical-compile-coverage.md: reduce aggregates section, without row fixed.
- count_values: a label that collides with a by or kept column makes that position a computed Utf8 label; a label the without list names is removed again, counting per kept group (IR schema and lowering). - One label-set predicate, ir::schema::is_label_set_column, read by the IR and resolve_without; one count_values label push for both paths. - compile_internal binds keys, filters, k and label once; the top-k and count_values lowerings return their operators. - promql_rows::series_identity_without: streaming rewrite of the canonical identity JSON, byte-equal to decode/filter/encode. - Sort with fetch: candidates compared with the heap's last row straight from the Arrow arrays, keys decoded only for kept rows, replace-top on a Vec heap, HeldRow without the sort-key slice, chunks gathered with one interleave (Batch::interleave). - OutputColumnSource::AddedLabel renamed ComputedLabel; nits.
- Sort{fetch}: a map sort key is compared by decoded value, chosen by
type (Arrow's map order differs); a map key column is NaN-checked
when its batch is read, as the full sort does. Randomized test of
Sort{fetch} against sort then limit, with map keys.
- count_values("__name__", v) without (...) drops __name__, as
Prometheus does (removed_by_without in the IR schema).
- series_identity_without: doc says the input must be canonical (a
canonical check cost 18-34 %); fuzz test against decode.
- Each group is listed once per batch (per-group stamp); the heap
compares numbers and times inline (compare_held_key).
- LabelIdentity over a dictionary rewrites the dictionary's texts once
per entry and keeps the keys.
- decode_cell shared by decode and KeptKeyValues; one NaN check
(key_column); count_values aggregate built once; redundant guard in
resolve_without dropped; READING sentinel replaced by chunks.len().
- OutputColumnSource doc lists the replaced-column case.
`count without (instance) (m)` failed to install with `PromQL vector requires labels, time and one value`: its output keeps a series identity, so the lowering adds the label rewrite that drops `__name__`, and that rewrite's layout accepted only a Float64 value, while a cross-series count is Int64 (as `count by` answers it). The layout now takes the one numeric column as the value; operators that compute on the value (binary operators, `histogram_quantile`, `label_replace`) read it through `float_layout`, which still requires Float64. Tests: planner `count without (instance)` and `(instance, zone)`, `group without` and `stddev without` with hand-computed answers; an executor test that a label rewrite carries an Int64 count and a binary operator refuses it. `sort_with_fetch_equals_sort_then_limit_over_random_rows` runs every seed with 40 cases per seed in a debug build (400 in release): 63 s to 2.4 s.
…tity
topk(1, sum without (instance) (m)) answered {job="b"} 2 instead of
{job="a"} 7: the frontend binds the sort key to Column(0) of the
aggregate's [sum] output, and identity realization then inserts the
identity column ahead of the measure without updating the parent.
sort, sort_desc and bottomk ordered by the identity text the same way.
with_promql_series_identity now compares each node's input scope before
and after realization and, when a column moved, renumbers every column
reference into that scope (NonASAPOp::map_input_columns, with
map_columns on GroupKeys, Reduction and AggIntent). A reference to a
column realization does not keep is an error.
- One generic column walk per IR argument type, with no catch-all arm: AggIntent::try_map_cols (map_columns, input_cols and the frontend's resolve_agg_intent use it) and GroupKeys::try_map_keys. A new column-carrying intent now fails to compile instead of being skipped. - NonASAPOp::map_parts walks every variant once; map_children and map_input_columns are thin wrappers. NonASAPOp::input_scope sits next to inputs and serves realization and Join validation. - Series identity realization renumbers an Aggregate's HAVING by how its output moves, and returns a typed SeriesIdentityError. The planner falls back to the unrealized root only when realization does not apply (limitk, info()); a dropped column is OptimizeError::SeriesIdentity. - Nested aggregates over a without aggregate install: the executor resolves a summary update's sample value as the planner does (the one numeric column after an aggregate renamed it). - avg without two or more labels plans: Pass 1's avg casts the last column, whatever the number of kept labels. - A grouping key that a without aggregate below removed is absent, as in Prometheus; a closed schema that carries the series identity no longer proves a label absent (it may be in the identity).
A schema error from rebuilding an operator after series identity renumbering was wrapped as NotApplicable, so promql_root silently planned the unrealized root. It is now SeriesIdentityError::InvalidRealizedOperator, a planning error like DroppedColumn. NotApplicable keeps only operators realization does not cover and sources that are not open PromQL sources.
zzylol
force-pushed
the
exec/x2-reduce-aggregates
branch
from
November 16, 2026 01:45
586df8b to
0b0e034
Compare
This was referenced Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
plan_installableofstddev by (job) (m)fails withinvalid DAG: node 2: invalid DAG: aggregate intent has no native implementation, andsum without (instance) (m)fails withgrouping without requires resolved label columns.StdDev,Variance,PearsonCorrandGroup, acount_valueslowering, an exactAggregate{Reduce, [TopK]}lowering onto a sort that keeps the first k rows per group (Sort { fetch }), andwithoutgrouping. The IR marks a series identity kept bywithoutas computed, and adds thecount_valueslabel underwithout(unless the grouping removes it, a label thewithoutlist names or__name__, as Prometheus removes it again). Series identity realization now renumbers the column references of every operator above a node it adds the identity column to, sotopk(1, sum without (instance) (m))ranks by the sum, not by the identity text. An aggregate over awithoutaggregate (sum by (job) (sum without (instance) (m))),avg withouttwo or more labels, and a grouping key awithoutbelow already removed now plan and answer as Prometheus does.stddev by (job) (m)gives{job="a"}1.2472 (√(14/9)) and{job="b"}0.25. This closes the executor half of [Bug] topk over rate/increase plans but fails installation: aggregate output names differ from measures #1303:topk by (region) (4, increase(data_total[5m]))now installs, both exact and approximate. An exact top-k holds groups × k rows instead of its whole input, and at 10^6 rows runs 5–17× faster than the full sort then limit it replaces (table below).Review guide
Base:
main(3e434c760).Read first:
crates/executor/src/operators/sort.rs:sort_with_fetch, the bounded per-group heap, withkey_comparatorandKeptKeyValues.crates/executor/src/physical_planner/mod.rs:lower_exact_topk,lower_count_values,top_k_sortandresolve_without, next to each other;compile_internalonly dispatches to them.crates/types/src/ir/schema/aggregate_schema.rs:without_output_schemaandpush_count_values_label.crates/executor/src/physical_planner/promql_rows.rs:series_identity_without.crates/executor/src/operators/aggregate/columnar.rs:MomentsandCoMoments.crates/types/src/ir/schema/series_identity.rs:positions_after_realization,renumberandSeriesIdentityError, and their use inwith_promql_series_identity, withNonASAPOp::map_parts(behindmap_childrenandmap_input_columns) incrates/types/src/ir/operator/non_asap.rs.crates/planner/src/run.rs:promql_root, the one place a realization error falls back or fails.crates/frontend-common/src/resolve.rs:resolve_group_keys_promqlandlabels_removed_by_without, the absent-label rules.Careful review: the NULL and NaN rules in
finish_measure,Operator::aggregate's nullability,non_asap.rs's SQL schema nullability; thewithoutrewrite over a series identity and thecount_valueslabel that replaces an identity label; tie order and memory accounting insort_with_fetch, and thatkey_comparatororders NULL, NaN and-0exactly ascompare_key(a map key is compared by decoded value, not by Arrow); the per-dictionary-entryLabelIdentitypath inexpressions/columnar.rs, whose output dictionary may repeat a text.Mechanical:
Operator::sortgains afetchargument. The SQL/PromQLSortbinding and the tests passNone; the two top-k lowerings passSome(k)throughtop_k_sort.planner/tests/time_series,integration-tests/tests/physical_common).topk::ranked_columnandranked_row_projectionbecomepub, because the executor now calls them.Overlap with open fix(executor): run each node by its plan Lifetime, so roots drain in any order #1300: both PRs touch
crates/executor/src/physical_planner/mod.rsandcrates/executor/tests/physical_semantics.rs. Whichever lands second adds theLifetimeargument to this PR's newPhysicalDAG::addcalls:Lifetime::Nonefor the helpers (the top-kSort, thecount_valuesProject, thewithoutprojection), andnode.lifetimefor the node's own id.Rebase onto main
3e434c760Rebased from
59365ce80ontomainat3e434c760, after the merge of the Stage 3 execution-contexts refactor (#1237). Same 8 commits; conflicts resolved as that PR's merge notes ask:crates/planner/tests/main.rs: bothmod exact_only_reads;andmod reduce_aggregates;.crates/types/src/ir/operator/non_asap.rs, theFilter,Join,Sort,SQLWindowFuncandPromqlRelabelarms ofmap_parts:fruns on the direct inputs before any scalar expression (Filterchild, thenpred;Joinleft, right, thenpred;Aggregatechild,filters, thenhaving;Sortchild, thenkeys;SQLWindowFuncchild,args, thenorder_by;PromqlRelabelchild, thenvalue;Projectchild, thencols). This PR's column renumbering is kept:map_pred(…, column, f),map_keys(…, column, f),partition_by.map_columns(column), andkeptfor aScan's predicates and anAggregate'shaving.ScalarExpr::map_partsmaps anInSubquery's subquery before itsexpr.Self::childrenorder …qthenc" sentence moved frommap_childrenontomap_parts;map_childrennow says it visits inmap_parts's order.non_asap::tests::map_children_calls_f_in_children_orderpasses.Checked on the rebased head, by test name against
main3e434c760:cargo check --workspace --all-targets,cargo fmt --all -- --checkandcargo clippy -p asap-types -p asap-executor -p asap-planner -p asap-integration-tests -p asap-frontend-common -p asap-stage1-pass1-per-computation-candidates --all-targets -- -D warningsare clean.cargo teston the same six crates: 877 passed, 8 failed, 6 ignored. All 8 fail onmaintoo;mainhas one more failure,runtime_install::check_compiles_reports_the_executors_error, which this PR fixes. Nothing newly fails.Behavior: before → after
The user writes one exact repeating PromQL query, every 10 s, over a gauge
m(10 series, sampled every 10 s, ingesting):stddev by (job) (m)aggregate intent has no native implementationa√(14/9),b0.25stdvar by (job) (m)a14/9,b0.0625group by (job) (m)a1,b1count_values("v", m)v="1"1,v="2"2,v="2.5"1,v="4"1count_values("v", m) without (instance), over{job=a} 1×2,{job=b} 2,{job=b,zone=z} 2{job=a,v=1}2,{job=b,v=2}1,{job=b,zone=z,v=2}1count_values("zone", m) without (instance), same data{job=a,zone=1}2,{job=b,zone=2}2:zoneis overwritten before grouping, as in Prometheuscount_values("zone", h) without (instance, zone), overh{job=a}2, 2, 2 (zonesz, none,y) and 5 (zonez){job=a}4: the excluded label is set and then removed, as in Prometheuscount_values("instance", m) without (instance){job=a}2,{job=b}1,{job=b,zone=z}1, noinstancelabelcount_values("__name__", h) without (instance), sameh{job=a}1,{job=a,zone=y}1,{job=a,zone=z}2:withoutalways drops__name__, as in Prometheussum without (instance) (m)grouping without requires resolved label columns{job="a"}7,{job="b"}2,{job="b",zone="z"}2.5count without (instance) (m){job="a"}3,{job="b"}1,{job="b",zone="z"}1 (Int64, ascount byanswers)count without (instance, zone) (m){job="a"}3,{job="b"}2group without (instance) (m)stddev without (instance) (m){job="a"}√(14/9),{job="b"}0,{job="b",zone="z"}0topk(1, sum without (instance) (m))grouping without requires resolved label columns(at this PR's previous head:{job="b"} 2, ranked by the identity text){job="a"}7bottomk(1, sum without (instance) (m)){job="b"}2sort(sum without (instance) (m))/sort_desc(...){job="b"}2,{job="b",zone="z"}2.5,{job="a"}7 / the reversetopk(1, count without (instance) (m)),sort_desc(count without (instance) (m)){job="a"}3 / 3, 1, 1topk by (job) (1, sum without (instance) (m)){job="a"}7,{job="b",zone="z"}2.5sum by (job) (sum without (instance) (m))grouping without requires resolved label columns(previous head:summary update column missing or ambiguous){job="a"}7,{job="b"}4.5max without (zone) (sum without (instance) (m)){job="a"}7,{job="b"}2.5avg without (instance, zone) (m)invalid scalar function signature: column outside scalar input scope){job="a"}7/3,{job="b"}2.25sum without (instance) (sum without (instance) (m))column instance not found in schema)sum without (instance) (m): the outerinstanceis already gone and removes nothingsum by (instance, job) (sum without (instance) (m) > 1){job="a"}7,{job="b"}4.5:instanceis the empty labelmax without (instance) ((10 * min without (zone) (m)) unless n)column instance not found; it was dropped as absent and answered withinstancekept wherever the rest plannedlimitk(2, up),info(up)topk(3, sum by (job) (rate(c[5m])))aggregate intent has no native implementationj20.5,j30.7,j40.9topk by (job) (1, increase(c[5m]))(the #1303 shape)SELECT k, STDDEV(x), VAR_SAMP(x), STDDEV_POP(x), VAR_POP(x) FROM t GROUP BY kk1(√(7/3), 7/3, √(14/9), 14/9);k2(NULL, NULL, 0, 0);k3all NULLSELECT corr(a, b) FROM tThis serves MVP correctness: every shape the frontends emit and the planner selects must run.
Interfaces
crates/executor/src/operators/mod.rs and sort.rs
enum Kind { Sort { keys: Vec<SortKey>, groups: Vec<usize>, + /// Keep only the first `fetch` rows of each group, in sort order + /// (`topk by (job) (3, m)` gives `Some(3)`); `None` sorts every row. + #[serde(default, skip_serializing_if = "Option::is_none")] + fetch: Option<u64>, }, } impl Operator { + /// Sort `input`'s rows by `keys` within each group of the `groups` + /// columns, groups in key order. `fetch` keeps only the first `fetch` + /// rows of each group, in sort order: `topk by (job) (3, m)` is a sort + /// by value descending, grouped by `job`, with `fetch` `Some(3)`, which + /// holds 3 rows per job instead of every input row. `None` sorts every + /// row. - pub fn sort(input: SchemaRef, keys: Vec<SortKey>, groups: Vec<usize>) -> Result<Self, Error> + pub fn sort( + input: SchemaRef, + keys: Vec<SortKey>, + groups: Vec<usize>, + fetch: Option<u64>, + ) -> Result<Self, Error> }crates/executor/src/operators/aggregate/mod.rs
pub enum Reduction { ... Percentile { column: usize, /// The fraction, in [0, 1]. q: f64, }, + /// Standard deviation of a numeric column's non-NULL values: the + /// square root of [`Reduction::Variance`]. + StdDev { + column: usize, + population: bool, + }, + /// Variance of a numeric column's non-NULL values, by Welford's + /// update as Prometheus `stdvar` and DataFusion `var_pop`/`var_samp` + /// compute it: the squared deviations' sum over n (population) or + /// n - 1 (sample); NaN once a value is NaN or infinite. SQL gives NULL + /// for no value, or one value of a sample; PromQL NaN. The values 1, 2 + /// and 4 have mean 7/3: population variance 14/9, sample variance 7/3. + Variance { + column: usize, + population: bool, + }, + /// SQL `corr(left, right)`, DataFusion's: Pearson's correlation over + /// the rows where both are non-NULL; NULL for fewer than two rows, a + /// constant input or one input's mean NaN, NaN when both are. + PearsonCorr { + left: usize, + right: usize, + }, + /// PromQL `group`: 1 for every group. + Group, }crates/executor/src/expressions/mod.rs
pub enum Expression { ... - /// Canonical encoding of a label map less `excluding`, identical to - /// `promql_rows::encode_series_identity`. + /// Canonical encoding of a label map, or of a series identity, less + /// `excluding`, identical to `promql_rows::encode_series_identity`. LabelIdentity { column: usize, #[serde(default, skip_serializing_if = "Vec::is_empty")] excluding: Vec<String>, }, Literal { value: Value, dtype: DataType, }, + /// A Float64 column's value as text, as Go's + /// `strconv.FormatFloat(v, 'f', -1, 64)` writes it (PromQL + /// `count_values`): the shortest decimal that reads back as `v`, + /// without exponent; `NaN`, `+Inf` and `-Inf`. `2.5` is `"2.5"`, `1e21` + /// is `"1000000000000000000000"` and `-0.0` is `"-0"`. + FloatText(usize), Negate(Box<Expression>), ... }crates/types/src/ir/schema/aggregate_schema.rs
crates/types/src/ir/schema/mod.rs
crates/executor/src/physical_planner/promql_rows.rs
crates/executor/src/values/batch/mod.rs
impl Batch { + /// The rows `indices` of `batches` (all in `schema`), each a (batch, + /// row) pair, in that order: one copy of each row, where [`Self::concat`] + /// then [`Self::take`] copy it twice. [...] + pub(crate) fn interleave( + schema: &SchemaRef, + batches: &[Self], + indices: &[(usize, usize)], + ) -> Result<Self, Error> }without_output_schema(private) now gives a kept series identity or label map the sourceComputedLabel(viais_label_set_column), and adds thecount_valueslabel column before the count, as thebypath does. On both paths acount_valueslabel that collides with abyor kept column turns that position into a computedUtf8label (push_count_values_label); underwithout, a label the grouping removes (one thewithoutlist names, or__name__, privateremoved_by_without) is not added. Its doc carries the example{__name__=m, job=a, instance=1, zone=z}→{job=a, zone=z}underwithout (instance).crates/types/src/ir/schema/topk/mod.rs
These two are
pubbecause the executor's top-k lowering builds the same ranked rows that the IR's schema derivation describes.crates/types/src/ir/operator/non_asap.rs, arguments.rs and agg_intent.rs
impl<C> NonASAPOp<C> { - /// The direct operator inputs, in field order, followed by the operator - /// nodes referenced from this operator's scalar expressions. + /// The direct operator inputs, followed by the operator nodes referenced + /// from this operator's scalar expressions. pub(crate) fn children(&self) -> Vec<&C> + /// The direct operator inputs, in field order: a `Join`'s left then + /// right. Their output columns, in this order, are the operator's + /// [`input_scope`](NonASAPOp::input_scope). + pub(crate) fn inputs(&self) -> Vec<&C> + /// The operator's input scope: the output columns of its direct + /// [`inputs`](NonASAPOp::inputs), in order, which its column references + /// index. A `Join` of `[a, b]` and `[c]` reads `c` as `Column(2)`; + /// [`map_input_columns`](NonASAPOp::map_input_columns) renumbers these + /// references. + pub(crate) fn input_scope(&self) -> Vec<&Field> + where + C: OperatorInput; pub(crate) fn map_children<D>(&self, f: impl FnMut(&C) -> D) -> NonASAPOp<D> + /// This operator with every column reference into its input scope + /// renumbered by `column`: the columns of its direct inputs, in + /// [`inputs`](NonASAPOp::inputs) order. A `Sort` by `Column(0)` over + /// `[sum]` whose input becomes `[$promql_series_identity, sum]` must sort + /// by `Column(1)`; the renumbering `0 → 1` gives that. Two references are + /// in another scope and kept: a `Scan`'s predicates index its own + /// schema, and an `Aggregate`'s `having` indexes its output. Operator + /// nodes read from scalar expressions keep their own scope. + pub(crate) fn map_input_columns(&self, column: &mut impl FnMut(ColumnId) -> ColumnId) -> Self + where + C: Clone; } impl<C> GroupKeys<C> { + /// These keys with every key mapped by `f`, `by` or `without` kept; the + /// first error stops the walk. A front end resolves `by ("job")` to + /// `by (2)` with it. + pub fn try_map_keys<D, E>( + &self, + f: &mut impl FnMut(&C) -> Result<D, E>, + ) -> Result<GroupKeys<D>, E> } impl<C: Clone> AggIntent<C> { + /// This intent with every column it holds mapped by `f`, in argument + /// order; the first error stops the walk. Every variant is listed, so a + /// new variant that holds a column must be mapped here: a front end + /// resolves `Sum { col: Some("value") }` to `Sum { col: Some(1) }` with + /// it, and series identity realization renumbers `Some(1)` to + /// `Some(2)` when it inserts a column before it. + pub fn try_map_cols<D, E>( + &self, + f: &mut impl FnMut(&C) -> Result<D, E>, + ) -> Result<AggIntent<D>, E> }GroupKeys::map_columns,Reduction::map_columnsandAggIntent::map_columns(pub(crate),ColumnIdonly) are one-line wrappers of these with an infallible map.AggIntent::input_colswalkstry_map_colstoo (lesshistogram_quantile'sle, a bucket bound, not a value).NonASAPOp::map_childrenkeeps its signature and is now a wrapper of the privatemap_parts.crates/types/src/ir/schema/series_identity.rs and crates/planner/src/pass.rs
pub fn with_promql_series_identity( root: &Rc<OperatorNode>, -) -> Result<Rc<OperatorNode>, String> +) -> Result<Rc<OperatorNode>, SeriesIdentityError> +/// Why [`with_promql_series_identity`] gives no root carrying +/// [`PROMQL_SERIES_IDENTITY`]. +#[derive(Debug, Clone, PartialEq, thiserror::Error)] +pub enum SeriesIdentityError { + /// The root holds an operator this realization does not cover, or a + /// source that is not an open PromQL source: `limitk(2, m)` lowers to + /// a `PromqlSeriesSample`, `info(m)` to a `PromqlInfoEnrich`, and a + /// closed scan has no dynamic labels to carry. The root is planned + /// over its label columns instead. + #[error("{0}")] + NotApplicable(String), + /// `operator` reads input column `column`, which realization does not + /// keep. Renumbering it would read another column and give a wrong + /// answer. + #[error("series identity realization does not keep column {column} that {operator} reads")] + DroppedColumn { + operator: &'static str, + column: ColumnId, + }, + /// An operator rebuilt over its realized children, with its column + /// references renumbered, has a schema that no longer derives. Its + /// unrealized form derived, and renumbering moves `Column(0)` of + /// `[sum]` to the same `sum` at `Column(1)` of + /// `[$promql_series_identity, sum]`, so this is a defect in + /// realization, not in the query. Planning the unrealized root would + /// hide it. + #[error("series identity realization built an invalid operator: {0}")] + InvalidRealizedOperator(SchemaDerivationError), +} pub enum OptimizeError { ... + /// Series identity realization would make an operator read a column it + /// does not keep, or rebuilt an operator whose schema no longer derives: + /// a defect in the realization, not in the query. + #[error("PromQL series identity realization: {0}")] + SeriesIdentity(SeriesIdentityError), } // crates/types/src/ir/schema/error.rs -#[derive(Debug, Error)] +#[derive(Debug, Clone, PartialEq, Error)] pub enum SchemaDerivationError {with_promql_series_identity's doc now says it renumbers every parent's column references, and anAggregate'shavingby how the aggregate's own output moves, and lists the three errors:NotApplicable(the caller plans the root over its label columns), andDroppedColumnandInvalidRealizedOperator(realization defects; the caller must not fall back).SchemaDerivationErrorgainsCloneandPartialEqsoSeriesIdentityErrorcan carry it.Usage
From crates/planner/tests/reduce_aggregates.rs. A runtime plans one exact repeating query with
plan_installable, then runs the installable plan's DAG with the executor over raw samples.Design scope
Every new type, field and file, placed in the engine:
Kind::Sort.fetch: Option<u64>,Operator::sort(.., fetch)topk/bottomk, SQLORDER BY .. LIMIT n, a current-series top-k.HeldRow,KeptKeyValues,key_comparator,heap_push,heap_sift_down_top,sort_order,key_column,compare_held_key(private,sort.rs)Vecmax-heap ordered by a closure, the one tie-break rule, a key column decoded with the one NaN-in-collection check, andcompare_keywith numbers and times inline for the heap.arrow_columns::decode_cell(pub(crate))Value;decodeandKeptKeyValuesboth read cells through it.identity_without_per_entry(private,expressions/columnar.rs)LabelIdentityover a dictionary column, evaluated once per dictionary entry as text, keys kept.removed_by_without(private,aggregate_schema.rs)withoutgrouping removes a label: an excluded column's name or__name__.Batch::interleave(pub(crate))ir::schema::is_label_set_columnwithoutschema and the executor'swithoutrewrite.promql_rows::series_identity_withoutencode/decode.Reduction::{StdDev, Variance, PearsonCorr, Group}AggIntent::{StdDev, Variance, PearsonCorr, Group}.Moments,CoMoments,MeasureStates::{Moments, CoMoments, Constant}(private)Expression::FloatTextwithout_kept_columnswithoutgrouping's columns, read by schema derivation and the executor.NonASAPOp::inputs,NonASAPOp::input_scope,NonASAPOp::map_input_columns(pub(crate)),NonASAPOp::map_parts(private)Joinvalidation), and one walk over every field that renumbers column references and maps children;map_childrenandmap_input_columnswrap it, asScalarExpr::map_partsdoes for expressions (nowpub(crate)).AggIntent::try_map_cols,GroupKeys::try_map_keys(pub)map_columns,pub(crate)wrappers), andinput_colsreads it.SeriesIdentityError(ir::schema::series_identity)positions_after_realization,moves,renumber(private,series_identity.rs)OptimizeError::SeriesIdentitypromql_root(private,run.rs)labels_removed_by_without(private,resolve.rs)withoutaggregate below removed from every series, which a grouping key may name.top_k_sort,lower_exact_topk,lower_count_values,check_grouping_columns,WITHOUT_PROJECTION_HELPER(private)compile_internal; the constant names the one fixed helper index.No new IR concept: a kept series identity under
withoutuses the existingOutputColumnSourcelabel variant, renamed fromAddedLabeltoComputedLabel(two files) since it now also covers a rewritten identity.Algorithm changes
Execution kernels: variance and correlation
stddev by (job) (m)andSELECT corr(a, b) FROM thad no kernel.Moments::addkeeps the count, the mean and the sum of squared deviations:delta = v - mean; mean += delta / n; m2 += delta * (v - mean).variance(population)divides by n or n - 1, andStdDevtakes the square root.CoMomentskeeps one count, both means, both sums of squared deviations and the co-momentdelta_a * (b - mean_b), and divides the population covariance by both population standard deviations.aggregation()STDDEV/STDVAR and DataFusion'sVarianceAccumulator. A non-finite value gives NaN, as in both.CoMomentswas flattened from twoMomentsplus its own count and means;co_moments_equal_two_moments_bit_for_bitchecks the old and new layouts give the same bits over 10,000 pairs from 1e-300 to 1e300 in scale.physical_semantics::{sql_variance_and_stddev_follow_their_definitions, promql_stddev_of_one_series, correlation_follows_datafusion}.Moments, 48 forCoMoments(was 80).corruses a sum-of-products form, so its last bits can differ. Tests compare within 1e-12.Execution kernel: sort with fetch
max.Sort { fetch: Some(k) }reads its input a batch at a time. Each group keeps a max-heap (aVecordered by a closure) of at most k rows, ordered by the sort keys and then by input position, so ties keep input order as the stable sort does.DynComparatorper key, wrapped to keep the operator's NULL and NaN order (NaN after every number in both directions,-0equal to0). A map key is chosen by its type for decodedcompare_key: Arrow does compare maps, but in another order ({a: NULL}before{a: 1},{a: -0}before{a: 0}).KeptKeyValues, throughdecode_cell). A map key's column is decoded when the batch is read, throughkey_column, the same NaN-in-collection check the full sort runs, so a NaN inside any row's map is an error, kept or not.compare_held_key: two numbers (neither NaN), integers or times inline, everything else throughcompare_key. The full sort keepscompare_key: the same arms there slowed a sort of 10^6 random numbers by 16–20 %.chunks.len(), the index that batch's chunk takes. Each group is listed once per batch (a per-group stamp of the batch number, no per-batch allocation), so re-pointing the batch's rows walks each touched heap once.interleave, and the output is oneinterleavein group and sort order.sort_with_fetch_equals_sort_then_limitcompares it with the full sort and a per-group limit, for fetch 0–8, both directions, NULLs first and last, NaN, ±0, ±Inf and ties, across batches, grouped and ungrouped.sort_with_fetch_over_ascending_input_equals_sort_then_limitdoes the same over ascending input (every row displaces one), with a nullable text key before the number, 18 batches.sort_with_fetch_equals_sort_then_limit_over_random_rowsruns 2,400 random cases (numbers with NULL, NaN, ±Inf and −0; nullable text; maps of nullable numbers with NULL, −0 and NaN inside; 1–3 keys, 0–2 grouping columns, fetch 0–1000) and compares rows or errors; before the map fix it failed on its first map case.Vec).sort_with_fetch_holds_groups_times_fetch_rowsruns 262,144 rows in a 1 MiB limit that the full sort exceeds.Compile: exact top-k
Aggregate{Reduce, [TopK]}was refused. Stage 2 no longer lowers it, so no path ran it.lower_exact_topk:top_k_sort(byranked_column, descending, NULLs and NaN last,fetch: Some(k)), then the ranked-rowProjectfromranked_row_projection. The current-series top-k (EvaluatePopulation{TopK}) uses the sametop_k_sortinstead of itsSortandLimit.reduce_aggregates::{promql_exact_topk_of_grouped_rates, promql_exact_topk_by_job_of_increases},current_series_heap.Compile:
withoutgroupinggroups()refused anywithout.resolve_withoutrewrites the grouping toby(without_kept_columns). Over a series identity (is_label_set_column) it first projects the identity less the excluded labels,__name__, and acount_valueslabel of the same name. Acount_valueslabel that the grouping removes (one thewithoutlist names, or__name__) is not in the IR's output, solower_count_valuescounts per kept group with no value-text column; the executor reads that fromoutputand does not re-derive it. A ranking (top-k,Sort,Limit) over a series identity is refused, because it keeps each row's labels.without_output_schemamarks that kept identity as computed (ComputedLabel), not copied from the input, so Stage 1 does not carry the input identity's column properties to it.reduce_aggregates::{promql_sum_without_instance, promql_count_values_without_instance, promql_count_values_without_overwrites_a_kept_label, promql_count_values_label_excluded_by_without_is_dropped, promql_count_values_without_replaces_a_kept_label_column, promql_count_values_name_label_under_without_is_dropped},exact_reduce_execution::without_partitions_over_a_table_rank_per_kept_column,aggregate_schema::tests::{a_without_aggregate_computes_its_kept_series_identity, count_values_adds_its_label_by_and_without_keys, a_count_values_label_replaces_a_kept_column_of_its_name, a_count_values_label_excluded_by_without_is_not_added, a_count_values_name_label_under_without_is_not_added}.withoutover a series identity. A Utf8 identity column is dictionary-encoded, so the projection rewrites the dictionary's texts, once per entry, as&strinto a new values array, and keeps the input's keys (identity_without_per_entry): noValueper identity and no re-hashing into a new dictionary. Two identities that differ only in an excluded label become one text twice in the dictionary; the group table resolves each entry by its text, so they group together (identities_equal_but_for_an_excluded_label_aggregate_in_one_group). A dictionary much larger than its batch, or a malformed text, takes the per-row path, so errors stay those of row-by-row evaluation.series_identity_withoutscans the canonical JSON's"name":"value"tokens, stepping over escapes, and copies the kept pairs' bytes into aStringsized to the input: no map, no decoding, no re-encoding. A name token without escapes is compared with the excluded names byte for byte; one with escapes is compared with each excluded name's encoding (so no name is encoded per call in the common case).series_identity_without_equals_decode_filter_encodechecks it byte for byte against decode/filter/encode over 145 identities (escaped quotes, backslashes, control characters, unicode, empty values) and 16 exclusion sets including none and every label; malformed input is an error;series_identity_without_equals_decode_on_mutated_identitiesfuzzes 20,000 mutated identities: no panic, and byte-equal to decode/filter/encode wherever decode accepts the input.IR: identity realization renumbers parent column references
withoutaggregate's output is still[sum]: the sort key oftopk(1, sum without (instance) (m))isColumn(0).with_promql_series_identitythen makes that output[$promql_series_identity, sum]and leaves the key atColumn(0), so the ranking read the identity text. It answered{job="b"} 2instead of{job="a"} 7;sort,sort_descandbottomkordered by the identity text too, forsumandcount.with_promql_series_identitycompares the input scope before and after its inputs were realized (positions_after_realization: the same columns in the same order, identity columns inserted). If any position moved,map_input_columnsrenumbers every column reference into that scope: sort keys and partitions, filter predicates, projections, limit partitions, aggregate keys, inputs and filters, relabel values, dedup columns, sample-by keys, window arguments, partitions and order keys, join predicates. AnAggregate'shavingindexes its own output, so it is renumbered by how that output moves (positions_after_realizationover the node's output before and after; PromQL emits nohaving). A reference to a column realization does not keep isSeriesIdentityError::DroppedColumn, not a read of another column.PromqlScalarOpandPromqlMaprealize the identity before binding (which is why filters and arithmetic over the same aggregate were right); the planner, executor and devtools call this one function;canonicalizeonly rewrites a node's columns when no parent reads it by position (read_by_position);OperatorNode::with_new_childrenkeeps positions and is used only with replacements of the same columns. None has this bug.reduce_aggregates::{promql_rankings_over_sum_without_rank_by_value, promql_rankings_over_count_without_rank_by_value}(topk,bottomk,sort,sort_desc, andtopkwith ajobmatcher, which bindsjobfirst), and the controlpromql_topk_by_job_ranks_within_each_job; unit testseries_identity::tests::realization_renumbers_parent_references_into_a_without_aggregate(sort key, filter, projection).Planner: when realization falls back
Stage1ToStage3Passreplaced every realization error by the unrealized root (unwrap_or_else(|_| Rc::clone(node))), so the new "a parent reads a dropped column" error would have been planned silently without labels.limitk/limit_ratio(PromqlSeriesSample) andinfo()(PromqlInfoEnrich). The other refusals (a closed source, awithoutranking) are not reached: the frontend rejectstopk withoutfirst.promql_rootfalls back only forSeriesIdentityError::NotApplicable(an operator realization does not cover, or a source that is not an open PromQL source).DroppedColumnandInvalidRealizedOperator(an operator rebuilt after renumbering whose schema no longer derives; it used to be wrapped asNotApplicableand fall back silently) areOptimizeError::SeriesIdentity. No well-formed input reachesInvalidRealizedOperatortoday: every renumbered reference points at the same-named, same-typed field, and the only inserted columns are Utf8 identity columns that no name or sample-value lookup picks. So it has no IR-level test;promql_rootis tested with it directly. Tests:run::tests::{a_promql_root_realization_does_not_apply_to_is_planned_as_is, a_realization_that_drops_a_read_column_fails_planning, a_realization_that_builds_an_invalid_operator_fails_planning},e2e_plan::plans_a_promql_root_series_identity_does_not_cover_over_its_labels; IR:series_identity::tests::{realization_renumbers_a_having_over_the_aggregate_output, renumbering_a_dropped_column_is_an_error}.Nested aggregates over
without,avg withoutseveral labels, absent labelssum by (job) (sum without (instance) (m))failed at install withsummary update column missing or ambiguous: the outer sum's state reads the sample value (ColumnRef::SampleValue), which the planner resolves as thevaluecolumn or, after an aggregate renamed it, the one numeric column (resolve_column_ref); the executor'snamed_columnlooked only forvalue. It now callsresolve_column_ref, so both readsumof[job, $promql_series_identity, sum]. Test first:reduce_aggregates::promql_aggregate_over_sum_without(alsomax without (zone)over it). 7 of the 21 corpus queries with this error now install; the other 14 stop at the next gap,native output type differs from Planner output(a global aggregate over an aggregate, A global PromQL aggregate over another aggregate answers one row for an empty input #1348) or a Float64-only binary operator over an Int64 count.avg without (instance, zone) (m)failed withcolumn outside scalar input scope: Pass 1's exactavgcasts its sum and count to Float64 with aProjectthat keptby.len()key columns. Underwithoutthe output has the kept labels (here only the identity), notby.len()columns.last_as_float64now keeps every column before the value. Test first:reduce_aggregates::promql_avg_without_two_labels.sum without (instance) (sum without (instance) (m))failed withcolumn instance not found: a grouping key absent from an open schema was an error. A key that awithoutaggregate below removed (labels_removed_by_without, through filters and label-keeping projections) is now absent, as in Prometheus: as awithoutkey it removes nothing, as abykey it is the empty label. Test first:reduce_aggregates::promql_without_an_absent_label_is_a_no_op; unit:resolve::tests::group_keys_promql_still_errors_on_open_schema.max without (instance) ((10 * min without (zone, region) (m)) unless (sum without (instance, zone) (m)))keepinginstance: the left side, realized early by the scalar operator, is the closed[$promql_series_identity, min], so the existing closed-schema rule droppedinstancefrom the grouping although the identity has it. That query now fails to plan instead (resolve::tests::group_keys_promql_keeps_an_absent_key_of_a_closed_identity_schema). The corpus plan diff shows no query that planned before stops planning.An Int64 count under a kept series identity. A cross-series
countanswers Int64, withbyorwithout. Underwithoutits output keeps a series identity, so the lowering adds the label rewrite that drops__name__(series_without_name). That rewrite only moves labels, so its layout now takes the one numeric column (Float64 or Int64) as the sample value. Operators that compute on the value read it as Float64 and go throughfloat_layout, which still refuses an Int64 value: a binary operator between vectors,histogram_quantileandlabel_replace. One rule: label rewrites carry any numeric sample, and computations need a Float64 sample.Performance
Measured with the efficiency review's round-3 harness: release build, pinned to one idle core, median of 5 runs, 8192-row batches, ns per row, peak MB from a counting allocator. Round 3 is the previous head (
3fa423f3c), round 4 this head, both built and run on the same host back to back. 10^6 rows unless noted; k = 5 unless noted.Exact top-k, random values:
Sort{fetch:5}round 3 → 4Sort{fetch:100}round 3 → 4maxExact top-k, ascending values (every row displaces its group's last kept row, as with monotone counters):
Sort{fetch:5}round 3 → 4Sort{fetch:100}round 3 → 4maxSortthenLimit, 10^6 random numbers: 1040 → 1058 at 10 partitions, 743 → 738 at 10^3).maxat fetch 5 and 5–7× at fetch 100 (open question below).sum without (instance)over a 4-label identity, ns per row:sum byidentity (reference)The identity rewrite alone (
series_identity_without) is unchanged: 146 ns per identity with 4 labels, 350 with 12.sum withoutat 10^4 or more series per batch is now 1.5×sum bythe identity, down from 3.4×; at 10^5 series it is faster thansum bythe identity, since it groups 30 kept label sets instead of 10^5 identities.Decisions and open questions
Coordination with #1241
Reduction::StdDev/Variancewith a Kahan-compensated kernel, as for Prometheusstddev_over_time. This PR's kernel is plain Welford, as for Prometheusstddevand DataFusion. This PR lands first; feat(executor): run exact per-series time functions (rate, changes, stddev_over_time, timestamp, irate state, …) #1241 then reuses theseStdDev/Variancevariants andMeasureStates::Momentsand drops its own definitions, so two variants with one name never coexist.stddevandstddev_over_timeneed different rounding (one compensated, one not) is decided at the feat(executor): run exact per-series time functions (rate, changes, stddev_over_time, timestamp, irate state, …) #1241 restack, after checking the Prometheus source. If they do, the variants gain a field saying which; if not,Momentstakes one update for both.Decided
The SQL tests declare the table
"at_rest": withingesting, Stage 3 selected an ingestion-time view over the whole table, which installation refused (An ingestion-time view with no time window below it (SQL table, PromQL offset over raw rows) is planned but refused at install #1288).series_identity_withoutchecks only structure, not canonical form (functional F3): a check that names strictly increase and escapes are the onesencode_series_identitywrites made the rewrite 18 % slower with 4 labels and 34 % with 12 (146 → 172 and 349 → 469 ns per identity in the review's microbenchmark; a first version cost 28 % and 42 %). That is above the 10 % budget, so the doc says the input must be canonical instead. Every identity the executor writes is canonical.The inline number comparison is on the heap only (
compare_held_key), not at the top ofcompare_key: at the top ofcompare_keyit made the unbounded sort of 10^6 random numbers 16–20 % slower (1040 → 1203–1236 ns per row) in four variants tried.count_valuesformats every row's value before grouping (efficiency review E3, not done). Grouping by the float first would format once per group, but the group table normalizes-0and0into one key, while Go writes them as"-0"and"0"; Prometheus keeps them apart. It is a constant factor (3.5–4×) oncount_valuesonly.Wrong ranking over a
withoutaggregate: fixed at the root, in identity realization (see Algorithm), not refused.Open
maxat fetch 5 and 5–7× at fetch 100. Left: a single key kept inline inHeldRowinstead of aBox<[Value]>(the review measured 10–23 % more, mostly at fetch 100 with many held rows), and the heap's own cost (an ascending row sinks from top to bottom, about 3.5 comparisons, plus about 15 % for decoding its key). The efficiency review recommends stopping after F1 and F2, so neither is done here.promql_rowsis a runtime codec insidephysical_planner(layering R3-L2); moving the identity codec tovalues/series_identity.rsis a follow-up, not done here.canonicalizecould renumber positional readers, as realization now does, instead of being skipped (round-5 A3).count(sum by (job) (m >= 1000))gives{} 0; it does on the previous head too). The executor tells SQL from PromQL by the identity column. The fuzz harness's two remaining mismatches are this bug, and it likely causes thenative output type differsinstall errors of 12 nested-aggregate corpus queries.withoutorbykey that only the series identity carries (the left side of(10 * min without (zone) (m)) unless n) is now a planning error rather than a wrong answer; resolving it from the identity is not done here.Review status
Round 1 (abstraction, layering, functional, efficiency):
count_values(...) without (...)did not install; acount_valueslabel colliding with an identity label grouped wronglynative output type differs from Planner output);without_output_schemaadds the label column; thewithoutprojection drops the label from the identitytopk withoutrefused by the frontendcorrlast bit vs DataFusionwithoutrecorded as an input copyAddedLabel, documented with the example; series_identity doc nameswithoutphysical-compile-coverage.mdwithoutclaim corrected; totals statedcompile_internallower_exact_topk,lower_count_valuesnext toresolve_without;compile_nodedoc notes the multi-operator loweringsFloatText,Moments,CoMoments,without_kept_columns,Variancesum_squared_deviations;output_has_one_label_set_per_group; "float text" inFloatTexterrors;numbersremoved (A8)CoMomentsstatetop_k_sortfor the exact and the current-series top-kphysical_common::{execute_series, series_labels, labels, RawDataSample}andtime_series::{run, sample, labels}; the integrationtopk,topk byandwithoutcases dropped (the planner tests run them end to end)check_grouping_columnsWITHOUT_PROJECTION_HELPER, the last helper index;helper_id's doc states the allocationVec<Option<f64>>float64_valuesreader returning a Float64 array; kernels loop over itSort { fetch }, measured abovewithoutidentity rewrite costcount_valuesformats per rowRound 2 (abstraction, layering, functional, efficiency):
count_valueslabel also in thewithoutlist split the groups and stayed in the outputpromql_count_values_label_excluded_by_without_is_droppedfailed with{job=a,zone=2} 3, {job=a,zone=5} 1;a_count_values_label_excluded_by_without_is_not_added): the IR adds no label,lower_count_valuescounts per kept group with no projectioncount_valueslabel colliding with abyor kept column was recorded as an input copya_count_values_label_replaces_a_kept_column_of_its_name,byandwithout, failed withInputColumn(2)): that position isComputedLabel,Utf8non-null. Planner testpromql_count_values_without_replaces_a_kept_label_columncovers the executor sideir::schema::is_label_set_column, read bywithout_output_schemaandresolve_withoutcount_valueslabel push copiedpush_count_values_labelfor both pathscompile_internalbinds keys, filters, k and label and checks one input and no filter once;lower_exact_topkandlower_count_valuesreturn their operators, and the four unreachable branches are gonepromql_rows::series_identity_without, a streaming copy;identity_lessdeleted; byte-equality test firstDynComparatoragainst the heap's last row, decode only kept rows, in-place replace with one sift down.peek_mutneedsOrd, which needs the sort keys in every row (E2), so the heap is aVecwith a closure and the replace is the same in-place overwrite plus sift. Key kept asBox<[Value]>, no new dependency. Ascending input 283 → 168 ns per row (10^6 rows, 10 partitions), short of the ~2× the review's model predictedHeldRowrepeats the sort keys; concat then takeHeldRowis 32 bytes;compactand the output useBatch::interleavefrom (chunk, row) indices. Peak at 10^5 rows × 10^4 partitions 11.9 → 9.1 MB, now below the full sort's 10.2sort_orderfloat64_valuesrepeatsfloatsfloats(column).or_else(..)TwoMomentsAddedLabelComputedLabel(two files)enum Kind {;Operator::sortcallers; DataFusion's TopK wording; #1288withoutunder row 2, SQL under row 29without_kept_columnsnames the identity less the excluded labelsRound 3 (abstraction and layering, functional, efficiency):
Sort{fetch}, the heap tookcompare_key's; a NaN inside a map was checked only on kept rowssort_with_fetch_equals_sort_then_limit_over_random_rowsfailed on seed 9, case 2, map key, fetch 5):key_comparatorpicks the decodedcompare_keyby type for a map; a map key's column is decoded and NaN-checked when its batch is read, by the samekey_columnas the full sort. Comment and Algorithm text correctedcount_values("__name__", v) without (...)kept__name__and split by valuepromql_count_values_name_label_under_without_is_droppedgave 4 rows with__name__;a_count_values_name_label_under_without_is_not_added):removed_by_withouttreats__name__as removed; the lowering reads that fromoutputand needs no change. Now{job=a} 1,{job=a,zone=y} 1,{job=a,zone=z} 2series_identity_withoutaccepts malformed and non-canonical inputtouchedheld one entry per kept rowtouchedreused across batchesValue::comparecompare_held_key), not incompare_key(see Decided). Equality tests cover NULL and NaN orderwithoutlimited by per-identityValueworkKeptKeyValues::writecopieddecode's cell table and NaN checkarrow_columns::decode_cell, read bydecode's number, bool, time, date and interval arms and bywrite; one NaN check inkey_columncount_valuesaggregate built twiceresolve_withoutREADINGsentinelchunks.len(), the index the batch's chunk takesOutputColumnSourcedoc missed the replaced columnpromql_rowsplacementRound 4 (verification of round 3): approved. These are the final fixes:
count without (instance) (m)failed at install withPromQL vector requires labels, time and one valuepromql_count_without_instancefailed with that error;group withoutandstddev withoutpassed before and after): the cause was theseries_without_namerewrite's Float64-only layout (see Algorithm). It also unblockscount without (...) * 2andcount without (...) > 1.sort,topkandbottomkover it now install, but rank wrongly (see Open).count_values ... withoutalready installed, since it lowers to anAggregatethat adds no rewritesort_with_fetch_equals_sort_then_limit_over_random_rowstook 63 s in a debug buildLabelIdentitydictionary becomes NULL, not""Wrong-ranking fix (
81d429edc):withoutaggregate ranked by the identity text (topk(1, sum without (instance) (m))gave{job="b"} 2)Round 5 (abstraction and layering, functional, of
81d429edc):AggIntentcolumn variants listed three times, catch-all armsAggIntent::try_map_colswith every variant and no catch-all;map_columns,input_colsand the frontend'sresolve_agg_intentuse it (75 lines deleted there).GroupKeys::try_map_keysforresolve_group_keysandmap_columns; the PromQL lenient path keeps its filtermap_input_columnsrepeatsmap_children's walkNonASAPOp::map_parts; both are one-line wrappers; the copied pred/key helpers are goneimpl Reductionblockshavingchecked by widthpositions_after_realizationover the aggregate's output; a dropped column is an error. Documented inwith_promql_series_identity's doc; testrealization_renumbers_a_having_over_the_aggregate_outputNonASAPOp::input_scopenext toinputs, used by realization andJoinvalidationwithoutfails to installavg withouttwo or more labels fails to planavgcast keeps every column before the valuewithoutbelow removed is absent (no-op forwithout, empty label forby)NotApplicable(todaylimitk,limit_ratio,info(): 54 of 5,982 corpus roots);DroppedColumnisOptimizeError::SeriesIdentity; tests for both pathsRound 6 (verification of
c462ccad5): approved. Final polish:NotApplicable, sopromql_rootfell back silentlySeriesIdentityError::InvalidRealizedOperator(SchemaDerivationError), a planning error likeDroppedColumn;NotApplicablekeeps only "operator not covered" and "not an open PromQL source"; both docs updated;promql_roottest. Not reachable from a well-formed input (see Algorithm), so no IR-level test. The 253-querywithoutcorpus plan diff is unchanged: all 506 entries identical to round 6withoutlevel downavg/stddev withoutplan-selection errorReady for review.
Verification
cargo nextest run -p asap-types -p asap-planner --no-fail-fastsummary_sharing::identical_ungrouped_queries_share_their_producers, which fails on main too. New:run::tests::a_realization_that_builds_an_invalid_operator_fails_planningcargo clippy -p asap-types -p asap-planner --all-targets --all-features -- -D warnings,cargo fmt --all --checkwithoutcorpus × exact and 5 % targets, DAG hash per entry, vsc462ccad5c462ccad5), by test namecargo nextest run --no-fail-fast -p asap-types -p asap-frontend-promql -p asap-frontend-common -p asap-planner -p asap-executor -p asap-integration-testsjoint_search_equivalence×2,materialized_views×1,plan_sharing×2,promql_to_post_asap×2,summary_sharing::identical_ungrouped_queries_share_their_producers). New:reduce_aggregates::{promql_aggregate_over_sum_without, promql_avg_without_two_labels, promql_without_an_absent_label_is_a_no_op}(all three failed on81d429edc),e2e_plan::plans_a_promql_root_series_identity_does_not_cover_over_its_labels, tworun::tests, twoseries_identity::tests, oneresolve::testscargo clippy --workspace --all-targets --all-features --locked -- -D warnings,cargo fmt --all --checkwithout/bypairs{} 0; itsbyrewrite gives the same wrong row; on the previous head too). Of the 63 cases that failed to plan only in thewithoutform on81d429edc, 58 now plan: all 35column outside scalar input scope(F2), all 13summary update column(F1), 10 of 15column not found(F3); the 5 left name a label only a binary operator's side or alabel_replacecarries. Fixed cases (205): 0 mismatches;without/bypairs (190): 108 same, 1 same values with labels differing, as before. The harness'svalues()panics on NaN (partial_cmp().unwrap()), which now stops seeds 2–5 early; those seeds ran withtotal_cmpthere, nothing else changedwithoutcorpus × exact and 5 % targets, DAG hash per entry, vs81d429edcsummary update columnqueries); none stops planning81d429edc) vs previous headc045e05e5, by test namecargo test -p asap-types -p asap-frontend-common -p asap-frontend-promql -p asap-planner -p asap-executor -p asap-integration-tests --no-fail-fastplanner39 passed, 1 failed (summary_sharing::identical_ungrouped_queries_share_their_producers, fails on the previous head too); the two new ranking tests failed on the previous head and pass. Integration 165 passed, the same 7 failing on bothcargo clippy -p asap-types -p asap-frontend-common -p asap-frontend-promql -p asap-executor -p asap-planner -p asap-integration-tests --all-features --all-targets -- -D warnings,cargo fmt --all -- --checkwithoutcorpus, exact and 5 % targets, DAG hash per workload, previous head vs fixtopk(3, sum without (device, fstype, mountpoint) (node_filesystem_size_bytes))(both targets), the bug's shape: same nodes, length and cost, its sort key now reads the sumcargo nextest run -p asap-executor -p asap-types --all-featuresa_label_rewrite_carries_an_int64_count_and_a_computation_refuses_it;sort_with_fetch_equals_sort_then_limit_over_random_rows2.4 s in debug)cargo test -p asap-executor --all-featuresseries_identity_without_equals_decode_filter_encode,series_identity_without_refuses_a_malformed_identity,series_identity_without_equals_decode_on_mutated_identities,co_moments_equal_two_moments_bit_for_bit);executor171 passed (incl.sort_with_fetch_equals_sort_then_limit,sort_with_fetch_over_ascending_input_equals_sort_then_limit,sort_with_fetch_equals_sort_then_limit_over_random_rows,identities_equal_but_for_an_excluded_label_aggregate_in_one_group,sort_with_fetch_holds_groups_times_fetch_rows); memory-limit binaries pass1beca789dcargo nextest run -p asap-integration-tests --all-features --no-fail-fastjoint_search_equivalence×2,materialized_views×1,plan_sharing×2,promql_to_post_asap×2); none newcargo clippy -p asap-executor -p asap-types -p asap-planner -p asap-integration-tests --all-features --all-targets -- -D warnings,cargo fmt --all -- --checkcargo nextest run -p asap-planner --all-features --test planner reduce_aggregatespromql_count_without_instance,promql_group_without_instance,promql_stddev_without_instance)cargo test -p asap-integration-tests --test integration -- exact_reduce_execution per_series_exact_executioncargo test -p asap-types --all-featurescargo clippy -p asap-executor -p asap-types -p asap-planner -p asap-integration-tests --all-features --all-targets -- -D warnings,cargo fmt --all -- --checkFrom the round-2 functional review (on
dec40d748):Sort{fetch}matched sort then limit on 400 random cases (1–30 batches including empty ones; no, one or two grouping columns; one or two keys in random directions, NULLs first or last; NaN, ±Inf, −0 and duplicates; fetch 0, 1, 2, 3, 5, 1000): identical rows in identical order. All 13 raw-series top-k cases were byte-identical to base.count_valueswithoutDAG byte-identical to base over 215 planned workloads (231 corpuswithoutqueries plus joint workloads, exact and 5 % targets); only the 9count_values ... without (instance)workloads changed (their IR schema gains the label column).Follow-ups
StdDev/Variance(see Coordination).withoutremoved still fails, loudly:withoutlevel down, sosum by (instance) (sum without (job) (sum without (instance) (m)))fails at lowering withcolumn \instance` not found`;avg without (instance) (avg without (instance) (m))and the same withstddevfail plan selection with "no valid candidate": the outerwithout([])is costed as an ungrouped aggregate;sum by (instance) (sum without (instance) (m))plans but does not install (native output type differs from Planner output).having, Math and native-histogram intents underReduce, andwithoutin a maintained population.