Skip to content

refactor: no silent defaults for kernel memory, operator rules and aggregate names - #1330

Merged
zzylol merged 4 commits into
mainfrom
refactor/no-silent-defaults
Nov 16, 2026
Merged

zzylol merged 4 commits into
mainfrom
refactor/no-silent-defaults

Conversation

@zzylol

@zzylol zzylol commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: several places gave an unknown a silent default. A new executor kernel was charged a made-up 4,096 bytes. A new operator read the rows at t in InputTimeAccess::of, and got a default answer from every _ arm in asap-cost's per-node usage, in edge_roles and in the raw-data window check. An Aggregate read from JSON without output_names got an empty list that matches none of its measures.
  • Change: each of these now has no default. Every kernel must implement approx_memory_bytes. The window questions are answered once on Operator (this PR's covered_window_ms, with main's chain_tumbling_window_width_ms and keeps_input_window), each listing every operator; the sites that asked them call these, including check_raw_data_windows and InputTimeAccess::of's window lengths. The other per-operator rules list each variant, and clippy::wildcard_enum_match_arm fails the -D warnings build on a new _. CostModel::per_node_unit_costs answers "which cost model prices each node" once. output_names is required when deserialized. A new per-family test checks the planner's state bytes against what the executor charges.
  • Effect: no plan changes, since every rule keeps its answer (checked on six workloads, below: selected plan and plan cost identical to main). A new operator or kernel now fails to build until someone decides what it needs. Example: on 10,000 seeded latencies, DDSketch at α = 0.01 is priced 18,752 B against 6,288 B charged (2.98×). The new test records that as a known gap on DDSketch's row under [Bug] DDSketch memory is charged at the 1e9 value-range bound, about 6.4× what the executor holds #1313, and fails once the gap closes or moves.

Principle: an unknown stays None (or unanswered) until code that knows the safe direction handles it. A rule that answers per operator, summary type or cost model lists every variant, so a new variant fails to build instead of silently getting a default. if let and matches! remain only for a check of one variant's identity ("is it a Scan", "is it an ASAP operator", "is it stored across evaluations"), where every other variant's answer is "no" by definition; the module comments name those sites.

Review guide

  • Base: main (3e434c760, after refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237). Not stacked. Head: 4185aff5b. Rebased onto main after refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237: see the section below.
  • Read first:
    • crates/types/src/ir/operator/node.rs: the three window methods and their table tests.
    • crates/cost/src/node/state_bytes/tests.rs: priced_bytes_agree_with_the_executor, KnownBytesGap, BYTES_AGREEMENT_FACTOR.
    • crates/types/src/plan/time_window.rs: InputTimeAccess::of.
  • Mechanical:
    • The wildcard-to-variant-list replacements in crates/cost/src/node/usage/mod.rs and edge_roles. Each arm keeps its old body.
    • The window sites, now calls to the Operator methods; check_raw_data_windows is now if covered_window_ms().is_some().
    • CostModel::per_node_unit_costs at its four call sites.
    • The two dag-viewer test fixtures: output_names: [] becomes [""], one name for their one top_k measure.
    • The label/labels/Rows helpers and Rows::update, shared by both byte tests.
  • Careful review:
    • The PromqlSubquery arm in InputTimeAccess::of, and covered_window_ms giving None for a subquery (see Decisions).
    • The gap ratios and the tolerance.

Rebased onto main after #1237

Base main 3e434c760 (was 59365ce80); head 4185aff5b (was 273cadbeb, kept as backup/refactor/no-silent-defaults-273cadbeb). #1237 added chain_tumbling_window_width_ms, keeps_input_window and request_anchor to Operator, and deleted maintained_window_ms, so this PR now adds only covered_window_ms of the three window methods.

File Conflict Resolution
crates/types/src/ir/operator/node.rs this PR's covered_window_ms and main's request_anchor at the same place; the variant lists of keeps_input_window (named keeps_input_time in this PR's second commit) both methods kept, with main's variant lists; keeps_input_window keeps main's name and this PR's round-2 doc comment; the table test calls keeps_input_window
crates/cost/src/node/usage/mod.rs, 4 hunks maintained_window_ms, the output interval, the tumbling-windows match, chain_width_ms main's side for all four: maintained_window_ms and this PR's edits to it are dropped (main replaced it by RefreshWindow::computed_from); main's output_interval_ms; main's payload if payload.keeps_input_window() arm; main's chain_tumbling_window_width_ms(producers). This PR's other changes to the file (the lint, state_family, unit costs, the if let sources rule, the listed matches) are kept

Two more changes, in their own commits:

  • 7b7d2e1ad (round-2 commit, amended): covered_window_ms's doc named the deleted maintained_window_ms; it now names windows_computed_from and so RefreshWindow::computed_from.
  • 4185aff5b: refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237 added two _ arms in modules this PR lints with clippy::wildcard_enum_match_arm, which fail -D warnings: shift_back_ms_below in time_window.rs is now if let TimeShift … else (the "is it a TimeShift" identity check the module comment already allows), and the last arm of the tumbling-windows match in node::usage is Payload::NonASAP(_) | Payload::ASAP(_) => 1 (keeps_input_window answers per operator).

Semantic check of #1237's names: this PR calls none of least_retention_ms_per_execution, ExecutionCosts::retention_ms_for_need, RefreshWindow::computed_from or the ErrorContract helpers; RefreshWindow::computed_from goes through windows_computed_from, which this PR changed only to read covered_window_ms (same answers).

Re-run after the rebase (release build, compared by test name with main 3e434c760):

Scope Command Result Newly failing vs main
check cargo check --workspace --all-targets clean –
fmt cargo fmt --all -- --check clean –
clippy cargo clippy -p asap-cost -p asap-types -p asap-executor -p asap-frontend-promql -p asap-stage2-physical-optimization -p asap-integration-tests -p asap-planner --all-targets -- -D warnings clean –
tests cargo test --no-fail-fast -p asap-cost -p asap-types -p asap-executor -p asap-frontend-promql -p asap-stage2-physical-optimization -p asap-stage3-plan-selection -p asap-planner -p asap-integration-tests -p asap-devtools --lib --tests 1071 passed, 65 failed, 9 ignored (main: 1065, 65, 9) none; the same 65 fail on main; the 6 new tests pass
doc tests cargo test -p asap-types -p asap-cost --doc 8 passed none
dag-viewer JS node --test tools/dag-viewer/*.test.mjs (Node 22) 34 passed, 10 skipped (main the same) none

Behavior: before → after

What the user or developer writes Before After
A new impl AggregateCore without approx_memory_bytes builds; each state is reserved at 4,096 B fails to build: E0046, missing trait item
A new NonASAPOp/ASAPOp variant builds; read as CurrentTimeOnly, given one Input edge role, and the _ answer in every usage rule fails to build at InputTimeAccess::of, the three window methods, edge_roles and each usage rule (non-exhaustive match); check_raw_data_windows follows covered_window_ms
A new CostModel variant builds; plan_cost gives it no unit costs fails to build at CostModel::per_node_unit_costs
A new _ => arm on an enum in ir::operator::node, plan::time_window or asap-cost node::usage builds cargo clippy -- -D warnings (CI's Run clippy step in .github/workflows/rust.yml) fails on wildcard_enum_match_arm or match_wildcard_for_single_variants
{"Aggregate": {"reduction": "PerEntity", "measures": ["Rate"], "child": 0, …}} with no output_names reads as output_names: [] missing field `output_names`
Any workload its plan the same plan

This serves design correctness: plan selection's resource charges should not rest on guesses that nobody chose.

Design scope

New production items, each a method on an existing type; no new types, files or fields:

Item Scope Why it is general
Operator::chain_tumbling_window_width_ms IR (ir::operator::node) On main since #1237 (exhaustive there); this PR adds only its table test.
Operator::covered_window_ms IR The window of input rows one output covers, for any operator; asked by windows_computed_from (and so main's RefreshWindow::computed_from), InputTimeAccess::of (the TimeRange and chain window lengths), check_raw_data_windows and the raw-data reader check in materialized_view::views.
Operator::keeps_input_window IR On main since #1237; this PR rewrites its doc comment (the window keeps its length under a TimeShift) and adds its table test.
CostModel::per_node_unit_costs cost model (asap-cost model) The unit costs each node's usage is priced with: only cost model 1 prices per node; cost model 2 prices the plan's peaks and cost model 3 has no prices. Data of CostModel itself, asked at four call sites.

Test-only: the Rows enum (now with Rows::update and Rows::fed), label, labels, next_uniform, BYTES_AGREEMENT_FACTOR and KnownBytesGap in cost::node::state_bytes::tests, shared by both byte tests.

Interfaces

crates/types/src/ir/operator/node.rs

+// The questions answered here per operator (e.g. the window it covers)
+// list every operator: a wildcard arm would give a new operator a default
+// answer instead of failing the build (under `-D warnings`). Clippy does
+// not check `if let` and `matches!`, so they stay only for a check of one
+// variant's identity, where every other operator's answer is "no" by
+// definition: "is it a `Scan`" (`scanned_sources`: only a scan reads a
+// source itself) and "is it an ASAP operator" (`is_asap`).
+#![warn(
+    clippy::wildcard_enum_match_arm,
+    clippy::match_wildcard_for_single_variants
+)]

 impl<C> Operator<C> {
     pub fn children(&self) -> Vec<&C>;
     pub fn map_children<D>(&self, f: impl FnMut(&C) -> D) -> Operator<D>;
     pub fn kind_name(&self) -> &'static str;
     pub fn chain_tumbling_window_width_ms(&self) -> Option<u64>;
+    /// The window of input rows one output covers, in ms: a `TimeRange`'s
+    /// range or a chain's tumbling window; `None` for every other operator.
+    /// E.g. `TimeRange(5m)`: `Some(300_000)`; a chain of 1-minute windows:
+    /// `Some(60_000)`; `Filter`: `None`. A `PromqlSubquery` evaluates its
+    /// input over a range, but each of its outputs is one evaluation, so it
+    /// covers none here.
+    ///
+    /// A `TimeRange` longer than `u64::MAX` ms (about 584 million years)
+    /// saturates to `Some(u64::MAX)`, so `Some` always means "covers a
+    /// window"; a caller that once gave `None` for such a range
+    /// (`windows_computed_from`, and so `RefreshWindow::computed_from`) now
+    /// carries on with `u64::MAX`. PromQL planning never reaches it: costing such a range
+    /// overflows first.
+    pub fn covered_window_ms(&self) -> Option<u64>;
     pub fn request_anchor(&self) -> Option<AtModifier>;
-    /// Whether each output row keeps its input row's time window, so the
-    /// window its input covers is also its own: `Filter` and `Project`
-    /// keep rows and times; `TimeShift` moves every row by one offset but
-    /// keeps the window's length. E.g. `Filter <- TimeRange(5m)` covers
-    /// 5 minutes, and `Project <- chain of 1-minute windows` holds
-    /// 1-minute windows; `Aggregate <- TimeRange(5m)` combines rows, so
-    /// `false`.
+    /// Whether the window its input covers is also its own, with the same
+    /// length: `Filter` and `Project` keep each row and its time;
+    /// `TimeShift` moves every row's time by one offset, so the window
+    /// moves but keeps its length. E.g. `Filter <- TimeRange(5m)` and
+    /// `TimeShift(1h) <- TimeRange(5m)` both cover 5 minutes;
+    /// `Aggregate <- TimeRange(5m)` combines rows, so `false`.
     pub fn keeps_input_window(&self) -> bool;
 }

crates/cost/src/model.rs

+impl CostModel {
+    /// The per-unit costs each node's usage is priced with. Only cost
+    /// model 1 prices per node: `UsageOverTime(u)` gives `Some(&u)`. Cost
+    /// model 2 prices the plan's peaks (in `plan_cost`), not each node, and
+    /// cost model 3 has no prices, so both give `None`.
+    pub fn per_node_unit_costs(&self) -> Option<&ResourceUnitCost>;
+}

crates/executor/src/summary_kernels/traits.rs

 pub trait AggregateCore: Send + Sync {
     fn clone_boxed_core(&self) -> Box<dyn AggregateCore>;
     fn as_any(&self) -> &dyn std::any::Any;
     fn as_any_mut(&mut self) -> &mut dyn std::any::Any;
     fn merge_with(&self, other: &dyn AggregateCore) -> Result<Box<dyn AggregateCore>, KernelError>;
     fn estimate(&self, query: &SketchStatistic) -> Result<f64, KernelError> { /* ... */ }
     fn is_empty(&self) -> bool { /* ... */ }
     /// Approximate in-memory footprint, used for execution memory reservations.
     /// A state shared with other rows (see [`Self::shared_state`]) is not
-    /// included.
-    fn approx_memory_bytes(&self) -> usize {
-        4096
-    }
+    /// included. No default: each kernel says what it holds, e.g. a KLL
+    /// state its sketch's `memory_bytes()`.
+    fn approx_memory_bytes(&self) -> usize;
     fn shared_state(&self) -> Option<(usize, usize)> { /* ... */ }
 }

All 11 kernels already override it, so none needed a change.

crates/types/src/ir/operator/non_asap.rs (file format: the Aggregate field becomes required)

 pub enum NonASAPOp<C = Rc<OperatorNode>> {
     Aggregate {
         reduction: Reduction,
         measures: Vec<AggIntent>,
         /// Output column names, exactly one entry per measure; an empty
         /// string means the default name (the intent's synthetic name).
-        #[serde(default)]
+        /// Required when deserialized: a missing list would not match the
+        /// measures.
         output_names: Vec<String>,
         #[serde(default)]
         filters: Vec<Option<Predicate<C>>>,
         #[serde(default)]
         having: Option<Predicate<C>>,
         child: C,
     },
 }

output_names is the only operator field holding output names. The NonASAPOp variants not shown are unchanged.

crates/types/src/plan/time_window.rs and crates/cost/src/node/usage/mod.rs: no signature changes. Both modules gain the two lints, with a comment naming their matches!, if let and let ... else identity checks, including the lifetime check matches!(.., Lifetime::AcrossEvaluations(_)). InputTimeAccess::of reads the TimeRange and chain window lengths from covered_window_ms, so the saturating ms conversion is written once. summary_type_name's Debug fallback keeps other => under an item-level #[allow(clippy::wildcard_enum_match_arm)]: any other type prints its Debug form, which names a new variant too. edge_roles gets the lint at item level. check_raw_data_windows asks covered_window_ms().is_some() and needs no list.

Usage, end to end: the PromQL query sum_over_time(m[5m] offset 1m), planned at ingestion time.

  1. The frontend lowers it to Aggregate(SumOverTime) <- TimeRange(5m) <- TimeShift(1m) <- Scan(m), with output_names: [""].
  2. Stage 3 and plan_cost call InputTimeAccess::of per node, through TimeIndexedPlan::input_time_access:
    • TimeShift(1m) at query time: WhatItsReadersRead, shifted 60,000 ms (its own arm);
    • TimeRange(5m): LastWindow { length_ms: 300_000 }, the length read from covered_window_ms;
    • Aggregate: CurrentTimeOnly, one of the listed operators.
  3. windows_computed_from(Aggregate) walks down: Aggregate.covered_window_ms() is None, TimeRange(5m) gives 300,000, over the 60,000 ms shift below it: {60_000, 300_000}, the same as before.
  4. RefreshWindow::computed_from(Aggregate) (main's refresh rule, the union of windows_computed_from): one window of 300,000 ms, so the view is refreshed every 5 minutes, as before.
  5. node_resource_usage_and_cost prices each node with settings.cost_model.per_node_unit_costs() (cost model 1 here), takes sources as the union over its producers (it is not a Scan), and its units and lookup arms name Aggregate explicitly, with the same values as before.

Algorithm changes

None. Every rewritten rule keeps its answer. One edge, unreachable in practice: a TimeRange longer than u64::MAX ms (about 584 million years) now saturates in covered_window_ms and is still a window; before, windows_computed_from (and so RefreshWindow::computed_from) gave None for it. The covered_window_ms doc comment states this. PromQL planning cannot reach it: sum_over_time(data[600000000y]), quantile_over_time(0.9, data[600000000y]) and sum_over_time(data[290000000y]) print byte-identical NoValidCandidate … analytical arithmetic overflowed on main and here, because costing overflows first.

What is listed now

Lines are on main before #1237 (59365ce80). After the rebase onto 3e434c760, four rows are main's code from #1237, marked (main).

Site (main) Matches on Now
crates/types/src/plan/time_window.rs:156 (InputTimeAccess::of) operator all 20 NonASAPOp + 11 ASAPOp; PromqlSubquery in its own arm, still CurrentTimeOnly
time_window.rs:341 (windows_computed_from) operator covered_window_ms(); None walks on down
crates/cost/src/node/usage/mod.rs:96 (state_family) FieldDataType Plain, SummarySample, Wavelet, StatModel → None
usage/mod.rs:103 (state_family) operator all operators but the builds, merges, estimates and finalizations → None
usage/mod.rs:223 and :237 (maintained_window_ms, the node and the walk) operator (main) deleted by #1237; the refresh window is RefreshWindow::computed_from, over windows_computed_from (above)
usage/mod.rs:272 (unit costs) CostModel CostModel::per_node_unit_costs()
usage/mod.rs:294 (output interval) operator (main) output_interval_ms: chain_tumbling_window_width_ms(), else window_ms or one second
usage/mod.rs:317 (tumbling windows) operator (main) the guarded TimeRange-over-chain arm, then payload if payload.keeps_input_window(); the last arm, _ on main, is now Payload::NonASAP(_) | Payload::ASAP(_) → 1
usage/mod.rs:321 (sources) operator if let Scan, else the union of its producers' sources (an identity check, the same form as scanned_sources)
usage/mod.rs:350 (measured unit cost) operator all but Aggregate → lookup
usage/mod.rs:524 (Aggregate grouping keys) Reduction PerEntity → 1
usage/mod.rs:674 (non-ASAP pass-through) NonASAPOp the 14 without their own arm → pass-through estimate
usage/mod.rs:805 (estimate rows per group) SketchStatistic FrequencyL2, FrequencyEntropy, Quantile, PointCount, Cardinality → 1
usage/mod.rs:824 (other ASAP operators)* operator MaintainPopulation, EvaluatePopulation, SummarySubtract, SummaryDelete, SummaryJoin, Extension → one op per row
usage/mod.rs:840 (units) operator all but Scan, SummaryMerge → input rows
usage/mod.rs:1047 (chain_width_ms) operator (main) chain_tumbling_window_width_ms(producers), read from ProducerOutput::chain_tumbling_window_width_ms
time_window.rs (shift_back_ms_below, added by #1237) operator if let TimeShift, else the path ends (an identity check, named in the module comment; _ on main)
crates/cost/src/plan/cost.rs:99 (plan_cost unit costs), :368 (request overhead), :388 (raw-data disk cost) CostModel CostModel::per_node_unit_costs()
crates/types/src/plan/physical_asap_dag.rs:567, :572 (edge_roles) NonASAPOp, ASAPOp the 13 and 8 single-input operators → one Input
crates/types/src/plan/materialized_view/raw_data.rs:115 (check_raw_data_windows) operator covered_window_ms().is_some() → check the raw window below; any other operator has no window to check
crates/types/src/plan/materialized_view/views.rs:232 (reads raw data) operator Scan → true; covered_window_ms().is_some() → false; any other operator passes on its input's answer
crates/types/src/ir/operator/node.rs (scanned_sources) operator if let Scan, else the union over its inputs (an identity check; was operator =>)

* The lint does not flag :824, because every top-level Operator variant is named elsewhere in that match. It is listed anyway for the principle.

The lint only checks top-level variants: it would accept Operator::NonASAP(_) | Operator::ASAP(_). So every operator site lists the leaf NonASAPOp/ASAPOp variants, which a new operator breaks. usage/mod.rs:512 matches a (bool, Lifetime) tuple and stays as it is.

Identity checks kept (matches!, if let, let ... else; named in each module comment):

  • time_window.rs current_time_read_needs_every_row: is it a Scan;
  • time_window.rs shift_back_ms_below: is it a TimeShift;
  • usage/mod.rs the retained raw bytes and sources: is it a Scan;
  • usage/mod.rs raw_data_samples_non_negative: is its producer a Scan or a TimeRange (any other producer's samples are not raw, the safe answer);
  • node.rs scanned_sources: is it a Scan; is_asap: is it an ASAP operator;
  • time_window.rs InputTimeAccess::of and usage/mod.rs: the lifetime check matches!(.., Lifetime::AcrossEvaluations(_)).

Not touched:

  • types::stages::logical::group (column_properties.rs): the sign-design PR handles it.
  • materialized_view/views.rs:436 (view_inputs): a walk down that stops at a raw TimeRange and errors at a Scan. Its _ keeps walking, which is right for any new operator: a walk that reaches a raw Scan still errors.
  • materialized_view/raw_data.rs:177 (below a raw range): its _ returns Unsupported, so a new operator fails loudly rather than getting a silent answer.

Known bytes gaps

priced_bytes_agree_with_the_executor covers every family the executor builds from a plan. Each family uses its parameters for a 1% error and is fed seeded rows, priced for the same count (Rows::fed):

  • 10,000 latencies log-uniform over 1 ms to 1 s, declared non-negative (KLL's charge reaches its priced size only past about 3,000 rows); or
  • 1,000 rows over 1,000 distinct label values or label sets (fixed-size grids: the ratio is the same at any count, and a heap of 10 still churns). At 10,000 rows Count Sketch with a heap alone took about 22 s in a debug build.

Every ratio below is the same as at 10,000 rows for every family.

The rows are seeded, but the executor builds KLL unseeded (Kll::with_k; its factory takes no seed), so KLL's charge varies by about 0.2% between runs (10,376 or 10,392 B; ratio 1.000 or 1.002). A comment on KLL's row says so; it is far inside the tolerance.

The test requires priced ÷ charged to lie in [1/1.5, 1.5]. A family outside it carries a KnownBytesGap on its own row (keyed by its FieldDataType), with its issue and measured ratio. The test fails in three cases:

  • a family with a gap comes within the tolerance;
  • its ratio moves by more than 5%;
  • a family without one falls outside the tolerance.
Family Priced B Charged B Ratio Issue
DDSketch (α = 0.01) 18,752 6,288 2.982 #1313 (priced for 9 decades; data spans 3)
exact Sum / Count / Min / Max / Increase / Rate 56 136 0.412 #1333 (priced as one state-column record; the accumulator object built here charges its whole struct)
HydraCms (272×5 per group, 3×64 grid, 10 groups) 11,040 per group 105,726 per group 0.104 #947 (priced as one Count-Min per group; the executor keeps one grid)

These agree within 1.5×:

  • KLL (k = 269), HLL (p = 18), KMV (k = 106,263);
  • Count-Min (272×5), Count Sketch (30,000×47) and both heap variants (heap 10);
  • UnivMon (256, 5, 1,024, 16).

HydraKll is left out: no plan builds it, since it has no parameter list and the factory refuses it. The pricing itself is not changed here.

Decisions and open questions

Needs reviewer decision

Decided

Verification

Scope Command Result Newly failing vs base
clippy cargo clippy -p asap-types -p asap-cost --all-targets -- -D warnings clean none
asap-types lib cargo test -p asap-types --lib 161 passed none
asap-types integration cargo test -p asap-types --tests 78 passed none
asap-types doc tests cargo test -p asap-types --doc 8 passed none
asap-cost lib cargo test -p asap-cost --lib 39 passed none
dag-viewer JS node --test tools/dag-viewer/*.test.mjs (Node 22) 34 passed, 10 skipped (need generated examples; the same before the fixture change) none
fmt cargo fmt --all -- --check clean –
plans asap_planner --run-until stage3 with a local, unpushed dump of the selected PhysicalASAPDAG and its plan_cost (the text and JSON printers end in todo!), this head against main 59365ce, on planner-layering/example1, promql/ratio-of-quantiles-5m, promql/rangeheavy-t5-q05-1h-60s-300s, promql/rangeheavy-t8-t5-1h, random-mixed-seed-862/workload-01 and -02 (and planner-layering/example2, which fails the same way on both: its SQL table is not declared) byte-identical output on all; the round-2 functional review also found plans and costs identical to main on 249 workloads –
bytes test time (debug) cargo test -p asap-cost --lib priced_bytes_agree_with_the_executor, alone, one test thread 2.84 s median (2.79 / 2.84 / 2.90 s) at load average about 23; was 29.3 s at load about 100 with 10,000 rows for every family –

New tests. The table says what each catches.

Test Catches On main
non_asap::tests::aggregate_output_names_are_required_when_deserialized An Aggregate without output_names being accepted again (the serde default coming back); and a broken round trip with names fails: deserialization succeeds
node::tests::only_a_chain_has_a_tumbling_window_width, a_time_range_and_a_chain_cover_a_window, filters_projections_and_shifts_keep_input_window One table per window method: the operators that answer, and representative ones that do not (e.g. a subquery covers no window) covered_window_ms does not exist; the other two methods are main's (#1237), untested there
time_window::tests::operators_with_a_time_access_decision The input time access of the operators with a decision: TimeShift by timing and lifetime, @ end, TimeRange, the chain at either timing, PromqlSubquery, and one Aggregate. A new operator fails to build in InputTimeAccess::of; its decision gets a row here passes: pins today's behaviour, which this PR keeps
state_bytes::tests::priced_bytes_agree_with_the_executor Plan selection and executor bytes drifting apart for any family; a known gap closing or moving without its row being updated; an executor-supported family with no row fails until the three families carry their gaps
clippy wildcard_enum_match_arm and match_wildcard_for_single_variants A new _ arm on an enum in the linted modules and functions. It runs in CI's Run clippy step (cargo clippy --workspace --all-targets --all-features --locked -- -D warnings) flagged 16 sites in usage and time_window on main, and scanned_sources in node.rs

Follow-ups and conflicts

Review status

  • Round 1: abstraction and layering reviews on a782ce3. Applied: A1 (window methods on Operator), A3 (lint gaps), A4/A5/A6/A7 (tests and doctests), L1 (CostModel unit costs), L2 (edge_roles, check_raw_data_windows; views.rs:232 asks covered_window_ms, so its decision is delegated to that exhaustive method; the other two listed under Not touched), L3–L7, N1–N3. A2 is the refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237 note above; A8 is settled by B2.
  • Efficiency review (round 1): F1 applied (per-row-kind counts in the bytes test; same ratios).
  • Round 2 (complete), on 1019fbc: abstraction, layering, functional and efficiency reviews.
    • Abstraction: approve. Applied B1 (check_raw_data_windows asks covered_window_ms, its list and lint attribute deleted), B2 (sources as if let Scan … else union), B3 (per_node_unit_costs), B4 (one saturating conversion).
    • Layering: approve with small changes. Applied R2-L1 (rename and doc), R2-L2 (refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237 note completed), R2-L3 (Design scope callers), R2-L4 (views.rs:232 wording), R2-L5 (= B4), R2-N1 (keeps_input_window, TimeShift doc), R2-N2 (lint comments name is_asap and the lifetime checks).
    • Functional: no blockers. Plans and costs match main on all 249 workloads (4 plan nondeterministically on main too); no test lost; no JSON relies on the output_names default; the saturation edge is unreachable through PromQL. Applied F1 (KLL row note; the executor exposes no seeded KLL) and F2 (dag-viewer fixtures).
    • Efficiency: approve. Planning time within noise of main on rangeheavy, rangeheavy-8users and rollupheavy, plan costs bit-identical; bytes test 2.84 s at load about 23. Applied its nit: the saturation change is stated in the covered_window_ms doc comment.

@zzylol zzylol added cost-model Cost formulas, statistics, ranking, and selection ir Pre-ASAP or post-ASAP intermediate representation refactor resource testing Corpus, regression, differential, or benchmark tests labels Nov 15, 2026
…gregate names

An unknown stays unanswered until code that knows the safe direction
handles it, and a match that returns a constant lists every operator,
so a new operator fails to build instead of getting a default.

- AggregateCore::approx_memory_bytes has no 4096-byte default; every
  kernel says what it holds (doc tests show a kernel without it fails
  to build).
- A per-family test checks plan selection's state bytes against the
  executor's charge on seeded rows, within 1.5x, with a known-gap list
  (DDSketch, exact accumulators, HydraCms) naming each issue.
- InputTimeAccess::of and windows_computed_from list every operator;
  behaviour unchanged, pinned by a per-operator table test.
- Aggregate output_names is required when deserialized.
- clippy::wildcard_enum_match_arm warns in types::plan::time_window and
  asap-cost node::usage; every wildcard there lists its variants.
- Operator::chain_tumbling_window_width_ms, covered_window_ms and
  keeps_input_time, each listing every operator, replace the five
  duplicated variant lists in time_window and node::usage; node.rs is
  linted, with a table test per method.
- CostModel::unit_costs, used by node usage, plan_cost, the request
  overhead and the raw-data disk cost.
- edge_roles and check_raw_data_windows list every operator; the
  raw-data reader check uses covered_window_ms.
- Lint comments name the identity checks kept as matches!/if let; add
  match_wildcard_for_single_variants; summary_type_name keeps its Debug
  fallback with an allow and a reason.
- Bytes test: known gaps on each family's row (KnownBytesGap, keyed by
  FieldDataType), the exact-accumulator gap under its own issue,
  Rows::update shared by both tests, BYTES_AGREEMENT_FACTOR everywhere,
  next_uniform, and per-row-kind counts (10,000 values, 1,000 label
  rows): about 1.8 s instead of 15-30 s in debug, same ratios.
- Time-access test trimmed to the operators with a decision; the two
  AggregateCore doctests dropped.
- check_raw_data_windows asks covered_window_ms instead of listing every
  operator; the cost usage `sources` rule takes the same `if let Scan`
  form as Operator::scanned_sources.
- CostModel::unit_costs -> per_node_unit_costs: only cost model 1
  prices per node.
- InputTimeAccess::of reads TimeRange and chain window lengths from
  covered_window_ms, so the saturating conversion is written once; the
  covered_window_ms doc states the saturation change.
- keeps_input_time -> keeps_input_window, with a TimeShift example.
- Lint comments name every identity check kept (is_asap, lifetime).
- Bytes test: note that the executor's unseeded KLL varies by ~0.2%.
- dag-viewer fixtures: one output name per measure.
…er the module lint

Main added a wildcard arm in shift_back_ms_below (time_window.rs) and in the tumbling-windows match (node::usage), both modules this PR lints with clippy::wildcard_enum_match_arm. The first is the 'is it a TimeShift' identity check the module comment already allows, now an if let; the second names both Operator halves, since keeps_input_window answers per operator.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cost-model Cost formulas, statistics, ranking, and selection ir Pre-ASAP or post-ASAP intermediate representation refactor resource testing Corpus, regression, differential, or benchmark tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant