Repository navigation
refactor: no silent defaults for kernel memory, operator rules and aggregate names - #1330
Merged
Merged
Conversation
…gregate names An unknown stays unanswered until code that knows the safe direction handles it, and a match that returns a constant lists every operator, so a new operator fails to build instead of getting a default. - AggregateCore::approx_memory_bytes has no 4096-byte default; every kernel says what it holds (doc tests show a kernel without it fails to build). - A per-family test checks plan selection's state bytes against the executor's charge on seeded rows, within 1.5x, with a known-gap list (DDSketch, exact accumulators, HydraCms) naming each issue. - InputTimeAccess::of and windows_computed_from list every operator; behaviour unchanged, pinned by a per-operator table test. - Aggregate output_names is required when deserialized. - clippy::wildcard_enum_match_arm warns in types::plan::time_window and asap-cost node::usage; every wildcard there lists its variants.
- Operator::chain_tumbling_window_width_ms, covered_window_ms and keeps_input_time, each listing every operator, replace the five duplicated variant lists in time_window and node::usage; node.rs is linted, with a table test per method. - CostModel::unit_costs, used by node usage, plan_cost, the request overhead and the raw-data disk cost. - edge_roles and check_raw_data_windows list every operator; the raw-data reader check uses covered_window_ms. - Lint comments name the identity checks kept as matches!/if let; add match_wildcard_for_single_variants; summary_type_name keeps its Debug fallback with an allow and a reason. - Bytes test: known gaps on each family's row (KnownBytesGap, keyed by FieldDataType), the exact-accumulator gap under its own issue, Rows::update shared by both tests, BYTES_AGREEMENT_FACTOR everywhere, next_uniform, and per-row-kind counts (10,000 values, 1,000 label rows): about 1.8 s instead of 15-30 s in debug, same ratios. - Time-access test trimmed to the operators with a decision; the two AggregateCore doctests dropped.
- check_raw_data_windows asks covered_window_ms instead of listing every operator; the cost usage `sources` rule takes the same `if let Scan` form as Operator::scanned_sources. - CostModel::unit_costs -> per_node_unit_costs: only cost model 1 prices per node. - InputTimeAccess::of reads TimeRange and chain window lengths from covered_window_ms, so the saturating conversion is written once; the covered_window_ms doc states the saturation change. - keeps_input_time -> keeps_input_window, with a TimeShift example. - Lint comments name every identity check kept (is_asap, lifetime). - Bytes test: note that the executor's unseeded KLL varies by ~0.2%. - dag-viewer fixtures: one output name per measure.
…er the module lint Main added a wildcard arm in shift_back_ms_below (time_window.rs) and in the tumbling-windows match (node::usage), both modules this PR lints with clippy::wildcard_enum_match_arm. The first is the 'is it a TimeShift' identity check the module comment already allows, now an if let; the second names both Operator halves, since keeps_input_window answers per operator.
zzylol
force-pushed
the
refactor/no-silent-defaults
branch
from
November 16, 2026 01:29
273cadb to
4185aff
Compare
This was referenced Nov 16, 2026
Merged
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
InputTimeAccess::of, and got a default answer from every_arm in asap-cost's per-node usage, inedge_rolesand in the raw-data window check. AnAggregateread from JSON withoutoutput_namesgot an empty list that matches none of its measures.approx_memory_bytes. The window questions are answered once onOperator(this PR'scovered_window_ms, with main'schain_tumbling_window_width_msandkeeps_input_window), each listing every operator; the sites that asked them call these, includingcheck_raw_data_windowsandInputTimeAccess::of's window lengths. The other per-operator rules list each variant, andclippy::wildcard_enum_match_armfails the-D warningsbuild on a new_.CostModel::per_node_unit_costsanswers "which cost model prices each node" once.output_namesis required when deserialized. A new per-family test checks the planner's state bytes against what the executor charges.Principle: an unknown stays
None(or unanswered) until code that knows the safe direction handles it. A rule that answers per operator, summary type or cost model lists every variant, so a new variant fails to build instead of silently getting a default.if letandmatches!remain only for a check of one variant's identity ("is it aScan", "is it an ASAP operator", "is it stored across evaluations"), where every other variant's answer is "no" by definition; the module comments name those sites.Review guide
main(3e434c760, after refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237). Not stacked. Head:4185aff5b. Rebased onto main after refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237: see the section below.crates/types/src/ir/operator/node.rs: the three window methods and their table tests.crates/cost/src/node/state_bytes/tests.rs:priced_bytes_agree_with_the_executor,KnownBytesGap,BYTES_AGREEMENT_FACTOR.crates/types/src/plan/time_window.rs:InputTimeAccess::of.crates/cost/src/node/usage/mod.rsandedge_roles. Each arm keeps its old body.Operatormethods;check_raw_data_windowsis nowif covered_window_ms().is_some().CostModel::per_node_unit_costsat its four call sites.output_names: []becomes[""], one name for their onetop_kmeasure.label/labels/Rowshelpers andRows::update, shared by both byte tests.PromqlSubqueryarm inInputTimeAccess::of, andcovered_window_msgivingNonefor a subquery (see Decisions).Rebased onto main after #1237
Base
main3e434c760(was59365ce80); head4185aff5b(was273cadbeb, kept asbackup/refactor/no-silent-defaults-273cadbeb). #1237 addedchain_tumbling_window_width_ms,keeps_input_windowandrequest_anchortoOperator, and deletedmaintained_window_ms, so this PR now adds onlycovered_window_msof the three window methods.crates/types/src/ir/operator/node.rscovered_window_msand main'srequest_anchorat the same place; the variant lists ofkeeps_input_window(namedkeeps_input_timein this PR's second commit)keeps_input_windowkeeps main's name and this PR's round-2 doc comment; the table test callskeeps_input_windowcrates/cost/src/node/usage/mod.rs, 4 hunksmaintained_window_ms, the output interval, the tumbling-windows match,chain_width_msmaintained_window_msand this PR's edits to it are dropped (main replaced it byRefreshWindow::computed_from); main'soutput_interval_ms; main'spayload if payload.keeps_input_window()arm; main'schain_tumbling_window_width_ms(producers). This PR's other changes to the file (the lint,state_family, unit costs, theif letsourcesrule, the listed matches) are keptTwo more changes, in their own commits:
7b7d2e1ad(round-2 commit, amended):covered_window_ms's doc named the deletedmaintained_window_ms; it now nameswindows_computed_fromand soRefreshWindow::computed_from.4185aff5b: refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237 added two_arms in modules this PR lints withclippy::wildcard_enum_match_arm, which fail-D warnings:shift_back_ms_belowintime_window.rsis nowif let TimeShift … else(the "is it aTimeShift" identity check the module comment already allows), and the last arm of the tumbling-windows match innode::usageisPayload::NonASAP(_) | Payload::ASAP(_) => 1(keeps_input_windowanswers per operator).Semantic check of #1237's names: this PR calls none of
least_retention_ms_per_execution,ExecutionCosts::retention_ms_for_need,RefreshWindow::computed_fromor theErrorContracthelpers;RefreshWindow::computed_fromgoes throughwindows_computed_from, which this PR changed only to readcovered_window_ms(same answers).Re-run after the rebase (release build, compared by test name with main
3e434c760):cargo check --workspace --all-targetscargo fmt --all -- --checkcargo clippy -p asap-cost -p asap-types -p asap-executor -p asap-frontend-promql -p asap-stage2-physical-optimization -p asap-integration-tests -p asap-planner --all-targets -- -D warningscargo test --no-fail-fast -p asap-cost -p asap-types -p asap-executor -p asap-frontend-promql -p asap-stage2-physical-optimization -p asap-stage3-plan-selection -p asap-planner -p asap-integration-tests -p asap-devtools --lib --testscargo test -p asap-types -p asap-cost --docnode --test tools/dag-viewer/*.test.mjs(Node 22)Behavior: before → after
impl AggregateCorewithoutapprox_memory_bytesNonASAPOp/ASAPOpvariantCurrentTimeOnly, given oneInputedge role, and the_answer in every usage ruleInputTimeAccess::of, the three window methods,edge_rolesand each usage rule (non-exhaustive match);check_raw_data_windowsfollowscovered_window_msCostModelvariantplan_costgives it no unit costsCostModel::per_node_unit_costs_ =>arm on an enum inir::operator::node,plan::time_windowor asap-costnode::usagecargo clippy -- -D warnings(CI'sRun clippystep in.github/workflows/rust.yml) fails onwildcard_enum_match_armormatch_wildcard_for_single_variants{"Aggregate": {"reduction": "PerEntity", "measures": ["Rate"], "child": 0, …}}with nooutput_namesoutput_names: []missing field `output_names`This serves design correctness: plan selection's resource charges should not rest on guesses that nobody chose.
Design scope
New production items, each a method on an existing type; no new types, files or fields:
Operator::chain_tumbling_window_width_msir::operator::node)Operator::covered_window_mswindows_computed_from(and so main'sRefreshWindow::computed_from),InputTimeAccess::of(theTimeRangeand chain window lengths),check_raw_data_windowsand the raw-data reader check inmaterialized_view::views.Operator::keeps_input_windowTimeShift) and adds its table test.CostModel::per_node_unit_costsasap-costmodel)CostModelitself, asked at four call sites.Test-only: the
Rowsenum (now withRows::updateandRows::fed),label,labels,next_uniform,BYTES_AGREEMENT_FACTORandKnownBytesGapincost::node::state_bytes::tests, shared by both byte tests.Interfaces
crates/types/src/ir/operator/node.rscrates/cost/src/model.rscrates/executor/src/summary_kernels/traits.rspub trait AggregateCore: Send + Sync { fn clone_boxed_core(&self) -> Box<dyn AggregateCore>; fn as_any(&self) -> &dyn std::any::Any; fn as_any_mut(&mut self) -> &mut dyn std::any::Any; fn merge_with(&self, other: &dyn AggregateCore) -> Result<Box<dyn AggregateCore>, KernelError>; fn estimate(&self, query: &SketchStatistic) -> Result<f64, KernelError> { /* ... */ } fn is_empty(&self) -> bool { /* ... */ } /// Approximate in-memory footprint, used for execution memory reservations. /// A state shared with other rows (see [`Self::shared_state`]) is not - /// included. - fn approx_memory_bytes(&self) -> usize { - 4096 - } + /// included. No default: each kernel says what it holds, e.g. a KLL + /// state its sketch's `memory_bytes()`. + fn approx_memory_bytes(&self) -> usize; fn shared_state(&self) -> Option<(usize, usize)> { /* ... */ } }All 11 kernels already override it, so none needed a change.
crates/types/src/ir/operator/non_asap.rs(file format: theAggregatefield becomes required)pub enum NonASAPOp<C = Rc<OperatorNode>> { Aggregate { reduction: Reduction, measures: Vec<AggIntent>, /// Output column names, exactly one entry per measure; an empty /// string means the default name (the intent's synthetic name). - #[serde(default)] + /// Required when deserialized: a missing list would not match the + /// measures. output_names: Vec<String>, #[serde(default)] filters: Vec<Option<Predicate<C>>>, #[serde(default)] having: Option<Predicate<C>>, child: C, }, }output_namesis the only operator field holding output names. TheNonASAPOpvariants not shown are unchanged.crates/types/src/plan/time_window.rsandcrates/cost/src/node/usage/mod.rs: no signature changes. Both modules gain the two lints, with a comment naming theirmatches!,if letandlet ... elseidentity checks, including the lifetime checkmatches!(.., Lifetime::AcrossEvaluations(_)).InputTimeAccess::ofreads theTimeRangeand chain window lengths fromcovered_window_ms, so the saturating ms conversion is written once.summary_type_name's Debug fallback keepsother =>under an item-level#[allow(clippy::wildcard_enum_match_arm)]: any other type prints its Debug form, which names a new variant too.edge_rolesgets the lint at item level.check_raw_data_windowsaskscovered_window_ms().is_some()and needs no list.Usage, end to end: the PromQL query
sum_over_time(m[5m] offset 1m), planned at ingestion time.Aggregate(SumOverTime) <- TimeRange(5m) <- TimeShift(1m) <- Scan(m), withoutput_names: [""].plan_costcallInputTimeAccess::ofper node, throughTimeIndexedPlan::input_time_access:TimeShift(1m)at query time:WhatItsReadersRead, shifted 60,000 ms (its own arm);TimeRange(5m):LastWindow { length_ms: 300_000 }, the length read fromcovered_window_ms;Aggregate:CurrentTimeOnly, one of the listed operators.windows_computed_from(Aggregate)walks down:Aggregate.covered_window_ms()isNone,TimeRange(5m)gives 300,000, over the 60,000 ms shift below it:{60_000, 300_000}, the same as before.RefreshWindow::computed_from(Aggregate)(main's refresh rule, the union ofwindows_computed_from): one window of 300,000 ms, so the view is refreshed every 5 minutes, as before.node_resource_usage_and_costprices each node withsettings.cost_model.per_node_unit_costs()(cost model 1 here), takessourcesas the union over its producers (it is not aScan), and itsunitsand lookup arms nameAggregateexplicitly, with the same values as before.Algorithm changes
None. Every rewritten rule keeps its answer. One edge, unreachable in practice: a
TimeRangelonger thanu64::MAXms (about 584 million years) now saturates incovered_window_msand is still a window; before,windows_computed_from(and soRefreshWindow::computed_from) gaveNonefor it. Thecovered_window_msdoc comment states this. PromQL planning cannot reach it:sum_over_time(data[600000000y]),quantile_over_time(0.9, data[600000000y])andsum_over_time(data[290000000y])print byte-identicalNoValidCandidate … analytical arithmetic overflowedon main and here, because costing overflows first.What is listed now
Lines are on
mainbefore #1237 (59365ce80). After the rebase onto3e434c760, four rows are main's code from #1237, marked (main).crates/types/src/plan/time_window.rs:156(InputTimeAccess::of)NonASAPOp+ 11ASAPOp;PromqlSubqueryin its own arm, stillCurrentTimeOnlytime_window.rs:341(windows_computed_from)covered_window_ms();Nonewalks on downcrates/cost/src/node/usage/mod.rs:96(state_family)FieldDataTypePlain,SummarySample,Wavelet,StatModel→Noneusage/mod.rs:103(state_family)Noneusage/mod.rs:223and:237(maintained_window_ms, the node and the walk)RefreshWindow::computed_from, overwindows_computed_from(above)usage/mod.rs:272(unit costs)CostModelCostModel::per_node_unit_costs()usage/mod.rs:294(output interval)output_interval_ms:chain_tumbling_window_width_ms(), elsewindow_msor one secondusage/mod.rs:317(tumbling windows)TimeRange-over-chain arm, thenpayload if payload.keeps_input_window(); the last arm,_on main, is nowPayload::NonASAP(_) | Payload::ASAP(_)→ 1usage/mod.rs:321(sources)if let Scan, else the union of its producers' sources (an identity check, the same form asscanned_sources)usage/mod.rs:350(measured unit cost)Aggregate→lookupusage/mod.rs:524(Aggregategrouping keys)ReductionPerEntity→ 1usage/mod.rs:674(non-ASAP pass-through)NonASAPOpusage/mod.rs:805(estimate rows per group)SketchStatisticFrequencyL2,FrequencyEntropy,Quantile,PointCount,Cardinality→ 1usage/mod.rs:824(other ASAP operators)*MaintainPopulation,EvaluatePopulation,SummarySubtract,SummaryDelete,SummaryJoin,Extension→ one op per rowusage/mod.rs:840(units)Scan,SummaryMerge→ input rowsusage/mod.rs:1047(chain_width_ms)chain_tumbling_window_width_ms(producers), read fromProducerOutput::chain_tumbling_window_width_mstime_window.rs(shift_back_ms_below, added by #1237)if let TimeShift, else the path ends (an identity check, named in the module comment;_on main)crates/cost/src/plan/cost.rs:99(plan_costunit costs),:368(request overhead),:388(raw-data disk cost)CostModelCostModel::per_node_unit_costs()crates/types/src/plan/physical_asap_dag.rs:567,:572(edge_roles)NonASAPOp,ASAPOpInputcrates/types/src/plan/materialized_view/raw_data.rs:115(check_raw_data_windows)covered_window_ms().is_some()→ check the raw window below; any other operator has no window to checkcrates/types/src/plan/materialized_view/views.rs:232(reads raw data)Scan→ true;covered_window_ms().is_some()→ false; any other operator passes on its input's answercrates/types/src/ir/operator/node.rs(scanned_sources)if let Scan, else the union over its inputs (an identity check; wasoperator =>)* The lint does not flag
:824, because every top-levelOperatorvariant is named elsewhere in that match. It is listed anyway for the principle.The lint only checks top-level variants: it would accept
Operator::NonASAP(_) | Operator::ASAP(_). So every operator site lists the leafNonASAPOp/ASAPOpvariants, which a new operator breaks.usage/mod.rs:512matches a(bool, Lifetime)tuple and stays as it is.Identity checks kept (
matches!,if let,let ... else; named in each module comment):time_window.rscurrent_time_read_needs_every_row: is it aScan;time_window.rsshift_back_ms_below: is it aTimeShift;usage/mod.rsthe retained raw bytes andsources: is it aScan;usage/mod.rsraw_data_samples_non_negative: is its producer aScanor aTimeRange(any other producer's samples are not raw, the safe answer);node.rsscanned_sources: is it aScan;is_asap: is it an ASAP operator;time_window.rsInputTimeAccess::ofandusage/mod.rs: the lifetime checkmatches!(.., Lifetime::AcrossEvaluations(_)).Not touched:
types::stages::logical::group(column_properties.rs): the sign-design PR handles it.materialized_view/views.rs:436(view_inputs): a walk down that stops at a rawTimeRangeand errors at aScan. Its_keeps walking, which is right for any new operator: a walk that reaches a rawScanstill errors.materialized_view/raw_data.rs:177(below a raw range): its_returnsUnsupported, so a new operator fails loudly rather than getting a silent answer.Known bytes gaps
priced_bytes_agree_with_the_executorcovers every family the executor builds from a plan. Each family uses its parameters for a 1% error and is fed seeded rows, priced for the same count (Rows::fed):Every ratio below is the same as at 10,000 rows for every family.
The rows are seeded, but the executor builds KLL unseeded (
Kll::with_k; its factory takes no seed), so KLL's charge varies by about 0.2% between runs (10,376 or 10,392 B; ratio 1.000 or 1.002). A comment on KLL's row says so; it is far inside the tolerance.The test requires priced ÷ charged to lie in [1/1.5, 1.5]. A family outside it carries a
KnownBytesGapon its own row (keyed by itsFieldDataType), with its issue and measured ratio. The test fails in three cases:These agree within 1.5×:
HydraKll is left out: no plan builds it, since it has no parameter list and the factory refuses it. The pricing itself is not changed here.
Decisions and open questions
Needs reviewer decision
PromqlSubquery.InputTimeAccess::ofreads it asCurrentTimeOnly, as before, andcovered_window_msgives itNone.materialized_view/views.rs(re_times_its_input) lists it as re-timing its input, so the two disagree. This PR keeps today's behaviour. Bounded scan windows, retention and Stage 3 pricing for subqueries, @ anchors and ranges over projected rows #1249 is where that gets decided.Decided
ir::operator::node,plan::time_windowandnode::usage, so new rule functions there are covered too; item level onedge_roles.covered_window_msandchain_tumbling_window_width_msstay two full lists, though they differ only in theTimeRangearm: building one from the other would let a new windowing operator skip one decision.OneEvaluationoutput rows; [Bug] An exact accumulator held as a kernel object is charged 136 bytes; plan selection prices 56 #1333 is the size of one state, record (56 B) against object (136 B).Verification
cargo clippy -p asap-types -p asap-cost --all-targets -- -D warningscargo test -p asap-types --libcargo test -p asap-types --testscargo test -p asap-types --doccargo test -p asap-cost --libnode --test tools/dag-viewer/*.test.mjs(Node 22)cargo fmt --all -- --checkasap_planner --run-until stage3with a local, unpushed dump of the selectedPhysicalASAPDAGand itsplan_cost(the text and JSON printers end intodo!), this head against main 59365ce, onplanner-layering/example1,promql/ratio-of-quantiles-5m,promql/rangeheavy-t5-q05-1h-60s-300s,promql/rangeheavy-t8-t5-1h,random-mixed-seed-862/workload-01and-02(andplanner-layering/example2, which fails the same way on both: its SQL table is not declared)cargo test -p asap-cost --lib priced_bytes_agree_with_the_executor, alone, one test threadNew tests. The table says what each catches.
non_asap::tests::aggregate_output_names_are_required_when_deserializedAggregatewithoutoutput_namesbeing accepted again (the serde default coming back); and a broken round trip with namesnode::tests::only_a_chain_has_a_tumbling_window_width,a_time_range_and_a_chain_cover_a_window,filters_projections_and_shifts_keep_input_windowcovered_window_msdoes not exist; the other two methods are main's (#1237), untested theretime_window::tests::operators_with_a_time_access_decisionTimeShiftby timing and lifetime,@ end,TimeRange, the chain at either timing,PromqlSubquery, and oneAggregate. A new operator fails to build inInputTimeAccess::of; its decision gets a row herestate_bytes::tests::priced_bytes_agree_with_the_executorwildcard_enum_match_armandmatch_wildcard_for_single_variants_arm on an enum in the linted modules and functions. It runs in CI'sRun clippystep (cargo clippy --workspace --all-targets --all-features --locked -- -D warnings)scanned_sourcesin node.rsFollow-ups and conflicts
exec/x2-reduce-aggregates) editsnon_asap.rs, away from this PR's lines: no textual conflict. Its new operators, if any, must be added to the listed matches.summary_kernels/factory near the byte test families. There is no textual conflict, but any byte change there must update the families' gaps.[profile.dev.package.asap_sketchlib] opt-level = 2in the workspace would speed every debug test that touches sketches; a workspace-wide choice, not made here.Operator::scanned_sourcesinnode.rsand thesourcesrule innode::usageencode the same rule (aScan's own source, else the union over its inputs). Both now take the sameif letform; they read different data (a source set vs the producers' names), so they are not merged.Review status
Operator), A3 (lint gaps), A4/A5/A6/A7 (tests and doctests), L1 (CostModelunit costs), L2 (edge_roles,check_raw_data_windows;views.rs:232askscovered_window_ms, so its decision is delegated to that exhaustive method; the other two listed under Not touched), L3–L7, N1–N3. A2 is the refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237 note above; A8 is settled by B2.check_raw_data_windowsaskscovered_window_ms, its list and lint attribute deleted), B2 (sourcesasif let Scan … else union), B3 (per_node_unit_costs), B4 (one saturating conversion).views.rs:232wording), R2-L5 (= B4), R2-N1 (keeps_input_window,TimeShiftdoc), R2-N2 (lint comments nameis_asapand the lifetime checks).output_namesdefault; the saturation edge is unreachable through PromQL. Applied F1 (KLL row note; the executor exposes no seeded KLL) and F2 (dag-viewer fixtures).covered_window_msdoc comment.