Skip to content

feat(stage0, types): memo hash-consing, group properties (column_sign, depends_on_evaluation_time, aggregate_coverage), @ end()/@ start() above their subtree (#1207 PR 2) - #1230

Merged
zzylol merged 12 commits into
mainfrom
refactor/1207-memo
Nov 10, 2026
Merged

zzylol merged 12 commits into
mainfrom
refactor/1207-memo

Conversation

@zzylol

@zzylol zzylol commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: PR 0 left the Memo methods, Stage 0 and the group properties todo!(), so every planning test stopped at Stage 0; and m[5m] @ end() lowered in place was a group of its own, sharing nothing with the same selector without @ (Range queries: evaluate @ end()/@ start() subtrees once per request and share them #1251).
  • Change: hash-consing (find_or_add_group), candidate storage (add_candidate), add_query with each query's accuracy target and $n_q$; column_signs, depends_on_evaluation_time, scanned_sources and aggregate_coverage derived once per group; PromQL @ end() / @ start() lifted above the subtree they anchor.
  • Effect: sum by (host) (rate(requests_total[1m])) over a declared counter now gets a NonNegative sum column, sum by (host) (gauge) an Unknown one (so Stage 2 later prunes its Count-Min heap top-k); a query over m with 1000 declared series enters with number_of_values_per_query_evaluation == Some(1000); latency[5m] @ end() shares the latency[5m] group with the instant query. Planning now passes Stage 0 and stops at Pass 1's target discovery (PR 3).

Review guide

  • Position: base main, after refactor(types): properties and requirements from #1207; chains as time-indexed state (#1207 PR 1) #1227 (merged as 14437bc47); refactor(types, frontends): column lineage replaces table qualifiers (#1207 PR 2b) #1233 (PR 2b, lineage) builds on it. Twelve commits, head bba5e221c: the restacked PR 2 (3cabdc3b5), the review fixes (77fc121d3), the coverage move with the aggregate-layout tests (c71be0331), the source facts and scanned sources (f458196ba), the expression check named by what it checks (0fef10001), the second review (1ae8acf91: D1, D2, D4, M1–M10), the third review (f68d414aa: D1 option A, items 1–12), the sign rules (c2f23d760: the ValueSign definition and sign items 1–5), the two-scope definition identity (73785fa07), the sign bit, view coverage on the plan node and review cleanups (288cd70d6), the memo query methods, Operator::scanned_sources, coverage wording and shared test helpers (0de33fd94), and coverage in the IR with the plan's content id by expansion (bba5e221c). This PR changes only core data structures, interfaces and definitions; case-specific fixes from review go to follow-up issues and PRs. Diff vs base: 67 files, +5,846 / −1,684 (most of it the coverage type and derivation moving out of summary_coverage.rs).
  • Restack (2026-11-10): the first commit is the previous PR 2 (36312cbaf, backup backup/1207-layout-memo-36312cbaf) in the scope layout, content unchanged: Memo methods in stages/logical/memo/mod.rs, hash-consing in stages/logical/memo/hash_consing.rs, group-property derivation in stages/logical/group.rs, the sign derivation in stages/logical/column_properties.rs (PR 2b adds lineage there), the memo unit tests in stages/logical/memo/tests.rs. The second commit applies the 2026-11-10 review (items 1–7, a–c under Decided); the third moves the coverage derivation to stages/logical/aggregate_coverage.rs (B1) and adds the aggregate_output_columns tests (B4), moves only. Review the three apart.
  • Reading order:
    1. types/src/stages/logical/memo/mod.rs: Memo::new, find_or_add_group, add_candidate, add_query; types/src/stages/logical/memo/hash_consing.rs.
    2. types/src/stages/logical/group.rs (validate_expression_over_input_groups, derive_group_properties) and the new types/src/stages/logical/column_properties.rs; types/src/ir/schema/aggregate_schema.rs (OutputColumnSource).
    3. stage0-memo/src/lib.rs: memo_of_queries.
    4. types/src/stages/logical/aggregate_coverage.rs (AggregateCoverage, aggregate_coverage, time_constraint, of_aggregate), types/src/ir/properties/selection.rs (SelectionBox, Constraint, constraint_of) and frontend-promql/src/promql.rs (anchored, selector).
  • Mechanical: SQL timestamp-literal folding and identity-cast dropping in frontend-common / frontend-sql; the top-k schema and Pass 1 read OperatorInput::scanned_sources / GroupProperties::scanned_sources instead of their own walks; test fixtures; concepts doc and asap-primitive-schema.md §4.2.2 updates.
  • Careful: the hash-consing equality (same_serialized_expression) and the shareability rule; the per-operator sign table; the coverage time intervals and where the walk stops (TimeShift { at: End | Start }); counter states merging only over intervals disjoint in time.

Behavior: before → after

Before (on PR 1), memo_of_queries, find_or_add_group, add_candidate and add_query panic at todo!(); m[5m] @ end() lowered as TimeRange(TimeShift(@ end())(Scan)) in a range query and as m[5m] in an instant one, through a range_evaluated flag (previous revision). It serves the design need that rules add groups and candidates only through these methods and derive every group property once.

Input Before After
sum by (host) (rate(requests_total[1m])), a counter panics at Stage 0 sum column NonNegative, depends_on_evaluation_time true
sum by (host) (gauge), nothing declared panics sum column Unknown
latency declared {"value": "non_negative"}, quantile_over_time(0.99, latency[1m]) panics state and quantile NonNegative
m[5m] @ 1700000000 panics depends_on_evaluation_time false; coverage ts ∈ (1699999700000, 1700000000000]
latency[5m] @ end() its own group TimeShift(@ end()) over the group of latency[5m], shared with instant queries
a query over m with 1000 declared series panics MemoQuery::number_of_values_per_query_evaluation == Some(1000)
KLL k = 200 and k = 345 over one input panics one group, two candidates; a duplicate returns the first id
quantile_over_time(0.9, latency[1h]) / quantile_over_time(0.5, latency[1h]), latency non-negative panics value NonNegative (Unknown at f68d414aa)

Interfaces

Where PR 2's code is (scope layout)

What File
Memo (with the data workload's source facts) and MemoQuery, the memo operations (new, find_or_add_group, add_candidate, add_query) and the hash-consing tests types/src/stages/logical/memo/mod.rs
The equality groups and candidates are looked up by, its hash, the expression serialized once types/src/stages/logical/memo/hash_consing.rs
Memo method unit tests types/src/stages/logical/memo/tests.rs
GroupProperties (incl. scanned_sources), MemoGroup (parents, queries crate-private behind their accessors), the expression check (validate_expression_over_input_groups), MemoGroupAsInput, derive_group_properties, depends_on_evaluation_time types/src/stages/logical/group.rs
ValueSign and the column_signs derivation types/src/stages/logical/column_properties.rs (PR 2b adds lineage)
AggregateCoverage, AggregateDefinition, CoverageError and their derivation (aggregate_coverage and its walk) types/src/stages/logical/aggregate_coverage.rs
The selection algebra: SelectionBox, Constraint, ColumnIdentity, IntervalConstraintLimit, constraint_of types/src/ir/properties/selection.rs
The aggregate layout (OutputColumnSource, aggregate_output, both crate-private) and its tests types/src/ir/schema/aggregate_schema.rs
A summary state's layout (summary_agg_layout) types/src/ir/operator/asap.rs
OperatorInput::scanned_sources, the rule sources_scanned_by (a Scan's source, else the union over its inputs) types/src/ir/operator/input.rs; Source::is_time_series in ir/operator/arguments.rs
Family lookup of a series' facts (series_facts, shared by DataWorkload::series_facts) types/src/workload/data.rs
Stage 0 (memo_of_queries, the declared signs, its tests) stage0-memo/src/lib.rs
Group-property and coverage tests; the memo fixtures they and the memo unit tests share types/tests/memo_group_properties.rs, memo_aggregate_coverage.rs; types/src/test_support/memo_fixtures.rs (feature test-support)

crates/types/src/stages/logical/memo/mod.rs · crates/types/src/stages/logical/memo/hash_consing.rs

 /// The memo, Stage 0's and Stage 1's one type: the groups with their candidates and each query's root. The type stays.
-#[derive(Debug, Clone, Default)]
+#[derive(Debug, Clone)]
 pub struct Memo {
     /// The groups, by id; added only through `find_or_add_group`.
     groups: Vec<MemoGroup>,
     /// The queries, by index in the workload.
-    pub queries: Vec<MemoQuery>,
+    queries: Vec<MemoQuery>,
+    /// The shareable groups (`shareable_when_equal`) by the hash of their serialized expression,
+    /// each listed with that serialization (computed once per expression).
+    groups_by_expression_hash: HashMap<u64, Vec<(MemoGroupId, String)>>,
+    /// The data workload's facts per source (`DataWorkload::sources`), fixed at construction:
+    /// a `Scan` group's column signs read them (a series through its family, `series_facts`).
+    pub(super) source_facts: BTreeMap<String, SourceFacts>,
 }
 /// One query of the memo: its root over groups, its accuracy target and n_q. The type stays.
 pub struct MemoQuery {
     /// Its root over groups.
     pub root: QueryRoot<MemoGroupId>,
     /// A_q; None for a query without one (every query when accuracy is not enforced).
     pub accuracy_target: Option<AccuracyTarget>,
     /// n_q, the values one evaluation returns; None when the data workload does not bound it.
     pub number_of_values_per_query_evaluation: Option<u64>,
 }
 impl Memo {
+    /// A memo without groups over the sources `data` describes; the only constructor (no `Default`).
+    pub fn new(data: &DataWorkload) -> Self;
+    /// The queries, by index in the workload.
+    pub fn queries(&self) -> &[MemoQuery];
     /// Adds `query`; its root's groups and every group below them list it (body filled in).
     pub fn add_query(&mut self, query: MemoQuery) -> usize;
     /// Hash-consing: the group whose expression (operator, arguments, input groups) equals `expression`,
     /// else a new group with derived `GroupProperties`; an unshareable expression always gets a new group.
     /// Fails, leaving the memo unchanged, when the expression is not valid over its input groups: its output
     /// schema does not derive, an input fails validation, or a merge reads states it may not merge (body filled in).
     pub fn find_or_add_group(
         &mut self,
         expression: Operator<MemoGroupId>,
     ) -> Result<MemoGroupId, SchemaDerivationError>;
     /// Adds `candidate` to `group`, or returns the id of the duplicate (same expression and θ); fails when its
     /// schema or kind differs from the group's (body filled in).
     pub fn add_candidate(
         &mut self,
         group: MemoGroupId,
         candidate: CandidateForMemoGroup,
     ) -> Result<CandidateForMemoGroupId, CandidateError>;
 }

crates/types/src/stages/logical/group.rs · crates/types/src/stages/logical/column_properties.rs

 /// What every candidate of a group produces, derived once from its expression and input groups. The type stays.
 pub struct GroupProperties {
     /// Output schema.
     pub schema: Schema,
     /// Output kind.
     pub result_kind: OperatorResultKind,
     /// Per output column: the sign of its values, from the input groups' and, for a `Scan`, the declared ones
     /// (now derived; was `todo!()`).
-    pub column_sign: Vec<ValueSign>,
+    pub column_signs: Vec<ValueSign>,
     /// Whether the result changes with the evaluation time: a `TimeRange`, `time()`/`now()`; false under `@ T`
     /// (now derived; was `todo!()`).
     pub depends_on_evaluation_time: bool,
     /// What an aggregate group covers: a summary state, a chain, a `TimeRange`/`TimeShift` over a chain or a
     /// merge over one, or an exact `Aggregate`; None otherwise (chains, window selections, exact Aggregates added).
-    pub aggregate_coverage: Option<SummaryCoverage>,
+    pub aggregate_coverage: Option<AggregateCoverage>,
     /// Whether two equal expressions return the same rows (not an unordered `LIMIT`).
     pub shareable_when_equal: bool,
+    /// The sources the `Scan`s below this group read: a `Scan`'s own, else the union over its input groups.
+    pub scanned_sources: BTreeSet<Source>,
 }
 pub enum CandidateError {
-    Schema(SchemaDerivationError),
+    /// The expression is not valid over its input groups.
+    NotValidOverInputGroups(SchemaDerivationError),
     OutputDiffersFromGroup { group: MemoGroupId },
 }
+/// Whether `operator` is a valid expression over its input groups: its output schema derives, its inputs pass
+/// validation, and a merge reads only states it may merge. Returns the output schema.
+pub(super) fn validate_expression_over_input_groups(operator: &Operator<MemoGroupAsInput<'_>>)
+    -> Result<Schema, SchemaDerivationError>;

crates/types/src/ir/operator/input.rs · crates/types/src/ir/operator/arguments.rs

 /// An input of an operator: a node of a tree or a memo group. The trait stays.
 pub trait OperatorInput: Sized + Clone {
     /// Its schema.
     fn schema(&self) -> &Schema;
     /// Its result kind.
     fn result_kind(&self) -> OperatorResultKind;
     /// The input's own operator, its children as inputs again.
     fn operator(&self) -> Operator<Self>;
     /// What the input covers when it is an aggregate group in the memo; None otherwise.
-    fn aggregate_coverage(&self) -> Option<&SummaryCoverage>;
+    fn aggregate_coverage(&self) -> Option<&AggregateCoverage>;
+    /// The sources the `Scan`s at or below the input read: a memo group's stored set, recomputed for a node tree.
+    fn scanned_sources(&self) -> BTreeSet<Source>;
 }
+/// A `Scan`'s source, else the union over its inputs' `scanned_sources`.
+pub(crate) fn sources_scanned_by<I: OperatorInput>(operator: &Operator<I>) -> BTreeSet<Source>;
-#[derive(Debug, Clone, PartialEq, Eq, Hash, Serialize, Deserialize)]
+#[derive(Debug, Clone, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize)]
 pub enum Source { TimeSeries { metric: String }, Table { table_ref: String } }
+impl Source {
+    /// Whether it is a time series (PromQL rows); the top-k schema and Pass 1 read `.all(Source::is_time_series)`.
+    pub fn is_time_series(&self) -> bool;
+}
+// workload/data.rs: `DataWorkload::series_facts` over a sources map; the method delegates to it.
+pub(crate) fn series_facts<'a>(sources: &'a BTreeMap<String, SourceFacts>, series: &str)
+    -> Option<(&'a SourceFacts, Option<PrometheusMetricType>)>;

crates/stage0-memo/src/lib.rs

 /// Stage 0: the memo groups of the roots, query `i` with `accuracy_targets[i]` and its n_q (`values_per_result`);
 /// `Scan`s read `data`'s declared signs. Fails with the first node whose expression is not valid over its children's groups (a frontend bug)
 /// (body filled in).
 pub fn memo_of_queries(
     roots: &[QueryRoot],
     accuracy_targets: &[Option<AccuracyTarget>],
     data: &DataWorkload,
 ) -> Result<Memo, SchemaDerivationError>;

PromQL lowering has no interface change; its output shape for @ end() / @ start() changes (TimeShift { at } above the selector). The top-k schema's and Pass 1's private walks for PromQL rows are replaced by scanned_sources.

Usage, end to end (Stage 0 tests a_scan_reads_the_declared_column_signs, each_query_enters_with_the_values_one_evaluation_returns; types test a_top_k_ranks_values_of_its_inputs_sign):

// What the user writes: sum by (host) (rate(requests_total[1m])) and topk(5, sum by (host) (gauge));
// the data declares `requests_total` a counter (series_count 1000) and nothing for `gauge`.
// `roots`: one operator root per query, from the PromQL frontend; Stage 0 reads them.
let roots: Vec<QueryRoot> = lower_promql_query_workload(&workload)?;
// `targets`: accuracy not enforced, so no target per query.
let targets = vec![None, None];
// `memo`: Stage 0's groups (find_or_add_group bottom-up) over Memo::new(&data) (requests_total a counter: value NonNegative),
// and each query added with add_query and its n_q; Pass 1 (PR 3) reads it next.
let memo: Memo = memo_of_queries(&roots, &targets, &data)?;
// What the user gets back from Stage 0:
// rate(requests_total[1m]) per series: value NonNegative (a counter's rate); the sum by host keeps it.
// sum by (host)(gauge): value Unknown, so Stage 2 (PR 5) prunes the Count-Min heap builds of the top-k
// (`CountMinWeightsNotProvenNonNegative`) and the query uses an exact top-k.
// The `[1m]` groups and everything above them have depends_on_evaluation_time == true.
// memo.queries[0].number_of_values_per_query_evaluation: bounded by the 1000 series it reads (values_per_result).

And @ end() (promql lib test at_start_and_at_end_are_lifted_above_the_selector_they_anchor, types test at_end_and_at_start_share_the_subtree_they_anchor_and_stay_unlifted):

range query   quantile_over_time(0.9, latency[5m] @ end())
instant query quantile_over_time(0.9, latency[5m])

G1 Scan latency;  G2 TimeRange[5m](G1)                       (shared by both)
G3 SummaryAgg[KLL](G2)          coverage ts ∈ (t − 5m, t], input_group G1
G4 TimeShift(@ end())(G2);  G5 SummaryAgg[KLL](G4)            coverage: input_group G4, no time constraint

crates/types/src/ir/schema/aggregate_schema.rs · crates/types/src/ir/operator/asap.rs · crates/types/src/stages/logical/aggregate_coverage.rs · crates/types/src/ir/properties/selection.rs

+/// Where one output column of an aggregate comes from: a kept input column, a measure's value with the input
+/// column the schema types it by, or a label the measure adds (`count_values`', `absent`'s).
+/// `sum by (job) (x)` → `[InputColumn(job), Measure { index: 0, input_column: Some(value) }]`.
+pub(crate) enum OutputColumnSource {
+    InputColumn(ColumnId),
+    Measure { index: usize, input_column: Option<ColumnId> },
+    AddedLabel,
+}
+/// `aggregate_output_schema`'s schema and where each of its columns comes from, in one pass.
+pub(crate) fn aggregate_output(in_schema, reduction, measures, output_names)
+    -> Result<(Schema, Vec<OutputColumnSource>), SchemaDerivationError>;
-pub(crate) fn summary_agg_schema(child: &Schema, summary_type: &FieldDataType, reduction: &Reduction)
-    -> Result<Schema, SchemaDerivationError>;
+/// A summary state's schema and its columns' sources: the state in the measure's place.
+pub(crate) fn summary_agg_layout(child: &Schema, summary_type: &FieldDataType, reduction: &Reduction)
+    -> Result<(Schema, Vec<OutputColumnSource>), SchemaDerivationError>;
// ir/properties/summary_coverage.rs → stages/logical/aggregate_coverage.rs (the type, its errors, its derivation)
-pub struct asap_types::ir::properties::summary_coverage::SummaryCoverage { pub definition: SummaryStateDefinition, pub selection: Vec<SelectionBox> }
+pub struct asap_types::stages::AggregateCoverage { pub definition: AggregateDefinition, pub selection: Vec<SelectionBox> }
-pub struct asap_types::ir::properties::summary_coverage::SummaryStateDefinition { .. }
+pub struct asap_types::stages::AggregateDefinition { .. }
-pub enum asap_types::ir::properties::summary_coverage::CoverageError { NotSummary, .. }
+pub enum asap_types::stages::CoverageError { NoCoverage, .. }
-impl SummaryCoverage { pub(crate) fn derive_from_inputs(operator: &Operator<MemoGroupAsInput<'_>>) -> Result<Self, CoverageError>; }
+pub(super) fn aggregate_coverage(operator: &Operator<MemoGroupAsInput<'_>>) -> Result<AggregateCoverage, CoverageError>;
// ir/properties/summary_coverage.rs → ir/properties/selection.rs (the selection algebra, unchanged)
-pub mod asap_types::ir::properties::summary_coverage;  // SelectionBox, Constraint, ColumnIdentity, IntervalConstraintLimit
+pub mod asap_types::ir::properties::selection;
// stages/logical/group.rs
-pub(super) fn validate_expression_over_input_groups(operator) -> Result<Schema, SchemaDerivationError>;
+pub(super) fn validate_expression_over_input_groups(operator)
+    -> Result<(Schema, Option<AggregateCoverage>), SchemaDerivationError>;  // a merge's coverage, reused

Algorithm changes

Stage 0 hash-consing

  • Problem: Stage 0's node equality (cse::same_node, deleted in PR 0) and the group lookup the rules need were two equalities that could disagree (on a guarantee, for one).
  • Before / after: before, nodes were interned bottom-up into an Rc table, then numbered and paired with groups. After: 1. group_of maps each node by address to its group (a diamond is visited once), children first, calling Memo::find_or_add_group(operator over child groups); it derives the properties, hashes the operator's arguments with the input group ids (hash_consing::expression_hash) and compares the bucket with same_expression (only shareable expressions are indexed). 2. add_query per root, with its target and $n_q$ = values_per_result(root, series_read(root, data), data) (None when data bounds nothing). add_candidate checks before changing anything, then appends, adds parents, and walks down from the inputs inserting the group's queries in sorted position (stopping below a group that already had them). declared_column_signs walks the roots once for the scanned sources and reads DataWorkload::series_facts (SourceFacts::column_signs), so a histogram's _sum/_count series get the family's declarations and a declared counter's value is non-negative without a declaration.
  • Correctness: two expressions are one group only if their arguments are equal and their input groups are the same, so equal groups compute equal values by induction; a candidate must reproduce its group's schema and result kind. Tests: Stage 0's 15, memo_tests 10.
  • Complexity: O(arguments) per lookup (children are ids); O(N) for N nodes; one extra walk over the roots for the scanned sources.
  • Assumptions and limits: a shareable parent over unshareable children is no longer shared.

Aggregate coverage with time intervals

  • Problem: PR 0 left time lifting, chain coverage and the counter merge check todo!(); @ T and SQL timestamp filters had no representation; exact aggregates had no coverage.
  • Before / after: before, one range TimeRange gave a separate relative time field and a chain expanded to a build over count × width. After, time_constraint sums the offsets of the TimeShifts on the path, takes the one range width and the one @ T, and gives (anchor − o − W, anchor − o] on the timestamp column with IntervalConstraintLimit ends (left-open, right-closed): m[5m] offset 1h → (t − 65m, t − 60m], @ T → fixed timestamps. A chain is a build over its input with time unconstrained, and a window selection over it adds the interval, so SummaryMerge(TimeRange(TimeShift(Chain[w](X)))) covers what SummaryAgg(TimeRange(TimeShift(X))) does and add_candidate accepts it in that group. DisjointInTime compares the timestamp column (counter states merge only over intervals disjoint in time). An exact Aggregate is a top unlifted operator. SQL folds a timestamp cast of a literal and resolution drops a cast to the operand's own type, so ts >= '2026-11-01' lifts. The walk in of_aggregate stops at TimeShift { at: End | Start }: it is the definition's input_group.
  • Correctness: a row is in a TimeRange's window iff its timestamp is in the interval; equal coverage means equal rows. Tests: memo_aggregate_coverage.rs (13).
  • Complexity: one pass over the liftable operators.
  • Assumptions and limits: time is not lifted from paths with several range TimeRanges or @ Ts.

Value sign, evaluation-time dependence, time-series reads

  • After: output_column_sign(operator, output, source_facts) per the doc's "Value sign" table: Scan: declared sign, else Unknown; Filter/TimeRange/TimeShift/Sort/Limit/Dedup/column Project: the input's; abs NonNegative, negation flips, numeric literals and casts; sum/avg/min/max/quantile/top-k/first/last keep the value's sign; counts and spreads NonNegative; rate/irate/increase NonNegative over a NonNegative input; a state records the sign of its update values; a merge keeps the sign its inputs agree on; a readout reads the state's (a distinct count, L2 norm or entropy is NonNegative); anything else Unknown. It follows each operator's output layout (aggregate_output_schema's; chain_schema's for a chain: window_end Unknown, the build's time column dropped; a merge over a chain's windows reads the chain's build); an unknown layout is Unknown everywhere; a debug_assert checks one sign per output column. depends_on_evaluation_time: true for a TimeRange and for a scalar expression reading time() / now(); false for TimeShift { at: Some(Timestamp(T)) }; otherwise true when an input group's is. reads_only_time_series: a Scan of a time series, or every input group reading only time series; OperatorInput::reads_only_time_series computes the same rule over node trees (one rule, ir::operator::input::reads_only_time_series).
  • Correctness: memo_group_properties.rs (10 tests; sign rules under their own section), Stage 0's a_scan_reads_the_declared_column_signs.
  • Complexity: one pass per group over its operator and its input groups' stored properties (O(output columns)).

Sign rules (commit c2f23d760)

  • Problem: two readouts claimed a sign they cannot guarantee (a Count Sketch or UnivMon point count over non-negative weights can be negative; a quantile with q outside [0, 1] is not a value of its input), and arithmetic dropped every sign, so x / 1000, x * 8 and a ratio of quantiles lost a known NonNegative.
  • Definition: NonNegative means every non-NaN value is ≥ 0, NonPositive every non-NaN value is ≤ 0 (documented on ValueSign); a NaN does not break either.
  • Before / after:
Rule Before (f68d414aa) After (c2f23d760)
point count / top-k readout of a Count Sketch, UnivMon or a variant (CountSketch, CountSketchFold, CountSketchWithHeap, CountSketchWithL2Norm, CountSketchWithSpaceSaving, HydraCountSketch, UnivMon, UnivMonQ, UnivMonTwoTier, HydraUnivMon) the state's sign Unknown
point count / top-k readout of Count-Min (and every other state) the state's sign unchanged
AggIntent::Quantile / SketchStatistic::Quantile with q outside [0, 1] the input's sign Unknown
quantile with q in [0, 1] the input's sign unchanged
vector BinaryOp arithmetic and ScalarExpr::Arithmetic (one rule, arithmetic_sign) Unknown +: both NonNegative → NonNegative, both NonPositive → NonPositive; * and /: same signs → NonNegative, opposite → NonPositive; -, %, ^, atan2 and anything unknown → Unknown
PromqlVectorFromScalar(expr) (vector(s)) Unknown value expression_sign(expr) (vector(1000) → NonNegative)
  • Placement: the per-algorithm fact is SummaryTypeCapability::estimates_may_flip_sign(summary_type) next to the other summary-type facts; no new trait. quantile_sign serves both quantile forms.
  • Correctness: a median of signed counters can be negative over non-negative updates; Count-Min only overestimates a non-negative count. A q outside [0, 1] is answered outside the input's values. The products and sums above keep their sign for every non-NaN result (a NaN is outside the definition). PromQL x / 1000 lowers to a Project of value / 1000 (ScalarExpr::Arithmetic), SQL value * 8 likewise, so both use the shared rule. Tests: the five sign tests listed under Verification in memo_group_properties.rs.

@ end() / @ start() lowering

  • Before / after: before (previous revision), lowered in place, differently for instant and range-evaluated queries (PromqlLowerer::range_evaluated). After, the selector or subquery X is lowered without the @ (its offset and an @ T stay in X), and TimeShift { offset_ms: 0, at: Some(End | Start) } is put above it; instant and range-evaluated queries lower alike.
  • Correctness: at_start_and_at_end_are_lifted_above_the_selector_they_anchor, at_end_and_at_start_share_the_subtree_they_anchor_and_stay_unlifted, conformance at_modifier_lowers_to_a_time_shift.

Decisions and open questions

Needs reviewer decision: None.

Decided:

Coverage design, 2026-11-10 (approved; commit bba5e221c):

  • Coverage in IR, generic over the input reference (supersedes the D1 move to stages). AggregateCoverage<R>, AggregateDefinition<R> (field input_group renamed input; unlifted_operators kept) and CoverageError are in ir/properties/aggregate_coverage.rs with the derivation, Operator::<I: OperatorInput>::aggregate_coverage, an IR rule like output_schema. OperatorInput gains type Reference: Clone + PartialEq and reference(), and keeps aggregate_coverage() (stored coverage). reference() returns Option<Self::Reference>: a rule's temporary build has none, so the wrapper InputOrTemporaryBuild<I> keeps Reference = I::Reference and still delegates stored coverage to the wrapped input. Node trees: Reference = (), no stored coverage. The memo: MemoGroupAsInput::Reference = MemoGroupId, GroupProperties::aggregate_coverage: Option<AggregateCoverage<MemoGroupId>>; merges still compare definitions with ==. SchemaDerivationError::Coverage(CoverageError) stays in ir, now ir-internal. Nothing under ir/ imports crate::stages (two doc links remain, to PhysicalProperties and PhysicalCandidate::properties).
  • Plan expands the memo's coverage into a content id. ViewCoverage is beside PhysicalASAPDAGNode in plan/physical_asap_dag.rs (the planner re-export keeps its name); stages::view_coverage(memo, group) (stages/logical/aggregate_coverage.rs, the only memo-scope glue left there) maps the group's coverage references to each input group's expression, recursively down to the Scans (ExpandedExpression, no ids), and sets definition_id to the 128-bit FNV-1a hash, in hex, of that definition's JSON (serde_f64 for non-finite floats; no new dependency); the selection is the memo's. Stage 3 setting the node field stays todo! in select_plan, which names view_coverage(memo, its group).
-// stages/logical/aggregate_coverage.rs
-pub struct AggregateDefinition { pub input_group: Option<MemoGroupId>, pub unlifted_operators: Vec<Operator<Option<MemoGroupId>>> }
-pub struct AggregateCoverage { pub(crate) definition: AggregateDefinition, pub(crate) selection: Vec<SelectionBox> }
-pub(super) fn aggregate_coverage(operator: &Operator<MemoGroupAsInput<'_>>) -> Result<AggregateCoverage, CoverageError>;
+// ir/properties/aggregate_coverage.rs
+pub struct AggregateDefinition<R> { pub input: Option<R>, pub unlifted_operators: Vec<Operator<Option<R>>> }
+pub struct AggregateCoverage<R> { pub(crate) definition: AggregateDefinition<R>, pub(crate) selection: Vec<SelectionBox> }
+impl<R: Clone> AggregateCoverage<R> { pub fn map_references<S>(self, f: impl FnMut(R) -> S) -> AggregateCoverage<S>; }
+impl<I: OperatorInput> Operator<I> { pub fn aggregate_coverage(&self) -> Result<AggregateCoverage<I::Reference>, CoverageError>; }
 pub trait OperatorInput: Sized + Clone {
+    type Reference: Clone + PartialEq;
+    fn reference(&self) -> Option<Self::Reference>;
-    fn aggregate_coverage(&self) -> Option<&AggregateCoverage>;
+    fn aggregate_coverage(&self) -> Option<&AggregateCoverage<Self::Reference>>;
 }
 pub struct GroupProperties {
-    pub aggregate_coverage: Option<AggregateCoverage>,
+    pub aggregate_coverage: Option<AggregateCoverage<MemoGroupId>>,
 }
-// plan/materialized_view.rs
-pub struct ViewCoverage { pub(crate) definition_id: String, pub(crate) selection: Vec<SelectionBox> }
+// plan/physical_asap_dag.rs
+pub struct ViewCoverage { pub(crate) definition_id: String, pub(crate) selection: Vec<SelectionBox> }
+// stages/logical/aggregate_coverage.rs
+pub fn view_coverage(memo: &Memo, group: MemoGroupId) -> Option<ViewCoverage>;
-pub use stages::{AggregateCoverage, AggregateDefinition, CoverageError};
+pub use ir::properties::{AggregateCoverage, AggregateDefinition, CoverageError};
+pub use stages::view_coverage;

Layout, 2026-11-10 (approved; commit 0de33fd94; items 1–3 of that review were replaced by the coverage design below, commit bba5e221c):

  • 4. root_groups and queries_rooted_at are Memo methods; the planner CLI's memo JSON calls memo.queries_rooted_at(group).
  • 5. sources_scanned_by (ir/operator/input.rs) is Operator::scanned_sources in ir/operator/node.rs, in the block of rules generic over OperatorInput; GroupProperties::scanned_sources and both OperatorInput impls call it.
  • 6. "summary coverage" → aggregate coverage: types/src/lib.rs, the SummaryMerge validation comment in ir/operator/asap.rs, frontend-sql/tests/sql_lowering.rs, docs/develop_docs/asap-aware-mapping-architecture.md, and the §6.6 row of asap-primitive-schema.md's section table. Left: docs/user_guide_docs/run-a-query.md's "Check summary coverage" (which summary candidates Pass 1 offers, another meaning) and planner_fuzz's campaign_writes_summary_coverage_and_drafts (the campaign's summary file).
  • 7. aggregate, aggregate_where and readout are in test_support::memo_fixtures; the memo unit tests (quantile_readout, quantile_agg) and memo_group_properties.rs (aggregate, readout) use them. quantile_agg without a filter now builds filters: vec![] (was vec![None]), as memo_group_properties did; its tests compare groups built the same way.
  • Experimental arms. Experimental SketchAlgorithm variants have no arms in point_counts_may_flip_sign: asap-types has no experimental feature (kept as is).
-pub fn root_groups(queries: &[MemoQuery]) -> BTreeSet<MemoGroupId>;
-pub fn queries_rooted_at(queries: &[MemoQuery], group: MemoGroupId) -> Vec<usize>;
+impl Memo {
+    pub fn root_groups(&self) -> BTreeSet<MemoGroupId>;
+    pub fn queries_rooted_at(&self, group: MemoGroupId) -> Vec<usize>;
+}
-// ir/operator/input.rs
-pub(crate) fn sources_scanned_by<I: OperatorInput>(operator: &Operator<I>) -> BTreeSet<Source>;
+// ir/operator/node.rs
+impl<I: OperatorInput> Operator<I> {
+    pub(crate) fn scanned_sources(&self) -> BTreeSet<Source>;
+}
 // test_support::memo_fixtures
+pub fn aggregate(reduction: Reduction, measure: AggIntent, child: MemoGroupId) -> Operator<MemoGroupId>;
+pub fn aggregate_where(reduction: Reduction, measure: AggIntent, filter: Option<Predicate<MemoGroupId>>, child: MemoGroupId) -> Operator<MemoGroupId>;
+pub fn readout(query: SketchStatistic, state: MemoGroupId) -> Operator<MemoGroupId>;

Sign bit, readout dtype, view coverage, 2026-11-10 (approved; commit 288cd70d6):

  • 1. ValueSign is the IEEE sign bit. NonNegative: every non-NaN value has its sign bit clear; NonPositive: set. sign_of_number: NaN → Unknown, is_sign_negative() → NonPositive, else NonNegative (-0.0 is NonPositive). Rules checked under it: + (two clear or two set sign bits keep it), * and / (XOR of sign bits, exact in IEEE), negation (flips), abs (clears), min / max / sum / avg / quantile interpolation (combine values of one sign bit), rate / irate / increase over a non-negative counter (x − x is +0): all hold, none adjusted. The concepts doc (docs(concepts): memo groups, properties, requirements, per-stage interfaces and rules #1224) states the old "≥ 0" definition and needs the sign-bit wording when it is synced.
  • 2. SummaryTypeCapability::point_counts_may_flip_sign (was estimates_may_flip_sign), pub(crate), an exhaustive match over SketchAlgorithm without _, same results. No arms for asap_sketchlib's experimental variants (CmsHll, MicroCm, UniformSampling): the planner never builds asap_sketchlib with experimental, and asap-types has no such feature, so #[cfg(feature = "experimental")] there is an unknown-cfg warning (an error under -D warnings); adding the feature to asap-types makes --all-features builds enable it and breaks the other exhaustive matches (parameter_list, asap-accuracy's guarantee table). The doc says so, as parameter_list already does by omission.
  • 3. View coverage on the plan node. PhysicalASAPDAGNode::view_coverage: Option<ViewCoverage> (serde default, skipped when None), set when the plan is built from the memo (Stage 3, where the node's group is known); select_plan's todo! names it: "… each materialized view node's view_coverage from its memo group's aggregate_coverage, with definition_id the content id of the expanded definition". The views code reads it (ViewWithoutCoverage when absent); coverage_of, the operator_nodes() IR built for it and coverage_by_node are deleted; operator_nodes is now test-only (query_roots reads it). No wire-version bump: the field is optional and absent from every fixture.
  • 4. readout_signs passes the state column's type to its rule (value(sign, &field.dtype)); the separate state-field lookup is deleted.
  • 5. Test-table rows drop the "(was Unknown)" notes.
  • 6. The content id is defined once, on ViewCoverage::definition_id; asap-primitive-schema.md and physical-planning-and-deployment.md link to it.
 pub enum ValueSign {
-    /// Every non-NaN value is ≥ 0.
+    /// Every non-NaN value has its sign bit clear (`+0.0`, positive, `+inf`).
     NonNegative,
-    /// Every non-NaN value is ≤ 0.
+    /// Every non-NaN value has its sign bit set (`-0.0`, negative, `-inf`).
     NonPositive,
     Unknown,
 }
 impl SummaryTypeCapability {
-    pub fn estimates_may_flip_sign(summary_type: &FieldDataType) -> bool;
+    pub(crate) fn point_counts_may_flip_sign(summary_type: &FieldDataType) -> bool;
 }
 pub struct PhysicalASAPDAGNode {
     …
     #[serde(skip)]
     pub selected_physical_candidate: Option<PhysicalCandidateId>,
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub view_coverage: Option<ViewCoverage>,
 }
 impl PhysicalASAPDAG {
-    pub(crate) fn operator_nodes(&self) -> Result<HashMap<PhysicalASAPNodeId, Rc<OperatorNode>>, PhysicalASAPDAGValidationError>;
+    #[cfg(any(test, feature = "test-support"))]
+    fn operator_nodes(&self) -> Result<HashMap<PhysicalASAPNodeId, Rc<OperatorNode>>, PhysicalASAPDAGValidationError>;
 }
-fn coverage_of(
-    id: PhysicalASAPNodeId,
-    inputs: &MaterializedViewInputs,
-    ir: &HashMap<PhysicalASAPNodeId, Rc<OperatorNode>>,
-    coverage_by_node: &HashMap<PhysicalASAPNodeId, ViewCoverage>,
-) -> Result<ViewCoverage, CoverageError>;

Definition identity, 2026-11-10 (approved; commit 73785fa07): two scopes, two identities.

  • Memo scope. Coverages compare by AggregateDefinition == (it holds MemoGroupIds); nothing in the memo needs an id. AggregateCoverage::definition_id() (an FNV hash of the definition's Debug text, ids included) and test_support::definition_id are deleted; states_with_equal_definitions_and_disjoint_selections_merge compares the definitions only.
  • Plan scope. ViewCoverage::definition_id is the content id of what the view computes: its definition with every input group expanded recursively to its expression down to the Scans, with no group or node ids, canonically serialized and hashed; equal across plans and processes that compute the same thing, different when they do not. coverage_of (private, todo! for PR 6) now returns the ViewCoverage: "Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 PR 6: the view's coverage from its memo group's aggregate_coverage, with definition_id the content id of the expanded definition". Design docs updated: asap-primitive-schema.md (the AggregateCoverage interface twice, worked example case 2, the fields list) and physical-planning-and-deployment.md (the read table, the matching paragraph, the materialized-views mapping, the store-keyed test row).
  • Fixtures. materialized_view_fixtures/*.json hold definition_id as opaque hex strings that tests only use as keys and compare; they stay.
  • Full-range interval (doc only). Constraint::Interval documents that (Unbounded, Unbounded) is still a constraint (it excludes NULL and NaN rows), not the absence of one.

Sign rules, 2026-11-10 (approved; commit c2f23d760):

  • Definition. NonNegative: every non-NaN value is ≥ 0; NonPositive: every non-NaN value is ≤ 0.
  • 1. Count Sketch and UnivMon (and variants) point-count and top-k readouts are Unknown; Count-Min keeps the state's sign. The fact is SummaryTypeCapability::estimates_may_flip_sign.
  • 2. A quantile with q outside [0, 1] is Unknown, for AggIntent::Quantile and SketchStatistic::Quantile.
  • 3. BinaryOp Arithmetic: the rule in the table under Algorithm changes, one shared fn (arithmetic_sign).
  • 4. PromqlVectorFromScalar(expr): the value column's sign is expression_sign(expr, &[]).
  • 5. ScalarExpr arithmetic in expression_sign uses arithmetic_sign.
  • No other behavior change.

Third review, 2026-11-10 (approved; commit f68d414aa; D2, deleting topological_order, set_candidate_input_requirements and root_groups, rejected: later PRs use them, first set_candidate_input_requirements in PR 3 (#1232) and PR 4, topological_order in PR 4 (Pass 2 requirements) and PR 6 (Stage 3 cost), root_groups in PR 4 (dead candidates) and PR 5 (pruning)):

  • D1 (option A; superseded by the coverage design, bba5e221c). AggregateCoverage, AggregateDefinition and CoverageError are in stages/logical/aggregate_coverage.rs with their derivation, the free function aggregate_coverage(operator) (pub(super)). ir/properties/selection.rs (renamed from summary_coverage.rs) keeps only the IR selection algebra: SelectionBox, Constraint, ColumnIdentity, IntervalConstraintLimit, constraint_of/flip, union/intersect/disjoint. Dependency direction: ir/operator/input.rs (OperatorInput::aggregate_coverage), ir/schema/error.rs (SchemaDerivationError::Coverage) and plan/materialized_view.rs import the type from stages (ir → stages, as for MemoGroupId already); stages/logical/aggregate_coverage.rs imports the selection algebra from ir/properties/selection.rs.
  • 1. An expression is serialized once with its group ids (serialized_expression, .expect); the hash reads that string; a group is equal by typed ==, then the cached strings (the same_value semantics for NaN and -0). operator_and_arguments, the extra same_expression path, the separate children hashing and the Option plumbing are deleted.
  • 2. A duplicate candidate is found by typed == first; the serializations are compared only when typed-equal.
  • 3. validate_expression_over_input_groups returns a merge's coverage, which derive_group_properties reuses: a merge's coverage is derived once.
  • 4. OutputColumnSource is pub(crate) and no longer re-exported; aggregate_output_columns is deleted; callers use aggregate_output.
  • 5–6. ranked_row_projection's doc restored; stale docs: find_or_add_group, ir/properties/mod.rs, the coverage error message, input.rs, materialized_view.rs, the design doc's §4.2 and §6.6 titles, AggregateDefinition's field docs (also exact Aggregate), the memo_aggregate_coverage.rs header.
  • 7. ValueSign is in column_properties.rs with its derivation, column_signs (was output_column_sign).
  • 8. Covered by D1.
  • 9. MemoGroup::parents and ::queries are pub(super) (read through parent_groups() and reached_by_queries()); Memo::queries is private, read through Memo::queries() (Stage 0, Pass 1, devtools and integration tests updated).
  • 10. The memo fixtures are in types/src/test_support/memo_fixtures.rs (feature test-support, and cfg(test)), shared by stages/logical/memo/tests.rs and the memo integration tests; the unit tests' scan and kll_state copies are deleted. Kept: the unit tests' service_scan and Stage 0's m scan, whose [ts, service, value, latency] column order the shared scan ([ts, value, columns…]) does not produce. group_properties_come_from_input_groups uses its file's helpers.
  • 11. aggregate_schema.rs computes a measure's input column once per measure on both paths.
  • 12. Source::is_time_series replaces scans_only_time_series; the top-k schema and Pass 1 read .iter().all(Source::is_time_series).

Second review, 2026-11-10 (approved; commit 1ae8acf91; D3, renaming column_sign away, rejected because PRs 4, 5 and 8 read it):

  • D1. OperatorInput::scanned_sources() (like aggregate_coverage()): a memo group returns its stored GroupProperties::scanned_sources; a node tree and a rule's temporary build compute it with sources_scanned_by. One scans_only_time_series, a free function over a set in ir/operator/input.rs, read by the top-k schema and Pass 1; GroupProperties::scans_only_time_series and the top-k schema's own walk are deleted.
  • D2. SummaryCoverage → AggregateCoverage, SummaryStateDefinition → AggregateDefinition, CoverageError::NotSummary → NoCoverage; the type's file is ir/properties/aggregate_coverage.rs and the derivation's stages/logical/aggregate_coverage.rs (same name, two scopes: the type in the IR's properties, its derivation over memo groups beside column_properties.rs); tests/memo_aggregate_coverage.rs.
  • D4. constraint_of and flip are beside Constraint in ir/properties/aggregate_coverage.rs; dedup and interval are private again. Crate visibility left: constraint_of, union, Constraint::intersect, Constraint::disjoint, SelectionBox::disjoint, AggregateCoverage::definition_id.
  • M1. One expression equality: same_serialized_expression (typed equality confirmed by the serialized operator, each computed once); same_expression is it with both sides serialized.
  • M2. OutputColumnSource::Measure { index, input_column }: the signs read the measure's input column from the layout; the re-derived probe column and measure input are deleted.
  • M3. summary_agg_layout (schema and sources in one pass) replaces summary_agg_schema and summary_agg_columns.
  • M4. validate_expression_over_input_groups returns the schema and derives coverage only for a SummaryMerge (its check); derive_group_properties derives the coverage itself.
  • M5–M6. Stale docs: asap-primitive-schema.md (@ T lifted, @ start()/@ end() not), the memo_group_properties.rs header, this body; doc links to GroupProperties::shareable_when_equal by full path (cargo doc checked).
  • M7. column_sign → column_signs; CandidateError::Schema → NotValidOverInputGroups; depends_on_it inlined.
  • M8. MemoGroupAsInput, its fields, group_properties(), output_column_sign and derive_from_inputs are pub(super) (only stages/logical reads them).
  • M9. Stage 0's hash-consing tests are in stages/logical/memo/tests.rs, built through find_or_add_group; a_scan_reads_the_declared_column_signs keeps one assert that memo_of_queries passes the data's facts through.
  • M10. Memo has no Default; tests use Memo::new(&DataWorkload::default()).

Layout decisions, 2026-11-10:

  • A (superseded the same day by option 1, review of memo/mod.rs:40). Memo holds the data workload's facts per source, fixed at construction (Memo::new(data)) and read-only afterwards, instead of signs declared during Stage 0's walk. Reason: the memo's data says what it is (the data workload's source facts), not a by-product of Stage 0's walk, and nothing mutates it after construction; Stage 0 needs no walk for it at all. A Scan group's signs read the facts where they are derived, a histogram's or summary's series through its family (series_facts, the one lookup DataWorkload::series_facts also uses) and a declared counter's value NonNegative. Stage 0's declared_column_signs and Memo::declare_column_signs are deleted. Compared with PR 8's final shape: the field is the same (source_facts: BTreeMap<String, SourceFacts>); PR 8 builds it in Stage 0 keyed by the scanned series name (family already resolved, counters' value already filled), and takes the map (Memo::new(source_facts)). Here the map is DataWorkload::sources keyed by family and resolved at the Scan, so at restack PR 8's readers of source_facts (max_distinct_per_evaluation, the lineage-keyed sign lookup) go through series_facts.
  • B (review of group.rs:83). GroupProperties::scanned_sources: BTreeSet<Source> with scans_only_time_series() replaces reads_only_time_series; OperatorInput::reads_only_time_series and its three impls are deleted. One rule derives it, scanned_sources in ir/operator/input.rs: a memo group stores it from its input groups' (group.rs), and the top-k schema in ir/schema/topk.rs, which also derives node trees' schemas (no group property there), applies the same rule down the tree. Pass 1 reads the aggregate group's scans_only_time_series(). Source gains PartialOrd, Ord (two String variants) for the set. For the restack: PR 8's Memo::max_distinct_per_evaluation walks the scans below a group; it could sum over scanned_sources instead (note: the walk counts each Scan group, the set each source once).
  • B1. SummaryCoverage's derivation over memo groups (derive_from_inputs and its walk helpers) is in stages/logical/aggregate_coverage.rs, next to column_properties.rs: like the column signs, it is a group property read from the input groups. The SummaryCoverage type and its other methods stay in ir/properties/summary_coverage.rs; the test of constraint_of moves with it. Visibility the move forces: interval, dedup, union, Constraint::intersect, Constraint::disjoint and SelectionBox::disjoint are pub(crate); the unused stages::MemoGroupAsInput re-export is deleted.
  • B4. aggregate_schema.rs tests aggregate_output_columns for by, without, count_values, absent and a per-entity reduction, each checking one source per output column.
  • C (review of group.rs:113). check_node_rules is validate_expression_over_input_groups: it checks that an expression is valid over its input groups (its output schema derives, its inputs pass validation, a merge reads only states it may merge); "node rules" named neither what nor over what. check_candidate_expression had one caller, so it is inlined into Memo::add_candidate. The CandidateError::Schema message and docs say what is checked; Stage 0's test is memo_of_queries_returns_the_error_of_a_node_invalid_over_its_input_groups.
  • B2 (moving the expression check out of group.rs) and B3 (the delivers_on tests' location): no change.

Review fixes, 2026-11-10 (approved; commit 77fc121d3):

  • 1. One aggregate layout. column_properties.rs re-derived the by/without/per-entity/count_values/absent layout aggregate_schema.rs owns, faked a Sum measure for summary states, and re-derived the chain layout. Now aggregate_output_columns (one pass with aggregate_output_schema) gives each output column's OutputColumnSource, which the signs read; a summary state's layout is summary_agg_columns beside summary_agg_schema; the chain keeps the build's layout without its time column, after window_end. A measure's input value is its first input column, else the column aggregate_output_schema types it by (the sample value, else the first column not kept). Bug fixed: count_values without (host) ("version", v) got a sign for a label column a without aggregate does not have (one sign too many, a debug_assert panic); regression test count_values_without_has_one_sign_per_output_column (failed before the fix, passes after).
  • 2. Look up before deriving. find_or_add_group looks the expression up first and derives properties only on a miss (a hit equals a group that was validated; only shareable groups are listed, and an equal expression is as shareable). add_candidate checks for a duplicate first, then only validates the expression over its input groups (validate_expression_over_input_groups: schema, input validation, merge coverage), not the group properties.
  • 3. of_summary_agg takes a SummaryAgg or a SummaryTumblingWindowChain; the chain arm that rebuilt a SummaryAgg is deleted.
  • 4. combined_with: one constraint intersection for the three lifted-column sites.
  • 5. of_window_selection walks to the chain with selected_from; a Filter on the way still gives no coverage.
  • 6. SourceColumn deleted (it was never serialized; supersedes Q1 below): the declared signs are keyed by source, then column (the shape of SourceFacts::column_signs), and Stage 0 declares a source's signs as its one walk (group_of) reaches the source's Scan (Memo::declare_column_signs replaces Memo::new); the separate walk is deleted.
  • 7. tests/memo_fixtures.rs holds scan(metric, columns, predicates), filter, range(width_ms, child), shift(offset_ms, at, child) and state(summary_type, child) for memo_group_properties.rs and memo_summary_coverage.rs (each keeps a one-line wrapper for its source, series / latency). group_properties_come_from_input_groups moved to memo_group_properties.rs, without its unshareable-LIMIT assertion. The unit tests in stages/logical/memo/tests.rs keep their local scan, kll_state and unordered_limit: unit tests in src/ cannot reach the integration tests' module.
  • a. A group's serialized operator is computed once and stored with its hash bucket entry; the new expression is serialized once per lookup.
  • b. depends_on_evaluation_time matches the operator's own reads, then checks the input groups once.
  • c. The SQL timestamp cast target is dtype_to_arrow(&DataType::Timestamp).
  • Visibility forced by the restack's moves (not by the fixes): LogicalProperties::parameters and Memo::source_facts are pub(super) (read by memo/hash_consing.rs, memo/tests.rs and group.rs), same_expression_and_logical_properties is pub(super) in memo/hash_consing.rs.
Item Decision Date
A group is its expression; both methods return Result; one validity check of an expression over its input groups for groups and candidates as implemented 2026-11-06
Exact Aggregate coverage, time as an interval with IntervalConstraintLimit, @ T fixed, SQL timestamp filters, DisjointInTime, chain coverage and the merge over a chain's windows as implemented 2026-11-07
Intervals (t − r, t] as implemented 2026-11-08
@ end()/@ start() lifted above the anchored subtree as TimeShift { at }(X), X without @; range_evaluated deleted; coverage leaves the TimeShift(@) unlifted as implemented; rejected: lowering in place (no sharing, #1251) 2026-11-08
ValueSign / column_sign derived per operator, Scan from declared source facts as implemented 2026-11-08
depends_on_evaluation_time derived in Stage 0 as implemented 2026-11-09
P6: the memo stores the declared signs (Memo::new, private declared_column_signs) as implemented 2026-11-09
MemoQuery::accuracy_target: Option<_> (P17), memo_of_queries(roots, accuracy_targets, data) (P5) as implemented 2026-11-09
Q1 (N2): SourceColumn { pub source: String, pub column: String } stays beside Source, re-exported by the planner facade as the SelectionBox key as implemented; a simpler alternative was asked for, keeping it was recommended and approved 2026-11-09
Q2 (R1): the extra value-sign rows (literals, casts, counts/spreads NonNegative, rates only over NonNegative input, comparisons, readouts of distinct count / L2 norm / entropy) as implemented 2026-11-09
Q3 (R2): time() / now() set depends_on_evaluation_time as implemented 2026-11-09
Q4 (R3): @ lifted above the selector only as implemented 2026-11-09
N7: GroupProperties::reads_only_time_series: bool, derived once in derive_group_properties; OperatorInput::reads_only_time_series the same rule over node trees; the recursive walks in ir/schema/topk.rs and Pass 1 deleted. Example: PromQL Scan latency → true; TimeRange(1m, Scan latency) → true; SQL Scan requests → false; Join(requests, hosts) → false (a SQL join reads relations only, so no join of a time series and a table exists to test) as implemented; rejected: a pub reads_only_time_series function with its own recursion 2026-11-09

Deleted besides the interfaces above: PromqlLowerer::range_evaluated and the range_evaluated parameter of lower_query, vs_parts, time_shift, extract_matrix (previous revision; replaced by anchored / selector); tests at_end_and_at_start_share_coverage_only_in_an_instant_query (types), at_start_and_at_end_stay_only_in_a_range_query (promql lib) and Stage 0's differing_accuracy_targets_are_two_groups_over_one_scan_group (measures carry no target since PR 0; replaced by queries_that_differ_only_in_their_accuracy_target_are_one_group); the todo!()s for column_sign, depends_on_evaluation_time, Stage 0, add_query, and #[allow(unreachable_code)] on derive_group_properties.

Verification

Coverage design (bba5e221c): 680 / 0 / 0, the same tests by name as 0de33fd94 plus three new ones in memo_aggregate_coverage.rs: a_view_coverage_is_equal_across_memos_whatever_the_group_ids (the latency scan is a different group id in each memo, the ids are equal), a_view_coverage_over_another_source_has_another_definition_id, a_group_without_aggregate_coverage_has_no_view_coverage. No test moved; adapted: memo_aggregate_coverage.rs imports the coverage types from ir::properties and reads definition().input. cargo check --workspace --all-features --tests, fmt --check, cargo clippy --workspace --all-features --all-targets -D warnings clean; cargo doc -p asap-types reports only the four older links.

Layout (0de33fd94): 677 / 0 / 0, the same tests by name as 288cd70d6 (no test moved; the memo unit tests and memo_group_properties call the shared fixtures). cargo check --workspace --all-features --tests, fmt --check, cargo clippy --workspace --all-features --all-targets -D warnings clean; cargo doc -p asap-types reports only the four older links.

Sign bit, view coverage (288cd70d6): 677 / 0 / 0, the same tests by name as 73785fa07. New case (failing first): scalar_arithmetic_follows_its_operands_signs row G7, latency / -0 — on 73785fa07 left [Unknown, NonNegative, Unknown], right [Unknown, NonPositive, Unknown]; passes. Adapted: the same test's and vector_arithmetic_follows_its_operands_signs's tables (history notes dropped); every PhysicalASAPDAGNode literal in tests (types/tests/physical_export.rs, 12 executor test files) gets view_coverage: None. cargo check --workspace --all-features --tests, fmt --check and cargo clippy --workspace --all-features --all-targets -D warnings clean; cargo doc -p asap-types reports only the four older links. View-path tests: integration::materialized_views::* and kll_tumbling_window_execution::* give the same results as on 73785fa07 (1 passed; 4 failing on both, on the retention todo! and a fixture's deserialization, before the views code).

Definition identity (73785fa07): 677 / 0 / 0, the same tests by name as c2f23d760; adapted: memo_aggregate_coverage::states_with_equal_definitions_and_disjoint_selections_merge (the definition_id equality assert removed, the definition == assert kept). cargo check --workspace --tests with no warnings, fmt --check and clippy --all-targets -D warnings (the test-set crates plus asap-integration-tests, which reads ViewCoverage::definition_id) clean.

Sign rules (c2f23d760): 677 / 0 / 0, the same tests by name as f68d414aa plus four new ones in memo_group_properties.rs, each failing on f68d414aa with only the test file applied:

  • a_count_sketch_point_count_has_no_sign_a_count_min_one_keeps_it (item 1): before, left NonNegative, right Unknown; passes.
  • a_quantile_outside_the_unit_interval_has_no_sign (item 2): before, left NonNegative, right Unknown; passes.
  • vector_arithmetic_follows_its_operands_signs (items 3, 4: the ratio of quantiles, +, -, vector(1000), vector(-1), products): before, the ratio's value left Unknown, right NonNegative; passes.
  • scalar_arithmetic_follows_its_operands_signs (item 5: PromQL x / 1000, x * 8, SQL value * 8, x - x, x * -8): before, x / 1000 left [Unknown, Unknown, Unknown], right [Unknown, NonNegative, Unknown]; passes.
  • scans_read_declared_signs_and_row_operators_keep_or_set_them: its G9 (value * 2 over non-negative latency) now expects NonNegative (was Unknown), and a G10 row checks value - value is Unknown.

cargo check, fmt --check, clippy --all-targets -D warnings clean; cargo doc -p asap-types reports only the four links older than this PR.

Third review (f68d414aa): 673 / 0 / 0, the same tests by name as 1ae8acf91 (a_value_comparison_is_an_interval_between_fixed_values and the selection tests run from ir::properties::selection); cargo check, fmt, clippy -D warnings clean; cargo doc -p asap-types with -D warnings reports only links older than this PR. Second review (1ae8acf91): 673 / 0 / 0, the same tests as 0fef10001 by name except equal_keyless_nodes_that_return_the_same_rows_are_one_group → equal_keyless_expressions_that_return_the_same_rows_are_one_group; moved from Stage 0 to stages::logical::memo::tests (same names): aggregates_that_differ_in_an_argument_are_different_groups, dedup_is_one_group_only_on_every_column, unordered_limits_and_their_parents_are_different_groups, limits_over_a_sort_with_their_partitions_are_one_group, signed_zero_and_non_finite_literals_are_different_groups; memo_summary_coverage::* → memo_aggregate_coverage::*. cargo check, fmt, clippy -D warnings clean; cargo doc -p asap-types with -D warnings reports only links older than this PR.

Restack and review fixes (2026-11-10): cargo test -p asap-types -p asap-stage0-memo -p asap-stage1-pass1-per-computation-candidates -p asap-stage1-pass2-reuse-candidates -p asap-frontend-common -p asap-frontend-promql -p asap-frontend-sql --all-features --no-fail-fast: previous head 36312cbaf 667 / 0 / 0; restacked commit 3cabdc3b5 667 / 0 / 0, the same tests by name (the two delivers_on tests run, no longer ignored); head 77fc121d3 668 / 0 / 0: the same plus count_values_without_has_one_sign_per_output_column, which failed on 3cabdc3b5 (one sign per output column of Aggregate, left 2, right 1) and passes; group_properties_come_from_input_groups runs from its new file. Head 0fef10001 (expression check renamed, check_candidate_expression inlined): 673 / 0 / 0, the same tests by name except memo_of_queries_returns_the_error_of_a_node_that_breaks_the_node_rules → memo_of_queries_returns_the_error_of_a_node_invalid_over_its_input_groups. Head f458196ba (source facts, scanned sources): 673 / 0 / 0, the same tests by name, a_group_reads_only_time_series_when_every_scan_below_reads_one renamed a_group_scans_the_sources_of_the_scans_below_it (it now also checks the sets); Stage 0's a_scan_reads_the_declared_column_signs (histogram family and counter) passes over Memo::new(data). Head c71be0331 (B1, B4): 673 / 0 / 0, the same plus the five aggregate_output_columns tests; the moved a_value_comparison_is_an_interval_between_fixed_values runs from aggregate_coverage.rs. cargo check --workspace --tests with no warnings, cargo clippy --workspace --all-targets -- -D warnings and cargo fmt --check clean at each head. The table below is from the earlier head.

cargo test -p <crate> --all-features --no-fail-fast. The full table below was measured at 8095a5145 (this PR before N7 added reads_only_time_series); at the head 49a145c26, asap-types and asap-stage0-memo were rerun: all pass, including a_group_reads_only_time_series_when_every_scan_below_reads_one. Every failure panics at a later PR's todo!() or is the stale fixture (PR 9); planning now stops at Pass 1's target discovery (PR 3). By test name against the base PR 1 (8030d3fe6): 0 newly failing; 45 new tests pass (asap-types 30, asap-stage0-memo 15, one each in asap-frontend-promql and asap-frontend-sql); PR 1's delivers_on_a_chain_when_its_width_divides_the_context_times and delivers_on_a_stored_result_on_its_group_grid are ignored there (their memo fixture needs this PR's find_or_add_group); this PR removes the two #[ignore] attributes and they run and pass: at the head 43f81fe01 (restacked onto PR 1 c2e717bf9; tree identical to 49a145c26) cargo test -p asap-types --no-fail-fast: 195 passed, 0 failed, 0 ignored. Tests that failed at PR 2's todo!() on PR 1 now fail at PR 3's (target discovery) or PR 6's. cargo clippy --workspace --all-targets --all-features -- -D warnings and cargo fmt --all --check clean; crate_graph: 6 passed.

Scope Passed / failed / ignored Newly failing vs base Notes (fails at)
asap-types 194 / 0 / 0 0
asap-accuracy 60 / 0 / 0 0
asap-cost 31 / 1 / 0 0 PR 6: check θ, which the exported node carries: 1
asap-frontend-common 49 / 0 / 0 0
asap-sql-function-catalog 10 / 0 / 0 0
metricsql_common 3 / 0 / 0 0
asap-frontend-promql 218 / 0 / 0 0
asap-frontend-metricsql 15 / 0 / 0 0
asap-frontend-sql 174 / 0 / 0 0
asap-executor 210 / 63 / 1 0 PR 6: θ on the exported node 32, Hydra grid sizes 13, retention_ms = $h_v$ 8, weighted-frequency θ 7 + 2; PR 3: target discovery over memo groups 1
asap-planner 8 / 18 / 1 0 PR 3: target discovery: 18
asap-devtools 37 / 56 / 3 0 PR 3: 48; PR 9: a group's node from its expression 5, label a group by its expression 1; asap_planner subprocess stopping at a todo!(): 2
asap-integration-tests 83 / 51 / 4 0 PR 3: 47; PR 6 retention_ms: 3; stale old-wire fixture (PR 9): 1
asap-stage3-plan-selection 3 / 0 / 0 0
asap-stage0-memo 15 / 0 / 0 0 all new
asap-stage2-physical-optimization 0 / 0 / 0 0
asap-stage1-pass2-reuse-candidates 0 / 0 / 0 0
asap-stage1-pass1-per-computation-candidates 11 / 0 / 0 0
Total 1121 / 189 / 9 0 at 8095a5145

New or rewritten tests:

  • memo_group_properties.rs (6): sign table rows for row operators (scans_read_declared_signs_and_row_operators_keep_or_set_them), aggregates (aggregates_keep_their_keys_and_derive_their_values), states/merges/readouts (states_record_their_input_sign_and_readouts_read_it), the doc's top-k example (a_top_k_ranks_values_of_its_inputs_sign), depends_on_evaluation_time true/false (a_result_depends_on_the_evaluation_time_under_a_time_range_or_now), a_group_reads_only_time_series_when_every_scan_below_reads_one.
  • at_end_and_at_start_share_the_subtree_they_anchor_and_stay_unlifted (types); promql lib at_start_and_at_end_are_lifted_above_the_selector_they_anchor; conformance at_modifier_lowers_to_a_time_shift (updated).
  • Stage 0: each_query_enters_with_the_values_one_evaluation_returns, a_scan_reads_the_declared_column_signs, queries_that_differ_only_in_their_accuracy_target_are_one_group, memo_of_queries_returns_the_error_of_a_node_that_breaks_the_node_rules.
  • stages::logical::memo::tests (was memo_tests in candidates.rs): find_or_add_group_* (equal expression, different inputs, unshareable, parents, no schema) and add_candidate_* (duplicate, parents and queries, output differs, inputs fail validation).
  • memo_aggregate_coverage.rs (13): coverage intervals, @ T, chains, merges over windows, counter disjointness.

End-to-end evidence: not applicable (planning stops at Pass 1's target discovery until PR 3).

Self-review: no trait or wrapper added (new: Memo::new(data) and private source_facts, private column_properties.rs functions, MemoGroupAsInput::group_properties() (pub(super), shared by the two derivations), and from the review OutputColumnSource with aggregate_output_columns); the sign derivation is not generic over OperatorInput because only memo groups have signs. The output-column layouts are aggregate_schema.rs's alone (review 1); PR 2b's column_lineage should read the same OutputColumnSource. No dead code (clippy -D warnings; the range_evaluated plumbing and stale #1207 PR 2 comments are gone). Fixed in this round: the coverage walk passed TimeShift { at: End | Start } and returned no interval for the whole path; it now stops there.

Follow-ups

@zzylol zzylol added optimizer Workload optimization and plan selection refactor labels Nov 6, 2026
@zzylol
zzylol force-pushed the refactor/1207-properties branch 2 times, most recently from 60a16ee to 107a77b Compare November 6, 2026 19:12
@zzylol
zzylol force-pushed the refactor/1207-memo branch from 9e7e404 to 7ea9152 Compare November 6, 2026 19:14
@zzylol
zzylol force-pushed the refactor/1207-properties branch 2 times, most recently from c7a0bc9 to 9e5ce62 Compare November 6, 2026 21:38
@zzylol
zzylol force-pushed the refactor/1207-memo branch from 54d8b30 to 60c5cfe Compare November 6, 2026 21:38
@zzylol
zzylol force-pushed the refactor/1207-properties branch from 9e5ce62 to 96d3457 Compare November 6, 2026 22:11
@zzylol
zzylol force-pushed the refactor/1207-memo branch from 60c5cfe to 4915a82 Compare November 6, 2026 22:11
@zzylol zzylol changed the title refactor(stage0, types): memo hash-consing and candidate storage (#1207 PR 2) refactor(stage0, types): memo hash-consing, candidate storage and group-coverage tests (#1207 PR 2) Nov 6, 2026
@zzylol
zzylol force-pushed the refactor/1207-properties branch from 96d3457 to b250852 Compare November 6, 2026 22:46
@zzylol
zzylol force-pushed the refactor/1207-memo branch from 4915a82 to e710906 Compare November 6, 2026 22:46
@zzylol
zzylol force-pushed the refactor/1207-properties branch 2 times, most recently from 7ceb7c3 to 12a4b96 Compare November 7, 2026 20:55
@zzylol
zzylol force-pushed the refactor/1207-memo branch from e710906 to ef0e774 Compare November 7, 2026 21:38
@zzylol zzylol changed the title refactor(stage0, types): memo hash-consing, candidate storage and group-coverage tests (#1207 PR 2) refactor(stage0, types): memo hash-consing, candidate storage, and aggregate coverage with timestamp intervals and chains (#1207 PR 2) Nov 7, 2026
@zzylol
zzylol force-pushed the refactor/1207-properties branch from 12a4b96 to cf93b37 Compare November 8, 2026 15:10
@zzylol
zzylol force-pushed the refactor/1207-memo branch from 30670f8 to fe0ff33 Compare November 8, 2026 15:18
@zzylol
zzylol force-pushed the refactor/1207-properties branch 3 times, most recently from 3c28259 to 8030d3f Compare November 9, 2026 04:33
@zzylol
zzylol force-pushed the refactor/1207-memo branch from fe0ff33 to 8095a51 Compare November 9, 2026 04:40
@zzylol zzylol changed the title refactor(stage0, types): memo hash-consing, candidate storage, and aggregate coverage with timestamp intervals and chains (#1207 PR 2) feat(stage0, types): memo hash-consing, group properties (column_sign, depends_on_evaluation_time, aggregate_coverage), @ end()/@ start() above their subtree (#1207 PR 2) Nov 9, 2026
@zzylol
zzylol force-pushed the refactor/1207-memo branch from 8095a51 to 49a145c Compare November 9, 2026 16:33
…ookups before derivation (#1207 PR 2 review)

1. Aggregate layout: aggregate_output_schema and the new
   aggregate_output_columns derive the schema and, per output column, its
   OutputColumnSource (InputColumn, Measure, AddedLabel) in one pass; the
   column signs read the sources, and a summary state's layout is
   summary_agg_columns, beside summary_agg_schema. Fixes a `without`
   count_values aggregate getting a sign for a label column its schema does
   not have (count_values_without_has_one_sign_per_output_column).
2. find_or_add_group looks the expression up before deriving its
   properties; add_candidate checks for a duplicate first and then only the
   node rules (check_candidate_expression), not the group properties.
3. A chain's coverage is of_summary_agg over the chain itself.
4. combined_with: one constraint intersection for the lifted columns.
5. of_window_selection walks to the chain with selected_from.
6. SourceColumn deleted: the memo's declared signs are keyed by source,
   then column, declared by Stage 0 as its walk reaches each source's
   Scan (Memo::declare_column_signs replaces Memo::new).
7. tests/memo_fixtures.rs: scan, filter, range, shift and state shared by
   the two memo test files; group_properties_come_from_input_groups moves
   to memo_group_properties.rs without its unshareable-LIMIT assertion.
a. A group's serialized operator is computed once and stored with its
   hash bucket entry.
b. depends_on_evaluation_time checks the input groups once.
c. The SQL timestamp cast target is dtype_to_arrow(&DataType::Timestamp).
@zzylol
zzylol force-pushed the refactor/1207-memo branch from 36312cb to 77fc121 Compare November 10, 2026 16:28
… properties; tests of the aggregate layout (#1207 PR 2 review)

- SummaryCoverage::derive_from_inputs and the walk helpers it uses move
  from ir/properties/summary_coverage.rs to the new
  stages/logical/aggregate_coverage.rs, next to column_properties.rs; the
  SummaryCoverage type and its other methods stay in ir/properties. The
  test of constraint_of moves with it.
- aggregate_schema.rs tests aggregate_output_columns for by, without,
  count_values, absent and a per-entity reduction.
@zzylol
zzylol marked this pull request as ready for review November 10, 2026 17:15
…cts; groups record the sources they scan (#1207 PR 2 review)

- Memo::new(data) keeps the data workload's facts per source, fixed at
  construction; a Scan group's column signs read them, a histogram's or
  summary's series through its family (series_facts, now shared with
  DataWorkload::series_facts) and a counter's value non-negative.
  Memo::declare_column_signs and Stage 0's declared_column_signs are
  deleted.
- GroupProperties::scanned_sources replaces reads_only_time_series, with
  scans_only_time_series(); OperatorInput::reads_only_time_series and its
  impls are deleted. One rule, scanned_sources in ir/operator/input.rs,
  derives it: stored per memo group, applied down a node tree by the
  top-k schema. Pass 1 reads the aggregate group's property. Source
  derives Ord.
…1207 PR 2 review)

check_node_rules -> validate_expression_over_input_groups (an expression
over input groups is valid when its output schema derives, its inputs pass
validation, and a merge reads only states it may merge). Its one other
caller, check_candidate_expression, is inlined into Memo::add_candidate.
Docs, the CandidateError::Schema message and a Stage 0 test name say what
is checked instead of "node rules".
@zzylol
zzylol force-pushed the refactor/1207-memo branch from c2cc5a1 to 0fef100 Compare November 10, 2026 17:30
…e names, one layout pass, one equality (#1207 PR 2 review)

- D1: OperatorInput::scanned_sources (a memo group's stored set, recomputed
  for node trees by sources_scanned_by); one scans_only_time_series over a
  set, read by the top-k schema and Pass 1; topk.rs's own walk is deleted.
- D2: SummaryCoverage -> AggregateCoverage, SummaryStateDefinition ->
  AggregateDefinition, CoverageError::NotSummary -> NoCoverage;
  ir/properties/aggregate_coverage.rs; tests/memo_aggregate_coverage.rs.
- D4: constraint_of and flip beside Constraint in ir/properties; dedup and
  interval private again.
- M1: one expression equality (same_serialized_expression).
- M2: OutputColumnSource::Measure { index, input_column }; the signs read
  the measure's input column from the layout.
- M3: summary_agg_layout replaces summary_agg_schema and summary_agg_columns.
- M4: validate_expression_over_input_groups returns the schema and derives
  coverage only to check a merge.
- M5, M6: stale docs and doc links.
- M7: column_sign -> column_signs; CandidateError::Schema ->
  NotValidOverInputGroups; depends_on_it inlined.
- M8: MemoGroupAsInput, its fields, group_properties, output_column_sign
  and derive_from_inputs are pub(super).
- M9: Stage 0's hash-consing tests move to stages/logical/memo/tests.rs,
  built through find_or_add_group.
- M10: Memo has no Default; Memo::new(&DataWorkload::default()) in tests.
…s derivation, one serialization per expression (#1207 PR 2 review)

- D1: AggregateCoverage, AggregateDefinition and CoverageError move to
  stages/logical/aggregate_coverage.rs with the derivation, now the free
  function aggregate_coverage; ir/properties/selection.rs keeps the
  selection algebra (SelectionBox, Constraint, ColumnIdentity,
  IntervalConstraintLimit, constraint_of).
- 1, 2: an expression is serialized once, with its group ids; the hash
  reads that string; a group is equal by typed equality, then the cached
  strings; a candidate by typed equality first.
- 3: validate_expression_over_input_groups returns a merge's coverage,
  which derive_group_properties reuses.
- 4: OutputColumnSource is crate-private; aggregate_output_columns deleted.
- 5, 6: ranked_row_projection's doc restored; stale docs fixed.
- 7: ValueSign and the sign derivation (column_signs) in
  column_properties.rs.
- 9: MemoGroup's parents and queries are pub(super); Memo::queries().
- 10: the memo fixtures are in test_support::memo_fixtures, shared by the
  memo's unit tests and the integration tests.
- 11: a measure's input column is computed once.
- 12: Source::is_time_series.
…ge quantiles; arithmetic signs

- ValueSign documents its meaning: NonNegative means every non-NaN value
  is >= 0, NonPositive every non-NaN value is <= 0.
- A point count or top-k read from a Count Sketch or UnivMon state (and
  their variants) is Unknown: a median of signed counters can be negative
  over non-negative weights. SummaryTypeCapability::estimates_may_flip_sign
  holds the per-algorithm fact; Count-Min keeps the state's sign.
- A quantile with q outside [0, 1] is Unknown, for both AggIntent::Quantile
  and SketchStatistic::Quantile.
- Vector BinaryOp arithmetic and ScalarExpr arithmetic share one rule
  (arithmetic_sign): a sum of two non-negatives (non-positives) is
  non-negative (non-positive); a product or quotient of same signs is
  non-negative and of opposite signs non-positive; anything else Unknown.
- PromqlVectorFromScalar(expr)'s value has expression_sign(expr).
- The memo compares coverages by AggregateDefinition == (it holds memo
  group ids); AggregateCoverage::definition_id and its test_support helper
  are deleted, and the memo coverage test compares definitions only.
- ViewCoverage::definition_id is documented as the plan-scope content id:
  the definition with every input group expanded recursively to its
  expression down to the Scans, with no group or node ids, canonically
  serialized and hashed. coverage_of (still todo for PR 6) returns the
  ViewCoverage and names that content id.
- Design docs (asap-primitive-schema.md, physical-planning-and-deployment.md)
  say the memo compares definitions and the plan carries the content id.
- Constraint::Interval documents that a full-range interval is still a
  constraint: it excludes NULL and NaN rows.
…the plan node

- ValueSign is the IEEE 754 sign bit: NonNegative means every non-NaN
  value has its sign bit clear, NonPositive set. sign_of_number reads the
  sign bit (-0.0 is NonPositive, NaN Unknown), so `x / -0` over a
  non-negative x is NonPositive, no longer NonNegative.
- SummaryTypeCapability::point_counts_may_flip_sign (was
  estimates_may_flip_sign) is pub(crate) and an exhaustive match over
  SketchAlgorithm.
- A readout's sign rule takes the state column's type
  (readout_signs's value(sign, dtype)); the separate state-field lookup is
  gone.
- PhysicalASAPDAGNode::view_coverage carries a materialized view's
  coverage into the plan; the views code reads it. coverage_of, the IR it
  needed and coverage_by_node are deleted; operator_nodes is test-only.
  Stage 3's todo names where it is filled (the content id of the expanded
  definition). Serde default, so plan files and fixtures are unchanged.
- The content id is defined once, on ViewCoverage::definition_id; the
  design docs link to it.
- Test rows without "(was Unknown)" notes.
…rding; shared memo test helpers

- root_groups and queries_rooted_at are Memo methods (root_groups(&self),
  queries_rooted_at(&self, group)) instead of free functions over
  &[MemoQuery]; the planner CLI calls the method.
- sources_scanned_by (ir/operator/input.rs) is Operator::scanned_sources
  in ir/operator/node.rs, with the other rules generic over the input.
- "summary coverage" wording left over from the rename now says aggregate
  coverage (crate docs, the merge-validation comment, a SQL lowering test
  comment, the architecture doc and the design doc's section 6.6 row).
- aggregate, aggregate_where and readout are in
  test_support::memo_fixtures, used by the memo unit tests and
  memo_group_properties instead of their own copies.
…lan's content id by expansion

- AggregateCoverage<R>, AggregateDefinition<R> (input_group renamed
  input) and CoverageError move to ir/properties/aggregate_coverage.rs
  with the derivation, now Operator::<I: OperatorInput>::aggregate_coverage,
  generic over OperatorInput::Reference (how a definition names an input).
  OperatorInput gains `type Reference` and `reference()`; node trees use
  `()` and store no coverage; the memo instantiates R = MemoGroupId.
  AggregateCoverage::map_references maps the references. Nothing under
  ir/ imports crate::stages.
- ViewCoverage moves beside PhysicalASAPDAGNode. stages::view_coverage
  expands a group's coverage (every input group to its expression, down
  to the Scans) and hashes its JSON (FNV-1a 128) into definition_id; the
  selection is the memo's. Stage 3's todo names it.
- Tests: equal definition_id across memos with different group ids,
  another source gives another id, no coverage gives None.
- asap-primitive-schema.md: the IR type and derivation, map_references,
  the content id by expansion; derive_from_inputs and input_group gone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Workload optimization and plan selection refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant