Skip to content

refactor(stage3): baseline selection MILP and its PhysicalASAPDAG; executor θ, native chains and a time-indexed store (#1207 PR 6) - #1236

Merged
zzylol merged 31 commits into
mainfrom
refactor/1207-stage3-baseline
Nov 14, 2026
Merged

zzylol merged 31 commits into
mainfrom
refactor/1207-stage3-baseline

Conversation

@zzylol

@zzylol zzylol commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Part of #1207 (PR 6 of the stack). Design: the concepts doc memo-groups-properties-requirements.md, section 3 "Stage 3: baseline selection and plan output" and section 5 "Cost and retention" (#1224) and planner-layering.md; this PR updates physical-planning-and-deployment.md.

Summary

  • Problem: on main, Stage 3's select_plan, the plan's retention, the installer and the executor's θ and chain handling are todo!(), so no workload can be planned or installed.
  • Change: Stage 3 selects with the baseline MILP over $x, y, z, t, r$ and builds the PhysicalASAPDAG itself, costing every candidate with asap-cost's own rules so the MILP's optimum equals the plan's plan_cost. Stage 3, asap-cost, the installer and the executor read time windows through one interface (asap_types::plan::time_window). The executor reads θ from the node, runs tumbling-window chains natively (built at ingestion or at query time), and stores views in one time-indexed store.
  • Effect: quantile_over_time(0.99, latency_ms[60m]) every 10 min (rank error 0.01, 1,000 series at 1 sample/s, no raw data kept) goes from todo!() to a selected, installed plan: a KLL chain (k = 1,404, width 600,000 ms, retention_ms 3,600,000) at 7.821 cost/s, 7.825 cost/s for the whole plan, equal to Stage 3's optimum.

Review guide

  • Base: main (PRs 1–5 of Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 merged). PR 7 (refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237) is stacked on this branch.
  • Read first:
    1. Stage 3 (crates/stage3-plan-selection/src): lib.rs (select_plan), selectable.rs, cost.rs ($w_v$, $c_{u,c,i}$), retention.rs, milp/model/mod.rs (solve_selection), milp/upper_bound.rs, plan.rs (plan_of).
    2. Time windows: types/src/plan/time_window.rs, then its users: cost/src/plan/read_back.rs, types/src/plan/materialized_view/views.rs (ViewIndex::read_back_ms, read_through_a_root), executor/src/physical_planner/time_window.rs (CompileGraph).
    3. Costing: cost/src/node/usage/mod.rs (node_resource_usage_and_cost, maintained_window_ms), cost/src/plan/execution_demand.rs, cost/src/plan/cost.rs (plan_cost).
    4. Executor: physical_planner/mod.rs (chains at ingestion and at query time), operators/time_range.rs, sources/memory.rs.
    5. Installer: planner/src/installable/mod.rs, types/src/plan/materialized_view/{mod,views,raw_data}.rs.
  • Mechanical: inline test modules moved to sibling tests.rs files; Stage 3, types and executor files split by scope; θ threaded into every executor summary constructor; NodeUsage → NodeResourceUsageAndCost; "price" → "cost" in the planner fuzz.
  • Careful: the one-consumer rule and the (B)/(D) trim in milp/model/mod.rs; the bound in milp/upper_bound.rs; InputTimeAccess::of (which reader reads what); the retained-output count in node_resource_usage_and_cost; computations at ingestion time; the query-time chain build (with_tumbling_window_ends, grouping by window end).

Behavior: before → after

Planner test runtime_install::the_plan_carries_stage3s_predicted_usage: p99 over 60 min, evaluated every 10 min at phase 0, rank error 0.01, 1,000 series at 1 sample/s, the deployment keeps no raw data. It serves the MVP need of planning, installing and running a repeated query end to end.

Before (main) After
Planning stops at select_plan (todo!()) Stage 3 selects and returns one PhysicalASAPDAG
Plan none 0 Scan (IT, None) → 1 SummaryTumblingWindowChain[KLL k=1404, w=600000] (IT, AcrossEvaluations, retention_ms 3,600,000) → 2 TimeRange(3600s) (QT, None) → 3 SummaryMerge (QT, OneEvaluation) → 4 SummaryEstimate(0.99) (QT, None) → 5 Project (QT, None)
Cost report none chain 7.821/s, scan 0.0042/s, each query-time node 1e-5/s or less; 7.825/s in all, equal to Selection::selected_plan_cost_per_second
Executor summaries panic without θ; the installer was to expand a chain into N SummaryAgg windows θ from the node; the chain runs natively, one window state per ingestion run, stored with its retention

Interfaces

Wire version stays 15: no node field changes (Stage 3 now sets view_coverage and parameters, which already existed). MaterializedView's JSON loses inputs.raw_data_windows (not part of the plan file).

crates/types/src/plan/time_window.rs (new) · crates/types/src/plan/mod.rs

The time-window interface, in four layers. Every caller (Stage 3, asap-cost, the installer, the executor) goes through it.

Node reads shifts_back_ms
query-time TimeShift(1h) WhatItsReadersRead 3,600,000
TimeShift(1h) stored at ingestion time CurrentTimeOnly 3,600,000
query-time chain WhatItsReadersRead 0
ingestion-time chain of width w LastWindow { w } 0
TimeRange(5m) LastWindow { 300_000 } 0
TimeShift with @ AllRows 0
any other operator CurrentTimeOnly 0
+// Layer 1 (value). Re-exported as `asap_types::plan::TimeWindow` and `asap_planner::TimeWindow`.
+/// The rows (t - ends_before_evaluation_ms - length_ms, t - ends_before_evaluation_ms] at evaluation time t.
+/// E.g. a 5m read with offset 1m = TimeWindow { ends_before_evaluation_ms: 60_000, length_ms: 300_000 }.
+pub struct TimeWindow { pub ends_before_evaluation_ms: u64, pub length_ms: u64 }
+impl TimeWindow {
+    /// How far back from t the window reaches: ends_before_evaluation_ms + length_ms (saturating).
+    pub fn reaches_back_ms(&self) -> u64;
+    /// The smallest window holding both.
+    pub fn union(self, other: Self) -> Self;
+    /// `(start, end]` at evaluation time `evaluation_time_ms`, in ms (saturating).
+    pub fn bounds_ms(&self, evaluation_time_ms: i64) -> (i64, i64);
+}
+// Layer 2 (one node).
+/// How one node reads its time-indexed input at its evaluation time t.
+pub struct InputTimeAccess {
+    /// Which input rows it reads.
+    pub reads: InputRows,
+    /// How much earlier its output rows are than the rows it read: a TimeShift's offset, else 0.
+    pub shifts_back_ms: i64,
+}
+/// Which input rows a node reads at its evaluation time t.
+pub enum InputRows {
+    /// Stores nothing itself: it reads exactly what its readers read (shifted back).
+    WhatItsReadersRead,
+    /// The window (t - length_ms, t]: a TimeRange's range, an ingestion-time chain's tumbling window.
+    LastWindow { length_ms: u64 },
+    /// Only the rows at t: any other operator, and a TimeShift stored at ingestion time.
+    CurrentTimeOnly,
+    /// Every row, because the time it reads does not depend on t (a TimeShift with `@`).
+    AllRows,
+}
+impl InputTimeAccess {
+    /// How `operator`, run at `timing` with `lifetime`, reads its input (the table above).
+    pub fn of<C>(operator: &Operator<C>, timing: ExecutionTiming, lifetime: Lifetime) -> Self;
+}
+// Layer 3 (graph).
+/// A plan the time walks run over: plan nodes, Stage 3 candidates or executor nodes.
+pub trait TimeIndexedPlan {
+    type Node: Copy + Ord;
+    /// The child type of the operators it holds.
+    type Child;
+    /// The nodes that read `node`.
+    fn readers_of(&self, node: Self::Node) -> Vec<Self::Node>;
+    /// The nodes `node` reads, in input order.
+    fn inputs_of(&self, node: Self::Node) -> Vec<Self::Node>;
+    /// How `node` reads its time-indexed input.
+    fn input_time_access(&self, node: Self::Node) -> InputTimeAccess;
+    /// Whether a reader of `node` that reads only the rows at its evaluation time (CurrentTimeOnly)
+    /// in fact needs every row of `node`: true for raw data, which is not stored by evaluation time.
+    fn current_time_read_needs_every_row(&self, node: Self::Node) -> bool;
+    /// The node's operator, which the walks down read (a TimeRange's range, a chain's tumbling
+    /// window, a TimeShift's offset); `InputTimeAccess` alone cannot tell a query-time chain from
+    /// a query-time TimeShift with offset 0.
+    fn operator_of(&self, node: Self::Node) -> &Operator<Self::Child>;
+}
+/// The plan as installed and costed: edges give readers and inputs, each node's
+/// `InputTimeAccess::of`, and a current-time read of a `Scan` needs every row. O(edges) per node asked.
+impl TimeIndexedPlan for PhysicalASAPDAG { /* Node = Child = PhysicalASAPNodeId */ }
+// Layer 4 (walks).
+/// Up: the window each reader of `producer` needs from its rows; None = unbounded. Starts at the
+/// producer's own shifts_back_ms; a WhatItsReadersRead reader adds its shifts_back_ms and the walk
+/// goes on; LastWindow ends the path with {offset, length}; CurrentTimeOnly with {offset, 0}, or
+/// None when the producer's current-time read needs every row; AllRows: None.
+pub fn windows_readers_need<P: TimeIndexedPlan>(plan: &P, producer: P::Node)
+    -> Option<Vec<(P::Node, TimeWindow)>>;
+/// Down: the windows of input rows `node`'s result is computed from (each TimeRange at or below it,
+/// each chain reached without passing one), each over the shifts directly below it.
+pub fn windows_computed_from<P: TimeIndexedPlan>(plan: &P, node: P::Node) -> Option<Vec<TimeWindow>>;
+/// The sum of the offsets of the TimeShifts directly below `node`; None with an `@` anchor.
+pub fn shift_back_ms_below<P: TimeIndexedPlan>(plan: &P, node: P::Node) -> Option<u64>;

The other implementations are private: Stage 3's Candidates (retention.rs: the readers its caller gives, selectable or selected; admissible producers as inputs; a Scan needs every row) and the executor's CompileGraph (compiled consumers and dependencies; a compile root reached through WhatItsReadersRead reads AllRows; every node needs every row, because today's executor store reads a slot whole outside a window, #1291). The installer adds a root rule on top of the walk in ViewIndex::read_through_a_root (types/src/plan/materialized_view/views.rs): a node a query root reads through WhatItsReadersRead nodes is read unbounded. asap-cost reads the plan without that rule, as Stage 3 reads its candidates.

crates/types/src/plan/physical_asap_dag.rs · crates/types/src/plan/materialized_view/ · crates/types/src/stages/physical/memo_candidates/mod.rs · crates/types/src/ir/properties/execution_data_state/mod.rs

 impl PhysicalASAPDAG {
+    /// The DAG of `nodes` (each after its inputs) with one edge per payload input, carrying the
+    /// producer's schema and data state; validated. Stage 3 builds its plan this way.
+    pub fn new(
+        nodes: Vec<PhysicalASAPDAGNode>,
+        roots: Vec<PhysicalASAPNodeId>,
+        scalar_roots: Vec<(usize, ScalarExpr<PhysicalASAPNodeId>)>,
+    ) -> Result<Self, PhysicalASAPDAGValidationError>;
     /// Each node's refresh window (signature unchanged; now implemented).
     pub fn refresh_windows(
         &self,
         nodes: impl IntoIterator<Item = PhysicalASAPNodeId>,
     ) -> HashMap<PhysicalASAPNodeId, Option<RefreshWindow>>;
-    /// Set every view's retention after export.
-    pub fn set_retention(
-        &mut self,
-        tumbling_window_retention_ms_by_node: &HashMap<PhysicalASAPNodeId, u64>,
-        reader_span_ms_by_node: &HashMap<PhysicalASAPNodeId, u64>,
-    ) -> Result<Vec<MaterializedView>, InstallableDagError>;
     /// Per source, the longest `retention_ms` of its raw-data `Scan`s (signature unchanged; now implemented).
     pub fn raw_data_retention(&self) -> Result<Vec<RawDataRetention>, InstallableDagError>;
 }
 impl PhysicalASAPDAGNode {
+    /// Whether it keeps time-indexed rows for its readers to read back, so carries `retention_ms`:
+    /// materialized across evaluations, or a raw-data `Scan` at query time.
+    pub fn retains_rows(&self) -> bool;
+    /// Whether it is a materialized view: materialized across evaluations and not a `Scan`, so it
+    /// keeps what its `retention_ms` covers.
+    pub fn is_view(&self) -> bool;
 }
 impl MaterializedView {
     /// Which run refreshes it; now public outside tests (the installer reads it).
     pub fn refresh(&self) -> MaterializedViewRefresh;
     /// How far back from the newest evaluation time its stored rows are kept; now public outside tests.
     pub fn retention_ms(&self) -> u64;
 }
 /// What a view's refresh reads. The type stays.
 pub struct MaterializedViewInputs {
-    /// The windows of raw data it scans directly.
-    pub(crate) raw_data_windows: Vec<RawDataWindow>,
     /// The views it reads, by node id.
     pub materialized_views: Vec<PhysicalASAPNodeId>,
 }
-/// A window of a raw-data source a view's refresh scans.
-pub struct RawDataWindow { source: Source, end_offset_ms: i64, width_ms: i64 }
 impl RefreshWindow {
-    pub fn of_window(end_offset_ms: i64, width_ms: i64) -> Option<Self>;
+    /// The refresh window of `window`: its length, and its end mod its length; `None` for an empty one.
+    pub fn of_window(window: TimeWindow) -> Option<Self>;
 }
 /// Why a runtime cannot install a DAG. The type stays.
 pub enum InstallableDagError {
     /// Bad input: the DAG is not a valid physical ASAP DAG.
     InvalidDag(PhysicalASAPDAGValidationError),
     /// Unsatisfiable request: a raw-data window whose shape no ingestion run refreshes in time.
     UnsupportedRawDataWindow { time_range_node_id: PhysicalASAPNodeId, reason: &'static str },
     /// Unsatisfiable request: a view reads a source `Scan` outside a raw-data window (e.g. a whole SQL table).
     ViewReadsUnwindowedSource { node_id: PhysicalASAPNodeId },
     /// Unsatisfiable request: a view with no aggregate coverage.
     ViewWithoutCoverage { node_id: PhysicalASAPNodeId, reason: String },
     /// Unsatisfiable request: an ingestion-time view reads windows ending at different offsets.
     IngestionBuildReadsWindowsOfDifferentOffsets {
         node_id: PhysicalASAPNodeId,
-        end_offsets_ms: Vec<i64>,
+        end_offsets_ms: Vec<u64>,
     },
     /// Bad input (a plan file): a view has no `retention_ms`.
     ViewWithoutRetention { node_id: PhysicalASAPNodeId },
     /// Bad input (a plan file): a view is retained for less than its readers read back.
     ViewRetentionTooShort { node_id: PhysicalASAPNodeId },
     /// Bad input (a plan file): a raw-data `Scan` read through a bounded window has no `retention_ms`.
     ScanWithoutRetention { node_id: PhysicalASAPNodeId },
     /// Bad input (a plan file): a raw-data `Scan` is retained for less than its window reaches back.
     ScanRetentionTooShort { node_id: PhysicalASAPNodeId },
 }
 impl PhysicalASAPMemoCandidates {
+    /// Physical edge `id`; Stage 3 reads the edges it allocates $y$ on.
+    pub fn edge(&self, id: PhysicalEdgeId) -> &PhysicalEdge;
+    /// The number of inputs of candidate `v` (e.g. 2 for a binary operator).
+    pub fn input_count(&self, v: PhysicalCandidateId) -> usize;
 }
 impl DataPrimitive {
+    /// What a node with `operator` and `result_kind` outputs: `SummaryState` for a summary
+    /// build/chain/join/maintained population or any `State` result, `Raw` otherwise.
+    pub fn of<C>(operator: &crate::ir::Operator<C>, result_kind: crate::ir::OperatorResultKind) -> Self;
 }
 #[cfg(any(test, feature = "test-support"))]
 impl ViewCoverage {
+    /// The coverage of definition `definition_id` over `selection`; for tests of hand-built plans.
+    pub fn new(definition_id: String, selection: Vec<SelectionBox>) -> Self;
 }
-/// `RawDataWindow::lookback_ms` (`test_support::forwarders`).
-pub fn lookback_ms(window: &RawDataWindow) -> u64;

RawDataWindow and raw_data_windows had no non-test reader; TimeWindow replaces them and the bare (i64, i64) windows.

crates/cost/src/node/usage/mod.rs · crates/cost/src/plan/cost.rs · crates/cost/src/plan/execution_demand.rs · crates/cost/src/plan/read_back.rs · crates/cost/src/node/state_bytes/mod.rs

 /// The whole plan's cost; how often each query runs and its range evaluation come from `entries`;
 /// formulas scaled by `settings.frozen_calibration_factor` (1 when unset).
 pub fn plan_cost(
     dag: &PhysicalASAPDAG,
-    nodes: &NodeMap<'_>,
-    recurrence: &[QueryRecurrence],
-    range_evaluations: &[Option<RangeEvaluation>],
+    entries: &[QueryWorkloadEntry],
     data: &DataWorkload,
     settings: &CostInputs,
-    formula_scale: f64,
 ) -> Result<(PlanCost, CalibrationFit), (PhysicalASAPNodeId, AnalyticalCostError)>;
+/// Cost model 1's cost per second of every query request's fixed overhead (`ns_per_evaluation`
+/// per request): what `plan_cost` charges every plan; 0 under cost models 2 and 3.
+pub fn request_overhead_cost_per_second(
+    entries: &[QueryWorkloadEntry],
+    settings: &CostInputs,
+) -> Result<f64, AnalyticalCostError>;
+/// Cost model 1's cost per second of keeping `bytes` of raw data on disk; a plan keeps each source
+/// once, for its scan that keeps the most. `None` under cost models 2 and 3.
+pub fn raw_data_disk_cost_per_second(bytes: u64, settings: &CostInputs) -> Option<f64>;
+/// How many times per second a node computed for the queries `reaching` it runs: once per second
+/// at ingestion time; their combined rate at query time; once per request when materialized across evaluations.
+pub fn executions_per_second<'e>(
+    timing: ExecutionTiming,
+    lifetime: Lifetime,
+    reaching: impl IntoIterator<Item = &'e QueryWorkloadEntry>,
+    non_repeating_query_horizon_seconds: f64,
+) -> Result<f64, AnalyticalCostError>;
 /// How often each node is computed: `(multiplier, per_second)`; a node computed at ingestion time
 /// runs once per second whatever views read it.
 pub fn computations(
     dag: &PhysicalASAPDAG,
-    recurrence: &[QueryRecurrence],
-    range_evaluations: &[Option<RangeEvaluation>],
+    entries: &[QueryWorkloadEntry],
     non_repeating_query_horizon_seconds: f64,
-    root_rates: Option<&[f64]>,
 ) -> Result<HashMap<PhysicalASAPNodeId, (u64, f64)>, AnalyticalCostError>;
 /// The window an ingestion-time node maintains: a chain's tumbling window, a `TimeRange`'s range,
 /// or the first `TimeRange` or chain below it through filters, projections and shifts (a view over
 /// an ingestion-time chain is refreshed once per chain window); over plan nodes or memo groups.
-pub fn maintained_window_ms(
-    node: PhysicalASAPNodeId,
-    nodes: &NodeMap<'_>,
-    producers_of: &HashMap<PhysicalASAPNodeId, Vec<PhysicalASAPNodeId>>,
-) -> Option<u64>;
+pub fn maintained_window_ms<'o, N: Copy + Ord, C: 'o>(
+    node: N,
+    operator_of: impl Fn(N) -> &'o Operator<C>,
+    input_of: impl Fn(N) -> Option<N>,
+) -> Option<u64>;
+/// The most any reader of `producer` reaches back: each window's `reaches_back_ms` plus its
+/// reader's `span_ms`; `None` when one reads it unbounded. Over `windows_readers_need`.
+pub fn read_back_ms<P: TimeIndexedPlan>(
+    plan: &P,
+    producer: P::Node,
+    span_ms: impl Fn(P::Node) -> u64,
+) -> Option<u64>;
 /// One producer of a node being costed. The type stays.
 pub struct ProducerOutput<'a> {
     /// Its payload.
     pub payload: &'a Payload,
     /// Its output schema.
     pub output_schema: &'a Schema,
     /// Its output rows and bytes.
     pub output: EdgeStatistics,
     /// The sources it reads.
     pub sources: BTreeSet<&'a str>,
     /// Its state family (with θ).
     pub family: StateFamily<'a>,
     /// Its output columns that still hold one source row's values unchanged.
     pub unchanged_columns: BTreeSet<&'a str>,
+    /// The tumbling windows its rows span: those a `TimeRange` selects from a chain, 1 otherwise.
+    pub tumbling_windows: u64,
 }
 /// What one node uses and costs.
-pub struct NodeUsage<'a> {
+pub struct NodeResourceUsageAndCost<'a> {
     /// Its output rows and bytes.
     pub output: EdgeStatistics,
     /// The sources it reads.
     pub sources: BTreeSet<&'a str>,
     /// Its state family.
     pub family: StateFamily<'a>,
     /// Its output columns that still hold one source row's values unchanged.
     pub unchanged_columns: BTreeSet<&'a str>,
+    /// The tumbling windows its rows span: those a `TimeRange` selects from a chain, 1 otherwise.
+    pub tumbling_windows: u64,
     /// Work of one execution, in ms on one core.
     pub work_ms_per_execution: f64,
     /// CPU operations per second.
     pub cpu_ops_per_second: f64,
     /// Bytes scanned per second.
     pub scanned_bytes_per_second: f64,
     /// Average memory, including memory a query-time node holds while it computes.
     pub average_memory_bytes: f64,
     /// Peak memory.
     pub peak_memory_bytes: u64,
     /// Disk bytes.
     pub disk_bytes: u64,
-    /// Query-time memory held while an evaluation runs: (bytes, computations per second, per-consumer copy).
-    pub held: Option<(u64, f64, bool)>,
     /// An in-memory ingestion-time state's bytes per window.
     pub maintained: Option<u64>,
     /// A query-time scan's source and the raw bytes it reads.
     pub raw_scan: Option<(&'a Source, u64)>,
     /// Its calibration fit.
     pub fit: CalibrationFit,
     /// Its cost.
     pub cost: NodeCost,
 }
+impl<'a> NodeResourceUsageAndCost<'a> {
+    /// What a consumer reads of the node with `payload` and `output_schema` that uses this.
+    pub fn producer_output(&self, payload: &'a Payload, output_schema: &'a Schema) -> ProducerOutput<'a>;
+}
 /// Measure and cost one node; Stage 3 calls it per candidate ($w_v$) and `plan_cost` per plan node.
-pub fn node_usage<'a>(
+pub fn node_resource_usage_and_cost<'a>(
     node: &'a PhysicalASAPDAGNode,
     producers: &[ProducerOutput<'a>],
     rate: (u64, f64),
-    retained_tumbling_window_count: u64,
-    ingestion_window_ms: Option<u64>,
+    window_ms: Option<u64>,
     data: &DataWorkload,
     settings: &'a CostInputs,
-    formula_scale: f64,
-) -> Result<NodeUsage<'a>, AnalyticalCostError>;
-pub fn tumbling_window_retention_ms(..) -> BTreeMap<PhysicalASAPNodeId, u64>;
-pub fn tumbling_window_retention_ms_of_plan(..) -> BTreeMap<PhysicalASAPNodeId, u64>;
-pub fn retained_tumbling_windows(..) -> u64;
-pub fn tumbling_windows_of(node: &PhysicalASAPDAGNode) -> u64;
-pub fn query_reach(..) -> Vec<HashSet<PhysicalASAPNodeId>>;
-pub fn node_map(dag: &PhysicalASAPDAG) -> NodeMap<'_>;
-pub type NodeMap<'a> = HashMap<PhysicalASAPNodeId, &'a PhysicalASAPDAGNode>;
+) -> Result<NodeResourceUsageAndCost<'a>, AnalyticalCostError>;
 /// Update operations per input row and bytes per state, at its θ; Err for a sketch without θ (a planner bug).
-pub fn summary_shape(family: &FieldDataType, input_rows: StateInputRows) -> (f64, u64);
+pub fn summary_shape(
+    family: &FieldDataType,
+    parameters: Option<&SketchParams>,
+    input_rows: StateInputRows,
+) -> Result<(f64, u64), AnalyticalCostError>;

Removed: held (held memory is in average_memory_bytes); the chain retention helpers, tumbling_windows_of and their file plan/retention.rs (retention is the node's retention_ms); query_reach (the latency-based held loop is gone); node_map and NodeMap (now private to plan/cost.rs, their only user).

crates/stage3-plan-selection/src/report.rs · crates/devtools/src/bin/asap_planner.rs

 /// What Stage 3 reports about its own run, beside the plan. The type stays.
 pub struct Selection {
     /// Per query, the accuracy Stage 3 accepted for the selected plan.
     pub accepted_accuracy_per_query: Vec<AcceptedAccuracy>,
     /// The candidates Stage 3 ruled out before the selection MILP.
     pub pruned_candidates: Vec<PrunedCandidate<PhysicalCandidate>>,
+    /// The MILP's optimum: the selected plan's cost per second under cost model 1, which
+    /// `asap_cost::plan_cost` of the plan reports too.
+    pub selected_plan_cost_per_second: f64,
-    pub milp_variables: usize,
-    pub milp_constraints: usize,
+    /// The selection MILP's variables (columns).
+    pub milp_variable_count: usize,
+    /// The selection MILP's constraint rows.
+    pub milp_constraint_count: usize,
     /// How long building and solving the selection MILP took.
     pub milp_construction_ms: u64,
     pub milp_solve_ms: u64,
 }
 // asap_planner CLI, `stage3_plan_selection` JSON section
+    "milp_objective_cost_per_second": selection.selected_plan_cost_per_second,
     "milp_variables": ..., "milp_constraints": ..., "milp_construction_ms": ..., "milp_solve_ms": ...

crates/executor/src/capability/mod.rs · summary_kernels/factory/mod.rs · summary_kernels/state_column_updater/mod.rs · operators/summary/mod.rs · operators/time_range.rs · plan/mod.rs · sources/memory.rs · physical_planner/test_support.rs

 /// Checks that a summary kernel accepts `input`, at the node's θ (bad input for θ the kernel does not build).
 pub fn validate_summary_kernel(
     family: &SummaryFamilyType,
+    parameters: Option<&SketchParams>,
     input: &SummaryUpdate,
 ) -> Result<(), String>;
 /// Builds a HydraCms grid at the node's θ.
-pub fn create_hydra_cms(family: &SummaryFamilyType) -> Result<HydraCms, String>;
+pub fn create_hydra_cms(
+    family: &SummaryFamilyType,
+    parameters: Option<&SketchParams>,
+) -> Result<HydraCms, String>;
 impl StateColumnUpdater {
     /// An updater for one summary type at θ; bad input for any other type or θ its kernel does not build.
-    pub fn new(family: &FieldDataType) -> Result<Self, Error>;
+    pub fn new(family: &FieldDataType, parameters: Option<&SketchParams>) -> Result<Self, Error>;
 }
 impl Operator {
     /// A keyed summary build per group, at θ.
     pub fn per_group_keyed_summary_build(
         input: SchemaRef,
         family: SummaryFamilyType,
+        parameters: Option<SketchParams>,
         items: Vec<usize>,
         row_weight: RowWeight,
         groups: Vec<usize>,
     ) -> Result<Self, Error>;
     /// One shared-structure build over all groups, at θ.
     pub fn all_groups_summary_build(
         input: SchemaRef,
         family: SummaryFamilyType,
+        parameters: Option<SketchParams>,
         item: usize,
         row_weight: RowWeight,
         groups: Vec<usize>,
     ) -> Result<Self, Error>;
     /// A value summary build per group, at θ.
     pub fn per_group_summary_build(
         input: SchemaRef,
         family: SummaryFamilyType,
+        parameters: Option<SketchParams>,
         value: usize,
         time: Option<usize>,
         groups: Vec<usize>,
     ) -> Result<Self, Error>;
     /// Emits an empty global state when no row arrives; the empty state is built at θ.
-    pub fn with_emit_empty_global(mut self) -> Result<Self, Error>;
+    pub fn with_emit_empty_global(
+        mut self,
+        parameters: Option<SketchParams>,
+    ) -> Result<Self, Error>;
+    /// A `TimeShift { offset_ms, at }`: evaluates its input once, at the request's end/start or T
+    /// minus `offset_ms`.
+    pub fn evaluate_at(input: SchemaRef, at: AtModifier, offset_ms: i64) -> Result<Self, Error>;
+    /// This range window, each kept row's time replaced by the end of its tumbling window of
+    /// `width_ms` counted back from the range's end E: `(E - (k+1)·w, E - k·w]` ends at `E - k·w`.
+    /// A tumbling-window chain built at query time groups its rows by that end.
+    pub fn with_tumbling_window_ends(mut self, width_ms: i64) -> Result<Self, Error>;
 }
 /// An executable operator.
 pub trait PhysicalOperator<V, S> {
     /// Its name.
     fn name(&self) -> &str;
     /// Its boundedness and emission given its inputs'.
     fn properties(&self, inputs: &[PlanProperties]) -> PlanProperties;
     /// Whether it needs bounded input.
     fn requires_bounded_input(&self) -> bool;
     /// Validate run-specific contracts before any source is opened.
     fn validate_context(&self, _context: &RunContext) -> Result<(), Error>;
+    /// The run context its inputs run in; default: this run's (a `TimeShift` with `@` overrides it).
+    fn input_context(&self, context: &RunContext) -> Result<RunContext, Error>;
     /// Its input schemas.
     fn input_schemas(&self) -> Vec<S>;
     /// Its output schema.
     fn output_schema(&self) -> S;
     /// Starts it over its inputs.
     fn start<'a>(
         &'a self,
         inputs: Vec<Input<'a, V>>,
         context: RunContext,
     ) -> Result<OutputStream<'a, V>, Error>;
     /// The bytes of one output value.
     fn output_bytes(&self, value: &V) -> usize;
 }
 impl InMemoryDataStore {
-    /// Stores a view's batches (no time index).
-    pub fn with_materialized_view(
-        mut self,
-        view: PhysicalASAPNodeId,
-        batches: Vec<Batch>,
-    ) -> Result<Self, Error>;
+    /// Appends `view`'s rows for `(start, end]`, then drops rows older than `view.retention_ms()`
+    /// (the rows appended last are always kept); bad input for an empty range or one starting
+    /// before the last appended end (append-only).
+    pub fn append_materialized_view(
+        &mut self,
+        view: &MaterializedView,
+        time_range_ms: (i64, i64),
+        batches: Vec<Batch>,
+    ) -> Result<(), Error>;
 }
 #[cfg(feature = "test-support")]
 pub mod test_support {
+    /// Sets θ on every sketch build among `nodes` (`parameters`, else its algorithm's first listed θ);
+    /// for tests of hand-built plans.
+    pub fn set_build_parameters<'a>(
+        nodes: impl IntoIterator<Item = &'a mut PhysicalASAPDAGNode>,
+        parameters: Option<&SketchParams>,
+    );
 }

crates/planner/src/installable/mod.rs · crates/planner/src/lib.rs

 /// Plan `input` for a deployment runtime: the selected DAG ready to install, and its cost report.
 pub async fn plan_installable(
     input: UserInput<'_>,
-) -> Result<(InstallablePlan, Option<SelectedPlanReport>), PlanError>;
+) -> Result<(InstallablePlan, SelectedPlanReport), PlanError>;
 /// The installable plan of `plan`, the plan Stage 3 built (a developer tool runs the stages itself).
 pub fn installable_plan_of(
-    plan: &PhysicalASAPDAG,
-    cost: Option<&PlanCost>,
+    plan: PhysicalASAPDAG,
     entries: &[QueryWorkloadEntry],
-) -> Result<(InstallablePlan, Option<SelectedPlanReport>), InstallablePlanError>;
+) -> Result<InstallablePlan, InstallablePlanError>;
 /// One operator node's predicted usage, beside the installed plan. The type stays.
 pub struct PredictedNodeUsage {
-    /// Its id in Stage 3's physical DAG of the selected candidate.
-    pub stage3_node_id: PhysicalASAPNodeId,
-    /// The ids in `dag` that run it (a chain's expanded windows).
-    pub installed_node_ids: Vec<PhysicalASAPNodeId>,
+    /// Its id in the installed DAG (chains are not expanded, so one id).
+    pub node_id: PhysicalASAPNodeId,
     /// Its operator, as the measured cost table names it.
     pub operator: OperatorKind,
     /// When it runs.
     pub timing: ExecutionTiming,
     /// Core-seconds of CPU per second.
     pub cpu_core_seconds_per_second: f64,
     /// Its cost per second under cost model 1; `None` under cost models 2 and 3.
     pub cost_per_second: Option<f64>,
     /// Whether its CPU came from the measured cost table or the formulas.
     pub cost_source: NodeCostSource,
 }
 /// Why an installable plan could not be built or is not consistent. The type stays.
 pub enum InstallablePlanError {
     /// Bad input (a plan file): another plan-file format.
     UnsupportedWireVersion(Option<u32>),
     /// Bad input (a plan file): not JSON of an `InstallablePlan`.
     Json(serde_json::Error),
     /// Bad input (a plan file): not a valid physical ASAP DAG.
     InvalidDag(PhysicalASAPDAGValidationError),
     /// Bad input (a plan file): not one query text per root.
     QueriesDoNotMatchRoots { roots: usize },
     /// Unsatisfiable request: a standalone scalar query.
     UnsupportedQueryRoot { entry_index: usize },
     /// Unsatisfiable request or bad input: views or raw-data windows cannot be derived.
     Uninstallable(InstallableDagError),
     /// Unsatisfiable request: the executor cannot refresh view `node_id`.
     MaterializedViewNotExecutable { node_id: PhysicalASAPNodeId, reason: String },
     /// Unsatisfiable request: the executor cannot compile the query run.
     QueryRunNotExecutable { reason: String },
+    /// Internal bug: `plan_cost` failed on the selected plan at node `node_id`.
+    PlanCostFailed { node_id: PhysicalASAPNodeId, reason: String },
 }
-pub use asap_types::plan::materialized_view::RawDataWindow;
+pub use asap_types::plan::TimeWindow;

installable_plan_of's cost was Some only from plan_installable; the developer tools passed None and ignored the report, so the report now comes from plan_installable alone.

crates/devtools/src/planner_fuzz/mod.rs · crates/integration-tests/src/lib.rs

-pub struct Priced {
-    pub selected: String,
-    pub cost_per_second: f64,
-    pub derived_root: PrivateNodes,
-}
+/// A plan costed by Stage 2 and Stage 3 outside the search: the cheapest physical properties
+/// assignment of given logical roots.
+pub struct Costed {
+    /// The selected plan, as text.
+    pub selected: String,
+    /// Its cost per second.
+    pub cost_per_second: f64,
+    /// The nodes only the last root reads: in a witness, the operators that derive `q'`.
+    pub derived_root: PrivateNodes,
+}
 impl Planner {
-    pub fn price(
-        &self,
-        workload: &FuzzWorkload,
-        roots: &[QueryRoot],
-        last_root_is_key_filter: bool,
-    ) -> Result<Priced, String>;
+    /// The planning stages over `roots`: the selected plan's cost (`plan_cost`) and the cost of the
+    /// nodes only its last root reads.
+    pub fn cost(&self, workload: &FuzzWorkload, roots: &[QueryRoot]) -> Result<Costed, String>;
-    pub fn reference_price(
-        &mut self,
-        workload: &FuzzWorkload,
-        roots: &[QueryRoot],
-    ) -> Result<Priced, String>;
+    /// `cost` of `workload`'s selected logical plan `roots`, cached per workload.
+    pub fn reference_cost(
+        &mut self,
+        workload: &FuzzWorkload,
+        roots: &[QueryRoot],
+    ) -> Result<Costed, String>;
 }
 pub mod planning {
+    /// The workload entry of the single query these tests plan: `accuracy`, everything else default.
+    pub fn single_query_entry(accuracy: AccuracyTarget) -> asap_types::workload::QueryWorkloadEntry;
 }

last_root_is_key_filter was unused.

Usage, end to end (planner test runtime_install::the_plan_carries_stage3s_predicted_usage):

// What the user writes: p99 over 60 min, every 10 min at phase 0 (fixed_interval_at {600000, 0}),
// rank error 0.01; the data: 1,000 series at 1 sample/s. Every stage below reads it.
let workload = workload(&["quantile_over_time(0.99, latency_ms[60m])"]);
// The deployment keeps no raw data, so Stage 2 prunes query-time scans.
let capabilities = DeploymentCapabilities { raw_data_retained: false, ..Default::default() };
// The models Stages 1-3 read: built-in accuracy, these capabilities, the deployment's cost inputs.
let models = PlanningModels::builtin().with_deployment_capabilities(&capabilities).with_cost_inputs(&settings);
// `plan`: the installable plan; `report`: its plan_cost report (not part of the plan file).
// Frontend → Stages 0–2 → select_plan:
//   selectable_edges keeps the admissible edges whose producer delivers to each consumer's evaluation context;
//   selection_costs calls node_resource_usage_and_cost once per candidate (w_v; rate from
//   executions_per_second, window from maintained_window_ms) and once per selectable edge (c_{u,c,i});
//   retention_need_ms (asap_cost::read_back_ms over windows_readers_need on Stage 3's Candidates)
//   gives the chain its retention;
//   solve_selection leaves out candidates dearer than a greedy feasible plan, scales, solves with HiGHS;
//   plan_of → PhysicalASAPDAG::new, with retention_ms from the selected readers.
// → plan_cost → installable_plan_of (materialized_views, refresh_windows, raw_data_retention checks,
//   executor compile checks).
let (plan, report) = plan_installable(UserInput::new(&workload, FrontendInput::Promql, models)).await?;
// plan.dag: 0 Scan (IT, None) → 1 SummaryTumblingWindowChain[KLL k=1404, w=600000] (IT, AcrossEvaluations,
//   retention_ms 3_600_000) → 2 TimeRange(3600s) → 3 SummaryMerge (OneEvaluation) → 4 SummaryEstimate(0.99) → 5 Project (QT).
// report.predicted_usage.per_node: chain 7.821/s, scan 0.0042/s, TimeRange 1.0e-5/s, merge 1.0e-5/s,
//   estimate and project 1.7e-6/s each; report.cost_per_second 7.825, equal to Stage 3's optimum.

The CLI shows the same optimum: asap_planner --run-until stage3 --format json prints stage3_plan_selection.milp_objective_cost_per_second.

Algorithm changes

Baseline selection (Stage 3)

  • Problem: select_plan was todo!().

  • After:

    1. selectable::selectable_edges: the admissible edges whose producer delivers to the evaluation context of every query reaching a query-time consumer (its grid, phase 0; unknown times use a 1 ms grid).

    2. cost::selection_costs: each candidate becomes one plan node (plan::candidate_node, shared with plan building) and is costed with node_resource_usage_and_cost over the largest logical output of each input among the producers it may read, computed once per candidate. That is $w_v$: its rate is executions_per_second, an ingestion-time node's window maintained_window_ms over memo groups, a query-time Scan's read per execution retention::scan_read_ms. For each selectable edge, $c_{u,c,i}$ = the consumer's cost with input $i$ read from $u$'s own output − $w_c$. A node that retains rows gets retention_need_ms, the most its selectable readers read back plus the range-evaluation span. Each query-time scan's raw data on disk ($d_v$) and the request overhead are kept apart.

    3. milp::model::solve_selection minimizes $\sum_v w_v x_v + \sum c_{u,c,i} y_{u,c,i} + \sum_s r_s$ under:

      • (A) $\sum_{u} y_{u,c,i} = x_c$ per input; (B) $y_{u,c,i} \le x_u$; (C) $\sum_{v \in G_q} z_{q,v} = 1$; (D) $z_{q,v} \le x_v$;
      • (E) $t_u + 1 \le t_c + n(1 - y_{u,c,i})$, only inside strongly connected components of the admissible edges (milp::graph::Components, $t_v$ only there);
      • (F) $x_v \le \sum y_{v,c,i} + \sum_q z_{q,v}$; for a query-time result that is not materialized ($\ell$ = None) this is one equality row, $\sum y_{v,c,i} + \sum_q z_{q,v} = x_v$, which also says it has at most one consumer;
      • one $r_s$ per raw-data source: $r_s \ge \mathrm{scale} \cdot d_v x_v$ for each query-time Scan of $s$ (raw_data_disk_constraints);
      • (G) and (H) hold because $y$ exists only on costed selectable edges.

      The (B)/(D) trim: for a query-time result that is not materialized, (B) and (D) are left out. They are implied: with $y, z \ge 0$, the (F) equality $\sum y + \sum z = x_v$ gives every $y_{v,c,i} \le x_v$ and $z_{q,v} \le x_v$. The feasible set is unchanged, so the optimum is the same; the model just has fewer rows.

      Bound and scaling: before scaling, milp::upper_bound::dearer_than_a_feasible_plan fixes $x_v = 0$ for every candidate whose $w_v + \sum_i \min(0, \min_u c_{u,v,i})$ exceeds the cost of a greedily built feasible plan. The coefficients left are then scaled so the largest is 1,000 (HiGHS's tolerances are absolute). HiGHS runs at gap 0, one thread.

      Objective scale in the raw-data row: the scale multiplies $d_v$ inside the row, and $r_s$'s objective coefficient is 1. $d_v$ is about 1e-12 to 2e-9 per second, below HiGHS's small_matrix_value (1e-9), so HiGHS dropped the unscaled coefficients and the disk cost left the model; in the row they are 42 to 1,000.

      The optimum, unscaled and plus the request overhead (milp::model::plan_cost_per_second, shared with the bound), is Selection::selected_plan_cost_per_second.

    4. plan::plan_of: Kahn order over ${y = 1}$, PhysicalASAPDAG::new derives the edges; retention_ms = the most the selected readers read back (at least the span of a query it answers); view_coverage of each AcrossEvaluations node from its group.

  • Correctness: the objective equals plan_cost because both call the same functions (node_resource_usage_and_cost, executions_per_second, maintained_window_ms, read_back_ms, raw_data_disk_cost_per_second, request_overhead_cost_per_second); pinned by stage3_objective::* and measured on the harness (Verification). The bound only removes candidates that no plan cheaper than a feasible one contains, assuming every other selected candidate costs at least 0 with its selected inputs: a_candidate_dearer_than_a_feasible_plan_does_not_hide_a_cheaper_plan (max by (job)(count_over_time(m[1m])): without the bound, an unused 2.55e6/s chain set the scale and HiGHS returned a plan 12.6% dearer; now the 0.0078/s plan). The disk-row placement: raw_data_on_disk_costing_less_than_the_solver_tolerance_still_steers_selection (at 3.1e-17 per disk byte-second, 2.542e-9/s selected; with the scale on $r_s$'s objective coefficient, the 4.001e-9/s plan selected when disk is free).

  • Complexity: one node_resource_usage_and_cost per candidate plus one per selectable edge; the bound is O(passes · edges), passes ≤ candidates (one or two in practice), at most 32 rebuilds; the MILP has |C| + |E_sel| + Σ|G_q| + |sources| variables plus $t_v$ inside cycles. On 148 workloads that plan (release build, HiGHS seed 0), the (B)/(D) trim and the row placement cut total solve time from 203.5 s to 111.7 s and constraint rows from 91,659 to 68,336, objectives equal (relative ≤ 1e-9) on 147 and lower by 5.5e-9 relative on y17 (solver tolerance). Solve time depends on HiGHS's random seed; medians over seeds 0–4, without → with the trim:

    Workload Rows Median solve time
    c09 topk(rate) 7,823 → 4,906 101.7 s → 64.3 s
    c16 topk(rate) over a chain 38.5 s → 30.0 s
  • Assumptions and limits: query latency and cluster capacity are not checked; an @ T read keeps its input unbounded; one $x_v$ per candidate and no accuracy constraint (limits under Follow-ups).

Time windows (types, cost, Stage 3, installer, executor)

  • Problem: the read-back walk existed five times (asap-cost, the installer's ViewIndex, the executor's sources/window.rs), over bare (i64, i64) windows, and each treated a chain as reading its newest window, which is wrong for a chain built at query time: sum_over_time(m[5m] offset 1m) + sum_over_time(m[5m]) would keep 60,000 ms of m instead of 360,000.
  • After: the four layers above. Callers fold the windows (largest reaches_back_ms for retention, union for the executor's slots). Behaviour that follows from the walk:
    • a query-time chain reads WhatItsReadersRead, so its input keeps what its readers read;
    • a TimeShift's retention_ms includes its offset: the up walk starts at the producer's own shifts_back_ms, so a stored TimeShift(1h) read by TimeRange(5m) keeps 1 h 5 m;
    • a TimeShift stored at ingestion time reads CurrentTimeOnly: the chain below it keeps only its newest window (retention 0; the store always keeps the rows appended last), not the offset;
    • a stored TimeShift directly over raw data reads it unbounded (raw data needs every row), so plan_cost refuses that Scan ("raw data retained without a bound"), as the installer does;
    • the installer reads a node that a query root reads through WhatItsReadersRead nodes unbounded (ViewIndex::read_through_a_root); asap-cost does not.
  • Correctness: time_window::tests (union, bounds, readers_read_a_producer_through_the_shifts_above_it: both roles of a stored shift and the raw-data case); runtime_install::shifted_and_unshifted_sums_over_a_shared_query_time_chain_install_and_answer_exactly (retention ≥ 360,000; answers equal the exact sums) and grouped_and_per_series_counts_over_a_shared_query_time_chain_install_and_answer; installable::tests::a_time_shift_view_keeps_its_offset_more (1 h retention for a TimeShift(1h) view read for 1 h is refused, 2 h installs; the chain below installs with one window); installable::tests::a_root_time_shift_is_read_whole_by_the_installer_not_by_the_cost (a hand-built TimeShift(5m) root over a query-time Scan: asap-cost reads it back 0, the installer keeps it unbounded; no frontend builds this shape directly).
  • Complexity: unchanged order, one walk per producer; the Stage 3 walk does not clone a Vec per step.
  • Assumptions and limits: a negative offset is read unbounded.

Costing and retention (asap-cost)

  • Problem: θ was not read from the node; chain retention came from separate helpers; a merge over a chain's windows output one state per state read; an ingestion-time scan feeding chains of two widths was counted once per width.
  • After:
    • θ from PhysicalASAPDAGNode::parameters; retention from retention_ms; held query-time memory inside average_memory_bytes; tumbling_windows on producers.
    • computations runs an ingestion-time node once per second, whatever views read it.
    • A query-time Scan reads per execution what its consumers read, once, and keeps its retention_ms on disk.
    • A node materialized at ingestion time keeps one output per refresh its retention_ms covers, plus the one being built: a chain per tumbling window; any other view per window it maintains (maintained_window_ms, which stops at a chain, so a view over a 60 s chain refreshes per 60 s), per second of rows when it maintains none. A TimeShift's offset is in its retention_ms, so the cost has no separate offset term.
    • A view (materialized across evaluations, not a Scan) without retention_ms is not costed (MissingOrZero("retention_ms")), so Stage 3 does not select it; these are exactly the views the installer refuses (ViewWithoutRetention). sum_over_time(m[5m] offset -5m) now ends at planning ("the selection MILP is infeasible; … retention_ms is missing or zero") instead of selecting an uninstallable view.
  • Correctness: usage::tests::{query_time_memory_is_held_only_while_the_node_computes, a_query_time_scan_reads_what_its_consumers_read_and_keeps_its_retention, a_merge_over_a_chain_s_windows_outputs_one_state_per_group, an_ingestion_time_view_of_raw_rows_keeps_the_rows_its_retention_covers, a_view_over_an_ingestion_time_chain_keeps_one_output_per_chain_window, a_view_without_retention_is_not_costed}; stage3_objective::{an_ingestion_time_scan_feeding_chains_of_two_widths_is_charged_once_per_second, the_objective_includes_request_overhead_and_charges_raw_data_on_disk_once_per_source, offset_windows_without_raw_data_select_installable_plans_costed_for_their_retention}. The last one: sum_over_time(m[5m] offset 90s) every 60 s without raw data selects an installable chain plan, not a TimeShift view over arriving rows costed as one second of rows.
  • Complexity: unchanged order.
  • Assumptions and limits: query latency is not costed.

Chains and the time-indexed store (executor)

  • Problem: summaries panicked without θ; a chain was to be expanded by the installer into N SummaryAgg windows; a query-time chain was refused.
  • After:
    • At ingestion, a run over (start, end] builds one window state (window_end = end); append_materialized_view appends it and drops rows older than retention_ms.
    • At query time, a chain reads the window its readers read (whole windows of width w); with_tumbling_window_ends(w) stamps each row with the end of its window counted back from that window's end, and the build groups by that end, so the TimeRange above selects windows by window_end as over stored ones. A query reads the windows with window_end ∈ (T − o − r, T − o] (w | r and w | o are checked).
    • A TimeShift with @ evaluates its input at the anchor (evaluate_at, input_context).
    • The refresh of a TimeShift stored over a chain is refused ("materialized view N is read unbounded"): the executor's walk reads every slot whole outside a window, and how a stored shift stamps its rows is not defined yet ([Bug] check_compiles accepts a plan whose view over an ingestion-time chain cannot be refreshed on its own #1291).
  • Correctness: executor/tests/tumbling_window_chain.rs (24 refreshes of a 10-min exact-sum chain; reads through TimeRange(1h) and TimeShift(1h)→TimeRange(1h) equal direct sums; a read past 2 h retention fails; a_chain_built_at_query_time_answers_queries_through_its_time_ranges; a_range_window_stamps_each_row_with_its_tumbling_windows_end); physical_planner::tests::a_stored_time_shift_over_a_chain_reads_the_chain_unbounded (as root and not).
  • Complexity: one pass over the input rows per query-time chain; one window state per ingestion run.
  • Assumptions and limits: non-chain AcrossEvaluations results are stored by the interval of data they summarize; rows from an @-anchored subtree keep the anchor time.

Pass 2: exact queries read exactly

ErrorMetricsOfMemo::exact_only_groups_of_query (asap-accuracy) gives an exact query no allowed error metric at its root, so every group below it is exact-only for it. Test: pass2_promql::exact_only_groups::an_exact_query_reads_every_sketch_readout_only_exactly. Stage 3 enforces this list per query in PR 7.

Deleted or replaced code

Deleted or replaced Replaced by
RawDataWindow, MaterializedViewInputs::raw_data_windows, the lookback_ms test forwarder, bare (i64, i64) windows TimeWindow
PhysicalASAPDAG::set_retention; the installer's reader_span_ms_by_node Stage 3 sets retention_ms in plan_of
The installer's chain expansion into N SummaryAgg windows (main: todo!()) native chains in the executor
asap-cost plan/retention.rs (tumbling_window_retention_ms, tumbling_window_retention_ms_of_plan, retained_tumbling_windows), tumbling_windows_of the node's retention_ms; ProducerOutput::tumbling_windows
NodeUsage::held, query_reach held memory inside average_memory_bytes
node_usage's retained_tumbling_window_count, ingestion_window_ms, formula_scale; plan_cost's formula_scale; computations's root_rates window_ms; settings.frozen_calibration_factor; rates from entries
Read-back walks in asap-cost, ViewIndex (windows_below, offset_below, its consumers map) and the executor's sources/window.rs (ScanWindow, time_range_window, scan_windows, view_windows) time_window walks over TimeIndexedPlan
InMemoryDataStore::with_materialized_view append_materialized_view (time-indexed, evicts by retention)
PredictedNodeUsage::{stage3_node_id, installed_node_ids} node_id
installable_plan_of's cost argument and report plan_installable returns the report
Stage 3's utilization_constraints stub and the other todo!() constraint bodies (A)–(F), the one-consumer rule, raw_data_disk_constraints
Selection::{milp_variables, milp_constraints} milp_variable_count, milp_constraint_count (the CLI JSON keeps the old keys)
Priced, Planner::price, reference_price, last_root_is_key_filter Costed, cost, reference_cost

Decisions and open questions

Needs reviewer decision: none.

Decided:

  • One time-window interface in four layers (value, one node, graph trait, walks); TimeIndexedPlan keeps operator_of for the walks down, and returns Vec (the walks touch a few nodes per call).
  • The root rule lives in the installer (ViewIndex::read_through_a_root) and the executor (CompileGraph), not in the shared PhysicalASAPDAG implementation, so asap-cost reads a plan as Stage 3 reads its candidates.
  • The executor keeps refusing a stored TimeShift's refresh until [Bug] check_compiles accepts a plan whose view over an ingestion-time chain cannot be refreshed on its own #1291; the installer does not encode that executor gap.
  • Query-time chains are built by the executor; Stage 2 does not prune them (an executor gap is not a planner invariant).
  • A candidate's raw data on disk is charged once per source in Stage 3; the request overhead is in the reported objective, not the MILP.
  • The objective is scaled after leaving out candidates dearer than a feasible plan; scaling by the median coefficient was rejected.
  • IngestionRefreshGroup::refresh_window stays public with only test readers: it is the window a runtime schedules the group's ingestion run by.
  • promql_to_post_asap::value_ranked_topk_over_binary_ratio_finalizes_both_summary_operands accepts either of two plans whose costs differ by 2e-11 relative (below HiGHS's tolerances); both finalize an exact summary per operand, which is what it checks.

Verification

  • cargo fmt --all -- --check and cargo clippy --workspace --all-targets --all-features -- -D warnings: clean at the head.
  • Covering suites at the head (asap-types, asap-cost, asap-stage3-plan-selection, asap-planner, asap-executor, asap-integration-tests, asap-devtools): 795 passed, 66 failed. All 66 also fail on main (which fails 183, with Stage 3 and the installer todo!()); none is newly failing.
Failing tests Count Why
devtools asap_planner CLI tests and asap_planner_acceptance 41 the CLI stops at a later todo!() before writing its output; PR 9 (#1239)
devtools planner_fuzz 15 todo!() "#1207 PR 9: the selected plan's cost (plan_cost) and roots from its PhysicalASAPDAG"
planner installable::tests::view_refreshes_compile_only_the_nodes_below_them 1 infeasible until PR 8 (#1238): Pass 2's error_contract lets avg/sum accept absolute error only, so every quantile sketch is cut and no chain exists
planner runtime_install::check_compiles_reports_the_executors_error 1 the executor has no native implementation of an aggregate intent (#1242)
planner summary_sharing::identical_ungrouped_queries_share_their_producers 1 pins a plan shape the baseline does not select
integration joint_search_equivalence 2 expect an error from capacity and latency checks the baseline does not have
integration materialized_views::materialized_views_equal_the_stored_outputs_main_listed 1 old-wire fixture; regenerated in PR 9
integration plan_sharing 2 expect the cheaper reuse plan (0.0083/s; the baseline selects 0.0096/s and 0.0124/s)
integration promql_to_post_asap 2 pin a heap topk candidate and a KLL over a rate accumulator that the baseline does not select
  • End-to-end harness (planning through plan_installable, plan_cost, installation, execution against an exact reference, and a brute force over the dumped MILP inputs; 197 workloads in sets of functional cases, offset and stored-shift cases, and windowed SQL):
Set Planned Objective = plan_cost (1e-9) Executed / within target
Functional (117) 98 98 54 / 52 (p17, x16: the shared-group limit below)
Offsets without raw data (25) 25 24 (y01: the scan-read gap below) 22 / 22
a01–a22 21 21 21 / 21
f01–f03 3 3 3 / 3
b01–b21 18 (b08, b09 end at planning: negative offsets) 18 13 / 13 (b17 hangs, #1290)

The brute force over 167 dumps finds no plan cheaper than the MILP's: 142 equal, y20 within 1.3e-9 relative, p29 and x06 time out with the MILP below the best found, 21 skipped (over 400 candidates). Nineteen forced stored-shift plans: chain retention 0, shift retention offset + range, refresh refused (#1291); a forced stored shift over a stored Scan is refused by plan_cost. After the time-window interface rework, a 98-workload sample (sets f, b, g, h, y03, the offset set and every fourth functional case) gives the same status, MILP objective, plan_cost and views on all 98, and the same answers on 96; b13 and y09 (KLL quantiles, randomized compaction) answer differently on every run and stay within their rank target. The 44 functional workloads refused at install are executor and frontend gaps the exact reference cannot run either (21 topk, 6 standalone scalar roots, 5 subquery value operations, 2 delta, 2 @ T raw windows, 1 vector shape) and 7 tracked in #1288 and #1289.

  • Timings: in Algorithm changes (medians over HiGHS seeds 0–4; solve time depends on the seed and repeats exactly for a fixed one).
  • Screenshots: not applicable.

Follow-ups

Gap Where
One $x_v$ per candidate and no accuracy constraint: a query can read a sketch sized for another query's target in a shared group (quantile_over_time(0.9, m[5m]) exact and at rank 0.01 both read KLL k = 2,863; p17, x16); Stage 3 does not yet enforce exact_only_groups_of_query per query PR 7 (#1237, execution contexts), PR 8 (#1238, constraint (K))
A query-time Scan's read and retention are costed for the most any selectable reader reads, not the selected readers (y01: objective 2.2565 vs plan_cost 2.2523; z02 selects 1.0377/s where 1.0279/s exists) PR 7 (#1237): $h_v$ and constraint (R)
view_refreshes_compile_only_the_nodes_below_them fails (avg/sum accept absolute error only, so no quantile chain) PR 8 (#1238)
Snapshot JSONs (crates/devtools/tests/{corpus_snapshots,sketch_plan_snapshots}) still carry the stale raw_data_windows keys; viewer and remaining devtools todo!()s PR 9 (#1239)
One query whose two branches read one stored chain hangs in the executor (b17, g10) #1290
The executor refuses a stored TimeShift's refresh; install checks compile a refresh group as one DAG, not each view's refresh on its own #1291
Executor counter_window, node_below_shifts, below_time_shifts re-walk shifts by hand instead of the shared walk #1292
Ingestion-time views over an unwindowed SQL table (s01, x13, c13, c14, s09, x04); refreshing an ingestion-time TimeShift view over raw rows #1288
count by (job) (count by (job, host) (m)): exact Count over a finalized count #1289
Exact Reduce TopK, topk output names #1242
Filtered builds at ingestion; ingestion reorder buffer for counter states #1247
stddev/deriv per entity, Reduce stddev, delta #1241, #1242
Subquery value operations, @ T raw windows, standalone scalar roots not filed
The MILP's weak LP relaxation (root bound about 2% of the optimum on c21); Stage 2's edges quadratic in alternatives per group deferred optimizer work
Query latency and cluster capacity are not checked by selection needs its own design
PRs 8–9 sit on an older base and need ports to this layout: Stage 3 select.rs → lib.rs/selectable.rs/report.rs, cost.rs → cost.rs/plan.rs/retention.rs, milp/model.rs → milp/model/mod.rs; asap_cost::read_back_ms and computations signatures; Planner::cost/Costed; installable_plan_of(plan, entries) PR 8 (#1238), PR 9 (#1239)

@zzylol zzylol added optimizer Workload optimization and plan selection refactor labels Nov 7, 2026
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from d11f12e to 616f765 Compare November 9, 2026 07:43
@zzylol
zzylol force-pushed the refactor/1207-stage3-baseline branch from 09d69f1 to 7d1bd55 Compare November 9, 2026 09:01
@zzylol zzylol changed the title refactor(stage3): baseline selection MILP over x, y, z, t with (A)-(I), realizer and plan export with θ and coverage (#1207 PR 6) refactor(stage3): baseline selection MILP and its PhysicalASAPDAG; executor θ, native chains and a time-indexed store (#1207 PR 6) Nov 9, 2026
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from 616f765 to f20177e Compare November 9, 2026 11:53
@zzylol
zzylol force-pushed the refactor/1207-stage3-baseline branch from 7d1bd55 to e53cf6e Compare November 9, 2026 12:11
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from f20177e to e05016c Compare November 9, 2026 17:44
@zzylol
zzylol force-pushed the refactor/1207-stage3-baseline branch from e53cf6e to 588d1a2 Compare November 9, 2026 17:48
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from e05016c to 513a1e7 Compare November 9, 2026 21:28
@zzylol
zzylol force-pushed the refactor/1207-stage3-baseline branch 3 times, most recently from 0723ff6 to e3938ec Compare November 10, 2026 00:40
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from 513a1e7 to 4c77ec6 Compare November 10, 2026 01:26
@zzylol
zzylol force-pushed the refactor/1207-stage3-baseline branch from e3938ec to db6d114 Compare November 10, 2026 01:26
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from 4c77ec6 to 8876072 Compare November 10, 2026 13:37
@zzylol
zzylol force-pushed the refactor/1207-stage3-baseline branch from db6d114 to c9e1579 Compare November 10, 2026 13:37
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch 3 times, most recently from 07678ec to 89bf901 Compare November 12, 2026 20:01
…ive tumbling-window chains and a time-indexed store in the executor (#1207 PR 6)

Stage 3 costs each physical candidate (w_v) and each selectable edge
(c_{u,c,i}) with asap-cost, solves the baseline MILP over x, y, z, t
(constraints (A)-(F), (E) only inside strongly connected components, a
query-time result that is not materialized read by at most one consumer),
and builds the PhysicalASAPDAG of the optimum directly, each node with its
timing, lifetime, θ, retention and, materialized, its view coverage
(asap_types::plan::view_coverage of its memo group).
asap-cost sizes summaries from the node's θ and prices chains and raw data
by the node's retention_ms. The executor builds and runs summaries with θ,
runs tumbling-window chains natively (TimeRange/TimeShift over a chain
select windows), keeps materialized rows in a time-indexed store with
retention, and evaluates TimeShift{at} once per request. Installation no
longer expands chains.
…the exact avg, through Stage 3's plan (#1207 PR 6)

Re-adds the three avg tests Pass 1's PR moved here: each plans the query
through every stage with the avg group's own expression left out, so Stage
3 selects the plan that computes avg as sum over count, and executes it
beside the exact plan (NULL and '' groups, a non-Utf8 key, an empty global
input, avg's count of its own input, an Int64 column).
…les (#1207 PR 6)

Move-only: each inline test module of a file PR 6 changes becomes a
sibling file (<name>/mod.rs + <name>/tests.rs, or a sibling of mod.rs):
the planner fuzz relations, the executor's compilation, state-column
updater, supported-summaries, batch and weighted-state tests, the
installer's tests and the execution data state's.
@zzylol
zzylol force-pushed the refactor/1207-stage3-baseline branch from c9e1579 to 78ec24c Compare November 12, 2026 20:36
@zzylol
zzylol changed the base branch from refactor/1207-stage2 to main November 12, 2026 20:36
…ers, not one run's (#1207 PR 6)

A materialized ingestion-time node that is neither a chain nor a Scan was
charged for one run's output and the one being built. A TimeShift over the
arriving rows keeps far more: `sum_over_time(m[5m] offset 90s)` without raw
data selected such a view at 0.01225/s, charged for one second of rows, and
the installer refused it. It now keeps one output per refresh that its
retention_ms covers (per window it maintains, per second of rows without
one), a TimeShift's offset more, plus the one being built; plan_cost and
Stage 3 share the rule, and the installable chain plan (0.10425/s) is
selected. Likewise `sum_over_time(m[1m] offset 1h)` hourly with a 10 s
`sum_over_time(m[1m])` now selects the chain plan (0.4925/s).
@zzylol
zzylol force-pushed the refactor/1207-stage3-baseline branch from 3718599 to 9ec3847 Compare November 13, 2026 17:37
#1207 PR 6)

A query-time result that is not materialized has the (F) equality
sum y + sum z = x_v, which with y, z >= 0 already implies its (B) rows
y <= x_u and its (D) rows z <= x_v. Those rows are no longer built:
c09 (one topk(rate) query) goes from 7,823 rows to 4,906.

The raw-data rows now carry the objective scale (r_s >= scale * d_v x_v)
and r_s has objective coefficient 1, so the objective's coefficients stay
in a narrow range instead of reaching scale (up to 1.7e5) on r_s.
… are finalized summaries (#1207 PR 6)

For topk(1, sum by(job)(increase(a[6h])) / sum by(job)(increase(b[6h])))
two plans' costs differ by about 2e-11 relative (0.03720476463786/s and
0.03720476463869/s), below the solver's tolerances: each operand
finalizes the sum by job itself, or finalizes an increase per series and
sums it by job. The test pinned the first; with the MILP's implied rows
left out the solver returns the second. It now accepts either, since both
finalize an exact summary for each operand, which is what it checks.
…w over a chain refreshes per window (#1207 PR 6)

The shared retention walk (windows_read_above) now starts at the
producer's own offset when the producer is itself a Through reader. An
ingestion-time TimeShift(1h) read by TimeRange(5m) gets retention_ms
1 h 5 m, as Stage 3, the installer's read_back_ms and the store's
eviction all read it, and the cost charges what retention_ms covers with
no separate offset term.

maintained_window_ms returns the chain's tumbling window when the walk
reaches a chain: a view over an ingestion-time chain is refreshed once
per chain window, not once per second. For sum_over_time(m[5m] offset 1h)
such a view was charged 218,456,000 bytes (3,901 one-second outputs); it
now keeps 66 windows, as the chain below it does.

The ingestion-time chain arm is gone: one retained output count (a
chain's windows, a view's refreshes) serves both timings, and one
ONE_SECOND_MS names the second of arriving rows.
…oc (#1207 PR 6)

The private EachNodeResourceUsageAndCost struct had one caller and only
returned two maps; its loop is now in plan_cost. The plan_cost doc says
that every node materialized across evaluations keeps what its
retention_ms covers, not only a chain.
…1207 PR 6)

Selection::selected_plan_cost_per_second is printed as
milp_objective_cost_per_second next to the MILP's size and times, and
milp_constraint_count has its own doc comment.
…ains, clearer names (#1207 PR 6)

- has_at_most_one_consumer is is_unmaterialized_query_time: it tests a
  query-time result that is not materialized, which is why it has at most
  one consumer. The bound's own copy of the test calls it.
- plan_cost_per_second sums a plan's candidates, edges and each source's
  raw data on disk once; the bound and the solved optimum both call it
  (the optimum adds the request overhead).
- Stage 3's cost_node no longer matches a chain before calling
  maintained_window_ms, which already returns a chain's width: Pass 2
  adds chain candidates only to the group that computes the chain.
- Nits: the retained-count alias, a rewrapped plan_cost doc, and the
  installer test helper's views as an Option instead of an empty string.
A view materialized across evaluations whose readers read it unbounded
(a negative offset, an `@` anchor) has no retention_ms. The cost charged
that as one output, so sum_over_time(m[5m] offset -5m) selected an
ingestion-time TimeShift(-5m) view at 0.01225/s that the installer then
refused (ViewWithoutRetention). Such a node now fails with
MissingOrZero("retention_ms"), the same set of nodes the installer
refuses (any view across evaluations but a Scan), so Stage 3 does not
select it.
…the chain below it does not (#1207 PR 6)

A TimeShift without `@` materialized across evaluations at ingestion
time keeps its own rows, which its readers read an offset later, and each
refresh reads the newest output of its input. The read-back walk passed
up through it as through a query-time shift, so the chain below it kept
the offset too: for sum_over_time(m[5m] offset 1h) with the shift stored
over a 60 s chain, both kept 3,900,000 ms (3,696,000 B each); the chain
needs its newest window and the one being built (112,000 B), 0.448/s
less.

TimeIndexedRead::of takes the node's lifetime and gives such a shift
StoredThrough { offset_ms }: the walk starts at its offset when it is the
producer (its retention is still 1 h 5 m) and ends at it with the empty
window when it is a reader. Stage 3's retention, the installer's
read_back_ms and asap_cost::read_back_ms all read it through this one
function.
…rance still steers selection (#1207 PR 6)

With a disk price of 3.1e-17 per byte-second, keeping raw data on disk
costs about 2e-9 per second, below the 1e-9 under which HiGHS drops a
matrix coefficient. With the objective scale on r_s's coefficient, the
unscaled raw-data rows lost those coefficients and Stage 3 selected the
plan it selects when disk is free: 4.001e-9/s for
sum_over_time(m[5m] offset 30m) + sum_over_time(m[5m]) every minute and
count_over_time(m[2m]) every hour. With the scale in the row it selects
the 2.542e-9/s plan. The test fails on the old placement.
…PR 6)

With StoredThrough, the executor's slot walk no longer mapped a stored
TimeShift at a compile root to Whole, so the refresh of a stored shift
over a chain compiled, read the chain's newest window, and the forced
sum_over_time(m[5m] offset 5m) plan answered nothing (0 of 12 series).
How a stored shift stamps the rows it stores is undefined until the
executor supports such a refresh, so it is refused as before: a Through
or StoredThrough reader at a compile root reads its input whole.
…ded (#1207 PR 6)

As a reader, TimeIndexedRead::StoredThrough reads its input at its
evaluation time, as AtEvaluationTime does, but gave {0, 0} even over raw
data, which has no newest output. Forced sum_over_time(m[5m] offset 1h)
with raw data kept and the shift stored over a stored Scan read the Scan
back 0 and costed the plan 0.0198/s against 60,855/s with the Scan kept
for the offset and range; the installer refuses that plan
(ViewReadsUnwindowedSource). The walk now reads raw data unbounded under
it, so such a Scan is not costable.

The executor's walk reads every consumer outside a window unbounded, so
a stored shift's refresh reads a chain slot unbounded and is refused
without its own arm; the arm is gone and the doc says what happens over
a chain (refused) and over a raw Scan (every row is read). New executor
test: the chain under a stored TimeShift(5m) is read unbounded, as the
root or not. The types walk test covers `of` and the raw-data case; the
cost test that repeated it is removed.
The cost's no-retention check and the installer's two view filters each
spelled "materialized across evaluations and not a raw-data Scan"; they
call is_view now, so they refuse the same nodes by construction. The
DAG check for a retention on a node that keeps no rows calls the
existing retains_rows. The installer's root guard says why a stored
shift at a root is not read whole. The bound's `unshared` alias is
inlined.
…lks over one plan trait (#1207 PR 6)

TimeIndexedRead answered two questions in one enum: which input rows a
node reads, and how much earlier its output rows are than those. That is
why a stored TimeShift needed its own variant. The module is now four
layers:

1. TimeWindow { ends_before_evaluation_ms, length_ms } (was end_offset_ms,
   width_ms), reaches_back_ms (was read_back_ms), union, bounds_ms.
2. InputTimeAccess { reads: InputRows, shifts_back_ms } with InputRows
   WhatItsReadersRead, LastWindow { length_ms }, CurrentTimeOnly, AllRows,
   replacing TimeIndexedRead (Through, StoredThrough, Window,
   AtEvaluationTime, Whole). A stored TimeShift is CurrentTimeOnly
   shifting back its offset.
3. TimeIndexedPlan: readers_of, inputs_of, input_time_access,
   is_raw_data and operator_of, replacing the closures and the
   at_evaluation_time_reads_whole flag. Implemented by PhysicalASAPDAG
   (installer and asap-cost), Stage 3's Candidates and the executor's
   CompileGraph.
4. windows_readers_need, windows_computed_from, shift_back_ms_below (were
   windows_read_above, windows_below, end_offset_below).

Behaviour is unchanged: the same walk rule, the same root rule (a root
read through reads all rows) in the installer and the executor, the
executor still counting every node as raw data. asap-cost's scan read
shares the installer's PhysicalASAPDAG plan, so it now applies the root
rule too: a Scan reached only through shifts or query-time chains up to a
query root is read whole. No workload of the review sets has such a root
(their objectives and plan costs are unchanged). asap_cost::read_back_ms
takes the plan. The installer's ViewIndex no longer keeps a consumers
map. No snapshot serializes a TimeWindow, so no snapshot changes.
…he root rule is the installer's (#1207 PR 6)

TimeIndexedPlan::is_raw_data is current_time_read_needs_every_row: it
answers whether a reader of only the current time in fact needs every
row, true for raw data. PhysicalASAPDAG and Stage 3's Candidates say so
for a Scan; the executor's CompileGraph for every node, since today's
executor store reads every slot whole outside a window.

PhysicalASAPDAG's input time access is plain InputTimeAccess::of again,
so asap-cost reads a plan as Stage 3 reads its candidates, as before the
redesign. The installer applies its root rule itself
(ViewIndex::read_through_a_root): a node a query root reads through
nodes that read what their readers read is read unbounded. New test: a
root TimeShift(5m) over a query-time Scan is read back 0 by asap-cost
and kept unbounded by the installer (it fails without the installer's
check).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Workload optimization and plan selection refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant