Repository navigation
refactor(stage3): baseline selection MILP and its PhysicalASAPDAG; executor θ, native chains and a time-indexed store (#1207 PR 6) - #1236
Merged
Conversation
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 9, 2026 07:43
d11f12e to
616f765
Compare
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
from
November 9, 2026 09:01
09d69f1 to
7d1bd55
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 9, 2026 11:53
616f765 to
f20177e
Compare
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
from
November 9, 2026 12:11
7d1bd55 to
e53cf6e
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 9, 2026 17:44
f20177e to
e05016c
Compare
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
from
November 9, 2026 17:48
e53cf6e to
588d1a2
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 9, 2026 21:28
e05016c to
513a1e7
Compare
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
3 times, most recently
from
November 10, 2026 00:40
0723ff6 to
e3938ec
Compare
This was referenced Nov 10, 2026
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 10, 2026 01:26
513a1e7 to
4c77ec6
Compare
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
from
November 10, 2026 01:26
e3938ec to
db6d114
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 10, 2026 13:37
4c77ec6 to
8876072
Compare
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
from
November 10, 2026 13:37
db6d114 to
c9e1579
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
3 times, most recently
from
November 12, 2026 20:01
07678ec to
89bf901
Compare
…ive tumbling-window chains and a time-indexed store in the executor (#1207 PR 6) Stage 3 costs each physical candidate (w_v) and each selectable edge (c_{u,c,i}) with asap-cost, solves the baseline MILP over x, y, z, t (constraints (A)-(F), (E) only inside strongly connected components, a query-time result that is not materialized read by at most one consumer), and builds the PhysicalASAPDAG of the optimum directly, each node with its timing, lifetime, θ, retention and, materialized, its view coverage (asap_types::plan::view_coverage of its memo group). asap-cost sizes summaries from the node's θ and prices chains and raw data by the node's retention_ms. The executor builds and runs summaries with θ, runs tumbling-window chains natively (TimeRange/TimeShift over a chain select windows), keeps materialized rows in a time-indexed store with retention, and evaluates TimeShift{at} once per request. Installation no longer expands chains.
…the exact avg, through Stage 3's plan (#1207 PR 6) Re-adds the three avg tests Pass 1's PR moved here: each plans the query through every stage with the avg group's own expression left out, so Stage 3 selects the plan that computes avg as sum over count, and executes it beside the exact plan (NULL and '' groups, a non-Utf8 key, an empty global input, avg's count of its own input, an Int64 column).
…les (#1207 PR 6) Move-only: each inline test module of a file PR 6 changes becomes a sibling file (<name>/mod.rs + <name>/tests.rs, or a sibling of mod.rs): the planner fuzz relations, the executor's compilation, state-column updater, supported-summaries, batch and weighted-state tests, the installer's tests and the execution data state's.
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
from
November 12, 2026 20:36
c9e1579 to
78ec24c
Compare
…", not "price" (#1207 PR 6)
…ers, not one run's (#1207 PR 6) A materialized ingestion-time node that is neither a chain nor a Scan was charged for one run's output and the one being built. A TimeShift over the arriving rows keeps far more: `sum_over_time(m[5m] offset 90s)` without raw data selected such a view at 0.01225/s, charged for one second of rows, and the installer refused it. It now keeps one output per refresh that its retention_ms covers (per window it maintains, per second of rows without one), a TimeShift's offset more, plus the one being built; plan_cost and Stage 3 share the rule, and the installable chain plan (0.10425/s) is selected. Likewise `sum_over_time(m[1m] offset 1h)` hourly with a 10 s `sum_over_time(m[1m])` now selects the chain plan (0.4925/s).
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
from
November 13, 2026 17:37
3718599 to
9ec3847
Compare
#1207 PR 6) A query-time result that is not materialized has the (F) equality sum y + sum z = x_v, which with y, z >= 0 already implies its (B) rows y <= x_u and its (D) rows z <= x_v. Those rows are no longer built: c09 (one topk(rate) query) goes from 7,823 rows to 4,906. The raw-data rows now carry the objective scale (r_s >= scale * d_v x_v) and r_s has objective coefficient 1, so the objective's coefficients stay in a narrow range instead of reaching scale (up to 1.7e5) on r_s.
… are finalized summaries (#1207 PR 6) For topk(1, sum by(job)(increase(a[6h])) / sum by(job)(increase(b[6h]))) two plans' costs differ by about 2e-11 relative (0.03720476463786/s and 0.03720476463869/s), below the solver's tolerances: each operand finalizes the sum by job itself, or finalizes an increase per series and sums it by job. The test pinned the first; with the MILP's implied rows left out the solver returns the second. It now accepts either, since both finalize an exact summary for each operand, which is what it checks.
…w over a chain refreshes per window (#1207 PR 6) The shared retention walk (windows_read_above) now starts at the producer's own offset when the producer is itself a Through reader. An ingestion-time TimeShift(1h) read by TimeRange(5m) gets retention_ms 1 h 5 m, as Stage 3, the installer's read_back_ms and the store's eviction all read it, and the cost charges what retention_ms covers with no separate offset term. maintained_window_ms returns the chain's tumbling window when the walk reaches a chain: a view over an ingestion-time chain is refreshed once per chain window, not once per second. For sum_over_time(m[5m] offset 1h) such a view was charged 218,456,000 bytes (3,901 one-second outputs); it now keeps 66 windows, as the chain below it does. The ingestion-time chain arm is gone: one retained output count (a chain's windows, a view's refreshes) serves both timings, and one ONE_SECOND_MS names the second of arriving rows.
…oc (#1207 PR 6) The private EachNodeResourceUsageAndCost struct had one caller and only returned two maps; its loop is now in plan_cost. The plan_cost doc says that every node materialized across evaluations keeps what its retention_ms covers, not only a chain.
…1207 PR 6) Selection::selected_plan_cost_per_second is printed as milp_objective_cost_per_second next to the MILP's size and times, and milp_constraint_count has its own doc comment.
…ains, clearer names (#1207 PR 6) - has_at_most_one_consumer is is_unmaterialized_query_time: it tests a query-time result that is not materialized, which is why it has at most one consumer. The bound's own copy of the test calls it. - plan_cost_per_second sums a plan's candidates, edges and each source's raw data on disk once; the bound and the solved optimum both call it (the optimum adds the request overhead). - Stage 3's cost_node no longer matches a chain before calling maintained_window_ms, which already returns a chain's width: Pass 2 adds chain candidates only to the group that computes the chain. - Nits: the retained-count alias, a rewrapped plan_cost doc, and the installer test helper's views as an Option instead of an empty string.
A view materialized across evaluations whose readers read it unbounded
(a negative offset, an `@` anchor) has no retention_ms. The cost charged
that as one output, so sum_over_time(m[5m] offset -5m) selected an
ingestion-time TimeShift(-5m) view at 0.01225/s that the installer then
refused (ViewWithoutRetention). Such a node now fails with
MissingOrZero("retention_ms"), the same set of nodes the installer
refuses (any view across evaluations but a Scan), so Stage 3 does not
select it.
…the chain below it does not (#1207 PR 6) A TimeShift without `@` materialized across evaluations at ingestion time keeps its own rows, which its readers read an offset later, and each refresh reads the newest output of its input. The read-back walk passed up through it as through a query-time shift, so the chain below it kept the offset too: for sum_over_time(m[5m] offset 1h) with the shift stored over a 60 s chain, both kept 3,900,000 ms (3,696,000 B each); the chain needs its newest window and the one being built (112,000 B), 0.448/s less. TimeIndexedRead::of takes the node's lifetime and gives such a shift StoredThrough { offset_ms }: the walk starts at its offset when it is the producer (its retention is still 1 h 5 m) and ends at it with the empty window when it is a reader. Stage 3's retention, the installer's read_back_ms and asap_cost::read_back_ms all read it through this one function.
…rance still steers selection (#1207 PR 6) With a disk price of 3.1e-17 per byte-second, keeping raw data on disk costs about 2e-9 per second, below the 1e-9 under which HiGHS drops a matrix coefficient. With the objective scale on r_s's coefficient, the unscaled raw-data rows lost those coefficients and Stage 3 selected the plan it selects when disk is free: 4.001e-9/s for sum_over_time(m[5m] offset 30m) + sum_over_time(m[5m]) every minute and count_over_time(m[2m]) every hour. With the scale in the row it selects the 2.542e-9/s plan. The test fails on the old placement.
…PR 6) With StoredThrough, the executor's slot walk no longer mapped a stored TimeShift at a compile root to Whole, so the refresh of a stored shift over a chain compiled, read the chain's newest window, and the forced sum_over_time(m[5m] offset 5m) plan answered nothing (0 of 12 series). How a stored shift stamps the rows it stores is undefined until the executor supports such a refresh, so it is refused as before: a Through or StoredThrough reader at a compile root reads its input whole.
…ded (#1207 PR 6) As a reader, TimeIndexedRead::StoredThrough reads its input at its evaluation time, as AtEvaluationTime does, but gave {0, 0} even over raw data, which has no newest output. Forced sum_over_time(m[5m] offset 1h) with raw data kept and the shift stored over a stored Scan read the Scan back 0 and costed the plan 0.0198/s against 60,855/s with the Scan kept for the offset and range; the installer refuses that plan (ViewReadsUnwindowedSource). The walk now reads raw data unbounded under it, so such a Scan is not costable. The executor's walk reads every consumer outside a window unbounded, so a stored shift's refresh reads a chain slot unbounded and is refused without its own arm; the arm is gone and the doc says what happens over a chain (refused) and over a raw Scan (every row is read). New executor test: the chain under a stored TimeShift(5m) is read unbounded, as the root or not. The types walk test covers `of` and the raw-data case; the cost test that repeated it is removed.
The cost's no-retention check and the installer's two view filters each spelled "materialized across evaluations and not a raw-data Scan"; they call is_view now, so they refuse the same nodes by construction. The DAG check for a retention on a node that keeps no rows calls the existing retains_rows. The installer's root guard says why a stored shift at a root is not read whole. The bound's `unshared` alias is inlined.
…lks over one plan trait (#1207 PR 6) TimeIndexedRead answered two questions in one enum: which input rows a node reads, and how much earlier its output rows are than those. That is why a stored TimeShift needed its own variant. The module is now four layers: 1. TimeWindow { ends_before_evaluation_ms, length_ms } (was end_offset_ms, width_ms), reaches_back_ms (was read_back_ms), union, bounds_ms. 2. InputTimeAccess { reads: InputRows, shifts_back_ms } with InputRows WhatItsReadersRead, LastWindow { length_ms }, CurrentTimeOnly, AllRows, replacing TimeIndexedRead (Through, StoredThrough, Window, AtEvaluationTime, Whole). A stored TimeShift is CurrentTimeOnly shifting back its offset. 3. TimeIndexedPlan: readers_of, inputs_of, input_time_access, is_raw_data and operator_of, replacing the closures and the at_evaluation_time_reads_whole flag. Implemented by PhysicalASAPDAG (installer and asap-cost), Stage 3's Candidates and the executor's CompileGraph. 4. windows_readers_need, windows_computed_from, shift_back_ms_below (were windows_read_above, windows_below, end_offset_below). Behaviour is unchanged: the same walk rule, the same root rule (a root read through reads all rows) in the installer and the executor, the executor still counting every node as raw data. asap-cost's scan read shares the installer's PhysicalASAPDAG plan, so it now applies the root rule too: a Scan reached only through shifts or query-time chains up to a query root is read whole. No workload of the review sets has such a root (their objectives and plan costs are unchanged). asap_cost::read_back_ms takes the plan. The installer's ViewIndex no longer keeps a consumers map. No snapshot serializes a TimeWindow, so no snapshot changes.
…he root rule is the installer's (#1207 PR 6) TimeIndexedPlan::is_raw_data is current_time_read_needs_every_row: it answers whether a reader of only the current time in fact needs every row, true for raw data. PhysicalASAPDAG and Stage 3's Candidates say so for a Scan; the executor's CompileGraph for every node, since today's executor store reads every slot whole outside a window. PhysicalASAPDAG's input time access is plain InputTimeAccess::of again, so asap-cost reads a plan as Stage 3 reads its candidates, as before the redesign. The installer applies its root rule itself (ViewIndex::read_through_a_root): a node a query root reads through nodes that read what their readers read is read unbounded. New test: a root TimeShift(5m) over a query-time Scan is read back 0 by asap-cost and kept unbounded by the installer (it fails without the installer's check).
zzylol
marked this pull request as ready for review
November 14, 2026 17:47
This was referenced Nov 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #1207 (PR 6 of the stack). Design: the concepts doc
memo-groups-properties-requirements.md, section 3 "Stage 3: baseline selection and plan output" and section 5 "Cost and retention" (#1224) andplanner-layering.md; this PR updatesphysical-planning-and-deployment.md.Summary
main, Stage 3'sselect_plan, the plan's retention, the installer and the executor's θ and chain handling aretodo!(), so no workload can be planned or installed.PhysicalASAPDAGitself, costing every candidate withasap-cost's own rules so the MILP's optimum equals the plan'splan_cost. Stage 3,asap-cost, the installer and the executor read time windows through one interface (asap_types::plan::time_window). The executor reads θ from the node, runs tumbling-window chains natively (built at ingestion or at query time), and stores views in one time-indexed store.quantile_over_time(0.99, latency_ms[60m])every 10 min (rank error 0.01, 1,000 series at 1 sample/s, no raw data kept) goes fromtodo!()to a selected, installed plan: a KLL chain (k = 1,404, width 600,000 ms,retention_ms3,600,000) at 7.821 cost/s, 7.825 cost/s for the whole plan, equal to Stage 3's optimum.Review guide
main(PRs 1–5 of Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 merged). PR 7 (refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237) is stacked on this branch.crates/stage3-plan-selection/src):lib.rs(select_plan),selectable.rs,cost.rs(retention.rs,milp/model/mod.rs(solve_selection),milp/upper_bound.rs,plan.rs(plan_of).types/src/plan/time_window.rs, then its users:cost/src/plan/read_back.rs,types/src/plan/materialized_view/views.rs(ViewIndex::read_back_ms,read_through_a_root),executor/src/physical_planner/time_window.rs(CompileGraph).cost/src/node/usage/mod.rs(node_resource_usage_and_cost,maintained_window_ms),cost/src/plan/execution_demand.rs,cost/src/plan/cost.rs(plan_cost).physical_planner/mod.rs(chains at ingestion and at query time),operators/time_range.rs,sources/memory.rs.planner/src/installable/mod.rs,types/src/plan/materialized_view/{mod,views,raw_data}.rs.tests.rsfiles; Stage 3, types and executor files split by scope; θ threaded into every executor summary constructor;NodeUsage→NodeResourceUsageAndCost; "price" → "cost" in the planner fuzz.milp/model/mod.rs; the bound inmilp/upper_bound.rs;InputTimeAccess::of(which reader reads what); the retained-output count innode_resource_usage_and_cost;computationsat ingestion time; the query-time chain build (with_tumbling_window_ends, grouping by window end).Behavior: before → after
Planner test
runtime_install::the_plan_carries_stage3s_predicted_usage: p99 over 60 min, evaluated every 10 min at phase 0, rank error 0.01, 1,000 series at 1 sample/s, the deployment keeps no raw data. It serves the MVP need of planning, installing and running a repeated query end to end.main)select_plan(todo!())PhysicalASAPDAGSummaryTumblingWindowChain[KLL k=1404, w=600000] (IT, AcrossEvaluations,retention_ms3,600,000) → 2 TimeRange(3600s) (QT, None) → 3 SummaryMerge (QT, OneEvaluation) → 4 SummaryEstimate(0.99) (QT, None) → 5 Project (QT, None)Selection::selected_plan_cost_per_secondSummaryAggwindowsInterfaces
Wire version stays 15: no node field changes (Stage 3 now sets
view_coverageandparameters, which already existed).MaterializedView's JSON losesinputs.raw_data_windows(not part of the plan file).crates/types/src/plan/time_window.rs(new) ·crates/types/src/plan/mod.rsThe time-window interface, in four layers. Every caller (Stage 3,
asap-cost, the installer, the executor) goes through it.readsshifts_back_msTimeShift(1h)WhatItsReadersReadTimeShift(1h)stored at ingestion timeCurrentTimeOnlyWhatItsReadersReadLastWindow { w }TimeRange(5m)LastWindow { 300_000 }TimeShiftwith@AllRowsCurrentTimeOnlyThe other implementations are private: Stage 3's
Candidates(retention.rs: the readers its caller gives, selectable or selected; admissible producers as inputs; aScanneeds every row) and the executor'sCompileGraph(compiled consumers and dependencies; a compile root reached throughWhatItsReadersReadreadsAllRows; every node needs every row, because today's executor store reads a slot whole outside a window, #1291). The installer adds a root rule on top of the walk inViewIndex::read_through_a_root(types/src/plan/materialized_view/views.rs): a node a query root reads throughWhatItsReadersReadnodes is read unbounded.asap-costreads the plan without that rule, as Stage 3 reads its candidates.crates/types/src/plan/physical_asap_dag.rs·crates/types/src/plan/materialized_view/·crates/types/src/stages/physical/memo_candidates/mod.rs·crates/types/src/ir/properties/execution_data_state/mod.rsimpl PhysicalASAPDAG { + /// The DAG of `nodes` (each after its inputs) with one edge per payload input, carrying the + /// producer's schema and data state; validated. Stage 3 builds its plan this way. + pub fn new( + nodes: Vec<PhysicalASAPDAGNode>, + roots: Vec<PhysicalASAPNodeId>, + scalar_roots: Vec<(usize, ScalarExpr<PhysicalASAPNodeId>)>, + ) -> Result<Self, PhysicalASAPDAGValidationError>; /// Each node's refresh window (signature unchanged; now implemented). pub fn refresh_windows( &self, nodes: impl IntoIterator<Item = PhysicalASAPNodeId>, ) -> HashMap<PhysicalASAPNodeId, Option<RefreshWindow>>; - /// Set every view's retention after export. - pub fn set_retention( - &mut self, - tumbling_window_retention_ms_by_node: &HashMap<PhysicalASAPNodeId, u64>, - reader_span_ms_by_node: &HashMap<PhysicalASAPNodeId, u64>, - ) -> Result<Vec<MaterializedView>, InstallableDagError>; /// Per source, the longest `retention_ms` of its raw-data `Scan`s (signature unchanged; now implemented). pub fn raw_data_retention(&self) -> Result<Vec<RawDataRetention>, InstallableDagError>; } impl PhysicalASAPDAGNode { + /// Whether it keeps time-indexed rows for its readers to read back, so carries `retention_ms`: + /// materialized across evaluations, or a raw-data `Scan` at query time. + pub fn retains_rows(&self) -> bool; + /// Whether it is a materialized view: materialized across evaluations and not a `Scan`, so it + /// keeps what its `retention_ms` covers. + pub fn is_view(&self) -> bool; } impl MaterializedView { /// Which run refreshes it; now public outside tests (the installer reads it). pub fn refresh(&self) -> MaterializedViewRefresh; /// How far back from the newest evaluation time its stored rows are kept; now public outside tests. pub fn retention_ms(&self) -> u64; } /// What a view's refresh reads. The type stays. pub struct MaterializedViewInputs { - /// The windows of raw data it scans directly. - pub(crate) raw_data_windows: Vec<RawDataWindow>, /// The views it reads, by node id. pub materialized_views: Vec<PhysicalASAPNodeId>, } -/// A window of a raw-data source a view's refresh scans. -pub struct RawDataWindow { source: Source, end_offset_ms: i64, width_ms: i64 } impl RefreshWindow { - pub fn of_window(end_offset_ms: i64, width_ms: i64) -> Option<Self>; + /// The refresh window of `window`: its length, and its end mod its length; `None` for an empty one. + pub fn of_window(window: TimeWindow) -> Option<Self>; } /// Why a runtime cannot install a DAG. The type stays. pub enum InstallableDagError { /// Bad input: the DAG is not a valid physical ASAP DAG. InvalidDag(PhysicalASAPDAGValidationError), /// Unsatisfiable request: a raw-data window whose shape no ingestion run refreshes in time. UnsupportedRawDataWindow { time_range_node_id: PhysicalASAPNodeId, reason: &'static str }, /// Unsatisfiable request: a view reads a source `Scan` outside a raw-data window (e.g. a whole SQL table). ViewReadsUnwindowedSource { node_id: PhysicalASAPNodeId }, /// Unsatisfiable request: a view with no aggregate coverage. ViewWithoutCoverage { node_id: PhysicalASAPNodeId, reason: String }, /// Unsatisfiable request: an ingestion-time view reads windows ending at different offsets. IngestionBuildReadsWindowsOfDifferentOffsets { node_id: PhysicalASAPNodeId, - end_offsets_ms: Vec<i64>, + end_offsets_ms: Vec<u64>, }, /// Bad input (a plan file): a view has no `retention_ms`. ViewWithoutRetention { node_id: PhysicalASAPNodeId }, /// Bad input (a plan file): a view is retained for less than its readers read back. ViewRetentionTooShort { node_id: PhysicalASAPNodeId }, /// Bad input (a plan file): a raw-data `Scan` read through a bounded window has no `retention_ms`. ScanWithoutRetention { node_id: PhysicalASAPNodeId }, /// Bad input (a plan file): a raw-data `Scan` is retained for less than its window reaches back. ScanRetentionTooShort { node_id: PhysicalASAPNodeId }, } impl PhysicalASAPMemoCandidates { + /// Physical edge `id`; Stage 3 reads the edges it allocates $y$ on. + pub fn edge(&self, id: PhysicalEdgeId) -> &PhysicalEdge; + /// The number of inputs of candidate `v` (e.g. 2 for a binary operator). + pub fn input_count(&self, v: PhysicalCandidateId) -> usize; } impl DataPrimitive { + /// What a node with `operator` and `result_kind` outputs: `SummaryState` for a summary + /// build/chain/join/maintained population or any `State` result, `Raw` otherwise. + pub fn of<C>(operator: &crate::ir::Operator<C>, result_kind: crate::ir::OperatorResultKind) -> Self; } #[cfg(any(test, feature = "test-support"))] impl ViewCoverage { + /// The coverage of definition `definition_id` over `selection`; for tests of hand-built plans. + pub fn new(definition_id: String, selection: Vec<SelectionBox>) -> Self; } -/// `RawDataWindow::lookback_ms` (`test_support::forwarders`). -pub fn lookback_ms(window: &RawDataWindow) -> u64;RawDataWindowandraw_data_windowshad no non-test reader;TimeWindowreplaces them and the bare(i64, i64)windows.crates/cost/src/node/usage/mod.rs·crates/cost/src/plan/cost.rs·crates/cost/src/plan/execution_demand.rs·crates/cost/src/plan/read_back.rs·crates/cost/src/node/state_bytes/mod.rs/// The whole plan's cost; how often each query runs and its range evaluation come from `entries`; /// formulas scaled by `settings.frozen_calibration_factor` (1 when unset). pub fn plan_cost( dag: &PhysicalASAPDAG, - nodes: &NodeMap<'_>, - recurrence: &[QueryRecurrence], - range_evaluations: &[Option<RangeEvaluation>], + entries: &[QueryWorkloadEntry], data: &DataWorkload, settings: &CostInputs, - formula_scale: f64, ) -> Result<(PlanCost, CalibrationFit), (PhysicalASAPNodeId, AnalyticalCostError)>; +/// Cost model 1's cost per second of every query request's fixed overhead (`ns_per_evaluation` +/// per request): what `plan_cost` charges every plan; 0 under cost models 2 and 3. +pub fn request_overhead_cost_per_second( + entries: &[QueryWorkloadEntry], + settings: &CostInputs, +) -> Result<f64, AnalyticalCostError>; +/// Cost model 1's cost per second of keeping `bytes` of raw data on disk; a plan keeps each source +/// once, for its scan that keeps the most. `None` under cost models 2 and 3. +pub fn raw_data_disk_cost_per_second(bytes: u64, settings: &CostInputs) -> Option<f64>; +/// How many times per second a node computed for the queries `reaching` it runs: once per second +/// at ingestion time; their combined rate at query time; once per request when materialized across evaluations. +pub fn executions_per_second<'e>( + timing: ExecutionTiming, + lifetime: Lifetime, + reaching: impl IntoIterator<Item = &'e QueryWorkloadEntry>, + non_repeating_query_horizon_seconds: f64, +) -> Result<f64, AnalyticalCostError>; /// How often each node is computed: `(multiplier, per_second)`; a node computed at ingestion time /// runs once per second whatever views read it. pub fn computations( dag: &PhysicalASAPDAG, - recurrence: &[QueryRecurrence], - range_evaluations: &[Option<RangeEvaluation>], + entries: &[QueryWorkloadEntry], non_repeating_query_horizon_seconds: f64, - root_rates: Option<&[f64]>, ) -> Result<HashMap<PhysicalASAPNodeId, (u64, f64)>, AnalyticalCostError>; /// The window an ingestion-time node maintains: a chain's tumbling window, a `TimeRange`'s range, /// or the first `TimeRange` or chain below it through filters, projections and shifts (a view over /// an ingestion-time chain is refreshed once per chain window); over plan nodes or memo groups. -pub fn maintained_window_ms( - node: PhysicalASAPNodeId, - nodes: &NodeMap<'_>, - producers_of: &HashMap<PhysicalASAPNodeId, Vec<PhysicalASAPNodeId>>, -) -> Option<u64>; +pub fn maintained_window_ms<'o, N: Copy + Ord, C: 'o>( + node: N, + operator_of: impl Fn(N) -> &'o Operator<C>, + input_of: impl Fn(N) -> Option<N>, +) -> Option<u64>; +/// The most any reader of `producer` reaches back: each window's `reaches_back_ms` plus its +/// reader's `span_ms`; `None` when one reads it unbounded. Over `windows_readers_need`. +pub fn read_back_ms<P: TimeIndexedPlan>( + plan: &P, + producer: P::Node, + span_ms: impl Fn(P::Node) -> u64, +) -> Option<u64>; /// One producer of a node being costed. The type stays. pub struct ProducerOutput<'a> { /// Its payload. pub payload: &'a Payload, /// Its output schema. pub output_schema: &'a Schema, /// Its output rows and bytes. pub output: EdgeStatistics, /// The sources it reads. pub sources: BTreeSet<&'a str>, /// Its state family (with θ). pub family: StateFamily<'a>, /// Its output columns that still hold one source row's values unchanged. pub unchanged_columns: BTreeSet<&'a str>, + /// The tumbling windows its rows span: those a `TimeRange` selects from a chain, 1 otherwise. + pub tumbling_windows: u64, } /// What one node uses and costs. -pub struct NodeUsage<'a> { +pub struct NodeResourceUsageAndCost<'a> { /// Its output rows and bytes. pub output: EdgeStatistics, /// The sources it reads. pub sources: BTreeSet<&'a str>, /// Its state family. pub family: StateFamily<'a>, /// Its output columns that still hold one source row's values unchanged. pub unchanged_columns: BTreeSet<&'a str>, + /// The tumbling windows its rows span: those a `TimeRange` selects from a chain, 1 otherwise. + pub tumbling_windows: u64, /// Work of one execution, in ms on one core. pub work_ms_per_execution: f64, /// CPU operations per second. pub cpu_ops_per_second: f64, /// Bytes scanned per second. pub scanned_bytes_per_second: f64, /// Average memory, including memory a query-time node holds while it computes. pub average_memory_bytes: f64, /// Peak memory. pub peak_memory_bytes: u64, /// Disk bytes. pub disk_bytes: u64, - /// Query-time memory held while an evaluation runs: (bytes, computations per second, per-consumer copy). - pub held: Option<(u64, f64, bool)>, /// An in-memory ingestion-time state's bytes per window. pub maintained: Option<u64>, /// A query-time scan's source and the raw bytes it reads. pub raw_scan: Option<(&'a Source, u64)>, /// Its calibration fit. pub fit: CalibrationFit, /// Its cost. pub cost: NodeCost, } +impl<'a> NodeResourceUsageAndCost<'a> { + /// What a consumer reads of the node with `payload` and `output_schema` that uses this. + pub fn producer_output(&self, payload: &'a Payload, output_schema: &'a Schema) -> ProducerOutput<'a>; +} /// Measure and cost one node; Stage 3 calls it per candidate ($w_v$) and `plan_cost` per plan node. -pub fn node_usage<'a>( +pub fn node_resource_usage_and_cost<'a>( node: &'a PhysicalASAPDAGNode, producers: &[ProducerOutput<'a>], rate: (u64, f64), - retained_tumbling_window_count: u64, - ingestion_window_ms: Option<u64>, + window_ms: Option<u64>, data: &DataWorkload, settings: &'a CostInputs, - formula_scale: f64, -) -> Result<NodeUsage<'a>, AnalyticalCostError>; -pub fn tumbling_window_retention_ms(..) -> BTreeMap<PhysicalASAPNodeId, u64>; -pub fn tumbling_window_retention_ms_of_plan(..) -> BTreeMap<PhysicalASAPNodeId, u64>; -pub fn retained_tumbling_windows(..) -> u64; -pub fn tumbling_windows_of(node: &PhysicalASAPDAGNode) -> u64; -pub fn query_reach(..) -> Vec<HashSet<PhysicalASAPNodeId>>; -pub fn node_map(dag: &PhysicalASAPDAG) -> NodeMap<'_>; -pub type NodeMap<'a> = HashMap<PhysicalASAPNodeId, &'a PhysicalASAPDAGNode>; +) -> Result<NodeResourceUsageAndCost<'a>, AnalyticalCostError>; /// Update operations per input row and bytes per state, at its θ; Err for a sketch without θ (a planner bug). -pub fn summary_shape(family: &FieldDataType, input_rows: StateInputRows) -> (f64, u64); +pub fn summary_shape( + family: &FieldDataType, + parameters: Option<&SketchParams>, + input_rows: StateInputRows, +) -> Result<(f64, u64), AnalyticalCostError>;Removed:
held(held memory is inaverage_memory_bytes); the chain retention helpers,tumbling_windows_ofand their fileplan/retention.rs(retention is the node'sretention_ms);query_reach(the latency-based held loop is gone);node_mapandNodeMap(now private toplan/cost.rs, their only user).crates/stage3-plan-selection/src/report.rs·crates/devtools/src/bin/asap_planner.rs/// What Stage 3 reports about its own run, beside the plan. The type stays. pub struct Selection { /// Per query, the accuracy Stage 3 accepted for the selected plan. pub accepted_accuracy_per_query: Vec<AcceptedAccuracy>, /// The candidates Stage 3 ruled out before the selection MILP. pub pruned_candidates: Vec<PrunedCandidate<PhysicalCandidate>>, + /// The MILP's optimum: the selected plan's cost per second under cost model 1, which + /// `asap_cost::plan_cost` of the plan reports too. + pub selected_plan_cost_per_second: f64, - pub milp_variables: usize, - pub milp_constraints: usize, + /// The selection MILP's variables (columns). + pub milp_variable_count: usize, + /// The selection MILP's constraint rows. + pub milp_constraint_count: usize, /// How long building and solving the selection MILP took. pub milp_construction_ms: u64, pub milp_solve_ms: u64, } // asap_planner CLI, `stage3_plan_selection` JSON section + "milp_objective_cost_per_second": selection.selected_plan_cost_per_second, "milp_variables": ..., "milp_constraints": ..., "milp_construction_ms": ..., "milp_solve_ms": ...crates/executor/src/capability/mod.rs·summary_kernels/factory/mod.rs·summary_kernels/state_column_updater/mod.rs·operators/summary/mod.rs·operators/time_range.rs·plan/mod.rs·sources/memory.rs·physical_planner/test_support.rs/// Checks that a summary kernel accepts `input`, at the node's θ (bad input for θ the kernel does not build). pub fn validate_summary_kernel( family: &SummaryFamilyType, + parameters: Option<&SketchParams>, input: &SummaryUpdate, ) -> Result<(), String>; /// Builds a HydraCms grid at the node's θ. -pub fn create_hydra_cms(family: &SummaryFamilyType) -> Result<HydraCms, String>; +pub fn create_hydra_cms( + family: &SummaryFamilyType, + parameters: Option<&SketchParams>, +) -> Result<HydraCms, String>; impl StateColumnUpdater { /// An updater for one summary type at θ; bad input for any other type or θ its kernel does not build. - pub fn new(family: &FieldDataType) -> Result<Self, Error>; + pub fn new(family: &FieldDataType, parameters: Option<&SketchParams>) -> Result<Self, Error>; } impl Operator { /// A keyed summary build per group, at θ. pub fn per_group_keyed_summary_build( input: SchemaRef, family: SummaryFamilyType, + parameters: Option<SketchParams>, items: Vec<usize>, row_weight: RowWeight, groups: Vec<usize>, ) -> Result<Self, Error>; /// One shared-structure build over all groups, at θ. pub fn all_groups_summary_build( input: SchemaRef, family: SummaryFamilyType, + parameters: Option<SketchParams>, item: usize, row_weight: RowWeight, groups: Vec<usize>, ) -> Result<Self, Error>; /// A value summary build per group, at θ. pub fn per_group_summary_build( input: SchemaRef, family: SummaryFamilyType, + parameters: Option<SketchParams>, value: usize, time: Option<usize>, groups: Vec<usize>, ) -> Result<Self, Error>; /// Emits an empty global state when no row arrives; the empty state is built at θ. - pub fn with_emit_empty_global(mut self) -> Result<Self, Error>; + pub fn with_emit_empty_global( + mut self, + parameters: Option<SketchParams>, + ) -> Result<Self, Error>; + /// A `TimeShift { offset_ms, at }`: evaluates its input once, at the request's end/start or T + /// minus `offset_ms`. + pub fn evaluate_at(input: SchemaRef, at: AtModifier, offset_ms: i64) -> Result<Self, Error>; + /// This range window, each kept row's time replaced by the end of its tumbling window of + /// `width_ms` counted back from the range's end E: `(E - (k+1)·w, E - k·w]` ends at `E - k·w`. + /// A tumbling-window chain built at query time groups its rows by that end. + pub fn with_tumbling_window_ends(mut self, width_ms: i64) -> Result<Self, Error>; } /// An executable operator. pub trait PhysicalOperator<V, S> { /// Its name. fn name(&self) -> &str; /// Its boundedness and emission given its inputs'. fn properties(&self, inputs: &[PlanProperties]) -> PlanProperties; /// Whether it needs bounded input. fn requires_bounded_input(&self) -> bool; /// Validate run-specific contracts before any source is opened. fn validate_context(&self, _context: &RunContext) -> Result<(), Error>; + /// The run context its inputs run in; default: this run's (a `TimeShift` with `@` overrides it). + fn input_context(&self, context: &RunContext) -> Result<RunContext, Error>; /// Its input schemas. fn input_schemas(&self) -> Vec<S>; /// Its output schema. fn output_schema(&self) -> S; /// Starts it over its inputs. fn start<'a>( &'a self, inputs: Vec<Input<'a, V>>, context: RunContext, ) -> Result<OutputStream<'a, V>, Error>; /// The bytes of one output value. fn output_bytes(&self, value: &V) -> usize; } impl InMemoryDataStore { - /// Stores a view's batches (no time index). - pub fn with_materialized_view( - mut self, - view: PhysicalASAPNodeId, - batches: Vec<Batch>, - ) -> Result<Self, Error>; + /// Appends `view`'s rows for `(start, end]`, then drops rows older than `view.retention_ms()` + /// (the rows appended last are always kept); bad input for an empty range or one starting + /// before the last appended end (append-only). + pub fn append_materialized_view( + &mut self, + view: &MaterializedView, + time_range_ms: (i64, i64), + batches: Vec<Batch>, + ) -> Result<(), Error>; } #[cfg(feature = "test-support")] pub mod test_support { + /// Sets θ on every sketch build among `nodes` (`parameters`, else its algorithm's first listed θ); + /// for tests of hand-built plans. + pub fn set_build_parameters<'a>( + nodes: impl IntoIterator<Item = &'a mut PhysicalASAPDAGNode>, + parameters: Option<&SketchParams>, + ); }crates/planner/src/installable/mod.rs·crates/planner/src/lib.rs/// Plan `input` for a deployment runtime: the selected DAG ready to install, and its cost report. pub async fn plan_installable( input: UserInput<'_>, -) -> Result<(InstallablePlan, Option<SelectedPlanReport>), PlanError>; +) -> Result<(InstallablePlan, SelectedPlanReport), PlanError>; /// The installable plan of `plan`, the plan Stage 3 built (a developer tool runs the stages itself). pub fn installable_plan_of( - plan: &PhysicalASAPDAG, - cost: Option<&PlanCost>, + plan: PhysicalASAPDAG, entries: &[QueryWorkloadEntry], -) -> Result<(InstallablePlan, Option<SelectedPlanReport>), InstallablePlanError>; +) -> Result<InstallablePlan, InstallablePlanError>; /// One operator node's predicted usage, beside the installed plan. The type stays. pub struct PredictedNodeUsage { - /// Its id in Stage 3's physical DAG of the selected candidate. - pub stage3_node_id: PhysicalASAPNodeId, - /// The ids in `dag` that run it (a chain's expanded windows). - pub installed_node_ids: Vec<PhysicalASAPNodeId>, + /// Its id in the installed DAG (chains are not expanded, so one id). + pub node_id: PhysicalASAPNodeId, /// Its operator, as the measured cost table names it. pub operator: OperatorKind, /// When it runs. pub timing: ExecutionTiming, /// Core-seconds of CPU per second. pub cpu_core_seconds_per_second: f64, /// Its cost per second under cost model 1; `None` under cost models 2 and 3. pub cost_per_second: Option<f64>, /// Whether its CPU came from the measured cost table or the formulas. pub cost_source: NodeCostSource, } /// Why an installable plan could not be built or is not consistent. The type stays. pub enum InstallablePlanError { /// Bad input (a plan file): another plan-file format. UnsupportedWireVersion(Option<u32>), /// Bad input (a plan file): not JSON of an `InstallablePlan`. Json(serde_json::Error), /// Bad input (a plan file): not a valid physical ASAP DAG. InvalidDag(PhysicalASAPDAGValidationError), /// Bad input (a plan file): not one query text per root. QueriesDoNotMatchRoots { roots: usize }, /// Unsatisfiable request: a standalone scalar query. UnsupportedQueryRoot { entry_index: usize }, /// Unsatisfiable request or bad input: views or raw-data windows cannot be derived. Uninstallable(InstallableDagError), /// Unsatisfiable request: the executor cannot refresh view `node_id`. MaterializedViewNotExecutable { node_id: PhysicalASAPNodeId, reason: String }, /// Unsatisfiable request: the executor cannot compile the query run. QueryRunNotExecutable { reason: String }, + /// Internal bug: `plan_cost` failed on the selected plan at node `node_id`. + PlanCostFailed { node_id: PhysicalASAPNodeId, reason: String }, } -pub use asap_types::plan::materialized_view::RawDataWindow; +pub use asap_types::plan::TimeWindow;installable_plan_of'scostwasSomeonly fromplan_installable; the developer tools passedNoneand ignored the report, so the report now comes fromplan_installablealone.crates/devtools/src/planner_fuzz/mod.rs·crates/integration-tests/src/lib.rslast_root_is_key_filterwas unused.Usage, end to end (planner test
runtime_install::the_plan_carries_stage3s_predicted_usage):The CLI shows the same optimum:
asap_planner --run-until stage3 --format jsonprintsstage3_plan_selection.milp_objective_cost_per_second.Algorithm changes
Baseline selection (Stage 3)
Problem:
select_planwastodo!().After:
selectable::selectable_edges: the admissible edges whose producer delivers to the evaluation context of every query reaching a query-time consumer (its grid, phase 0; unknown times use a 1 ms grid).cost::selection_costs: each candidate becomes one plan node (plan::candidate_node, shared with plan building) and is costed withnode_resource_usage_and_costover the largest logical output of each input among the producers it may read, computed once per candidate. That isexecutions_per_second, an ingestion-time node's windowmaintained_window_msover memo groups, a query-timeScan's read per executionretention::scan_read_ms. For each selectable edge,retention_need_ms, the most its selectable readers read back plus the range-evaluation span. Each query-time scan's raw data on disk (milp::model::solve_selectionminimizesmilp::graph::Components,Scanofraw_data_disk_constraints);The (B)/(D) trim: for a query-time result that is not materialized, (B) and (D) are left out. They are implied: with$y, z \ge 0$ , the (F) equality $\sum y + \sum z = x_v$ gives every $y_{v,c,i} \le x_v$ and $z_{q,v} \le x_v$ . The feasible set is unchanged, so the optimum is the same; the model just has fewer rows.
Bound and scaling: before scaling,$x_v = 0$ for every candidate whose $w_v + \sum_i \min(0, \min_u c_{u,v,i})$ exceeds the cost of a greedily built feasible plan. The coefficients left are then scaled so the largest is 1,000 (HiGHS's tolerances are absolute). HiGHS runs at gap 0, one thread.
milp::upper_bound::dearer_than_a_feasible_planfixesObjective scale in the raw-data row: the scale multiplies$d_v$ inside the row, and $r_s$ 's objective coefficient is 1. $d_v$ is about 1e-12 to 2e-9 per second, below HiGHS's
small_matrix_value(1e-9), so HiGHS dropped the unscaled coefficients and the disk cost left the model; in the row they are 42 to 1,000.The optimum, unscaled and plus the request overhead (
milp::model::plan_cost_per_second, shared with the bound), isSelection::selected_plan_cost_per_second.plan::plan_of: Kahn order overPhysicalASAPDAG::newderives the edges;retention_ms= the most the selected readers read back (at least the span of a query it answers);view_coverageof eachAcrossEvaluationsnode from its group.Correctness: the objective equals$r_s$ 's objective coefficient, the 4.001e-9/s plan selected when disk is free).
plan_costbecause both call the same functions (node_resource_usage_and_cost,executions_per_second,maintained_window_ms,read_back_ms,raw_data_disk_cost_per_second,request_overhead_cost_per_second); pinned bystage3_objective::*and measured on the harness (Verification). The bound only removes candidates that no plan cheaper than a feasible one contains, assuming every other selected candidate costs at least 0 with its selected inputs:a_candidate_dearer_than_a_feasible_plan_does_not_hide_a_cheaper_plan(max by (job)(count_over_time(m[1m])): without the bound, an unused 2.55e6/s chain set the scale and HiGHS returned a plan 12.6% dearer; now the 0.0078/s plan). The disk-row placement:raw_data_on_disk_costing_less_than_the_solver_tolerance_still_steers_selection(at 3.1e-17 per disk byte-second, 2.542e-9/s selected; with the scale onComplexity: one$t_v$ inside cycles. On 148 workloads that plan (release build, HiGHS seed 0), the (B)/(D) trim and the row placement cut total solve time from 203.5 s to 111.7 s and constraint rows from 91,659 to 68,336, objectives equal (relative ≤ 1e-9) on 147 and lower by 5.5e-9 relative on y17 (solver tolerance). Solve time depends on HiGHS's random seed; medians over seeds 0–4, without → with the trim:
node_resource_usage_and_costper candidate plus one per selectable edge; the bound is O(passes · edges), passes ≤ candidates (one or two in practice), at most 32 rebuilds; the MILP has |C| + |E_sel| + Σ|G_q| + |sources| variables plustopk(rate)topk(rate)over a chainAssumptions and limits: query latency and cluster capacity are not checked; an$x_v$ per candidate and no accuracy constraint (limits under Follow-ups).
@ Tread keeps its input unbounded; oneTime windows (types, cost, Stage 3, installer, executor)
asap-cost, the installer'sViewIndex, the executor'ssources/window.rs), over bare(i64, i64)windows, and each treated a chain as reading its newest window, which is wrong for a chain built at query time:sum_over_time(m[5m] offset 1m)+sum_over_time(m[5m])would keep 60,000 ms ofminstead of 360,000.reaches_back_msfor retention,unionfor the executor's slots). Behaviour that follows from the walk:WhatItsReadersRead, so its input keeps what its readers read;TimeShift'sretention_msincludes its offset: the up walk starts at the producer's ownshifts_back_ms, so a storedTimeShift(1h)read byTimeRange(5m)keeps 1 h 5 m;TimeShiftstored at ingestion time readsCurrentTimeOnly: the chain below it keeps only its newest window (retention 0; the store always keeps the rows appended last), not the offset;TimeShiftdirectly over raw data reads it unbounded (raw data needs every row), soplan_costrefuses thatScan("raw data retained without a bound"), as the installer does;WhatItsReadersReadnodes unbounded (ViewIndex::read_through_a_root);asap-costdoes not.time_window::tests(union, bounds,readers_read_a_producer_through_the_shifts_above_it: both roles of a stored shift and the raw-data case);runtime_install::shifted_and_unshifted_sums_over_a_shared_query_time_chain_install_and_answer_exactly(retention ≥ 360,000; answers equal the exact sums) andgrouped_and_per_series_counts_over_a_shared_query_time_chain_install_and_answer;installable::tests::a_time_shift_view_keeps_its_offset_more(1 h retention for aTimeShift(1h)view read for 1 h is refused, 2 h installs; the chain below installs with one window);installable::tests::a_root_time_shift_is_read_whole_by_the_installer_not_by_the_cost(a hand-builtTimeShift(5m)root over a query-timeScan:asap-costreads it back 0, the installer keeps it unbounded; no frontend builds this shape directly).Vecper step.Costing and retention (asap-cost)
PhysicalASAPDAGNode::parameters; retention fromretention_ms; held query-time memory insideaverage_memory_bytes;tumbling_windowson producers.computationsruns an ingestion-time node once per second, whatever views read it.Scanreads per execution what its consumers read, once, and keeps itsretention_mson disk.retention_mscovers, plus the one being built: a chain per tumbling window; any other view per window it maintains (maintained_window_ms, which stops at a chain, so a view over a 60 s chain refreshes per 60 s), per second of rows when it maintains none. ATimeShift's offset is in itsretention_ms, so the cost has no separate offset term.Scan) withoutretention_msis not costed (MissingOrZero("retention_ms")), so Stage 3 does not select it; these are exactly the views the installer refuses (ViewWithoutRetention).sum_over_time(m[5m] offset -5m)now ends at planning ("the selection MILP is infeasible; … retention_ms is missing or zero") instead of selecting an uninstallable view.usage::tests::{query_time_memory_is_held_only_while_the_node_computes, a_query_time_scan_reads_what_its_consumers_read_and_keeps_its_retention, a_merge_over_a_chain_s_windows_outputs_one_state_per_group, an_ingestion_time_view_of_raw_rows_keeps_the_rows_its_retention_covers, a_view_over_an_ingestion_time_chain_keeps_one_output_per_chain_window, a_view_without_retention_is_not_costed};stage3_objective::{an_ingestion_time_scan_feeding_chains_of_two_widths_is_charged_once_per_second, the_objective_includes_request_overhead_and_charges_raw_data_on_disk_once_per_source, offset_windows_without_raw_data_select_installable_plans_costed_for_their_retention}. The last one:sum_over_time(m[5m] offset 90s)every 60 s without raw data selects an installable chain plan, not aTimeShiftview over arriving rows costed as one second of rows.Chains and the time-indexed store (executor)
SummaryAggwindows; a query-time chain was refused.window_end = end);append_materialized_viewappends it and drops rows older thanretention_ms.with_tumbling_window_ends(w)stamps each row with the end of its window counted back from that window's end, and the build groups by that end, so theTimeRangeabove selects windows bywindow_endas over stored ones. A query reads the windows withwindow_end ∈ (T − o − r, T − o](w | r and w | o are checked).TimeShiftwith@evaluates its input at the anchor (evaluate_at,input_context).TimeShiftstored over a chain is refused ("materialized view N is read unbounded"): the executor's walk reads every slot whole outside a window, and how a stored shift stamps its rows is not defined yet ([Bug] check_compiles accepts a plan whose view over an ingestion-time chain cannot be refreshed on its own #1291).executor/tests/tumbling_window_chain.rs(24 refreshes of a 10-min exact-sum chain; reads throughTimeRange(1h)andTimeShift(1h)→TimeRange(1h)equal direct sums; a read past 2 h retention fails;a_chain_built_at_query_time_answers_queries_through_its_time_ranges;a_range_window_stamps_each_row_with_its_tumbling_windows_end);physical_planner::tests::a_stored_time_shift_over_a_chain_reads_the_chain_unbounded(as root and not).AcrossEvaluationsresults are stored by the interval of data they summarize; rows from an@-anchored subtree keep the anchor time.Pass 2: exact queries read exactly
ErrorMetricsOfMemo::exact_only_groups_of_query(asap-accuracy) gives an exact query no allowed error metric at its root, so every group below it is exact-only for it. Test:pass2_promql::exact_only_groups::an_exact_query_reads_every_sketch_readout_only_exactly. Stage 3 enforces this list per query in PR 7.Deleted or replaced code
RawDataWindow,MaterializedViewInputs::raw_data_windows, thelookback_mstest forwarder, bare(i64, i64)windowsTimeWindowPhysicalASAPDAG::set_retention; the installer'sreader_span_ms_by_noderetention_msinplan_ofSummaryAggwindows (main:todo!())asap-costplan/retention.rs(tumbling_window_retention_ms,tumbling_window_retention_ms_of_plan,retained_tumbling_windows),tumbling_windows_ofretention_ms;ProducerOutput::tumbling_windowsNodeUsage::held,query_reachaverage_memory_bytesnode_usage'sretained_tumbling_window_count,ingestion_window_ms,formula_scale;plan_cost'sformula_scale;computations'sroot_rateswindow_ms;settings.frozen_calibration_factor; rates fromentriesasap-cost,ViewIndex(windows_below,offset_below, its consumers map) and the executor'ssources/window.rs(ScanWindow,time_range_window,scan_windows,view_windows)time_windowwalks overTimeIndexedPlanInMemoryDataStore::with_materialized_viewappend_materialized_view(time-indexed, evicts by retention)PredictedNodeUsage::{stage3_node_id, installed_node_ids}node_idinstallable_plan_of'scostargument and reportplan_installablereturns the reportutilization_constraintsstub and the othertodo!()constraint bodiesraw_data_disk_constraintsSelection::{milp_variables, milp_constraints}milp_variable_count,milp_constraint_count(the CLI JSON keeps the old keys)Priced,Planner::price,reference_price,last_root_is_key_filterCosted,cost,reference_costDecisions and open questions
Needs reviewer decision: none.
Decided:
TimeIndexedPlankeepsoperator_offor the walks down, and returnsVec(the walks touch a few nodes per call).ViewIndex::read_through_a_root) and the executor (CompileGraph), not in the sharedPhysicalASAPDAGimplementation, soasap-costreads a plan as Stage 3 reads its candidates.TimeShift's refresh until [Bug] check_compiles accepts a plan whose view over an ingestion-time chain cannot be refreshed on its own #1291; the installer does not encode that executor gap.IngestionRefreshGroup::refresh_windowstays public with only test readers: it is the window a runtime schedules the group's ingestion run by.promql_to_post_asap::value_ranked_topk_over_binary_ratio_finalizes_both_summary_operandsaccepts either of two plans whose costs differ by 2e-11 relative (below HiGHS's tolerances); both finalize an exact summary per operand, which is what it checks.Verification
cargo fmt --all -- --checkandcargo clippy --workspace --all-targets --all-features -- -D warnings: clean at the head.main(which fails 183, with Stage 3 and the installertodo!()); none is newly failing.asap_plannerCLI tests andasap_planner_acceptancetodo!()before writing its output; PR 9 (#1239)planner_fuzztodo!()"#1207 PR 9: the selected plan's cost (plan_cost) and roots from its PhysicalASAPDAG"installable::tests::view_refreshes_compile_only_the_nodes_below_themruntime_install::check_compiles_reports_the_executors_errorsummary_sharing::identical_ungrouped_queries_share_their_producersjoint_search_equivalencematerialized_views::materialized_views_equal_the_stored_outputs_main_listedplan_sharingpromql_to_post_asaptopkcandidate and a KLL over a rate accumulator that the baseline does not selectplan_installable,plan_cost, installation, execution against an exact reference, and a brute force over the dumped MILP inputs; 197 workloads in sets of functional cases, offset and stored-shift cases, and windowed SQL):plan_cost(1e-9)The brute force over 167 dumps finds no plan cheaper than the MILP's: 142 equal, y20 within 1.3e-9 relative, p29 and x06 time out with the MILP below the best found, 21 skipped (over 400 candidates). Nineteen forced stored-shift plans: chain retention 0, shift retention offset + range, refresh refused (#1291); a forced stored shift over a stored
Scanis refused byplan_cost. After the time-window interface rework, a 98-workload sample (sets f, b, g, h, y03, the offset set and every fourth functional case) gives the same status, MILP objective,plan_costand views on all 98, and the same answers on 96; b13 and y09 (KLL quantiles, randomized compaction) answer differently on every run and stay within their rank target. The 44 functional workloads refused at install are executor and frontend gaps the exact reference cannot run either (21topk, 6 standalone scalar roots, 5 subquery value operations, 2delta, 2@ Traw windows, 1 vector shape) and 7 tracked in #1288 and #1289.Follow-ups
quantile_over_time(0.9, m[5m])exact and at rank 0.01 both read KLL k = 2,863; p17, x16); Stage 3 does not yet enforceexact_only_groups_of_queryper queryScan's read and retention are costed for the most any selectable reader reads, not the selected readers (y01: objective 2.2565 vsplan_cost2.2523; z02 selects 1.0377/s where 1.0279/s exists)view_refreshes_compile_only_the_nodes_below_themfails (avg/sum accept absolute error only, so no quantile chain)crates/devtools/tests/{corpus_snapshots,sketch_plan_snapshots}) still carry the staleraw_data_windowskeys; viewer and remaining devtoolstodo!()sTimeShift's refresh; install checks compile a refresh group as one DAG, not each view's refresh on its owncounter_window,node_below_shifts,below_time_shiftsre-walk shifts by hand instead of the shared walkTimeShiftview over raw rowscount by (job) (count by (job, host) (m)): exact Count over a finalized counttopkoutput namesdelta@ Traw windows, standalone scalar rootsselect.rs→lib.rs/selectable.rs/report.rs,cost.rs→cost.rs/plan.rs/retention.rs,milp/model.rs→milp/model/mod.rs;asap_cost::read_back_msandcomputationssignatures;Planner::cost/Costed;installable_plan_of(plan, entries)