Repository navigation
refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) - #1237
Merged
Merged
Conversation
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
from
November 9, 2026 09:01
09d69f1 to
7d1bd55
Compare
zzylol
force-pushed
the
refactor/1207-contexts
branch
from
November 9, 2026 09:35
8a466a4 to
3b831f9
Compare
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
from
November 9, 2026 12:11
7d1bd55 to
e53cf6e
Compare
zzylol
force-pushed
the
refactor/1207-contexts
branch
from
November 9, 2026 12:22
3b831f9 to
6e46d1f
Compare
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
from
November 9, 2026 17:48
e53cf6e to
588d1a2
Compare
zzylol
force-pushed
the
refactor/1207-contexts
branch
2 times, most recently
from
November 9, 2026 19:13
f7299f5 to
4216ad0
Compare
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
from
November 9, 2026 21:28
588d1a2 to
6e44c2b
Compare
zzylol
force-pushed
the
refactor/1207-contexts
branch
from
November 9, 2026 21:36
4216ad0 to
a1493df
Compare
zzylol
force-pushed
the
refactor/1207-contexts
branch
from
November 9, 2026 23:55
a1493df to
0816428
Compare
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
from
November 10, 2026 00:19
6e44c2b to
0723ff6
Compare
zzylol
force-pushed
the
refactor/1207-contexts
branch
from
November 10, 2026 00:19
0816428 to
e678f70
Compare
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
from
November 10, 2026 00:40
0723ff6 to
e3938ec
Compare
zzylol
force-pushed
the
refactor/1207-contexts
branch
from
November 10, 2026 00:45
e678f70 to
2a95559
Compare
This was referenced Nov 10, 2026
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
from
November 10, 2026 01:26
e3938ec to
db6d114
Compare
zzylol
force-pushed
the
refactor/1207-contexts
branch
from
November 10, 2026 01:26
2a95559 to
6e7a516
Compare
zzylol
force-pushed
the
refactor/1207-stage3-baseline
branch
from
November 10, 2026 13:37
db6d114 to
c9e1579
Compare
zzylol
force-pushed
the
refactor/1207-contexts
branch
from
November 10, 2026 13:37
6e7a516 to
e9cc34a
Compare
…text's queries, dominance in its own module (#1207 PR 7) - `crate::retention::reads_back`: every read of an execution's rows (edge needs and query-output spans), folded by costing, (R), `least_retention_ms` and the greedy bound; `ms_to_seconds` and `ExecutionCosts::retention_cost_per_second` for the ms-to-seconds step. - `asap_cost::plan::execution_demand::queries_of_context`: Stage 3 and `computations` pick a context's queries with it. - `milp/dominance.rs` holds `dominated_by_one_evaluation_twin`, over two lists per (context, logical candidate) and the selectable edges. - `SelectableExecutions`: `SelectableEdge`, `edges_out_of` next to `edges_into` (used by the components, (X) and dominance), private fields with getters, a map of context ids. - `raw_data_disk_rows` gives the raw-data rows to the scale and the constraints; `has_one_reader`; the scaled `raw_data_on_disk` doc. - `delivers_on` compares with `own_schedule` (no `% 0`). - asap-cost: the `Scan` arm and the retained raw rows share `raw_rows`.
…; @ start() per span (#1207 PR 7) - `asap_cost::plan::execution_demand::query_reads`: how each query reads the nodes below it, per (query, anchor), over a plan or a memo. `consumers` (plan_cost) walks plan nodes with it, Stage 3 memo groups (`SelectableExecutions::readers_of`). A run in an evaluation context is rated by the queries reading its group in their evaluation context, not by every query reaching the group: an instant `m[5m] @ end()` beside a range `m[5m] @ end() offset 10m` cost 0.3642/s in the MILP for a plan `plan_cost` charged 0.1542/s. Stage 3 walks every candidate of a group, `plan_cost` the selected plan; the difference is documented. - `@ start()`: `evaluation_rate` of the request schedules per span, summed over spans; `anchored_request_recurrence` deleted. - `ExecutionCosts::least_need_ms_per_execution`, computed once, read by the greedy bound and the h_v >= need * x rows. - The bound uses `has_one_reader`; the query-time `Scan` reads `raw_rows_back`; `Candidates` holds its edge; `SelectedEdge`; `is_across_evaluations` wrapper inlined; module docs and planner-layering.md list dominance.rs.
…e; one raw-row-store reader rule (#1207 PR 7) - IR: `Operator::chain_tumbling_window_width_ms` and `Operator::request_anchor`; Stage 3, asap-cost and Pass 2 call them. - Logical: `EvaluationGrid`, `LogicalASAPMemoCandidates::own_schedule` (an `EvaluationGrid`) and `query_must_not_read`; `Memo::output_groups`. `PhysicalCandidate` keeps `delivers_on` only. - Physical: `PhysicalProperties::has_one_reader_per_context`, `PhysicalASAPMemoCandidates::operator`, `ExecutionContext::EVERY_ARRIVING_ROW`, and the context rules `queries_of_context`, `anchor_is_evaluation_time` and `query_reads` (moved from asap-cost, which keeps depending on types only). - Workload: `QueryWorkloadEntry::evaluation_grid_ms` and `request_time_grid_ms`. - Plan: `PhysicalASAPDAGNode::keeps_raw_rows_at_ingestion` (a `Scan`, or a `TimeShift` without `@` over one), with `asap_cost::reads_one_batch_of`: an ingestion-time reader of a raw-row store reads one batch, in Stage 3 and now in `plan_cost` too (it read the whole store there); the cost model's raw-row rule uses the same predicate. - asap-cost: `output_interval_ms` is public; Stage 3's Δ of π_v calls it. - Stage 3: `SelectableExecutions::reads_back` and `Reader`; the `is_across_evaluations` wrapper and `is_query_time_scan`'s memo walk replaced by `PhysicalASAPMemoCandidates::operator`.
…ingestion reads one batch; two queries reading one anchored group (#1207 PR 7)
…riginal's variables (#1207 PR 7) The same queries on several grids (one user every 60 s, another every 300 s) give one evaluation context per grid with the same executions and edges, each cost a fixed multiple of the other's. The MILP chose each copy freely, so HiGHS branched over both: c16 130 s, RangeHeavy-8users 222 s on seed 0. copies_in_equivalent_contexts finds such copies (same candidates, kept alike, same memo edges from corresponding producers, paired queries with the same outputs, exact-only groups and span, costs proportional), and selection_variables gives a copy's executions, edges and queries the original's x, y and z. Exact: some optimum selects alike in both copies (give both the cheaper one's selection; retention and raw data do not grow). Plan costs are unchanged; c16 drops to about 10 s.
…tored result (#1207 PR 7) Three rules said when an ingestion-time stored result refreshes and disagreed: Stage 3 (own_schedule: a chain at phase 0, anything else its group's evaluation grid), the installed plan (refresh_window: a chain (w, s mod w), anything else the union of the windows below it) and the cost model (maintained_window_ms: the first range or chain below). An ingestion-time sum(TimeRange(5m)) read every 60 s ran its inputs on a 60 s pipeline and served the 60 s reader, while the plan refreshed it every 5 min; a chain over TimeShift(30s) had phase 0 in Stage 3 and 30 s in the plan. The plan's rule is the one rule: RefreshWindow::computed_from over any TimeIndexedPlan. PhysicalASAPDAG::refresh_window, Stage 3's ingestion pipeline context, the stored times delivers_on checks (PhysicalASAPMemoCandidates::stored_times; delivers_on moves to the candidates, whose data it reads) and both cost paths (plan_cost and Stage 3's ingestion_window_ms, its width) use it. The candidates implement TimeIndexedPlan; Stage 3's per-edge Candidates delegates to it. shift_back_ms_below follows every alternative producer and needs them to agree. maintained_window_ms and refresh_windows are deleted.
… stored-times rule; review fixes (#1207 PR 7) A query is evaluated in its own evaluation context, but the context-copy search put it in the context of its first answering execution. With an AcrossEvaluations candidate listed first that is Materialized(v), so two evaluation contexts whose queries read different exact-only groups had empty query lists, matched, and shared their variables: Q0 every 10 s and Q1 every 20 s that must read the exact quantile then selected alike, at 9 instead of 8. SelectableExecutions now records each query's evaluation context in new (evaluation_context), the copy search reads it, and selection_variables asserts that every shared z is a real (query, answering execution) pair. The copy search moves out of dominance into milp/equivalent_contexts.rs (symmetry reduction) and returns SharedVariables. Contexts are bucketed by a canonical key (candidates, kept, least need, in-edges and query signatures), so only the cost proportionality check compares two contexts; the out-edge and input-count checks it implied are a debug_assert. Dominance and the copy search share one relative tolerance (COST_RELATIVE_TOLERANCE). stored_times uses RefreshWindow::computed_from for any chain, at either timing (a query-time chain over TimeShift(30s) is stored at phase 30 s, as at ingestion time), and its group's evaluation grid for any other query-time result; own_schedule is deleted. EvaluationGrid gets From<RefreshWindow>, which delivers_on uses. A Filter or Project between a chain and a TimeRange keeps the chain's tumbling-window width in the cost model, as a TimeShift does: the rule is Operator::keeps_input_window (exhaustive), which also passes the windows a TimeRange selected through them. Operator::chain_tumbling_window_width_ms and Operator::request_anchor list every operator. Also: QueryWorkloadEntry::range_span_ms at the three span sites; Memo::output_groups reads QueryRoot::operator_refs and output_references is deleted; PhysicalASAPMemoCandidates::is_query_time_scan; one ONE_SECOND_MS in asap_types::workload; docs name Stage 2's output where they said Stage 3 candidates; comments state rules, not process.
… output; targets in context copies (#1207 PR 7) avg by (job)(quantile_over_time(0.99, data[300s])) at relative 0.05 over non-negative data had no plan without raw data: Pass 2 listed every DDSketch group as exact-only for it, and Stage 3 now enforces the list (X). The avg's exact-accumulator path reads count(quantile): a Project above the count passes no metric, so the count's group got none, and the walk and its closure marked the quantile and every sketch below it exact-only. A count's value does not depend on its input's error (error factor 0 in every metric), so a consumer now passes every metric in which an input's error does not reach its output, and the exact-only closure does not cross such an input. The DDSketch plan is valid: #1302 made a relative bound on avg hold for values of one sign, and data is declared non-negative. A store always keeps a stored result's newest output, and plan_cost charges at least one retained output, but Stage 3 let a stored result read only at its newest output retain 0 (an ingestion-time chain read by a stored ingestion-time TimeShift: 0.007/s apart). Each candidate with a retention cost now retains at least its output interval (ExecutionCosts::newest_output_retention_ms): the linear retention cost is taken at no less than one output, (R)'s least need, the bound and the reported retention are floored at it. Context copies also compare each query's accuracy target (metric, ε, δ, scope), so the accuracy constraint (K) holds alike in both copies: the same queries at rank 0.01 every 10 s and rank 0.05 every 20 s are not copies.
… to the context's largest cost, one metric rule per input, plan children order (#1207 PR 7) - Context copies compare costs to COST_RELATIVE_TOLERANCE of the copy's largest cost, alpha from that pair: a small edge cost, a difference of large costs, keeps their rounding (RollupHeavy-8users flows: -2.36e-15 and -4.71e-16, ratio 0.200000000026), which stopped copies sharing. - One metric rule per input: ErrorContract::input_error_reaches_output and ErrorMetricsOfMemo::error_metrics_allowed_on_input, used by the exact-only walk and closure, the edges and the readout caps; a count's input edge from a readout holds in any metric. - Newest-output floor applied once: most_read_ms floored where built, ExecutionCosts::retention_ms_for_need for edge and output needs. - Operator::map_children applies f in children() order, so a plan's Project over q with scalar(count(q)) keeps q as its child; plan_cost takes a node's producers in its payload's input order. - Layering: From<RefreshWindow> for EvaluationGrid in plan; ExecutionContext::is_outside_evaluations; QuerySignature with named fields; QueryWorkloadEntry::evaluation_grid; exact-only field docs. - Abstraction: key_of returns queries in signature order; tolerance comments; range_span_ms for @ start() spans.
zzylol
force-pushed
the
refactor/1207-contexts
branch
from
November 15, 2026 15:54
796d435 to
44a223d
Compare
… compare kept costs only, IN subquery order, producers from the payload (#1207 PR 7) - E1-b: proportional_costs pairs only kept executions and edges between kept ones, so a left-out execution no longer sets the copy tolerance; alpha must be positive. Test: a left-out range costing 1e12 merged two contexts (optimum 2.5 instead of 2.421). - N1 gap: ScalarExpr::map_parts maps an IN's subquery before its left side, as operator_refs lists them; a test checks map_children follows children() for every operator with scalar references. - L-1: least_need_ms_per_execution -> least_retention_ms_per_execution. - A2: plan_cost reads producers from PhysicalASAPDAGNode::producers instead of sorting edges. A3: unretained nodes built once. A4: outputs_of[&query]. L-2: input_error_reaches_output is pub(crate) and indexes directly.
zzylol
marked this pull request as ready for review
November 16, 2026 00:45
This was referenced Nov 16, 2026
Merged
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
exact_only_groups_of_querywas computed but not enforced).TimeRangeper context (retention_ms3,600,000, chosen by (R));quantile_over_time(0.9, m[5m])at relative 0.05 no longer reads the rank-0.01 query's KLL, and an exact p90 no longer reads a sketch; the MILP's optimum (Selection::selected_plan_cost_per_second) equalsplan_coston every objective pin (q21 0.9020/s) and on every planned harness workload (round 1: 98 of 98, round 2: 25 of 25, this PR's review: 43 of 43); an@ end()/@ start()subtree runs once per request in the MILP and inplan_costalike (two queries sharingm[5m] @ end(): 0.0473/s, whereplan_costcharged 0.1513/s); z05 (count_over_time(m[1m])every 10 s +max_over_time(m[1d])every 600 s) costs 0.9301/s, the brute-force optimum (was 1.0590/s); c16 plans in about 10 s (128 s before review round 4) and RangeHeavy-8users in about 9 s (222 s), because a copy of an equivalent evaluation context shares the original's MILP variables (round 5 fixed which contexts count as copies, H1: two contexts whose queries read different exact-only groups were wrongly copies when anAcrossEvaluationscandidate answered first); an ingestion-time stored result refreshes, delivers and is costed on one rule, the installed plan's refresh window (M4); a query-timeScanis no longer costed for a reader it does not have (PR 6's F2 (a)):count_over_time(m[1m])every 60 s next tomax_over_time(m[1d])costs 0.0062/s, as alone; Pass 2 no longer makes acount's input exact-only, soavg by (job)(quantile_over_time(0.99, data[300s]))at relative 0.05 over non-negative data plans again without raw data, as onmain(round 5, F1: 30 random-mixed workloads now plan, 6 others cost 8–18% less); a stored result retains at least its newest output, asplan_costcharges.Review guide
main, rebased onto59365ce80(with fix(frontend): name every aggregate measure and validate names match measures #1308; PR 6 refactor(stage3): baseline selection MILP and its PhysicalASAPDAG; executor θ, native chains and a time-indexed store (#1207 PR 6) #1236 and fix(types): a stored result delivers on every context only when final; @ T pins only its own time (#1264) #1299 below it); refactor(stage3): constraint (K) per execution and metric; sign-conditioned DDSketch guarantee, classic HLL certificate from the source's distinct bound (#1207 PR 8) #1238 (PR 8, accuracy) builds on it. Head0b5ec4653. Commits: the PR 7 model and (X) (d233d7323,7060891de), review round 1 (c86e08497…527f80b11), R1 (e054797f3), review round 2 (571581460,51b17a046,f144b76b7), review round 3 (69d205a2b,d96f41ec8), layering (a040fe2b1,4cca6a110), review round 4 (0d04e4d90context copies,813ecaef5M4), review round 5 (rebased:6ca90a0ac,3417d63be; before the rebase21e473690,796d43503), the round-5 reviews (44a223d93), the round-6 review (0b5ec4653). Backups of previous heads:backup/1237-1ef03212f,backup/1237-baab8969a,backup/1237-fa47deaab,backup/1237-c7465133b,backup/1237-33e7d6d56,backup/1237-77f403791,backup/refactor/1207-contexts-4cca6a110,backup/refactor/1207-contexts-813ecaef5,backup/refactor/1207-contexts-21e473690,backup/refactor/1207-contexts-796d43503.stage3-plan-selection/src/selectable.rs:SelectableExecutions::new(contexts, executions, edges by execution position,answering),input_context(request contexts per anchor); the rules it reads live in asap-types:ExecutionContext::EVERY_ARRIVING_ROW(types/src/stages/physical/execution_context.rs) andPhysicalProperties::has_one_reader_per_context(types/src/stages/physical/properties/mod.rs).stage3-plan-selection/src/cost.rs:execution_costs(executions_per_second;retention.rs:edge_retention_need_ms,edge_scan_read_ms.stage3-plan-selection/src/milp/model/mod.rs(variables per execution,kept, the scale),milp/retention.rs(milp/exact_only.rs((X)),milp/upper_bound.rs(per-query greedy bound with retention,dominated_by_one_evaluation_twin),milp/graph.rs.cost/src/plan/execution_demand/mod.rs(context_executions_per_second,computationsper context,Consumer::QueryRequest { at }),cost/src/node/usage/mod.rs(raw rows materialized at ingestion,chain_tumbling_window_width_ms).types/src/stages/physical/memo_candidates/mod.rs:delivers_on(moved fromPhysicalCandidate),stored_times, the candidates'TimeIndexedPlan;types/src/plan/materialized_view/mod.rs:RefreshWindow::computed_from;types/src/plan/time_window.rs:shift_back_ms_belowover alternatives.stage3-plan-selection/src/milp/equivalent_contexts.rs:copies_in_equivalent_contexts(SharedVariables, contexts bucketed byContextKey),SelectableExecutions::evaluation_context, andselection_variablesinmilp/model/mod.rs.plan.rs(one node per selected execution,retention_msfromlib.rs/report.rs(executions instead of candidates), tests in sibling files (execution_demand/tests.rs,cost/tests.rs,report/tests.rs), the objective pins instage3_objective.rs.copies_in_equivalent_contexts(which contexts count as copies); the request contexts ininput_context; the one shared rate function;File organization (by crate and scope)
Behavior: before → after
Planner test
runtime_install::one_chain_serves_two_evaluation_contexts: two queries over the last hour of 1,000 latency series (1 sample/s), rank error 0.01, p99 every 10 min and p50 every 30 min, phase 0; the deployment keeps no raw data.Materialized(chain)retention_ms3,600,000TimeRange(3600s)nodes (one per context), two rootsPlanner tests
exact_only_reads(the PR 6 review's repros):m)quantile_over_time(0.9, m[5m])at rank 0.01 (Q0) and at relative 0.05 (Q1)Interfaces
ir/operator/node.rs·stages/logical/memo_candidates.rs·stages/physical/execution_context.rs·plan/materialized_view/raw_data.rs·workload/query.rs·stages/physical/memo_candidates/mod.rs·plan/materialized_view/mod.rs·plan/materialized_view/views.rs·plan/time_window.rs// asap-types, IR impl<C> Operator<C> { - /// `f` may change the reference type, e.g. from `Rc<OperatorNode>` to a node id. + /// `self` with `f` applied to each child, in [`Self::children`] order (direct inputs, then the + /// nodes its scalar expressions read), so a caller may hand out replacements by position. pub fn map_children<D>(&self, f: impl FnMut(&C) -> D) -> Operator<D>; + /// Its tumbling-window width when it is a `SummaryTumblingWindowChain`. + pub fn chain_tumbling_window_width_ms(&self) -> Option<u64>; + /// The anchor of a `TimeShift` with `@ end()` / `@ start()`; None otherwise (`@ <ts>` included). + pub fn request_anchor(&self) -> Option<AtModifier>; + /// Whether each output row keeps its input row's window: `Filter`, `Project`, `TimeShift`. + pub fn keeps_input_window(&self) -> bool; } -#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, Serialize, Deserialize)] pub enum AtModifier { Start, End, Timestamp(i64) } // asap-types, workload +impl QueryWorkloadEntry { + /// The grid in ms its evaluations fall on; 1 when not known. + pub fn evaluation_grid_ms(&self) -> u64; + /// The grid in ms of its request ends (`@ end()`) or starts (`@ start()`); 1 when not known. + pub fn request_time_grid_ms(&self, at: AtModifier) -> u64; + /// Its range-evaluation span in ms, 0 for one evaluation per run. + pub fn range_span_ms(&self) -> u64; + /// The evaluation grid of its context: every `evaluation_grid_ms`, phase 0 (was Stage 3's free + /// `evaluation_grid(entry)`). + pub fn evaluation_grid(&self) -> EvaluationGrid; +} +/// One second, in ms (moved from asap-cost; `EVERY_ARRIVING_ROW` is one second wide). +pub const ONE_SECOND_MS: u64 = 1_000; // asap-types, logical stage (EvaluationGrid moved here from stages::physical, re-exported from stages) pub struct EvaluationGrid { pub period_ms: u64, pub phase_ms: u64 } +impl LogicalASAPMemoCandidates { + /// Whether `query` must not read `candidate`: it has θ and its group is on the query's exact-only list. + pub fn query_must_not_read(&self, query: usize, candidate: CandidateForMemoGroupId) -> bool; +} +impl Memo { pub fn output_groups(&self) -> Vec<(usize, MemoGroupId)>; } -/// Each operator reference of `query_roots` (now `QueryRoot::operator_refs` in `output_groups`). -pub fn output_references<C: Clone>(query_roots: &[QueryRoot<C>]) -> Vec<(usize, C)>; // asap-types, physical stage impl PhysicalCandidate { - /// Whether a consumer in `context` can read this producer. - pub fn delivers_on(&self, candidates: &PhysicalASAPMemoCandidates, context: &ExecutionContext) -> bool; } impl PhysicalProperties { + pub fn is_across_evaluations(&self) -> bool; + /// At most one consumer per context: query time and ℓ = None. + pub fn has_one_reader_per_context(&self) -> bool; } impl PhysicalASAPMemoCandidates { + /// The operator of candidate `v`'s logical candidate. + pub fn operator(&self, v: PhysicalCandidateId) -> &Operator<MemoGroupId>; + /// The times `v`, materialized across evaluations, has a stored result for: a chain at either + /// timing and any ingestion-time result on its refresh window (`RefreshWindow::computed_from`, + /// width as period), any other query-time result on its group's evaluation grid. + pub fn stored_times(&self, v: PhysicalCandidateId) -> Option<EvaluationGrid>; + /// Whether `v` is a `Scan` run at query time (moved from Stage 3's cost.rs). + pub fn is_query_time_scan(&self, v: PhysicalCandidateId) -> bool; + /// Whether a consumer in `context` can read producer `v` (moved from `PhysicalCandidate`, + /// whose data it does not hold); `Materialized(u)` uses u's stored times; finality from main. + pub fn delivers_on(&self, v: PhysicalCandidateId, context: &ExecutionContext) -> bool; } +/// The candidates as the time walks see them: admissible producers as inputs, admissible +/// consumers as readers; Stage 3's per-edge view delegates to it. +impl TimeIndexedPlan for PhysicalASAPMemoCandidates { type Node = PhysicalCandidateId; type Child = MemoGroupId; /* ... */ } impl ExecutionContext { + /// Every ingestion-time execution that is not materialized: each second of arriving rows. + pub const EVERY_ARRIVING_ROW: Self; + /// `IngestionPipeline` or `Materialized`: every evaluation context reading its executions + /// shares them. + pub fn is_outside_evaluations(&self) -> bool; } +/// The queries on evaluation grid `grid_ms` among `reaching`. +pub fn queries_of_context(grid_ms: u64, reaching: impl IntoIterator<Item = usize>, entries: &[QueryWorkloadEntry]) -> Vec<usize>; +/// Whether an `@ end()` / `@ start()` input read by `queries` runs in their evaluation context +/// (`@ end()` with an instant query among them). +pub fn anchor_is_evaluation_time<'e>(at: AtModifier, queries: impl IntoIterator<Item = &'e QueryWorkloadEntry>) -> bool; +/// How each query reads the nodes below it, per (query, anchor), over a plan or a memo; Stage 3 and +/// `plan_cost` rate a context by it. +pub fn query_reads<N: Copy + Eq + Hash>(starts: &[Vec<N>], inputs_of: impl Fn(N) -> Vec<N>, + anchor_of: impl Fn(N) -> Option<AtModifier>, stops_at: impl Fn(N) -> bool, entries: &[QueryWorkloadEntry]) + -> HashMap<N, Vec<(usize, Option<AtModifier>)>>; // asap-types, plan impl RefreshWindow { + /// The one refresh rule of an ingestion-time stored `node` in any `TimeIndexedPlan` (plan + /// nodes, Stage 2's output `PhysicalASAPMemoCandidates`): the union of the windows it is + /// computed from; a chain of width w over shifts s is (w, s mod w). + pub fn computed_from<P: TimeIndexedPlan>(plan: &P, node: P::Node) -> Option<Self>; } +/// The times a refresh window ends at: period = width, same phase (next to `RefreshWindow`). +impl From<RefreshWindow> for EvaluationGrid; impl PhysicalASAPDAG { /// `RefreshWindow::computed_from` over this DAG. pub fn refresh_window(&self, node: PhysicalASAPNodeId) -> Option<RefreshWindow>; - /// `refresh_window` of each of `nodes`, the DAG indexed once. - pub fn refresh_windows(&self, nodes: impl IntoIterator<Item = PhysicalASAPNodeId>) -> HashMap<PhysicalASAPNodeId, Option<RefreshWindow>>; } /// Now follows every input (over candidates, the alternative producers of one input); the paths /// must agree, else `None`. pub fn shift_back_ms_below<P: TimeIndexedPlan>(plan: &P, node: P::Node) -> Option<u64>; impl PhysicalASAPDAGNode { + /// A `Scan`, or a `TimeShift` without `@` over one (`input`), stored across evaluations at ingestion. + pub fn keeps_raw_rows_at_ingestion<C>(&self, input: Option<&Operator<C>>) -> bool; + /// The nodes it reads, one per edge into it in a validated DAG, in its payload's input order + /// (`Operator::children`: direct inputs, then each node its scalar expressions read, once). + pub fn producers(&self) -> Vec<PhysicalASAPNodeId>; }crates/cost/src/plan/execution_demand/mod.rs·crates/cost/src/node/usage/mod.rspub enum Consumer { Query(usize), + /// Query `query` below a `TimeShift` anchored with `at`: once per request. + QueryRequest { query: usize, at: AtModifier }, Materialized(PhysicalASAPNodeId), } +/// The run rate of a node in one context of `queries` (Stage 3's and `plan_cost`'s one function): +/// no anchor, at their evaluations; `@ end()`, once per request end; `@ start()`, per span, summed. +pub fn context_executions_per_second<'e>(anchor: Option<AtModifier>, + queries: impl IntoIterator<Item = &'e QueryWorkloadEntry>, horizon_seconds: f64) -> Result<f64, AnalyticalCostError>; /// Signature changed: the workload entries, for `query_reads`. -pub fn consumers(dag: &PhysicalASAPDAG) -> HashMap<PhysicalASAPNodeId, Vec<Consumer>>; +pub fn consumers(dag: &PhysicalASAPDAG, entries: &[QueryWorkloadEntry]) -> HashMap<PhysicalASAPNodeId, Vec<Consumer>>; +/// The time between two outputs of a node that keeps rows (π_v's Δ in Stage 3). +pub fn output_interval_ms<C>(payload: &Operator<C>, window_ms: Option<u64>) -> u64; -/// The window an ingestion-time grouping maintains (the first range or chain below it); replaced -/// by the refresh window's width in `plan_cost` and Stage 3. -pub fn maintained_window_ms<'o, N: Copy + Ord, C: 'o>(node: N, operator_of: impl Fn(N) -> &'o Operator<C>, - input_of: impl Fn(N) -> Option<N>) -> Option<u64>; +/// An ingestion-time reader of a raw-row store reads one batch; Stage 3 and `plan_cost` both use it. +pub fn reads_one_batch_of<C>(consumer: &PhysicalASAPDAGNode, producer: &PhysicalASAPDAGNode, + producer_input: Option<&Operator<C>>) -> bool; pub struct ProducerOutput<'a> { ..., pub tumbling_windows: u64, + pub chain_tumbling_window_width_ms: Option<u64> } pub struct NodeResourceUsageAndCost<'a> { ..., + pub chain_tumbling_window_width_ms: Option<u64> } /// Behavior: a node's producers are `PhysicalASAPDAGNode::producers` (its payload's input order), /// not the DAG's edge order. pub fn plan_cost(dag, entries, data, settings) -> Result<(PlanCost, CalibrationFit), ..>;asap-accuracy (
error_contract/mod.rs·metric_rule/mod.rs):Crate-private, Stage 3 (
selectable.rs·cost.rs·retention.rs·milp/):Removals (items on
main): in Stage 3,SelectionCosts/selection_costs,selectable_edges,query_outputs, the freeevaluation_context,delivers_to_consumerandoutput_groups. In asap-cost,maintained_window_msand the privateONE_SECOND_MS(nowasap_types::workload::ONE_SECOND_MS). In asap-types,PhysicalCandidate::delivers_on(now onPhysicalASAPMemoCandidates),PhysicalASAPDAG::refresh_windowsandstages::output_references. In asap-accuracy,ErrorMetricsOfMemo::{error_metrics_allowed_on_inputs, error_metrics_passed_to_inputs}(nowerror_metrics_allowed_on_input(_edge)).Usage
The two review-round-4 changes, end to end, as the tests run them (
cargo test -p asap-types a_stored_chain_and_an_ingestion_time_result_are_stored_on_their_refresh_window,cargo test -p asap-stage3-plan-selection a_context_equivalent_to_another_shares_its_variables) and as the CLI runs c16:Review round 1: what changed
Functional, quality and efficiency reviews of
1ef03212f; all approved items are in.a_candidate_dearer_than_a_feasible_plan_does_not_hide_a_cheaper_plan(fails without the fix)plan_coststopped atOneEvaluationnodes, so an@subtree was charged per step (a07: objective 0.0473/s,plan_cost0.1513/s)asap_cost::context_executions_per_second, called by Stage 3 and bycomputations;computationswalks throughOneEvaluationnodes;@ start()counts distinct request startsplan_cost;stage3_objective::the_objective_is_the_plan_cost_on_a_shared_at_end_subtree(fails on1ef03212f),..._on_at_start_subtrees_of_two_spans;execution_demand::tests::{a_shared_one_evaluation_at_end_shift_runs_its_input_once_per_request, at_start_inputs_run_once_per_distinct_request_start}@ start()input shared oneOneEvaluationresult with instant evaluations (a10) and with@ end()(a04, a09)@ end()/@ start()shift run in a context of their own, keyed by the anchor and the queries that reach the shift; its rate counts those queries' requests onlyan_at_start_subtree_shares_no_evaluation_result_with_an_instant_query(fails on1ef03212f);..._on_anchored_and_plain_queries_on_one_grid. a04 0.1364/s, a09 0.0525/s, a10 0.0325/s, each =plan_cost(on1ef03212ftheirplan_costwas 0.2353, 0.1514, 0.1313/s)the_bound_leaves_out_of_each_query_only_what_it_must_not_read(fails with the union)the_bound_counts_the_least_retention_an_execution_paysLifetime::None/OneEvaluationtwinsOneEvaluationtwin (same logical candidate and θ, same context) costs no more, reads each of its producers and is read by each of its consumers on the same input at no more, and answers every query output it answers; none witha_none_execution_its_one_evaluation_twin_dominates_is_left_out(left out at equal cost; kept when the twin costs more or cannot be read by its consumer)plan_cost8.89e7/s): with #1236's retention of aTimeShiftview including its offset,ingestion_window_ms, shared with costing)plan_cost;stage3_objective::the_objective_is_the_plan_cost_on_one_off_queries_over_years(fails without the fix)Scan's read per executionplan_coston every planned harness workloadExecutionCosts::retention_cost_per_retained_second(ExecutionCosts::retention_need_ms_per_edge, computed once per memo edge; (R),least_retention_msand the bound read itbaseline_scan_read_ms)retention.rs:edge_retention_need_msandedge_scan_read_msover oneread_backedges_intobuilt once inSelectableExecutions; (A), the bound,graph.rs,exact_only.rsuse them;index_ofdeletedPhysicalCandidate::own_schedule;PhysicalProperties::is_across_evaluations;has_one_reader_per_contextshared by (F), (B)/(D) and the boundselect_plan; the forwarder deleted, asap-cost takes the workload entry;Consumer::QueryRequest { at }; the anchors looked up in a mapSelectableExecutionswithansweringas a field;milp/retention.rs;QueryIdinselectable.rsselect_plan(noz <= 0rows); the ingestion arm reachable once the rate matches on the context first; the test helper usesoutput_groupsplanner-layering.mdandasap-aware-mapping-architecture.mdlistexact_only.rs,milp/retention.rsand (X)Planning time, before (
1ef03212f) → after (this head), HiGHS seeds 0–4, median (min–max):1ef03212ftopk(5, sum by (host)(rate(c[5m])))every 60 s and 120 s; debug harness, wall)asap_planner, wall)tools/dag-viewerexamples, one run, seed 0 (release, wall)Solve times are seed-dependent; the objective is the same for every seed (example 3a within 1.4e-10 relative). The host was shared with other jobs.
Review round 2: what changed
@ start()counted each distinct start time once, but two queries requested together with different spans need two runs (h02b: charged 1/120 s, needs 2/120 s)context_executions_per_second(the rate Stage 3 andplan_costshare) counts@ start()once per request of each distinct (schedule, span)execution_demand::tests::at_start_inputs_run_once_per_request_of_each_span(spans 1 h and 2 m every 60 s: 2/60, was 1/60; fails without the fix). h02b 0.00667 → 0.00927/s, h02c 0.00870 → 0.01130/s, k19 0.0812 → 0.1072/s, each =plan_costasap_cost::plan::execution_demand::queries_of_context: Stage 3 (queries reaching a memo group, or an anchored shift) andcomputations(queries whose walk reaches a plan node) pick a context's queries with one rule. Each side still reaches queries its own way (the memo group before solving, the plan after); objective =plan_coston every workload belowstage3_objective::the_objective_is_the_plan_cost_on_two_phases_of_one_gridmilp/dominance.rsholdsdominated_by_one_evaluation_twin, over two lists (ℓ = None,OneEvaluation) per (context, logical candidate);type Twinsdeleteda_none_execution_its_one_evaluation_twin_dominates_is_left_outcrate::retention::reads_back: every read of an execution's rows (edge needs and query-output spans); costing, (R),least_retention_ms, the greedy bound and F6′ fold itSelectableExecutions::edges_out_ofnext toedges_into; the components, (X) and dominance walk themSelectableEdge { consumer, edge, producer }; all fields private with getters (answering()); context ids in aHashMapretention::ms_to_secondsandExecutionCosts::retention_cost_per_second;has_one_reader; theraw_data_on_diskdoc says it is scaledraw_data_disk_rowsgives the raw-data rows to the scale and to the constraintsdelivers_oncompares withown_schedule(no% 0)per_executionasif/else if letScanarm and the retained raw rows shareraw_rowsRe-verified on
33e7d6d56(same tree as the round-2 head before the rebase ontomain):-D warnings) clean;cargo test --workspace: 1415 passed, 66 failed, the same 66 names as before.plan_cost: round 1 98/98, round 2 25/25, this PR's set 43/43, the round-2 review's h/k set 32/32. Objectives equal toc7465133bexcept h02b, h02c, k19 (N1, above) and g05 (−6e-9). Executed answers: 53/54 (x16 is PR 8's), 25/25, 34/34, 11/11.Timings with F6′ on and off on this head, release
asap_planner, HiGHS seeds 0–4, median (min–max); solve times are seed-dependent:tools/dag-viewerexamples, MILP solve (sum of medians)topkoverrate, one context), walltopk(5, sum by (host)(rate(c[5m])))every 60 s and 120 s), wallc16 is not helped by F6′: its time is the θ fan-out across two contexts (efficiency F4, a follow-up).
Review round 3 and layering: what changed
mainwith #1299physical/candidate.rs(3 commits) andmemo_candidates/tests.rs(both sides add tests, both kept); #1299's finality rule kept indelivers_onwith this PR's schedule; theMaterialized(v)sentence says "no non-final producer"memo_candidates::tests::an_ingestion_time_view_without_a_window_delivers_on_every_contextasap_types::stages::query_reads, the walkplan_costtakes over the plan; Stage 3 walks memo groups), not by every query reaching the group; closes quality R3plan_cost;stage3_objective::the_objective_is_the_plan_cost_on_an_instant_and_a_shifted_range_at_end(fails without the fix); a03, a04, k01, k02, k07, k09, k10, x1–x3 lower, each =plan_cost@ start()rate:evaluation_rateof the request schedules per span, summed over spans;anchored_request_recurrencedeletedat_start_inputs_run_once_per_request_of_each_span(with phases 10 s and 70 s of one schedule: 1/60)ExecutionCosts::least_need_ms_per_execution, one function over costed reads, read by the greedy bound and F6′has_one_readerScanreadsraw_rows_backCandidatesholds its edge;readers_ofin the trait impldominance.rsis_across_evaluationswrapper inlined;SolvedSelection::edgesisVec<SelectedEdge>Operator::chain_tumbling_window_width_ms(types, Stage 3, asap-cost, Pass 2);EvaluationGridmoves to the logical layer (round 4 moveddelivers_ontoPhysicalASAPMemoCandidates; round 5 deletedown_schedule)PhysicalASAPDAGNode::keeps_raw_rows_at_ingestion(aScan, or aTimeShiftwithout@over one) andasap_cost::reads_one_batch_of, used by Stage 3 and now byplan_cost(which charged the reader the whole store) and the cost model's raw-row ruleexecution_demand::tests::an_ingestion_time_reader_of_raw_rows_kept_at_ingestion_reads_one_batch(fails without the fix); no harness workload selects such an edgeasap_cost::output_interval_msis public; Stage 3's Δ of π_v calls itOperator::request_anchorstage3_objectivecase with two queries reading one anchored groupthe_objective_is_the_plan_cost_when_two_queries_read_one_anchored_groupquery_must_not_read,has_one_reader_per_context,ExecutionContext::EVERY_ARRIVING_ROW,QueryWorkloadEntry::{evaluation_grid_ms, request_time_grid_ms},queries_of_context/anchor_is_evaluation_time/query_readsintypes::stages(Stage 3 imports no context rule from asap-cost),PhysicalASAPMemoCandidates::operator,SelectableExecutions::reads_back,Memo::output_groupsReview round 4: M4 and the context-doubling slowdown
RefreshWindow::computed_from: a chain over shifts s is (w, s mod w), anything else the union of the windows below it), over anyTimeIndexedPlan.PhysicalASAPDAG::refresh_window, Stage 3'sIngestionPipelinecontext,delivers_on(throughstored_times) and both cost paths (plan_costand Stage 3'singestion_window_ms, the window's width) call it.own_schedulestayed for query-time results (round 5 deletes it, M-A);maintained_window_msandrefresh_windowsare deletedmemo_candidates::tests::a_stored_chain_and_an_ingestion_time_result_are_stored_on_their_refresh_window(sum(TimeRange(5m))on a 60 s grid: (5 min, 0), no longer delivers on 60 s; a chain overTimeShift(30s): phase 30 s; fails on the old rule); 197 set and 34 examples: costs unchanged (table below)copies_in_equivalent_contexts); exact to within 1e-12 of the largest kept coefficient (Algorithm changes)milp::model::tests::a_context_equivalent_to_another_shares_its_variables; timings belowMedian over HiGHS seeds 0–4 (release, one thread; host load 30–90 from other jobs, before and after run side by side), MILP variables / rows as built, and the cheapest plan cost over the seeds:
Costs differ by at most 5e-10 relative, in both directions: HiGHS stops within its tolerances (the per-seed optima of one model already spread by up to 2e-6 relative on http-p99). On RangeHeavy-8users L1 quantiles (seed 0), the unshared model's own κ60 selection, copied into κ300, costs 1.42518739e-9/s there, less than the 1.42518742e-9/s it selected for κ300: the unshared optimum is itself a near-optimum within HiGHS's tolerance.
Both changes against
4cca6a110, seed 0: of the round-4 set of 197, 168 ran on both heads (the rest, random-mixed workloads with MVP caps and accuracy enforced that plan nothing on either head, were stopped), 142 plan on both, 141 at a bitwise-equal cost and o11y_all14 within 1.4e-16, none planned on one head only; the 34tools/dag-viewerexamples all plan, 28 bitwise equal and the rest within 1.9e-16 (46.9 s → 50.8 s wall, none multi-grid). So M4 changes no selected plan in these sets.Review round 5: what changed
Reviews of
813ecaef5: abstraction, layering and functional (applied here); efficiency is still running and goes into the next round. Commits21e473690(abstraction, layering, #1330 alignment) and796d43503(functional F1, F2 (a), F3). Each fix's test was written first and fails on the code before it.SelectableExecutions::newrecords each query's evaluation context (evaluation_context_of_query, read byevaluation_context(q)); the copy search reads it and theanswering.first()derivation is deleted;selection_variablesdebug-asserts that every sharedcontexts_whose_queries_read_different_groups_are_not_copies(on813ecaef5: the two contexts shared 6 executions with the queries unpaired);queries_reading_different_groups_select_differently_in_contexts_alike(optimum 8; on813ecaef59, Q0 forced onto the exact quantile); 197 set and 34 examples unchangedstored_timesusesRefreshWindow::computed_fromfor any chain, at either timing, and the group's evaluation grid for any other query-time result;own_scheduleis deleted. A query-timeAcrossEvaluationschain overTimeShift(30s)is stored at phase 30 s (was 0)a_stored_chain_and_an_ingestion_time_result_are_stored_on_their_refresh_window(query-time chain: (60 s, 30 s), delivers on 60 s @ 30 s and 120 s @ 90 s, not on 60 s @ 0)debug_assertContextKey(per execution: candidate, kept, costed, least need, in-edges by memo edge and producer; sorted query signatures); only the cost proportionality check compares two contexts, and queries pair in signature order (noswap_removematching)a_context_equivalent_to_another_shares_its_variablesunchangedCOST_RELATIVE_TOLERANCE(1e-12) for dominance and the copies;QueryWorkloadEntry::range_span_msat the three span sitesFrom<RefreshWindow> for EvaluationGrid, used bystored_timesanddelivers_on;EvaluationGrid's doc covers stored timesmilp/equivalent_contexts.rs(MILP symmetry reduction) and returnsSharedVariables;SelectableExecutions::is_root_evaluation_contextreplaces the closuresstages::output_referencesis deleted:Memo::output_groupsreads the existingQueryRoot::operator_refsPhysicalASAPMemoCandidatesPhysicalASAPMemoCandidates::is_query_time_scanasap_types::workload::ONE_SECOND_MS, used byEVERY_ARRIVING_ROWand the cost modeldelivers_on, no "as the user decided" inanchor_is_evaluation_timeequivalent_contexts.rs; the architecture doc's Stage 2 row namesstored_times,delivers_onand the candidates'TimeIndexedPlaninstallable::tests::view_refreshes_compile_only_the_nodes_below_them(passes onmain) failed:avg by (job)(quantile_over_time(0.99, data[300s]))every 1 s at relative 0.05, no raw data, had no planmain's DDSketch plan is valid (#1302: a relative bound onavgholds for values of one sign;datais declarednon_negative). Theavggroup's exact-accumulator path readscount(quantile); aProjectabove the count passes no metric, so the count's group got none, and the walk and its exact-only closure marked the quantile and every sketch below it exact-only. A count's value does not depend on its input's error (error factor 0 in every metric): a consumer now passes every metric in which an input's error does not reach its output, and the closure does not cross such an input. Computing the list per path over alternatives instead was rejected: (X) is per query, and a group reached through an exact alternative and an approximate one would then admit a sketch below the exact onemetric_rule::tests::a_count_reads_its_input_in_any_metric(fails before: every group exact-only); the planner test passes againplan_cost(m04n, m05n, m06n, m08, m10n on non-selected plans, +0.007/s)InMemoryDataStore::append_materialized_viewkeeps the rows ending at the newest end whateverretention_ms), soplan_cost's one output is right. Stage 3 now retains at least the output interval Δ of each candidate with a retention cost (ExecutionCosts::newest_output_retention_ms): its linear retention cost is taken at no less than Δ, and (R)'s least need, the greedy bound and the reportedretention_msare floored at ita_stored_result_read_at_its_newest_output_retains_one_output(retains 10 s and pays 2.01; without the floor 0 and 2.00)OneEvaluationquery-timeScan, are priced for every possible reader (v04n, v13n miss theplan_costoptimum by 7.3e-10/s)a_context_equivalent_to_another_shares_its_variables(rank 0.01 every 10 s and 0.05 every 20 s are not copies; fails before)main)evaluation_gridkeeps phase 0); not changed heredelivers_on's home, renamed functions, Design scopeOperator::chain_tumbling_window_width_mslists every operator, with #1330's signature and doc;Operator::request_anchorlists every operator and everyAtModifier(no_ =>), so #1330'swildcard_enum_match_armlint onnode.rspassesTimeShiftonly (matches!)FilterorProjectbetween a chain and aTimeRangekeeps each row's window end, so it keeps the chain's width, as aTimeShiftdoes. The rule ischain_tumbling_window_width_ms()or the input's width whenOperator::keeps_input_window()(exhaustive, #1330'skeeps_input_timebody under the name its review proposes); the windows aTimeRangeselected pass through the same operators. Before, aTimeRange(5m)overFilterover a 60 s chain was costed as one window: the merge above it read 1,000 states instead of 5,000an_operator_keeping_its_input_window_over_a_chain_or_its_windows_selects_what_the_range_does(fails before: the merge read 1000 states, not 5000); 197 set and 34 examples unchangedFor #1330: #1330 (
refactor/no-silent-defaults,273cadbeb) ownsOperator::keeps_input_windowand the exhaustiveOperator::chain_tumbling_window_width_ms(with itswildcard_enum_match_armlint and their tests), and is recommended to merge first. This PR keeps identical copies of both bodies so that it builds and tests on its own. Once #1330 merges, the rebase drops these copies and keeps #1330's; the conflict is textual only (the same lines intypes/src/ir/operator/node.rs), and this PR's two callers incost/src/node/usage/mod.rsand its deletion ofmaintained_window_msstay.Executor finding (not changed here, #1336): the executor does not apply a
TimeShiftbelow a query-time chain. Its window filter takes its offset from the readers above (window_read_above) while theScanbelow reads the shifted range, so the two windows miss each other. A throwaway executor test,sumper job over one sample per minute,TimeRange(5m)over a 60 s chain overTimeShift(30s)over the scan: at T = 4 h it returned a = 950 where the shifted window gives 1185 (the samples at T − 4m … T − 1m only). At T = 4 h − 30 s it returned 1185. M-A does not depend on the executor; it makes Stage 3 offer such a chain only to readers at phase 30 s.The functional reviewer's harness was re-run on
796d43503for the m* and v* workloads. The only changes to it were three hook edits for the new names.plan_coston all 37.813ecaef5except v15n. There both contexts now keep their own KLL: k = 269 for the rank-0.01 query (it was k = 52, shared), which is within its band on 12 of 12 series (was 9 of 12). Its MILP has 218 variables, as with copies off; it was 120.plan_costoptimum on all 18 finished workloads: m01n–m11n, v01n, v02n, v03n, v11n, v12n and v17n.plan_coston every enumerated plan there. On813ecaef5it differed on 8 to 2,304 plans of m04n, m05n, m06n, m08 and m10n (F2 (a)). For example, m04n plan 11 was 0.04128 in the model against 0.04828 inplan_cost; both are now 0.0482833.plan_cost, by up to 0.8%, and a plan 7e-10/s cheaper exists.plan_cost.Timings: release build, one thread, 5 repeats per head of the same deterministic model, run side by side on pinned cores (host load average 25–33 from other jobs). The MILP size is as built.
813ecaef5median (range)796d43503median (range)The model has the same size on both heads. On
21e473690it is also the same cost to the bit, yet a first run gave RangeHeavy 13.8 → 15.5 s. So the wall differences are host load and HiGHS's path, not a larger model.Plan costs against
813ecaef5, with the same binary build and instrumentation for both heads:tools/dag-viewerexamples (accuracy ignored, as before): all plan, with the same MILP sizes; costs are bitwise equal or within 1.9e-16.21e473690(round-5 layering and H1) changes no cost: all 148 workloads that plan are within 1.4e-16, and none plans on one head only.796d43503(functional fixes):plan_costalready charged that output.avg by (...)(quantile_over_time(φ, data[100s]))at relative 0.05 overdata, which is declarednon_negative.With MVP caps (no raw data kept), 30 workloads now plan where
813ecaef5found no plan: random-mixed-seed-862 workloads 01–10, -x5 01–10 and -long 01–10. Before, the MILP was infeasible or stopped.With default caps, 6 workloads cost less:
813ecaef5796d43503The
avgof quantiles now reads DDSketch quantiles, which F1 found valid, instead of the exact quantile over raw rows.Round-5 reviews: what changed
Abstraction, layering, functional and efficiency reviews of
796d43503. The branch is rebased ontomain59365ce80(with #1308's tests; no conflicts), and one commit,44a223d93, applies the fixes. Each fix's test was written first and fails on the code before it.COST_RELATIVE_TOLERANCEof the copy context's largest cost, with α from that paircontexts_whose_small_edge_costs_differ_by_rounding_are_copies(those two edge costs, 60 s and 300 s; fails on796d43503). L1 flows 1,526 → 786 variables (258 executions shared), http-distinct 8,510 → 4,342 (1,366), RollupHeavy-8users 15,630 → 8,003 (2,542); costs: RollupHeavy-8users and http-distinct bitwise equal to round 4's, flows within 1.5e-12 of796d43503ErrorContract::input_error_reaches_outputandErrorMetricsOfMemo::error_metrics_allowed_on_input= (output_allowed ∩ passed) ∪ ignored, used by both exact-only loops (the closure is now.is_empty()), the edges (error_metrics_allowed_on_input_edge, per input) andreadout_caps. The two field docs say the closure stops below acount's inputa_counts_input_edge_from_a_readout_holds_in_a_metric_its_output_does_not_allow(countof a KLL readout at relative 0.05:MetricRule { producer_metric: Rank }before). Widening a group's allowed metrics for a count was tried and dropped (see "A count's input is not exact-only")most_read_msis floored at Δ where it is built;ExecutionCosts::retention_ms_for_needfloors each edge and output need, used byleast_need_ms_per_execution(now over&ExecutionCosts) andleast_retention_ms.newest_output_retention_msholds Δ for every candidate that retains rowsNonASAPOp::map_childrenappliedfto scalar references before the direct child, whilechildren()lists the child first, socandidate_node(which hands out node ids by position) builtq + scalar(count(q))'sProjectover the count, withqas the scalar reference.map_childrennow appliesfinchildren()order (Filter,Project,Aggregate,Join,Sort,SQLWindowFunc,PromqlRelabel).plan_costalso takes a node's producers in its payload's input order, whatever order the DAG lists its edges instage3_objective::the_objective_is_the_plan_cost_on_a_scalar_operand(f07r's shape: objective 0.0360364 vsplan_cost0.0360197 before; equal after, and equal with the plan's edges reversed, which fails without the producer order)From<RefreshWindow> for EvaluationGridnext toRefreshWindowinplan;ExecutionContext::is_outside_evaluations;QuerySignaturewith named fields;QueryWorkloadEntry::evaluation_grid()replaces Stage 3's freeevaluation_gridkey_ofreturns each context's queries in signature order (no recomputed signatures, noexpect); queries are iterated directly. The greedy bound's 1e-9 slack has a comment (leaving an execution out cannot be undone), its relaxation usesCOST_RELATIVE_TOLERANCE, and dominance's absolute floor is commented.@ start()spans userange_span_ms()computed_fromandleast_need_ms_per_executioninterfaces, Removals and M4 "Before" (onlymain's items), reading-order paths, c16 timing, the broken formulasmainon some workloads)Planning time against
mainand796d43503: releaseasap_planner --run-until stage3, one thread, each workload pinned to its own core, 5 runs alternating the three trees, median (range); host load 5–65 from other jobs. Variables are as built; "shared" is the executions a copy takes from its original.main59365ce80796d43503796d43503/ headmainmain)main(#1352)The head changes no timing beyond noise except where copies share again (E1).
mainselects differently on several of these, so its time buys a different plan: RollupHeavy-8users costs 2.2477e-5/s onmainand 2.1459e-5/s here; of the default-capability workloads, 03 and -x5 03 cost the same on all three trees (230.42/s, 1154.86/s), while 01, 02, -x5 01 and -x5 02 cost 685.0, 1115.0, 3427.6 and 5577.8/s onmainagainst 208.0, 303.1, 1042.6 and 1519.5/s here. The slowdown againstmainis in HiGHS (weak LP relaxation, little presolve; HiGHS logs in #1352), not in building the model.These timings depend on host load. Three rows moved by up to ±50% between measurements, in both directions; the ratios hold, the absolute times do not. Release
asap_planner --run-until stage3, one core per workload, runs alternating the two trees, median (range):44a223d93(load 13–47)0b5ec4653(load 29–53)main60.3 s, head 49.4 smain29.80 s, head 24.44 smain42.8 s (42.4–58.1), head 28.4 s (28.3–32.1)main5.54 s, head 0.74 smain8.68 s, head 1.08 smain6.44 s (5.70–6.94), head 0.76 s (0.76–1.05)main8.70 s, head 3.18 smain13.63 s, head 4.42 smain18.1 s (18.0–19.0), head 6.19 s (6.17–7.04)Variables and plan costs are the same in every measurement. RollupHeavy-8users is bimodal on both trees (runs near 30 s and near 42–62 s), so its row reads "about equal" to 1.5× faster depending on the run; the two L1 rows are 2.7–3.1× (http-distinct) and 7.5–8.5× (flows) faster on the head in all three.
RangeHeavy-8users plan cost,
main9.6355e-3/s → head 9.5295e-5/s. The 100× is inmain's reported cost, not in the plan:main'sselected_plan_cost_per_secondis its MILP objective, which for this workload charges about 9.54e-3/s thatplan_costdoes not. Priced bymain's ownplan_cost,main's plan costs 9.5430e-5/s, so the head's plan is 0.14% cheaper, and the head's objective equals itsplan_cost(9.52950391423e-5/s). Measured on0b5ec4653with a temporary per-node dump ofplan_cost(not committed).Plan costs against
796d43503, the same instrumented build of both trees, one run each (the 197 set with its capabilities, the 34 examples with accuracy ignored): 212 of 231 workloads plan on both and none on one only (19 plan on neither, as before). All 212 have the same MILP size and cost within 1e-9; 195 are bitwise equal and 17 differ by at most 2.2e-16. So no plan changes on these sets; themap_childrenfix changes plans only where an operator reads a scalar reference, which none of them selects (f07r is the pinned case). Copies share executions on 4 of them, as before.Round-6 review: what changed
Abstraction and layering review of
44a223d93; one commit,0b5ec4653, applies the fixes. Each behavior fix's test was written first and fails on44a223d93.proportional_costspairs only kept executions and edges whose two ends are kept (the key already makeskeptequal in both contexts), so the tolerance is 1e-12 of the largest kept coefficient; it also requiresa_left_out_execution_does_not_set_the_tolerance_of_context_copies:m[5m]every 60 s and every 300 s with a left-outOneEvaluationrange costing 1e12 (2e11). On44a223d93the contexts merged and the optimum was 2.5; now they do not, and it is 2.421. RollupHeavy-8users still shares: L1 flows 786 variables (258 executions shared), L1 http-distinct 4,342 (1,366), whole 8,003 (2,542), the same as on44a223d93least_need_ms_per_execution(field and function) →least_retention_ms_per_execution: it holds each execution's least retention, each need floored at ΔScalarExpr::map_partsmaps anIN (SELECT …)'s subquery before its left side, asoperator_refslists them, somap_childrenfollowschildren()order for every operatornon_asap::tests::map_children_calls_f_in_children_order: every operator with scalar references (Scan,Values,Filter,Project,Aggregate,Join,Sort,SQLWindowFunc,PromqlVectorFromScalar,PromqlRelabel), each with anINsubquery; fails on44a223d93(Scan: 11, 12, … against 12, 11, …)plan_costreads each node's producers from its payload (PhysicalASAPDAGNode::producers, fromedge_roles, which a validated DAG's edges equal) instead of sorting the edges, so no producer sorts first silentlystage3_objective::the_objective_is_the_plan_cost_on_a_scalar_operand(edges reversed)execution_costsbuilds each candidate's unretained node once and reuses it forretains_rows, Δ and the retained nodessignatureindexesoutputs_of[&query]: a query has an evaluation context exactly when it has an output referenceErrorContract::input_error_reaches_outputispub(crate)and indexes its input directly (contracts are sized tochildren().len())map_childrenorder andproducersin the interface diffTests run on
0b5ec4653(only those covering the changed code):cargo nextest run -p asap-types -p asap-accuracy -p asap-cost -p asap-stage3-plan-selection --lib256 passed;-p asap-types -p asap-stage3-plan-selection -p asap-planner --tests292 passed, 2 failed (summary_sharing::identical_ungrouped_queries_share_their_producers,runtime_install::check_compiles_reports_the_executors_error, both failing onmaintoo);-p asap-integration-testsstage3_objective16 passed;cargo fmt --checkandcargo clippy --all-targets -D warningson the four changed crates clean.For #1242: both PRs change
NonASAPOp::map_children/map_parts(#1242 renames it and adds column renumbering), so the PR that merges second must keep:crates/planner/tests/main.rsneeds bothmod exact_only_reads;andmod reduce_aggregates;. Innon_asap.rstheFilter,Join,Sort,SQLWindowFuncandPromqlRelabelarms conflict;ProjectandAggregatemerge without a conflict and come out right (f(child)first, thenmap_parts(column, f)).map_parts,fon the direct inputs before any scalar expression of the arm:Filterchild, thenpred;Joinleft, right, thenpred;Aggregatechild,filters, thenhaving;Sortchild, thenkeys;SQLWindowFuncchild,args, thenorder_by;PromqlRelabelchild, thenvalue;Projectchild, thencols. InScalarExpr::map_parts, anInSubquery's subquery before itsexpr.map_pred(…, column, f),map_keys(…, column, f),partition_by.map_columns(column), andkeptfor aScan's predicates and anAggregate'shaving; this PR's&mut fcalls becomef, sincemap_partsalready takesfby reference.Self::childrenorder …qthenc" sentence moves ontomap_parts, asmap_childrenthen delegates to it.map_children_calls_f_in_children_orderstays and must pass after the merge.map_input_columns(C::clone) does not depend on the order; the only positional caller,plan.rs:41, is the same in both PRs.Round-7 verification
The round-6 functional and efficiency review (of
44a223d93), re-checked on0b5ec4653; no code change, so the head stays0b5ec4653. Release builds, review harness not committed.RV_BIG(half in the copy), small scan costs whose preferred alternative flips between the contextsRV_BIGfrom 1e4 to 1e15 (1e6 gave 6.0e-7 on44a223d93). Not added as a test: it is the same case asa_left_out_execution_does_not_set_the_tolerance_of_context_copies(a left-out execution setting the tolerance), at smaller absolute costsplan_of+plan_coston every enumerated selection), m/v/h/f and scalar-reference sets, 119 workloadsplan_coston all 108 (largest relative difference 3.7e-16). The selection is the enumerated optimum on 106; v04n (6.85e-9) and v13n (4.13e-9) are the #1335 twins. Among unselected selections, the model only overprices (#1335) or differs by at most 6.6e-12 relativecopies_in_equivalent_contextsreplaced by the identity), 197 set + 34 viewer workloadsReview status
4cca6a1100d04e4d90,813ecaef5813ecaef5→796d43503796d43503→44a223d93(rebased ontomain59365ce80)44a223d93→0b5ec465344a223d93, verified on0b5ec4653(no code change)0b5ec4653), efficiency (E-a, L1, plan-cost note)Design scope
New file, type and field (all crate-private to Stage 3 unless noted):
stage3-plan-selection/src/milp/equivalent_contexts.rs. Scope: MILP symmetry reduction, which contexts' variables the MILP may share. It holdsSharedVariables(per execution, edge and query, the original whose variables it takes) and the privateContextKey,ExecutionKeyandExecutionInContextit buckets contexts by. These are general: any workload asking the same queries on several grids.SelectableExecutions::evaluation_context_of_query(read byevaluation_context). Scope: the Stage 3 selection index; it records which context each query is evaluated in.ExecutionCosts::newest_output_retention_ms. Scope: Stage 3 costs; per candidate that retains rows, its output interval Δ, the newest output a store always keeps; read throughExecutionCosts::retention_ms_for_need, the one place a read's need is floored at it.const COST_RELATIVE_TOLERANCE(milp/mod.rs). Scope: Stage 3 MILP reductions; one tolerance for "equal cost": dominance, the greedy bound's relaxation, and the copies, where it is taken of the largest cost the MILP reads in the copy context (kept executions, edges between kept ones).QuerySignature(private,equivalent_contexts.rs) now has named fields:answering_executions_per_output,exact_only_groups,range_span_ms,accuracy_target_index.New public items on existing types:
Operator::{chain_tumbling_window_width_ms, request_anchor, keeps_input_window}(IR). Each answers one per-operator question and lists every operator.QueryWorkloadEntry::{evaluation_grid_ms, evaluation_grid, request_time_grid_ms, range_span_ms}andasap_types::workload::ONE_SECOND_MS(workload;evaluation_gridreads only the entry, so it moved from Stage 3).EvaluationGrid, moved to the logical stage;From<RefreshWindow> for EvaluationGridsits next toRefreshWindowinplan, sostages::logicalimports nothing fromplan. AlsoLogicalASAPMemoCandidates::query_must_not_readandMemo::output_groups.PhysicalASAPMemoCandidates::{operator, stored_times, delivers_on (moved from PhysicalCandidate), is_query_time_scan}, andTimeIndexedPlan for PhysicalASAPMemoCandidates. Scope: Stage 2's output, physical stage; Stage 3's per-edge view delegates to it.PhysicalProperties::{is_across_evaluations, has_one_reader_per_context}andExecutionContext::{EVERY_ARRIVING_ROW, is_outside_evaluations}(the context's kind decides whether evaluation contexts share its executions).ErrorContract::input_error_reaches_outputandErrorMetricsOfMemo::error_metrics_allowed_on_input(_edge)(asap-accuracy, the metric rule): one rule for "an input's error reaches the output", used by the exact-only lists, the edges and the readout caps.stages::physical::execution_context: the free functionsquery_reads,queries_of_contextandanchor_is_evaluation_time, used over both memo groups and plan nodes.RefreshWindow::computed_from(plan),PhysicalASAPDAGNode::{keeps_raw_rows_at_ingestion, producers}(plan;producersreads only the node's payload, soplan_costtakes producer order from it instead of sorting edges) andshift_back_ms_belowover alternatives.asap_cost::{output_interval_ms, reads_one_batch_of, context_executions_per_second}andConsumer::QueryRequest.The context-copy sharing is a formulation change, accepted by the reviewer (see "Decided").
Algorithm changes
A copy of an equivalent evaluation context shares the original's variables
topk(5, sum by (host)(rate(c[5m])))every 60 s and every 120 s) took 107–239 s, against 15 s for either query alone; RangeHeavy-8users 222 s on seed 0.copies_in_equivalent_contexts(milp/equivalent_contexts.rs) walks the root evaluation contexts (no request anchor) in id order, buckets them by a canonical key, and within a bucket compares only costs; a context is a copy of an earlier one when their executions are of the same candidates, kept alike, with the same least need, their edges the same memo edges from producers that correspond (in the context, or the same execution outside every evaluation context; a producer in another evaluation or request context rules it out), their queries (each in its own evaluation context,SelectableExecutions::evaluation_context, wherever its answering executions run) pair up with the same output executions, exact-only groups and range span, and every cost the MILP reads (of a kept execution, or of an edge between two kept ones) is the same multipleCOST_RELATIVE_TOLERANCE) of the largest such cost in the copy context. A left-out execution's cost does not count: it is not in the MILP, and one costing 1e12 would otherwise allow each kept cost an error of 1 (round 6, E1-b).selection_variablesgives the copy's executions, edges and queries the original's variables; the objective adds their costs on one variable. All rows stay (duplicates of the original's), HiGHS's presolve drops them.One refresh rule for an ingestion-time stored result (M4)
sum(TimeRange(5m))view on a 60 s group grid ran its inputs on a 60 s pipeline and served a 60 s reader in Stage 3, while the installed plan refreshed it every 5 minutes (tumbling) and the cost model charged one output per 5 minutes; a chain overTimeShift(30s)had phase 0 in Stage 3 and 30 s in the plan.PhysicalCandidate::delivers_on): a chain stored at every multiple of its width at phase 0, any other stored result on its group's evaluation grid; planrefresh_window; costmaintained_window_ms(the first range or chain below through filters, projections and shifts).RefreshWindow::computed_fromoverTimeIndexedPlan, the plan's rule, for all three. Over candidates the producers of one input are alternatives:windows_computed_fromunions their windows andshift_back_ms_belowneeds their shift sums to agree.stored_timesuses this rule for any chain at either timing too (a query-time chain overTimeShift(30s)is stored at phase 30 s), and its group's evaluation grid for any other query-time result. Today no reader is offered such a chain: the frontends strip the shifts below a chain and re-wrap them above it (add_chain_groups), so every chain reads unshifted data, and a 60 s query at phase 30 s runs on a 30 s grid at phase 0 (P4). The gate is correct (a review-only test of a query-time 60 s chain overTimeShift(30s): stored at (60 s, 30 s), no edge to a 60 s reader at phase 0), but the gain it describes cannot happen yet, and Executor: a query-time chain over a TimeShift ignores the shift in its own window #1336 cannot be triggered by a real workload.None(only final producers deliver).A count's input is not exact-only (round 5, F1)
avg by (job)(quantile_over_time(0.99, data[300s]))at relative 0.05 over non-negativedata, with no raw data kept, had no plan. Pass 2 listed every DDSketch group as exact-only for it, and (X) enforces the list.exact_only_groups_of_querypasses to an input only the metrics allowed at the consumer's output that its contract passes. Theavggroup's exact-accumulator path readscount(quantile)under aProjectthat passes no metric, so the count group allowed none. The walk then gave the quantile group no metric (the intersection over paths), and the closure "every group below an exact-only group" covered every sketch below.Countaccumulator or acountmeasure). One function holds the rule,ErrorMetricsOfMemo::error_metrics_allowed_on_input= (allowed at the output ∩ passed by the contract) ∪ the metrics whereErrorContract::input_error_reaches_outputis false; the walk, its closure (an input below an exact-only group is exact-only when that set is empty under an output allowing none) and the edges' metric rule use it, and the readout caps useinput_error_reaches_output. The metrics a group's output allows (readout_caps) are not widened by a count: widening the quantile group underavg = sum / countadmitted every alternative readout of it and grew models with no cheaper plan (avgof a relative quantile 62 → 455 MILP variables, random-mixed-seed-862 workload-01 875 → 1,164 and 60 ms → 12.7 s, four defcaps workloads that plan in 8–25 s did not plan within 300 s in one run on a loaded host), so a count's input edge holds only where it reads a readout directly. The intersection over paths, and over a group's alternative candidates, stays: (X) is per query, so a group reached through an exact alternative must stay exact-only below it.metric_rule::tests::a_count_reads_its_input_in_any_metric;installable::tests::view_refreshes_compile_only_the_nodes_below_them.A stored result retains at least its newest output (round 5, F2 (a))
TimeShift(which reads its newest window) gotplan_costcharges one retained output (retained_output_count … .max(1)), and the store keeps the newest output whateverretention_ms.ExecutionCosts::newest_output_retention_msholds Δ, the output interval, for each candidate that retains rows. The most any consumer reads (most_read_ms) is floored at Δ once, where it is built, so the linear retention cost is taken atExecutionCosts::retention_ms_for_need, which (R)'s least need per execution (and so the greedy bound) andleast_retention_msuse. A selected stored result retains at least Δ and pays for it.a_stored_result_read_at_its_newest_output_retains_one_output. On the 197 set only y18 changes, by 2.7e-11 relative.Selection over execution contexts
SelectableExecutions::newwalks from each query output: the query's context isEvaluation { grid }. An input of an ingestion-timeAcrossEvaluationsconsumer must deliver on its refresh window (IngestionPipeline); aTimeShift { at: End | Start }'s input runs on the grid of its queries' request end (start) times; anAcrossEvaluationsproducer runs inMaterialized(u); any other ingestion-time producer in the one ingestion contextEVERY_ARRIVING_ROW(each arriving row computed once, asasap_cost::computationscharges it); any other in its consumer's context.cost::execution_costs: each execution costed withnode_resource_usage_and_costatexecutions_per_second(ingestion 1; evaluation context:evaluation_rateof the context's queries reaching the group, the@-anchored subtree once per request;Materialized(m): as often asasap_cost::plan::execution_demand::executions_per_second). Edge cost = the consumer's cost in its context with inputedge_retention_need_ms);Scanmaterialized at ingestion.milp: (A)–(F) and the one-consumer rule per execution; (R):plan::plan_ofbuilds one node per selected execution in Kahn order;retention_ms= the leaststage3_objectivepins the optimum toplan_cost(1e-9 relative) on five workloads (two chain widths without raw data, request overhead and disk on two filtered scans and on one query, the shared scan, q21). Model tests: one-consumer rule per context, a None scan once per context on two grids, a materialized chain once retaining 300 s for 1 m and 5 m readers, (E) on a cycle, the bound's left-out candidate;cost::tests::no_execution_reading_a_producer_costs_less_than_nothing; the planner test above.node_resource_usage_and_costper candidate, per execution, per context edge and per retaining candidate (one more). Example 3a: 1,560 variables, 2,130 rows, solved in 0.34–0.54 s over seeds 0–4 (release); timings per seed in "Review round 1: what changed".Exact-only reads per query, constraint (X) (second commit)
exact_only_groups_of_querylists, per query, the groups it may read only exactly, while other queries may still read their sketches; PR 6's selection did not enforce it, so a relative-error query read a rank-error query's KLL, and (with PR 6's exact-target fix) an exact query read a sketch readout.LogicalASAPMemoCandidates::query_must_not_read: its candidate has sketch parameters θ and its group is onmilp::model::tests::a_query_reads_its_exact_only_groups_through_exact_candidates_only(Q0 every 10 s, Q1 every 30 s on one p90: with Pass 2's list Q1 reads the exact quantile at the output; with only the state group listed, along the edge; with no list, the readout); plannerexact_only_reads::a_relative_query_reads_no_rank_sketch_of_a_query_sharing_its_groups(R3) and..::an_exact_query_sharing_groups_with_an_approximate_one_reads_no_sketch(R2). All three fail with (X) removed and pass with it. Sketch sizing for another query's target (R2(b)) stays for PR 8.asap-cost per context, raw rows at ingestion, shifted chains
computations: a node not materialized across evaluations is computed once per execution context of its consumers, per (evaluation grid, anchor) (multiplier = number of contexts), atcontext_executions_per_second, the rate Stage 3 uses; an@ end()/@ start()subtree once per request (Consumer::QueryRequest { at }). Tests:a_not_materialized_node_is_computed_once_per_evaluation_grid,a_range_query_computes_its_at_end_subtree_once_per_request,a_shared_one_evaluation_at_end_shift_runs_its_input_once_per_request,at_start_inputs_run_once_per_distinct_request_start.Scanor aTimeShiftby an offset over one; both output and keep the rows of theirretention_ms, which for a stored shift includes its offset (refactor(stage3): baseline selection MILP and its PhysicalASAPDAG; executor θ, native chains and a time-indexed store (#1207 PR 6) #1236'sStoredThroughread), as PR 6's F1 rule keeps for any other ingestion-time view (q21: the objective was 0.326/s for a planplan_costcharged 0.0219/s). Test:usage::tests::an_ingestion_time_view_of_raw_rows_keeps_the_rows_its_retention_covers(a 390 s retention behind a 90 s offset keeps 390,000 rows).chain_tumbling_window_width_mspasses through every operator that keeps its input's window (Operator::keeps_input_window:TimeShift,Filter,Project), which also pass ontumbling_windows, so aTimeRangeover a shifted, filtered or projected chain selects windows. Test:usage::tests::an_operator_keeping_its_input_window_over_a_chain_or_its_windows_selects_what_the_range_does(round 5 addedFilterandProject; failed before).PR 6 round-2 findings (measured on
1ef03212f)The numbers below are from before review round 1; it changed z05 to 0.9301/s ("Review round 1: what changed").
With PR 6's F1 rule (an ingestion-time view keeps what its retention covers, a$h_v$ plus the offset), any other ingestion-time view takes PR 6's rule. PR 6's F2 (a), a query-time $h_v$ and (R): the scan's read and retention follow the selected readers. Measured with the functional review's harness on its round-2 workloads (default unit costs, executed against the exact reference):
TimeShift's offset more) and this PR's raw rows at ingestion are kept together; aTimeShiftby an offset over aScanstays raw rows (output and keep the rows of their retentionScancosted for the longest reader it could have, is closed here by9ec3847fc: optimum /plan_costplan_costcount_over_time(m[1m])every 60 s alonemax_over_time(m[1d])every 600 scount_over_time(m[1m])every 10 s alonemax_over_time(m[1d])every 600 s[1m]every 1 h +[1h]every 10 sAll 24 planned round-2 workloads: optimum =
plan_cost(PR 6: 23 of 24); none selects an ingestion-timeTimeShiftview; 23 executed answer within target (z05 reads a 1-day chain the harness's 2 hours of data cannot fill; y16 is the drain-order deadlock the review notes, not run). The F1 workloads: y13 0.10455/s, y05 0.0924/s, y08 0.19813/s, y18 0.04160/s, y02 0.64055/s, z03 0.15300/s, all installed and exact. The 117 round-1 workloads (planned, not executed): optimum =plan_coston all 98 planned, the same statuses as PR 6.Decisions and open questions
Needs reviewer decision:
main. Some workloads plan much more slowly than onmain(RangeHeavy L1 quantiles 0.12 s → 6.78 s, default-capability workload-03 2.73 s → 25.6 s; table in "Round-5 reviews"), from a weak LP relaxation and little presolve over execution contexts (Stage 3 over execution contexts plans some workloads 3–37× slower than main: weak LP relaxation, little presolve (RangeHeavy L1 quantiles 0.21 s → 7.7 s) #1352). Others are faster (c16, c09, the RollupHeavy L1 flows and http-distinct components); RollupHeavy-8users as a whole is about equal. You decide whether this PR merges before the formulation is tightened in Stage 3 over execution contexts plans some workloads 3–37× slower than main: weak LP relaxation, little presolve (RangeHeavy L1 quantiles 0.21 s → 7.7 s) #1352.Decided:
copies_in_equivalent_contextsabove (milp/equivalent_contexts.rs, returning the crate-privateSharedVariables); the MILP for a workload with the same queries on several grids has one set ofmain). Round 5 found and fixed a fault in it (H1): a query was put in the context of its first answering execution, which for anAcrossEvaluationscandidate isMaterialized(v), so two contexts whose queries read different exact-only groups counted as copies; with it, Q0 every 10 s and Q1 every 20 s that must read the exact quantile selected alike, at 9 instead of the optimum 8 (queries_reading_different_groups_select_differently_in_contexts_alike, fails before). No harness workload changed (197 set, 34 examples). Since round 5 the pairing also compares each query's accuracy target (F3), so (K) in refactor(stage3): constraint (K) per execution and metric; sign-conditioned DDSketch guarantee, classic HLL certificate from the source's distinct bound (#1207 PR 8) #1238 keeps it exact.delivers_on, Stage 3's ingestion pipeline and the cost model use the plan's rule; only readers on a grid the refresh window divides read such a view. Rejected: (a) refreshing on the readers' grid (needs a new window type).@ end()runs in that query's evaluation context, since its anchor is its evaluation time, and shares results with the plain reads on its grid, in Stage 3's contexts and inplan_costtogether (asap_types::stages::anchor_is_evaluation_time; when an instant and a range query on one grid share the shift, the input runs at every evaluation of the grid, which covers the range query's request ends). Range queries and@ start()keep their per-(anchor, queries) request contexts. a06 back to 0.0360/s, objective =plan_cost. Tests:stage3_objective::an_instant_at_end_input_shares_the_evaluation_context,execution_demand::tests::an_instant_at_end_input_runs_in_the_evaluation_context(both fail without the change).EVERY_ARRIVING_ROW, anIngestionPipelineof 1 s), ascomputationscomputes an ingestion-time node once per second whatever views read it. A producer of an ingestion-time chain is still checked against the chain's refresh window (delivers_on).plan_costdoes;SolvedSelection::retention_msis the leastmilp/exact_only.rswithread_by_query) and constraint (X).Materialized(m)runs as often asasap-costrunsSelection::selected_plan_cost_per_secondis the optimum; (B)/(D) left out where (F)'s equality implies them; the scale in the raw-data rows.Verification
On
44a223d93(round-5 reviews), rebased ontomain59365ce80:cargo test --no-fail-fastfor asap-accuracy, -types, -cost, the three frontends, stage0, Pass 1, Pass 2, Stage 2, Stage 3, -planner, -devtools, -integration-tests; and the same onmain59365ce80main1051 passed, 65 failedtodo!("#1207 PR 9")and the known planner/integration ones); none fails on the head only. Names only onmain: the renameda_query_time_result_that_is_not_materialized_has_one_consumerand doctests whose line numbers movedcontexts_whose_small_edge_costs_differ_by_rounding_are_copies,a_counts_input_edge_from_a_readout_holds_in_a_metric_its_output_does_not_allow,stage3_objective::the_objective_is_the_plan_cost_on_a_scalar_operandcargo fmt --check,cargo clippy --workspace --all-targets -- -D warningsOn
796d43503(review round 5):cargo test --no-fail-fast -p asap-types -p asap-cost -p asap-accuracy -p asap-stage1-pass2-reuse-candidates -p asap-stage2-physical-optimization -p asap-stage3-plan-selectiona_count_reads_its_input_in_any_metric,contexts_whose_queries_read_different_groups_are_not_copies,queries_reading_different_groups_select_differently_in_contexts_alike,a_stored_result_read_at_its_newest_output_retains_one_output,a_stored_chain_and_an_ingestion_time_result_are_stored_on_their_refresh_window,an_operator_keeping_its_input_window_over_a_chain_or_its_windows_selects_what_the_range_does; the copy test also checks targetscargo test --no-fail-fast -p asap-planner -p asap-integration-testsmain59365ce80813ecaef5had a 10th,installable::tests::view_refreshes_compile_only_the_nodes_below_them(F1; the round-4 claim "none newly failing" compared with4cca6a110, notmain), now fixedcargo fmt --check,cargo clippy --workspace --all-targets -- -D warningsOn
813ecaef5(review round 4):cargo test -p asap-types -p asap-cost -p asap-stage2-physical-optimization -p asap-stage3-plan-selectionan_ingestion_time_result_is_stored_on_its_refresh_window(M4's two examples),a_context_equivalent_to_another_shares_its_variables(the copy mapping, optimum and variable count)cargo test --no-fail-fast -p asap-executor -p asap-planner -p asap-integration-tests4cca6a110main(view_refreshes_compile_only_the_nodes_below_them, F1 of round 5)cargo fmt,cargo clippy --all-targetson the four changed cratesOn
4cca6a110(onmain4de407569):cargo fmt --check;cargo clippy --workspace --all-targets -D warningsclean.cargo test --workspace: 1425 passed, 66 failed, the same 66 names as before.plan_costObjectives on the round-3 head (
77f403791) against33e7d6d56: equal except N-k19's set (k19_C/C2 0.3642 → 0.1542/s, a03 0.1296 → 0.1122, a04 0.1364 → 0.1146, k01/k02 0.2149 → 0.1624, k07 0.1800 → 0.1346, k09 0.0891 → 0.0717, k10 0.1103 → 0.0884, x1–x3 0.1296 → 0.1122) and p12/p13 (@ <ts>final since #1299: 1.208 → 0.0132, 1.252 → 0.0574), each =plan_cost. The layering commits change no objective on any workload.tools/dag-viewerexamples: objectives equal to77f403791within 3e-16; 37.8 s wall (release, seed 0); example 3a 21,686.84/s.stage3_objective::{..._on_an_instant_and_a_shifted_range_at_end, ..._when_two_queries_read_one_anchored_group},execution_demand::tests::an_ingestion_time_reader_of_raw_rows_kept_at_ingestion_reads_one_batch,memo_candidates::tests::an_ingestion_time_view_without_a_window_delivers_on_every_context; the N1 and N-k19 tests fail without their fixes.Follow-ups
Stage 3 overprices unselected plans with split twin readers or a shared OneEvaluation scan #1335: Stage 3 overprices unselected plans whose twin executions split their readers, or whose shared
OneEvaluationquery-timeScanhas readers of different windows (round 5, F2 (b), (c)).Executor: a query-time chain over a TimeShift ignores the shift in its own window #1336: the executor's query-time chain over a
TimeShiftignores the shift in its own window (round 5, executor finding).Stage 3 over execution contexts plans some workloads 3–37× slower than main: weak LP relaxation, little presolve (RangeHeavy L1 quantiles 0.21 s → 7.7 s) #1352: Stage 3 over execution contexts plans some workloads 3–37× slower than
main(weak LP relaxation, little presolve; round-5 efficiency review, E2).Stage 3 MILP: scaling the objective by its largest coefficient lets HiGHS stop 5e-8 relative above the optimum (RangeHeavy-8users L1 quantiles) #1356: the objective is scaled by its largest kept coefficient, which can belong to an execution the optimum does not select; then the optimum is about 1 scaled unit and plans about 1e-7 relative apart fall inside HiGHS's absolute tolerances (RangeHeavy-8users L1 quantiles: 9.1106875266e-5 with copies, 9.1106870769e-5 without, 4.9e-8 relative; the same before this PR's copy check; round-6 functional review, L1).
Accuracy constraint (K) per execution and metric: PR 8 (refactor(stage3): constraint (K) per execution and metric; sign-conditioned DDSketch guarantee, classic HLL certificate from the source's distinct bound (#1207 PR 8) #1238), including sketch sizing for another query's target. MILP sizes and solve times on the snapshot workloads: PR 9 (refactor: tools and snapshots over the #1207 memo, edges and contexts (#1207 PR 9) #1239).
Planning time (efficiency review F4 and F5, not in this PR; F6′ and context copies are in): an aggregated linking cut$\sum_{c \in G} y_{\kappa,u,c,i} \le N_{\kappa,G},x_u$ for the θ fan-out bound, tree-cost pruning with a better greedy bound, and tighter (R)/(X) linking under fractional $y$ .
The concepts doc §5 (on its own branch) catches up with: the one ingestion context$r_s \ge \pi_v h_v$ ; the $z$ -term of (R), $h_v \ge \mathrm{span}q z{q,v}$; the ingestion exemption from the one-consumer rule; (X) and $a_{q,e}$ ; request contexts per anchor (an $\pi_v$ per output interval.
EVERY_ARRIVING_ROW; the per-source raw-data variable@ end()/@ start()input apart from plain evaluations, rated by the requests of the queries reaching its shift);