Repository navigation
Conversation
This was referenced Nov 15, 2026
Contributor
Author
|
Closing as superseded. Main's The still-missing pieces are tracked on top of
The branch |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The #1207 refactor removed the planner's executor-support checks, so the executor must run every shape the planner can choose. Three time-related shapes that the PromQL and SQL frontends emit today could not be planned and run (executor gaps 25, 29 and 30 of the gap inventory). On main,
plan_installablefails on each:Behind those planner errors, the executor had no lowering for any of them, and a counter's window dropped an
@anchor and its offset (#748).What
These shapes now plan and run exactly, with Prometheus semantics (a range
(t - r, t]is left-open; a subquery[R:s]evaluates at the multiples ofsin(t - R, t];@pins the evaluation time, withstart()/end()the query range's bounds):f(m[r])[R:s]andm[R:s], withoffset/@on the subquery or on its selector, under any outer per-series function (max_over_time,rate, …). The inner function can be a per-entityAggregateor an exact accumulator (FinalizeExactAccumulator(SummaryAgg)) over a range. A subquery without a step uses Prometheus' default evaluation interval, 1 minute.@anchors:m @ t,m offset 1m @ t,f(m[r] @ t),@ start(),@ end(), both for selector windows and for exact counter windows (closes Counter window ignores the @ anchor (and its offset): rate(m[5m] @ 300) reads the window ending at the evaluation time #748). With no query range given,start()andend()are the evaluation time, the same as a Prometheus instant query.TimeRangeover any time-indexed input (e.g. SQLasap_rate, which is lowered asTimeRange(Project(Scan))), and a per-entitySummaryAggover such a range or over a subquery.Scanread under a subquery, because it readsR + rback, not justr.plan_costnow prices a read through an@anchor as unwindowed, which is how the selection MILP already priced it. This removes the MILP/plan_costmismatch above.TimeRange/TimeShift(Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 PR 6).How
Operator::time_rangetakes the window's@anchor.time_range::anchor_timeresolves@ <t>,start()andend()from the run's query range, or from the evaluation time when the run has none.series_windowand the counterevaluation_rangeuse the same resolution. An anchored window in an ingestion run is an error.physical_planner:shifted_endfolds theTimeShifts below a range, as Prometheus nests them: the deepest@wins, and only the offsets from that anchor down count. It replacesnode_below_shifts(which gave up on@) and the typestime_range_windowcall.TimeRangeinput: aTimeRangelowers over any input. It treats the rows as a time series when theScanit reaches through first inputs is a time series.TimeShiftlowering: aTimeShiftpasses rows through only over aScan's rows (throughProject/Filter). Over evaluated rows it is now refused instead of silently passing them through.PromqlSubquery, or aTimeShiftover one) lowers to the existingOperator::series_windowwithsteps, reading the inner selector'sScandirectly. The compile walk rewires the node's input to thatScan, so the inner nodes are not compiled. TheScan's read window is widened to the subquery's span:(T - o - off - R - r, T - o - off], or unbounded with an@. A counter evaluation over a subquery takes the subquery's grid as its window.counter_windowreturns the anchor, andExactEvaluationcarries it.Subqueryis a private compile-time description.series_windowalready existed but was not wired.Overlap with X1 (#1241). X1's per-entity
Operator::windowpath (rate-like functions as a per-entityAggregate) readstime_range_window, so it still refuses an@-anchored range or a subquery input. Whichever PR lands second should useshifted_end/Subqueryfor that window. Textual overlaps:operators/mod.rsnearis_counter_evaluation/with_counter_windowsummary_kernels/exact.rsnearExactEvaluationoperators/series_window.rs: different hunksplanner/tests/main.rs: one line each, also with X2 (feat(executor): grouped stddev/variance, corr, group, count_values, exact topk and without grouping #1242)Key code interfaces added / modified
crates/executor/src/operators/time_range.rs
crates/executor/src/summary_kernels/exact.rs
crates/types/src/ir/materialized_view.rs
raw_data_retention(): a query-timeScanread through aTimeRangethat a subquery or shift re-times is kept unbounded (lookback_ms: None).Usage
From
crates/planner/tests/time_operators.rs(subquery_of_a_rate_evaluates_at_steps_and_moves_with_offset), run at this commit:The SQL query
SELECT service, asap_rate(latency, ts, 300000) AS r FROM metrics GROUP BY servicereturnsa 0.2,b 0.3(sql_asap_rate_reads_its_window_of_projected_rows).Major algorithm changes
Subquery evaluation (executor lowering)
max_over_time(rate(c[1m])[10m:1m])had no lowering.PromqlSubqueryfell through to the generic node compile and was refused.subquery(id)matches the subquery (and theTimeShifts above it) and its inner instant selector or per-series range function.Scan.Subquery::operatorsbuildsseries_window(function, r, inner offset/@, steps {R, s, outer offset/@}), plusseries_without_namefor a name-dropping function.Scan's window and the subquery's span.series_window's grid is the multiples ofsin(e - R, e], as in Prometheus'evalSubquery. Each step reads(t' - r, t']witht' = (@ or t) - offset. Tests check step alignment with an evaluation time off the grid, the left-open grid start,offseton the subquery, and a 10 s instant lookback against hand-computed Prometheus values.series_window's existing cost. Steps are capped at 100,000.Shift folding and anchored windows
rate(m[5m] @ 300)read the window ending at the evaluation time (Counter window ignores the @ anchor (and its offset): rate(m[5m] @ 300) reads the window ending at the evaluation time #748), andm @ thad no lowering.@gave up (time_range_window→Unbounded→ no lowering) or was skipped (counter_window).shifted_endfolds top-down. An anchored shift resets the end toanchor - its offset, and later unanchored shifts add their offsets. The anchor is resolved at run time.@/offset. Tests compare an anchored rate with the unanchored rate at the anchor (with and without offset), and@ start()/@ end()at every step of a query range.Before this PR
Subqueries,
@anchors and SQLasap_ratefailed in planning: either the installable raw-window check refused them, or Stage 3's@pricing disagreed between the MILP andplan_cost. Behind that, the executor had no lowering, and exact counters ignored@.After this PR
The queries above plan, install and return Prometheus' values. Every answer is exact, but some reads are wider than needed:
Scanunder a subquery keeps raw data unbounded for the deployment.@-anchored read is unbounded and needs the source'srows_heldto be priced.asap_rate) reads its whole table.All four are tracked in #1249.
Verification
crates/planner/tests/time_operators.rs(5 tests: frontend →plan_installable→ executor over raw data). On origin/main all 5 fail with the errors in Why; at this commit all pass.tumbling_window_merge::anchored_counter_window_ends_at_the_anchor(Counter window ignores the @ anchor (and its offset): rate(m[5m] @ 300) reads the window ending at the evaluation time #748) is un-ignored and passes.runtime_handoff::raw_data_windows_read_at_query_time_install: a subquery and a negative offset install at query time, with their retention.installable::tests::a_time_shift_above_the_range_is_refused_in_a_viewdevtools memo_pricing::subquery_choices_install: replaces the old "not installable" subquery example. Stage 2 now prunes every remaining non-executor installation refusal I tried (negative offset or subquery without raw retention), so that path has no PromQL example left.cargo test -p asap-executor -p asap-types -p asap-cost: passcargo test -p asap-plan-selection --lib: 142 passcargo test -p asap-planner: passcargo test -p asap-integration-tests: 366 pass, 24 ignoredmemo_pricing: passcargo fmt --all;cargo clippy -p asap-executor -p asap-types -p asap-cost -p asap-planner -p asap-integration-tests -p asap-devtools --all-features --all-targets -- -D warnings: clean.Follow-ups: #1248 (step-grid evaluator for general subqueries), #1249 (bounded windows, retention and pricing).