Skip to content

feat(executor): PromQL subqueries, @ anchors and time ranges over computed rows - #1250

Closed
zzylol wants to merge 1 commit into
mainfrom
exec/x4-time-operators
Closed

zzylol wants to merge 1 commit into
mainfrom
exec/x4-time-operators

Conversation

@zzylol

@zzylol zzylol commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Why

The #1207 refactor removed the planner's executor-support checks, so the executor must run every shape the planner can choose. Three time-related shapes that the PromQL and SQL frontends emit today could not be planned and run (executor gaps 25, 29 and 30 of the gap inventory). On main, plan_installable fails on each:

max_over_time(rate(c[1m])[10m:1m]): installable plan: raw-data window of node 1 is unsupported: a subquery or a TimeShift above the range
rate(c[5m] @ 1799999760): plan selection: plan selection: the selection MILP's objective 4.81e-4 differs from plan_cost 1.87e-4 (per second, without request fees)
SELECT service, asap_rate(latency, ts, 300000) AS r FROM metrics GROUP BY service: installable plan: raw-data window of node 2 is unsupported: an operator other than TimeShift below the range

Behind those planner errors, the executor had no lowering for any of them, and a counter's window dropped an @ anchor and its offset (#748).

What

These shapes now plan and run exactly, with Prometheus semantics (a range (t - r, t] is left-open; a subquery [R:s] evaluates at the multiples of s in (t - R, t]; @ pins the evaluation time, with start()/end() the query range's bounds):

  • Gap 25, PromQL subqueries: f(m[r])[R:s] and m[R:s], with offset/@ on the subquery or on its selector, under any outer per-series function (max_over_time, rate, …). The inner function can be a per-entity Aggregate or an exact accumulator (FinalizeExactAccumulator(SummaryAgg)) over a range. A subquery without a step uses Prometheus' default evaluation interval, 1 minute.
  • Gap 29, @ anchors: m @ t, m offset 1m @ t, f(m[r] @ t), @ start(), @ end(), both for selector windows and for exact counter windows (closes Counter window ignores the @ anchor (and its offset): rate(m[5m] @ 300) reads the window ending at the evaluation time #748). With no query range given, start() and end() are the evaluation time, the same as a Prometheus instant query.
  • Gap 30, ranges over computed rows: a TimeRange over any time-indexed input (e.g. SQL asap_rate, which is lowered as TimeRange(Project(Scan))), and a per-entity SummaryAgg over such a range or over a subquery.
  • Planner side, kept consistent with the executor:
    • The installable plan's raw-data window check now applies only to windows that a materialized view's refresh reads. At query time the executor reads any shape.
    • The retention handed to the deployment is unbounded for a Scan read under a subquery, because it reads R + r back, not just r.
    • plan_cost now prices a read through an @ anchor as unwindowed, which is how the selection MILP already priced it. This removes the MILP/plan_cost mismatch above.
  • Out of scope: chain TimeRange/TimeShift (Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 PR 6).

How

  • Operator::time_range takes the window's @ anchor. time_range::anchor_time resolves @ <t>, start() and end() from the run's query range, or from the evaluation time when the run has none. series_window and the counter evaluation_range use the same resolution. An anchored window in an ingestion run is an error.
  • physical_planner:
    • Shifts: shifted_end folds the TimeShifts below a range, as Prometheus nests them: the deepest @ wins, and only the offsets from that anchor down count. It replaces node_below_shifts (which gave up on @) and the types time_range_window call.
    • TimeRange input: a TimeRange lowers over any input. It treats the rows as a time series when the Scan it reaches through first inputs is a time series.
    • TimeShift lowering: a TimeShift passes rows through only over a Scan's rows (through Project/Filter). Over evaluated rows it is now refused instead of silently passing them through.
    • Subqueries: a subquery (the PromqlSubquery, or a TimeShift over one) lowers to the existing Operator::series_window with steps, reading the inner selector's Scan directly. The compile walk rewires the node's input to that Scan, so the inner nodes are not compiled. The Scan's read window is widened to the subquery's span: (T - o - off - R - r, T - o - off], or unbounded with an @. A counter evaluation over a subquery takes the subquery's grid as its window.
    • Counter windows: counter_window returns the anchor, and ExactEvaluation carries it.
  • No new abstraction. Subquery is a private compile-time description. series_window already existed but was not wired.

Overlap with X1 (#1241). X1's per-entity Operator::window path (rate-like functions as a per-entity Aggregate) reads time_range_window, so it still refuses an @-anchored range or a subquery input. Whichever PR lands second should use shifted_end / Subquery for that window. Textual overlaps:

Key code interfaces added / modified

crates/executor/src/operators/time_range.rs

impl Operator {
    /// The rows of `input` whose `time` lies in `(e - width_ms, e]`, with `e` the end the
    /// shifts give: `T - end_offset_ms`, or the `@` anchor's time minus `end_offset_ms`.
    /// Now over any time-indexed input, not only a `Scan`.
    pub fn time_range(
        input: SchemaRef,
        time: usize,
        at: Option<AtModifier>, // new
        end_offset_ms: i64,
        width_ms: i64,
        sample_value: Option<usize>,
        latest_per_series: Option<Vec<usize>>,
    ) -> Result<Self, Error>;
}

crates/executor/src/summary_kernels/exact.rs

/// Planned evaluation of an exact summary (the type stays).
pub struct ExactEvaluation {
    /// The exact statistic read.
    pub statistic: ExactKind, // unchanged
    /// The logical PromQL counter window's width.
    pub lookback_ms: Option<i64>, // unchanged
    /// How long before the reference time the counter window ends (PromQL `offset`).
    pub offset_ms: i64, // unchanged
    /// The `@` anchor whose time the window ends `offset_ms` before; `None` = the
    /// evaluation time. Serialized only when set.
    pub at: Option<AtModifier>, // new
}

crates/types/src/ir/materialized_view.rs

impl PhysicalASAPDAG {
    /// Checks the shape of every raw-data window a materialized view's refresh reads
    /// (was: every raw-data window, query-time ones included).
    pub fn check_raw_data_windows(&self) -> Result<(), InstallableDagError>; // semantics changed
}

pub enum InstallableDagError {
    /// Unsatisfiable request: a view's refresh reads raw data through a `TimeRange` whose
    /// window is not a fixed offset from the evaluation time. Now produced only below views.
    UnsupportedRawDataWindow { time_range_node_id: PhysicalASAPNodeId, reason: &'static str }, // narrowed
    // every other variant unchanged
}

raw_data_retention(): a query-time Scan read through a TimeRange that a subquery or shift re-times is kept unbounded (lookback_ms: None).

Usage

From crates/planner/tests/time_operators.rs (subquery_of_a_rate_evaluates_at_steps_and_moves_with_offset), run at this commit:

// The workload: one exact repeating PromQL query every 10 s over `c` (10 series,
// a sample every 10 s, 1,000 rows held).
let workload = workload(r#""promql""#, "max_over_time(rate(c[1m])[10m:1m])", "c");
// Stages 1–3 select Scan -> TimeRange(1m) -> SummaryAgg(Exact Rate) -> Finalize
// -> PromqlSubquery(10m, 1m) -> SummaryAgg(Exact Max) -> Finalize; installation
// compiles it with the executor (it used to refuse the subquery's raw window).
let plan = plan_installable(UserInput::new(&workload, FrontendInput::Promql, PlanningModels::builtin()))
    .await?.0;
// The executor lowers the subquery to one series window with steps over `c`'s
// Scan, which reads (T - 11m, T]; the outer max builds over its step rows.
let compiled = compile(&query_time_reference(&plan)?.dag, &[], &roots)?;
// `c` of job a: +1 per 10 s sample, +6 per sample in (T-300s, T-240s], +100 per sample
// in (T-660s, T-600s]. Run at T.
let answers = run(&compiled, T, None, samples);
// → [("a", 0.6)]: the rate at step T-240s (6 per 10 s); the step T-600s (rate 10/s)
// is not on the left-open grid (T-600s, T]. With `offset 5m` the answer is 10.0.

The SQL query SELECT service, asap_rate(latency, ts, 300000) AS r FROM metrics GROUP BY service returns a 0.2, b 0.3 (sql_asap_rate_reads_its_window_of_projected_rows).

Major algorithm changes

Subquery evaluation (executor lowering)

  • Problem: max_over_time(rate(c[1m])[10m:1m]) had no lowering. PromqlSubquery fell through to the generic node compile and was refused.
  • Before / after:
    • Before: refused.
    • After:
      1. subquery(id) matches the subquery (and the TimeShifts above it) and its inner instant selector or per-series range function.
      2. The compile walk replaces the node's inputs with the selector's Scan.
      3. Subquery::operators builds series_window(function, r, inner offset/@, steps {R, s, outer offset/@}), plus series_without_name for a name-dropping function.
      4. The raw slot window becomes the hull of the Scan's window and the subquery's span.
  • Correctness: series_window's grid is the multiples of s in (e - R, e], as in Prometheus' evalSubquery. Each step reads (t' - r, t'] with t' = (@ or t) - offset. Tests check step alignment with an evaluation time off the grid, the left-open grid start, offset on the subquery, and a 10 s instant lookback against hand-computed Prometheus values.
  • Complexity: O(rows log rows + steps × series × window) per evaluation, which is series_window's existing cost. Steps are capped at 100,000.
  • Assumptions and limits: other inner expressions (cross-series aggregates, binary operations, nested subqueries) are refused (Executor: subqueries over expressions other than a selector or a per-series range function (step-grid evaluator) #1248).

Shift folding and anchored windows

  • Problem: rate(m[5m] @ 300) read the window ending at the evaluation time (Counter window ignores the @ anchor (and its offset): rate(m[5m] @ 300) reads the window ending at the evaluation time #748), and m @ t had no lowering.
  • Before / after:
    • Before: offsets summed, and an @ gave up (time_range_window → Unbounded → no lowering) or was skipped (counter_window).
    • After: shifted_end folds top-down. An anchored shift resets the end to anchor - its offset, and later unanchored shifts add their offsets. The anchor is resolved at run time.
  • Correctness: this matches Prometheus' nesting of @/offset. Tests compare an anchored rate with the unanchored rate at the anchor (with and without offset), and @ start()/@ end() at every step of a query range.
  • Complexity: unchanged.

Before this PR

Subqueries, @ anchors and SQL asap_rate failed in planning: either the installable raw-window check refused them, or Stage 3's @ pricing disagreed between the MILP and plan_cost. Behind that, the executor had no lowering, and exact counters ignored @.

After this PR

The queries above plan, install and return Prometheus' values. Every answer is exact, but some reads are wider than needed:

  • A Scan under a subquery keeps raw data unbounded for the deployment.
  • An @-anchored read is unbounded and needs the source's rows_held to be priced.
  • A range over projected rows (SQL asap_rate) reads its whole table.
  • Stage 3 still prices a subquery's scan at the inner range only.

All four are tracked in #1249.

Verification

  • New e2e tests: crates/planner/tests/time_operators.rs (5 tests: frontend → plan_installable → executor over raw data). On origin/main all 5 fail with the errors in Why; at this commit all pass.
  • tumbling_window_merge::anchored_counter_window_ends_at_the_anchor (Counter window ignores the @ anchor (and its offset): rate(m[5m] @ 300) reads the window ending at the evaluation time #748) is un-ignored and passes.
  • Updated tests:
    • runtime_handoff::raw_data_windows_read_at_query_time_install: a subquery and a negative offset install at query time, with their retention.
    • installable::tests::a_time_shift_above_the_range_is_refused_in_a_view
    • devtools memo_pricing::subquery_choices_install: replaces the old "not installable" subquery example. Stage 2 now prunes every remaining non-executor installation refusal I tried (negative offset or subquery without raw retention), so that path has no PromQL example left.
  • Suites run:
    • cargo test -p asap-executor -p asap-types -p asap-cost: pass
    • cargo test -p asap-plan-selection --lib: 142 pass
    • cargo test -p asap-planner: pass
    • cargo test -p asap-integration-tests: 366 pass, 24 ignored
    • devtools memo_pricing: pass
  • cargo fmt --all; cargo clippy -p asap-executor -p asap-types -p asap-cost -p asap-planner -p asap-integration-tests -p asap-devtools --all-features --all-targets -- -D warnings: clean.
  • Not applicable: screenshots, performance measurements.

Follow-ups: #1248 (step-grid evaluator for general subqueries), #1249 (bounded windows, retention and pricing).

@zzylol

zzylol commented Nov 15, 2026

Copy link
Copy Markdown
Contributor Author

Closing as superseded. Main's @ lowering (Operator::evaluate_at, anchored_context, the time-window module) replaced the code this PR changes, with a different design. This PR also adds executor time walks (shifted_end, source_below, window_hull), and the approved design for #1291/#1292 deletes those walks in favour of the edge read_window. Its planner-side changes (retention, @ pricing) now belong to #1249, #1294 and #1295.

The still-missing pieces are tracked on top of read_window:

The branch exec/x4-time-operators is kept. Its crates/planner/tests/time_operators.rs tests are named in those issues as acceptance tests to port.

@zzylol zzylol closed this Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request promql PromQL front-end lowering (L1→L2) sql SQL front-end lowering (DataFusion → L2) window Window functions, frames, panes, or temporal semantics

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Counter window ignores the @ anchor (and its offset): rate(m[5m] @ 300) reads the window ending at the evaluation time

1 participant