Skip to content

fix(executor): read an unbound ingestion-time node as at query time, keeping sample times - #1344

Merged
zzylol merged 4 commits into
mainfrom
fix/1338-chain-in-query-run
Nov 16, 2026
Merged

zzylol merged 4 commits into
mainfrom
fix/1338-chain-in-query-run

Conversation

@zzylol

@zzylol zzylol commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: sum by (job)(rate(requests_total[5m])) and sum by (job)(increase(requests_total[5m])) answer no rows when their plan runs in a query run with no materialized view bound. The plan builds a 10 s tumbling-window chain at ingestion time.
  • Change: in the executor, an unbound view (a non-root AcrossEvaluations ingestion-time node) is now read as if placed at query time, because the compile computes it for its readers. Such a chain is built for its readers: every window they read, with each sample's own time.
  • Effect: on the 4-series, 15 s data of sum by (job)(rate|increase(counter[5m])) plans and installs but answers no rows #1338, increase answers [] → j1: 60, j2: 140 and rate answers [] → j1: 0.2, j2: 0.4667. Plans run with their views bound (as installed) give the same answers as before. Installed query-time chains over a counter, which used to fail, now answer exactly.

Closes #1338.

Review guide

Rebased onto #1343 697178e05 (main 863153e3c)

Head f3d63d072 (was 5bf4b132e) is rebased onto #1343's new head 697178e05, which is on main 863153e3c (after #1237, #1242, #1309, #1339, #1330, #1324 and #1300). The base is still test/1338-counter-rate-answers. Conflicts and resolutions:

Behavior: before → after

The workload: sum by (job)(increase(requests_total[5m])) every 10 s, relative error 0.01, over 1,000 counter series sampled every second. Built-in models; raw data kept or not.

The selected plan: Scan → SummaryTumblingWindowChain(Increase, PerEntity, 10 s, ingestion time) → TimeRange 300s → SummaryMerge → Finalize → SummaryAgg Sum by job → Finalize.

Data: series a..d, jobs j1 = {a, b} and j2 = {c, d}, one sample every 15 s over 8 minutes. Series i counts (i + 1) * step.

Run before after
compile(&plan.dag, &[], roots), query run at T [] j1: 60, j2: 140
same, rate [] j1: 0.2, j2: 0.4667
same, increase(requests_total[5m]) on the plan without raw data [] a: 20, b: 40, c: 60, d: 80
installed: chain views from 30 ingestion runs, then the query run j1: 60, j2: 140 unchanged
installed query-time counter chain: rate(requests_total[5m] offset 1m) and rate(requests_total[5m]), each every minute, raw data kept (a shared 1-minute chain at query time) error: counter samples must have strictly increasing timestamps within each group a: 0.0667, b: 0.1333, c: 0.2, d: 0.2667 for both queries
Scan → chain(60s, view) → TimeShift(1h, view) → TimeRange(5m), one query run, no view bound, at T = 4h error: built for its readers is read outside the time ranges above it (on the first round of this PR) a: 890, b: 1780
same plan, chain view bound (windows from ingestion runs), TimeShift view unbound, offsets 1h / 5m / 2h error: materialized view 1 is read unbounded compiles and answers exactly

The first three rows are the run that tests, reference runs and the MetricsQL workload test use: a plan run from raw samples, with nothing installed first. No plan the planner produces today has a stored TimeShift (all 20 TimeShifts in the 109 installable corpus plans run at query time), so the executor tests are what exercise that path. Plans also choose query-time chains, for example a chain shared by rate(c[5m] offset 1m) and rate(c[5m]). Over a counter with more than one sample per window, those chains failed because of the second bug below.

Cause

Two executor bugs on the same path:

  1. Only the newest window was built. In physical_planner/mod.rs, the chain lowering picked its form from the node's planned timing alone. A chain placed at ingestion time was always lowered the way an ingestion run builds it: TimeRange(0, 10 s), which keeps only (T − 10 s, T]. That held even when it was compiled for readers in a query run with its view not bound. The window walk (CompileGraph::input_time_access) likewise read its input as LastWindow(10 s). With samples every 15 s, each series has at most one sample in that window. The exact finalize then drops every counter row (fewer than two samples), and sum by (job) gets an empty input. With 1 s samples, the answer was not empty but wrong: one 10 s window, 28.5 instead of 900. A stored TimeShift at ingestion time had the same problem: it read CurrentTimeOnly, as its refresh does.
  2. Sample times were lost. A chain built for its readers groups rows by window end. Operator::with_tumbling_window_ends did this by replacing each row's time with its window's end. The build then updated counter states at window-end times. With one sample per window, a counter's rate/increase extrapolated from the wrong first and last sample times: 19.5 instead of 20 per series (half-interval extrapolation, because the first "sample" sat 20 s after the range start). With several samples per window, they shared one time, and the counter build failed with counter samples must have strictly increasing timestamps within each group. This hit every chain the planner places at query time over a counter, installed or not.

Fix

  1. CompileGraph::input_time_access works out the timing each node runs at in this compile: an unbound view (a non-root AcrossEvaluations ingestion-time node) is treated as a query-time node, because the compile computes it for its readers. It passes that timing to InputTimeAccess::of, so the plan layer's mapping from timing to read applies unchanged.

    • An unbound chain reads WhatItsReadersRead and builds every tumbling window its readers read.
    • A stored TimeShift reads its readers' windows, shifted back by its offset.

    The chain lowering matches on input_time_access(id).reads: LastWindow { length_ms } gives the refresh form, the window (T − length_ms, T]; any other read builds for its readers. A chain that is a compile root (an ingestion run refreshing its view) is unchanged: it builds the window ending at the run's end.

  2. with_tumbling_window_ends keeps each row's time and appends the window end as a trailing $tumbling_window_end column. The chain build finds that column by name, groups by it, and updates states with the sample time. The code that replaced the time is deleted.

The override in input_time_access is temporary. The planned per-edge read_window (#1291, #1292) replaces it: an unbound node then reads the union of its outgoing edges' read_window, with no executor-side timing rule. The code comment says so.

Interfaces

crates/executor/src/operators/time_range.rs

 impl Operator {
-    /// This range window, each kept row's time replaced by the end of the
-    /// tumbling window of `width_ms` it lies in.
+    /// This range window, each kept row followed by the end of the tumbling
+    /// window of `width_ms` its time lies in, in a trailing
+    /// `$tumbling_window_end` column; the row's own time is kept. The
+    /// output schema gains that one field; an input that already has it
+    /// is an error.
     pub fn with_tumbling_window_ends(mut self, width_ms: i64) -> Result<Self, Error>;
 }

crates/executor/src/physical_planner/time_window.rs (crate-private; no signature change)

 impl TimeIndexedPlan for CompileGraph<'_> {
+    /// An unbound view (a non-root `AcrossEvaluations` ingestion-time node)
+    /// reads as if placed at query time: this compile computes it for the
+    /// compiled nodes reading it.
     fn input_time_access(&self, node: NodeId) -> InputTimeAccess;
 }

Usage (the new test, end to end):

// The plan of the grouped counter query on a deployment that keeps raw data:
// a 10 s Increase chain at ingestion time under TimeRange(5m).
let plan =
    plan_counter_queries(&["sum by (job)(increase(requests_total[5m]))"], 10_000, true).await;
// The executor compiles the root with no view bound: the chain is an
// unbound view, so input_time_access reads it as at query time, WhatItsReadersRead.
// Its Scan is read through (T - 5m, T] and the chain builds 30 windows,
// each row followed by its $tumbling_window_end and updated at its sample
// time.
let answered = query_answers(&plan, &counter_samples());
// [({"job": "j1"}, 60.0), ({"job": "j2"}, 140.0)]
assert_eq!(answered[0], vec![(job("j1"), 60.0), (job("j2"), 140.0)]);

Algorithm changes

Chain lowering and window walk (executor).

  • Problem: see Cause. The run had an unbound ingestion-time chain under TimeRange(5m), and the answer was [] instead of j1: 60, j2: 140.
  • Before:
    1. Each node's read came from its planned timing.
    2. An ingestion-time chain → TimeRange(0, width), one window.
    3. A query-time chain → TimeRange(read) plus window ends that replace the time column, grouped by that column and updated at it.
  • After:
    1. Each node's read comes from the timing it runs at in this compile: an unbound view (ingestion time, AcrossEvaluations, not a root) → query time.
    2. A chain whose read is not LastWindow is built for its readers → TimeRange(read) plus an appended window-end column, grouped by it and updated at the sample time.
    3. A chain read as LastWindow { length_ms } (a compile root at ingestion time) builds the one window (T − length_ms, T], as before.
  • Correctness: a chain built for its readers yields the same per-window states as the stored windows of its view: same rows per window, same sample times. So reading it equals reading the installed view, except over a TimeShift below the chain (see Limits). The tests check this on 12 counter cases and the stored-shift case against hand-worked answers, and check that the installed path is unchanged.
  • Complexity: unchanged order. An unbound chain read through (T − 5m, T] now reads 5 m of input instead of 10 s; the old behavior was simply wrong. Compiling with no view bound costs the same as a query-time chain over the same rows: measured 333 ms vs 322 ms at 10^6 rows and 10^4 series, about 0.33 µs per row read. Installed plans, with views bound, are unchanged. Each row of a chain built for readers carries one extra timestamp column. That column costs 1–3% in time, within noise, and no measurable memory.
  • Limits:

Design scope

  • No new type, method or file. The rule lives in CompileGraph::input_time_access (executor scope, physical-plan lowering, the window walk's view of a compile), the one place that knows this compile's roots. It feeds the plan layer's InputTimeAccess::of, so the mapping from timing to read stays in one place, asap_types::plan::time_window. The per-edge read_window replaces it later.
  • TUMBLING_WINDOW_END_COLUMN (pub(crate)): executor scope, operator kernel (TimeRange). It names the column with_tumbling_window_ends appends, which every chain built for its readers groups by, whatever its summary family.
  • No planner, IR or plan-format change. The plan in sum by (job)(rate|increase(counter[5m])) plans and installs but answers no rows #1338 was right.

Decisions and open questions

Review status

Round 2 (abstraction and layering; functional and efficiency): both approved. This head has the final fixes from round 2 and is ready to merge:

  • LY1: the override is narrowed to an unbound view, (IngestionTime, AcrossEvaluations(_)) if !root, and the doc says exactly that. No behavior changes: an ingestion-time chain that is not AcrossEvaluations is rejected by PhysicalASAPDAG::validate before any time walk (OneEvaluationAtIngestion, or a summary build not materialized at ingestion time), and InputTimeAccess::of reads the other operators the same at either timing.
  • AB1: the chain lowering matches on input_time_access(id).reads; LastWindow { length_ms } gives the refresh form with that length_ms.
  • AB2: the missing window-end error is neutral ("a chain built for its readers has no window-end column").
  • AB3: the one-line plan_counter_query wrapper is inlined into its caller.
  • LY2: ExecutionTiming is imported at the top of time_window.rs; the long doc line is re-wrapped.
  • N1: Behavior row added (bound chain view, unbound stored TimeShift). N2: noted under Behavior. N3: needs no change.

Round 1 (abstraction and layering, functional, efficiency), all addressed on this head:

Verification

On f3d63d072, compared by test name with main 863153e3c and with #1343 (697178e05), each built the same way:

Scope This PR #1343 main Newly failing
cargo check --workspace --all-targets clean clean clean
cargo fmt --all -- --check clean clean clean
cargo clippy -p asap-executor -p asap-planner --all-targets -- -D warnings clean clean clean
cargo nextest run -p asap-executor 312 passed (the new stored-shift test and the moved unit test included) 311 passed 311 passed none
cargo nextest run -p asap-planner --test planner (runtime_install, metricsql_workloads, reduce_aggregates over time_series) 53 passed, 1 failed 51 passed, 2 failed 50 passed, 1 failed none. Fixed: counter_rates_and_increases_answer_at_query_time_from_raw_samples. Fails on all three: summary_sharing::identical_ungrouped_queries_share_their_producers

The integration-test and corpus rows of the earlier round were measured on 5bf4b132e over main 3e434c760 and were not rerun after this rebase.

Tests:

  • runtime_install::counter_rates_and_increases_answer_at_query_time_from_raw_samples (from test(planner): counter rate and increase answers, installed and at query time #1343): rate/increase per series, per job and in total, raw data kept or not, with the whole plan in one query run. 8 of 12 cases failed before; all pass now.
  • runtime_install::counter_rates_and_increases_answer_from_views_built_at_ingestion_time (from test(planner): counter rate and increase answers, installed and at query time #1343): the same 12 cases as installed. Passes before and after.
  • runtime_install::counter_rates_over_a_shared_query_time_chain_answer_as_installed (new): the F2 row. Fails on the base with counter samples must have strictly increasing timestamps within each group.
  • tumbling_window_chain::an_unbound_stored_time_shift_over_an_unbound_chain_reads_its_readers_windows (new): the stored-shift row, against the hand-worked a: 890, b: 1780. Failed on the previous head.
  • operators::time_range::tests::a_range_window_stamps_each_row_with_its_tumbling_windows_end (moved): each kept row keeps its time and gets its window end appended.

Follow-ups

@zzylol zzylol added bug Something isn't working promql PromQL front-end lowering (L1→L2) window Window functions, frames, panes, or temporal semantics labels Nov 15, 2026
@zzylol zzylol changed the title fix(executor): build an unbound chain's windows for its readers, with sample times fix(executor): read an unbound ingestion-time node as at query time, keeping sample times Nov 15, 2026
@zzylol
zzylol force-pushed the test/1338-counter-rate-answers branch from 4970361 to 697178e Compare November 16, 2026 04:42
@zzylol
zzylol force-pushed the fix/1338-chain-in-query-run branch from 5bf4b13 to f3d63d0 Compare November 16, 2026 04:42
… sample times

A plan run without its materialized views bound lost the rows of
sum by (job)(rate(c[5m])) and sum by (job)(increase(c[5m])):

- A tumbling-window chain placed at ingestion time and compiled for its
  readers (not a compile root, its view not bound) was built as in an
  ingestion run: only its newest window (T - 10s, T]. A counter has at
  most one sample there, so the finalize dropped every series. Such a
  chain now builds every window its readers read, as a query-time chain
  does, and its input is read through those windows.
- A chain built for its readers replaced each sample's time with its
  window's end before the build, so a counter's rate and increase
  extrapolated from window ends instead of sample times (19.5 for 20).
  The range window now appends the window end as its own column and
  keeps the sample time.
A node placed at ingestion time that is not a compile root is a view the
compile does not bind, so it is computed for its readers. The window walk
(CompileGraph::input_time_access) now reads it with query-time timing
through InputTimeAccess::of, which replaces the chain-only
chain_builds_windows_readers_read and also covers a stored TimeShift.
The chain lowering is built for its readers when its read is not
LastWindow.

The window-end column is found by TUMBLING_WINDOW_END_COLUMN (now
pub(crate)) in both places; the range-window test moves to a unit test
that uses it.

Tests: an unbound stored TimeShift(1h) over an unbound 1-minute chain
answers the hand-worked a: 890, b: 1780; two counter rates sharing an
installed query-time chain answer exactly (the base fails with "counter
samples must have strictly increasing timestamps within each group").
Narrow the timing override in CompileGraph::input_time_access to an
unbound view (a non-root AcrossEvaluations ingestion-time node). The
chain lowering matches on its input time access: LastWindow gives the
refresh form over that window, any other read builds for its readers.
Neutral missing-window-end error; inline the one-line plan_counter_query
wrapper; import ExecutionTiming at the top level.
…computed for its readers

A compile with a view slot not bound computes that view for its readers
in the query run, so a computed AcrossEvaluations node with several
consumers is no longer only a hand-built DAG's.
@zzylol
zzylol changed the base branch from test/1338-counter-rate-answers to main November 16, 2026 13:42
@zzylol
zzylol force-pushed the fix/1338-chain-in-query-run branch from f3d63d0 to 3ae606c Compare November 16, 2026 13:42
@zzylol
zzylol merged commit d0f7db1 into main Nov 16, 2026
2 of 3 checks passed
@zzylol

zzylol commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main 52f9d9ee8 (after #1343 squash-merged) with --onto, dropping #1343's pre-squash commit; no conflicts, tree unchanged. Head 3ae606ccd (was f3d63d072); backup branch backup/fix/1338-chain-in-query-run-f3d63d072. Retargeted to main.

Validation on the merged tree: cargo fmt --all -- --check, cargo check --workspace --all-targets and cargo clippy --workspace --all-targets -- -D warnings pass. cargo test -p asap-executor and cargo test -p asap-planner --test planner compared by name with main: no new failures; runtime_install::counter_rates_and_increases_answer_at_query_time_from_raw_samples now passes. Only pre-existing failure: summary_sharing::identical_ungrouped_queries_share_their_producers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working promql PromQL front-end lowering (L1→L2) window Window functions, frames, panes, or temporal semantics

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sum by (job)(rate|increase(counter[5m])) plans and installs but answers no rows

1 participant