Repository navigation
fix(executor): read an unbound ingestion-time node as at query time, keeping sample times - #1344
Merged
Merged
Conversation
zzylol
force-pushed
the
test/1338-counter-rate-answers
branch
from
November 16, 2026 04:42
4970361 to
697178e
Compare
zzylol
force-pushed
the
fix/1338-chain-in-query-run
branch
from
November 16, 2026 04:42
5bf4b13 to
f3d63d0
Compare
… sample times A plan run without its materialized views bound lost the rows of sum by (job)(rate(c[5m])) and sum by (job)(increase(c[5m])): - A tumbling-window chain placed at ingestion time and compiled for its readers (not a compile root, its view not bound) was built as in an ingestion run: only its newest window (T - 10s, T]. A counter has at most one sample there, so the finalize dropped every series. Such a chain now builds every window its readers read, as a query-time chain does, and its input is read through those windows. - A chain built for its readers replaced each sample's time with its window's end before the build, so a counter's rate and increase extrapolated from window ends instead of sample times (19.5 for 20). The range window now appends the window end as its own column and keeps the sample time.
A node placed at ingestion time that is not a compile root is a view the compile does not bind, so it is computed for its readers. The window walk (CompileGraph::input_time_access) now reads it with query-time timing through InputTimeAccess::of, which replaces the chain-only chain_builds_windows_readers_read and also covers a stored TimeShift. The chain lowering is built for its readers when its read is not LastWindow. The window-end column is found by TUMBLING_WINDOW_END_COLUMN (now pub(crate)) in both places; the range-window test moves to a unit test that uses it. Tests: an unbound stored TimeShift(1h) over an unbound 1-minute chain answers the hand-worked a: 890, b: 1780; two counter rates sharing an installed query-time chain answer exactly (the base fails with "counter samples must have strictly increasing timestamps within each group").
Narrow the timing override in CompileGraph::input_time_access to an unbound view (a non-root AcrossEvaluations ingestion-time node). The chain lowering matches on its input time access: LastWindow gives the refresh form over that window, any other read builds for its readers. Neutral missing-window-end error; inline the one-line plan_counter_query wrapper; import ExecutionTiming at the top level.
…computed for its readers A compile with a view slot not bound computes that view for its readers in the query run, so a computed AcrossEvaluations node with several consumers is no longer only a hand-built DAG's.
zzylol
force-pushed
the
fix/1338-chain-in-query-run
branch
from
November 16, 2026 13:42
f3d63d0 to
3ae606c
Compare
Contributor
Author
|
Rebased onto Validation on the merged tree: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
sum by (job)(rate(requests_total[5m]))andsum by (job)(increase(requests_total[5m]))answer no rows when their plan runs in a query run with no materialized view bound. The plan builds a 10 s tumbling-window chain at ingestion time.AcrossEvaluationsingestion-time node) is now read as if placed at query time, because the compile computes it for its readers. Such a chain is built for its readers: every window they read, with each sample's own time.increaseanswers[]→j1: 60, j2: 140andrateanswers[]→j1: 0.2, j2: 0.4667. Plans run with their views bound (as installed) give the same answers as before. Installed query-time chains over a counter, which used to fail, now answer exactly.Closes #1338.
Review guide
test/1338-counter-rate-answers(test(planner): counter rate and increase answers, installed and at query time #1343, the failing tests). This PR is the fix on top.CompileGraph::input_time_accessinphysical_planner/time_window.rs. Then read the chain lowering inphysical_planner/mod.rs(searchbuilt_for_readers), and thenOperator::with_tumbling_window_ends.a_range_window_stamps_each_row_with_its_tumbling_windows_endmoved fromcrates/executor/tests/tumbling_window_chain.rsinto a unit test inoperators/time_range.rs. It now finds the window end byTUMBLING_WINDOW_END_COLUMN, which is crate-private.Rebased onto #1343
697178e05(main863153e3c)Head
f3d63d072(was5bf4b132e) is rebased onto #1343's new head697178e05, which is onmain863153e3c(after #1237, #1242, #1309, #1339, #1330, #1324 and #1300). The base is stilltest/1338-counter-rate-answers. Conflicts and resolutions:operators/mod.rs,operators/time_range.rs,physical_planner/mod.rs: the context changed (fix(accuracy): relative error through negation, abs, set operators and relabels; readouts compose only through error-carrying inputs; exact accumulators take non-finite values and reject stale markers; finalize sign of counters #1309 changedSTALE_MARKERhandling and theTimeRangestale-marker comment; refactor: no silent defaults for kernel memory, operator rules and aggregate names #1330 added the no-wildcard-arm lint). The PR's hunks applied unchanged; clippy-D warningsis clean.crates/executor/tests/tumbling_window_chain.rs: fix(executor): run each node by its plan Lifetime, so roots drain in any order #1300 gavePhysicalDAG::addaLifetimeargument. The range-window test this PR moves out of the file was updated there by fix(executor): run each node by its plan Lifetime, so roots drain in any order #1300; the move keeps this PR's version, and the unit test inoperators/time_range.rsnow passesLifetime::None. The file's now-unusedLifetimeimport is removed.crates/planner/tests/runtime_install.rs: follows test(planner): counter rate and increase answers, installed and at query time #1343's resolution (sharedtime_serieshelpers,run(dag, store, views, ids, scope)).NodeRun::Helddoc (and the executor README line): a computedAcrossEvaluationsnode with several consumers is now also an unbound view that a compile computes for its readers, not only a hand-built DAG's node. New commitf3d63d072.Behavior: before → after
The workload:
sum by (job)(increase(requests_total[5m]))every 10 s, relative error 0.01, over 1,000 counter series sampled every second. Built-in models; raw data kept or not.The selected plan:
Scan → SummaryTumblingWindowChain(Increase, PerEntity, 10 s, ingestion time) → TimeRange 300s → SummaryMerge → Finalize → SummaryAgg Sum by job → Finalize.Data: series
a..d, jobsj1= {a, b} andj2= {c, d}, one sample every 15 s over 8 minutes. Seriesicounts(i + 1) * step.compile(&plan.dag, &[], roots), query run at T[]j1: 60, j2: 140rate[]j1: 0.2, j2: 0.4667increase(requests_total[5m])on the plan without raw data[]a: 20, b: 40, c: 60, d: 80j1: 60, j2: 140rate(requests_total[5m] offset 1m)andrate(requests_total[5m]), each every minute, raw data kept (a shared 1-minute chain at query time)counter samples must have strictly increasing timestamps within each groupa: 0.0667, b: 0.1333, c: 0.2, d: 0.2667for both queriesScan → chain(60s, view) → TimeShift(1h, view) → TimeRange(5m), one query run, no view bound, at T = 4hbuilt for its readers is read outside the time ranges above it(on the first round of this PR)a: 890, b: 1780TimeShiftview unbound, offsets 1h / 5m / 2hmaterialized view 1 is read unboundedThe first three rows are the run that tests, reference runs and the MetricsQL workload test use: a plan run from raw samples, with nothing installed first. No plan the planner produces today has a stored
TimeShift(all 20TimeShifts in the 109 installable corpus plans run at query time), so the executor tests are what exercise that path. Plans also choose query-time chains, for example a chain shared byrate(c[5m] offset 1m)andrate(c[5m]). Over a counter with more than one sample per window, those chains failed because of the second bug below.Cause
Two executor bugs on the same path:
physical_planner/mod.rs, the chain lowering picked its form from the node's planned timing alone. A chain placed at ingestion time was always lowered the way an ingestion run builds it:TimeRange(0, 10 s), which keeps only(T − 10 s, T]. That held even when it was compiled for readers in a query run with its view not bound. The window walk (CompileGraph::input_time_access) likewise read its input asLastWindow(10 s). With samples every 15 s, each series has at most one sample in that window. The exact finalize then drops every counter row (fewer than two samples), andsum by (job)gets an empty input. With 1 s samples, the answer was not empty but wrong: one 10 s window, 28.5 instead of 900. A storedTimeShiftat ingestion time had the same problem: it readCurrentTimeOnly, as its refresh does.Operator::with_tumbling_window_endsdid this by replacing each row's time with its window's end. The build then updated counter states at window-end times. With one sample per window, a counter'srate/increaseextrapolated from the wrong first and last sample times: 19.5 instead of 20 per series (half-interval extrapolation, because the first "sample" sat 20 s after the range start). With several samples per window, they shared one time, and the counter build failed withcounter samples must have strictly increasing timestamps within each group. This hit every chain the planner places at query time over a counter, installed or not.Fix
CompileGraph::input_time_accessworks out the timing each node runs at in this compile: an unbound view (a non-rootAcrossEvaluationsingestion-time node) is treated as a query-time node, because the compile computes it for its readers. It passes that timing toInputTimeAccess::of, so the plan layer's mapping from timing to read applies unchanged.WhatItsReadersReadand builds every tumbling window its readers read.TimeShiftreads its readers' windows, shifted back by its offset.The chain lowering matches on
input_time_access(id).reads:LastWindow { length_ms }gives the refresh form, the window(T − length_ms, T]; any other read builds for its readers. A chain that is a compile root (an ingestion run refreshing its view) is unchanged: it builds the window ending at the run's end.with_tumbling_window_endskeeps each row's time and appends the window end as a trailing$tumbling_window_endcolumn. The chain build finds that column by name, groups by it, and updates states with the sample time. The code that replaced the time is deleted.The override in
input_time_accessis temporary. The planned per-edgeread_window(#1291, #1292) replaces it: an unbound node then reads the union of its outgoing edges'read_window, with no executor-side timing rule. The code comment says so.Interfaces
crates/executor/src/operators/time_range.rsimpl Operator { - /// This range window, each kept row's time replaced by the end of the - /// tumbling window of `width_ms` it lies in. + /// This range window, each kept row followed by the end of the tumbling + /// window of `width_ms` its time lies in, in a trailing + /// `$tumbling_window_end` column; the row's own time is kept. The + /// output schema gains that one field; an input that already has it + /// is an error. pub fn with_tumbling_window_ends(mut self, width_ms: i64) -> Result<Self, Error>; }crates/executor/src/physical_planner/time_window.rs(crate-private; no signature change)impl TimeIndexedPlan for CompileGraph<'_> { + /// An unbound view (a non-root `AcrossEvaluations` ingestion-time node) + /// reads as if placed at query time: this compile computes it for the + /// compiled nodes reading it. fn input_time_access(&self, node: NodeId) -> InputTimeAccess; }Usage (the new test, end to end):
Algorithm changes
Chain lowering and window walk (executor).
TimeRange(5m), and the answer was[]instead ofj1: 60, j2: 140.TimeRange(0, width), one window.TimeRange(read)plus window ends that replace the time column, grouped by that column and updated at it.AcrossEvaluations, not a root) → query time.LastWindowis built for its readers →TimeRange(read)plus an appended window-end column, grouped by it and updated at the sample time.LastWindow { length_ms }(a compile root at ingestion time) builds the one window(T − length_ms, T], as before.TimeShiftbelow the chain (see Limits). The tests check this on 12 counter cases and the stored-shift case against hand-worked answers, and check that the installed path is unchanged.(T − 5m, T]now reads 5 m of input instead of 10 s; the old behavior was simply wrong. Compiling with no view bound costs the same as a query-time chain over the same rows: measured 333 ms vs 322 ms at 10^6 rows and 10^4 series, about 0.33 µs per row read. Installed plans, with views bound, are unchanged. Each row of a chain built for readers carries one extra timestamp column. That column costs 1–3% in time, within noise, and no measurable memory.TimeShiftwhose offset is not a multiple of the window width ignores the shift in its own window) now also affects an unbound ingestion-time chain, because it goes down the same path. Example:Finalize(Merge(TimeRange(5m, Chain(1m, TimeShift(30s, Scan)))))at T = 4h givesa = 950, where 1185 is right. The base gave{}. Not fixed here; the repro is on Executor: a query-time chain over a TimeShift ignores the shift in its own window #1336.Design scope
CompileGraph::input_time_access(executor scope, physical-plan lowering, the window walk's view of a compile), the one place that knows this compile's roots. It feeds the plan layer'sInputTimeAccess::of, so the mapping from timing to read stays in one place,asap_types::plan::time_window. The per-edgeread_windowreplaces it later.TUMBLING_WINDOW_END_COLUMN(pub(crate)): executor scope, operator kernel (TimeRange). It names the columnwith_tumbling_window_endsappends, which every chain built for its readers groups by, whatever its summary family.Decisions and open questions
compilerefuse a query root that reaches an unbound ingestion-time view. That would make every plan-from-raw-samples run (tests, reference runs, the MetricsQL workload test) bind views first. Building the windows gives the same answer as the installed view.NodeRun::Helddoc: fix(executor): run each node by its plan Lifetime, so roots drain in any order #1300 merged first, so this PR updates it (and the executor README): a computedAcrossEvaluationsnode with several consumers is also an unbound view computed for its readers.Review status
Round 2 (abstraction and layering; functional and efficiency): both approved. This head has the final fixes from round 2 and is ready to merge:
(IngestionTime, AcrossEvaluations(_)) if !root, and the doc says exactly that. No behavior changes: an ingestion-time chain that is notAcrossEvaluationsis rejected byPhysicalASAPDAG::validatebefore any time walk (OneEvaluationAtIngestion, or a summary build not materialized at ingestion time), andInputTimeAccess::ofreads the other operators the same at either timing.input_time_access(id).reads;LastWindow { length_ms }gives the refresh form with thatlength_ms.plan_counter_querywrapper is inlined into its caller.ExecutionTimingis imported at the top oftime_window.rs; the long doc line is re-wrapped.TimeShift). N2: noted under Behavior. N3: needs no change.Round 1 (abstraction and layering, functional, efficiency), all addressed on this head:
chain_builds_windows_readers_readis deleted. Its rule is now general ininput_time_accessand also covers a storedTimeShift(new testan_unbound_stored_time_shift_over_an_unbound_chain_reads_its_readers_windows, failing on the previous head). A note points toread_window.TUMBLING_WINDOW_END_COLUMNispub(crate), dropped from Interfaces, and used by the moved test.counter_rates_over_a_shared_query_time_chain_answer_as_installed, which fails on the base.irate,resets,changeshave no executor kernel) and KLL quantile_over_time over a few samples returns a sample (50) where Prometheus interpolates (46) #1346 (KLLquantile_over_timeover a few samples returns a sample where Prometheus interpolates).Verification
On
f3d63d072, compared by test name withmain863153e3cand with #1343 (697178e05), each built the same way:maincargo check --workspace --all-targetscargo fmt --all -- --checkcargo clippy -p asap-executor -p asap-planner --all-targets -- -D warningscargo nextest run -p asap-executorcargo nextest run -p asap-planner --test planner(runtime_install,metricsql_workloads,reduce_aggregatesovertime_series)counter_rates_and_increases_answer_at_query_time_from_raw_samples. Fails on all three:summary_sharing::identical_ungrouped_queries_share_their_producersThe integration-test and corpus rows of the earlier round were measured on
5bf4b132eovermain3e434c760and were not rerun after this rebase.Tests:
runtime_install::counter_rates_and_increases_answer_at_query_time_from_raw_samples(from test(planner): counter rate and increase answers, installed and at query time #1343): rate/increase per series, per job and in total, raw data kept or not, with the whole plan in one query run. 8 of 12 cases failed before; all pass now.runtime_install::counter_rates_and_increases_answer_from_views_built_at_ingestion_time(from test(planner): counter rate and increase answers, installed and at query time #1343): the same 12 cases as installed. Passes before and after.runtime_install::counter_rates_over_a_shared_query_time_chain_answer_as_installed(new): the F2 row. Fails on the base withcounter samples must have strictly increasing timestamps within each group.tumbling_window_chain::an_unbound_stored_time_shift_over_an_unbound_chain_reads_its_readers_windows(new): the stored-shift row, against the hand-workeda: 890, b: 1780. Failed on the previous head.operators::time_range::tests::a_range_window_stamps_each_row_with_its_tumbling_windows_end(moved): each kept row keeps its time and gets its window end appended.Follow-ups
j1: 0.2, j2: 0.4667) instead of only checking that the two languages agree.sum by (job)(max_over_time(latency[5m]))→j1: 18, j2: 58) came from the same one-window bug. Over 5 m the hand-worked answer isj1: 22, j2: 62, which this fix gives.