Repository navigation
fix(executor): run each node by its plan Lifetime, so roots drain in any order - #1300
Merged
Merged
Conversation
zzylol
force-pushed
the
fix/1290-executor-shared-producer
branch
from
November 14, 2026 20:00
35f5e57 to
48819b4
Compare
This was referenced Nov 14, 2026
zzylol
force-pushed
the
fix/1290-executor-shared-producer
branch
from
November 15, 2026 07:02
cacb7a0 to
4c64ca3
Compare
…any order A shared producer kept one queue bounded by `max_buffered_batches`; a reader ahead of the others waited for them, so draining root streams one after another (y16), or a SeriesBinary collecting its left branch before its right over one stored chain (b17, g10), hung forever. The runtime now runs each node by the plan's `Lifetime` and the run's scope: - query time, a leaf (raw data or view slot): one store read per consumer; - query time, OneEvaluation or AcrossEvaluations: computed to the end once, then read by every consumer, charged to the byte budget; - query time, None: one consumer; more is an error naming the node; - ingestion time: a node with several consumers runs once and each batch reaches every consumer; `execute_ingestion` polls every root and returns one future with each root's batches. `PhysicalDAG::add` takes the node's `Lifetime`; compiling a PhysicalASAPDAG passes each node's own. `Limits::max_buffered_batches`, reader positions and the batch-count backpressure are removed. Closes #1290
Address the quality review of the Lifetime runtime:
- `run_as` returns `NodeRun { ReadPerConsumer, Held, Streamed }`, the one
rule for per-consumer leaf reads, held outputs and the query-time fan-out
check; holding now happens at query time only, so ingestion streams.
- `start` walks the DAG once (`reach`): validates each node's run context,
counts its consumers and records its inputs' context per evaluation time.
- Error messages name the call for a wrong scope, and the fan-out error
gives the evaluation time and says the node should be OneEvaluation.
- The LIMIT test runs at ingestion and checks that the LIMIT consumer of a
fan-out node drops its read; scope rejections get their own test.
- Docs on `add`, `add_boxed`, `execute`, `execute_ingestion` and the README
state the rejections; test helper and doc nits.
… partial held output Address the functional review: - An operator may drop its input in `start` before the node's next consumer subscribes; the stream is now dropped early only once every expected consumer has subscribed and left, instead of panicking later. - A cancellation seen by one producer cancels every producer of the run, releasing batches a held node keeps for consumers nobody polls. - A held node that fails partway clears its consumers' queues, so each gets only the error rather than a prefix that looks like success.
Address the efficiency review: - A raw data slot carries its Scan node's plan lifetime. At query time a OneEvaluation scan with several consumers is read once and held for all of them, as the planner charges one read whose output is held; only a Lifetime::None scan (and a view slot, whose states the store keeps) is read once per consumer. - A materialized node is held only when it has more than one consumer at its evaluation time; with one consumer it streams, since no sibling can wait on that consumer. - README and runtime docs say which leaves are read per consumer, and that ingestion queues are bounded only by the byte budget.
… drop on cancel - A query-time leaf whose rows the store keeps (`None` or `AcrossEvaluations`) is read per consumer; a `OneEvaluation` node, or a computed `AcrossEvaluations` node, with more than one consumer is held. View slots carry their plan lifetime; the view-slot lifetime constant is gone. An `AcrossEvaluations` raw scan with several consumers is no longer held, which the planner never charged. - `CompiledPhysicalDAG` keeps every Planner node's lifetime in one map; `RawDataSlot::lifetime`, the operator node's lifetime and the extra lowering argument are gone, and `from_operators` takes main's pairs. - Cancelling the other producers of a run no longer drops their streams while the producer that saw the cancellation is borrowed, which panicked with "already borrowed" when an operator cancelled inside its own poll. - The fan-out check states its condition directly; `ReachedNode` names the walk's per-node facts; `Producer::consumers`; `IngestionOutputs` is the batches, used inside `Result`. - The query-time test helper marks every shared node `OneEvaluation`, leaves included; `views_at` makes a `OneEvaluation` node it moves to ingestion time `None`. - Docs: the read rule lives in `NodeRun` and the README; the design doc's slot table has a lifetime column; the executing-a-plan tutorial names `execute_ingestion` and `execute`.
- Build CompiledPhysicalDAG::lifetimes from the nodes the roots reach. - from_operators doc: every node runs as Lifetime::None, so a shared computed node needs PhysicalDAG::add with a lifetime. - NodeRun, Producer, README: a leaf read per consumer runs once per consumer; constants hold their own rows; a held computed AcrossEvaluations node occurs only in a hand-built DAG. - cancel doc: operator state in a kept stream is freed when the caller drops the run's inputs. - views_at doc gives the validation reason; tutorial uses Scope::Ingestion / Scope::Query consistently.
The grouped-limit, exact top-k and chain tests added on main build a PhysicalDAG with three-argument add calls; each node they add is read by one consumer, so it runs with Lifetime::None.
zzylol
force-pushed
the
fix/1290-executor-shared-producer
branch
from
November 16, 2026 04:11
d2d5f60 to
88c83a0
Compare
This was referenced Nov 16, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
max_buffered_batches(8). A reader that got ahead waited for the others, so a run whose consumers do not all advance together hung with no error. Forsum(sum_over_time(m[5m] offset 5m)) / sum(sum_over_time(m[5m]))(b17), the query run never returned: the harness killed it after 900 s.Lifetime. At query time no output is streamed to several consumers, so no reader waits on another. A leaf whose rows the store keeps (Lifetime::NoneorAcrossEvaluations: a raw data scan or a view slot) is read once per consumer. AOneEvaluationoutput with several consumers, whether a computed node or a raw dataScan, runs once and is held for all of them. Ingestion runs, the only place left with fan-out, are driven by the runtime itself through the newexecute_ingestion.Review guide
mainat9c752ddc6(rebased after feat(executor): grouped stddev/variance, corr, group, count_values, exact topk and without grouping #1242, fix(accuracy): relative error through negation, abs, set operators and relabels; readouts compose only through error-carrying inputs; exact accumulators take non-finite values and reject stale markers; finalize sign of counters #1309, fix(planner): give MetricsQL roots the series identity #1339 and refactor: no silent defaults for kernel memory, operator rules and aggregate names #1330). Head:88c83a059. This PR is not stacked.crates/executor/src/runtime/mod.rs. Start withNodeRunandrun_as, the one statement of the run rule. Thenexecute,execute_ingestionandstart: they count consumers per (node, evaluation time) and reject fan-out of a computed node that is not materialized. ThenProducerandInput::poll_next, where each reader has its own unread queue and a held output is read only once it has finished.crates/executor/src/plan/mod.rs(addtakes aLifetime), andcrates/executor/src/physical_planner/compiled.rs:CompiledPhysicalDAG::lifetimesholds the lifetime of every compiled Planner node, filled once incompile_internal(physical_planner/mod.rs).unboundandinstantiategive each runtime node its lifetime from that map; a helper node the lowering adds is not in it and getsLifetime::None.add(Lifetime::Noneadded);max_buffered_batches;execute_ingestionthrough one helper,tests/common::run_roots, and inphysical_commonandsql_table_scans.physical_asap_dag_at_query_time_with_node_ids(crates/types) now marks every node with more than one consumerOneEvaluation, leaves included, as Stage 3 does for a shared query-time output.views_at(physical_planner/test_support.rs) makes aOneEvaluationnode it moves to ingestion timeLifetime::None, since ingestion has no evaluation to hold it for.OneEvaluationon a shared node, which used to be implicit:physical_dag,physical_semantics,univmon_execution,kll_tumbling_window_execution, and the runtime unit tests.Rebase onto main
9c752ddc6Rebased from
59365ce80ontomain9c752ddc6(after #1242, #1309, #1339 and #1330). The previous headd2d5f6077is kept asbackup/fix/1290-executor-shared-producer-d2d5f6077. One new commit.crates/executor/src/physical_planner/mod.rs(commits 1 and 3)top_k_sortnode; this PR's sort and limit pair with aLifetimeargument, which a later commit replaces by thelifetimesmaptop_k_sortnode, with noLifetimeargument once the map takes overcrates/executor/tests/stored_tumbling_window_memory_limit.rsOperator::sort(…, None)(newfetchargument); this PR'sLifetime::NoneNew commit
88c83a059: the tests #1242 added (blocking_resources.rs, and the grouped-limit, exact top-k and chain tests inphysical_semantics.rs) build aPhysicalDAGwith three-argumentadd. Each node they add has one consumer, so it now getsLifetime::None.#1242's lowering does not need the
Lifetimerun rule or thelifetimesmap. Its new helper nodes are thewithoutprojection, the top-k /count_valueshelper and the summary-merge union. Each has one consumer, so it runs asLifetime::None, the default for ids the map does not hold. The Planner node itself keepsnode.lifetimethrough the map.Checked on
88c83a059, by test name againstmain9c752ddc6:cargo check --workspace --all-targets,cargo fmt --all -- --checkandcargo clippy -p asap-executor -p asap-integration-tests -p asap-planner -p asap-types -p asap-cost --all-targets -- -D warnings(with #1330's wildcard lint) are clean.cargo test --no-fail-fastonasap-executor,asap-integration-tests,asap-plannerandasap-types: 814 passed, 8 failed, 6 ignored. All 8 fail onmain9c752ddc6too (joint_search_equivalence×2,materialized_views_equal_the_stored_outputs_main_listed,plan_sharing×2,promql_to_post_asap×2,identical_ungrouped_queries_share_their_producers). Nothing newly fails.Checked against main
3e434c760After the Stage 3 execution-contexts refactor (#1237) merged, this head (
d2d5f6077) was rebased locally ontomain3e434c760with no conflicts and built there; it needs no new push.cargo check --workspace --all-targets,cargo fmt --all -- --checkandcargo clippy -p asap-executor -p asap-integration-tests -p asap-planner -p asap-types --all-targets -- -D warningsare clean.cargo teston those four crates: 750 passed, 9 failed, 6 ignored; all 9 fail onmain3e434c760too, by name. Nothing newly fails.Behavior: before → after
The workload is b17. One query is evaluated every 60 s, raw data is not kept, and there are 1,000 gauge series at one sample per second:
Stage 3 selects one stored ingestion-time chain (node 1, exact
Sum, 15 s windows). Both sides of the division read it at query time. ATimeShiftwithout@compiles to aProjectat the same evaluation time, so both branches read the same producer key(1, T).SeriesBinarycollects its left input to the end before its right. The left branch pulled 8 window batches while the right reader stayed at 0. The queue was full, the left reader returnedPending, and nothing woke it.AcrossEvaluationsleaf, so a store read per consumer. Each consumer opens its own read (read_materialized_view), so neither waits on the other.execute ids=[11])join_allThis serves correctness: a plan Stage 3 selects must run whatever order the caller drains its roots in.
Interfaces
crates/executor/src/plan/mod.rsimpl<'a, V: 'a, S: Clone + PartialEq + Debug + 'a> PhysicalDAG<'a, V, S> { + /// Add node `id`, reading `inputs`, run by its plan `lifetime`. pub fn add( &mut self, id: NodeId, inputs: Vec<NodeId>, operator: impl PhysicalOperator<V, S> + 'a, + lifetime: Lifetime, ) -> Result<(), Error>; + /// [`Self::add`] with an already boxed operator. pub fn add_boxed( &mut self, id: NodeId, inputs: Vec<NodeId>, operator: Box<dyn PhysicalOperator<V, S> + 'a>, + lifetime: Lifetime, ) -> Result<(), Error>; pub fn validate(&self, roots: &[NodeId]) -> Result<(), Error>; pub fn properties(&self, roots: &[NodeId]) -> Result<BTreeMap<NodeId, PlanProperties>, Error>; + /// A query run: one output stream per root, which the caller may drain + /// in any order. Rejects an ingestion scope (run it with + /// [`Self::execute_ingestion`]), and a computed node with + /// `Lifetime::None` and more than one consumer at one evaluation time (a + /// planner bug), before any operator starts. pub fn execute<'r>( &'r self, roots: &[NodeId], context: RunContext, ) -> Result<Vec<Input<'r, V>>, Error> where 'a: 'r; + /// An ingestion run: one future that resolves to every root's batches, + /// in `roots` order, for the caller to write. Rejects a query scope (run + /// it with [`Self::execute`]). + pub fn execute_ingestion<'r>( + &'r self, + roots: &[NodeId], + context: RunContext, + ) -> Result<impl std::future::Future<Output = Result<IngestionOutputs<V>, Error>> + 'r, Error> + where + 'a: 'r; } pub(crate) struct Node<'a, V, S> { pub(crate) inputs: Vec<NodeId>, pub(crate) operator: Box<dyn PhysicalOperator<V, S> + 'a>, + /// How long its output is kept, from the plan: what the runtime runs it + /// by (see the crate README). Example: `OneEvaluation`, computed once + /// and held for its consumers at query time. + pub(crate) lifetime: Lifetime, }crates/executor/src/runtime/mod.rscrates/executor/src/runtime/context.rs#[derive(Clone, Debug)] pub struct Limits { - pub max_buffered_batches: usize, pub max_bytes: usize, }max_buffered_batchesis removed because no reader waits on another any more. Each reader of an ingestion fan-out has its own unbounded queue: the byte budget is the only bound, so a lagging consumer holds what the others have read, and over the budget the run fails withMemoryLimitinstead of hanging.crates/executor/src/physical_planner/compiled.rspub struct CompiledPhysicalDAG { nodes: BTreeMap<NodeId, Node>, roots: Vec<NodeId>, + /// The plan lifetime of each compiled Planner node, which the runtime + /// runs it by (see `NodeRun` in the runtime). A helper node the lowering + /// adds is not in it and has `Lifetime::None`. Example: node 1, a stored view, + /// maps to `AcrossEvaluations(Memory)`, and node 3, a query-time output + /// two queries read, to `OneEvaluation(Memory)`. + lifetimes: BTreeMap<NodeId, Lifetime>, }The field is private; it is shown because it is where the compiled DAG keeps the plan's lifetimes.
from_operatorskeepsmain's signature: its nodes are not in the map, so every node runs asLifetime::None, which every caller passed. So at query time a computed node built withfrom_operatorsmay have at most one consumer; a shared one needs aPhysicalDAGbuilt withadd(…, lifetime). Its doc says so.Usage: the b17 query run as the review harness runs it, roots drained one after another.
Design scope
Every new or changed item, where it sits in the engine, and why it is general:
PhysicalDAG::add/add_boxedtakelifetime, stored onplan::Node::lifetime(executor runtime DAG).PhysicalDAGis the generic runtime DAG, and a hand-built one (tests, the README example, any embedder) has no plan to read lifetimes from, so the publicaddkeeps the argument. Every runtime node has a lifetime; it reusesplanner_types::stages::Lifetime, no new lifetime type.CompiledPhysicalDAG::lifetimes: BTreeMap<NodeId, Lifetime>(executor lowering, private), filled once incompile_internalfrom the plan nodes the roots reach (the nodes that are compiled), read byunboundandinstantiatethrough the privatelifetime(id). It is "the plan lifetime of each compiled Planner node", which every slot and operator has. It replacesRawDataSlot::lifetime, the lifetime oncompiled::Node::Operator, the view-slot lifetime constant and the extra argument of the crate-private loweringadd, which are deleted.from_operatorsis back tomain's pairs.NodeRun { ReadPerConsumer, Held, Streamed }andrun_as(executor runtime, private): how any node runs, from the run's scope, its lifetime, whether it is a leaf and its consumer count. The one statement of the read rule; the README describes it for callers, and other docs link to them.ReachedNode { consumers, input_context }andReached(executor runtime, private): what the pre-run walk records for each (node, evaluation time) any run reaches.RunProducers(executor runtime, private): every producer of one run, so a cancellation seen by one stops the others.Producer::consumers(renamed fromexpected_readers) andProducer::end(private): the consumer count, asNodeRunand the error text call it;endis the end of a node's output shared byfinish(which drops the stream) andcancel(which keeps it).execute_ingestionandIngestionOutputs<V>(executor runtime API): an ingestion run of any DAG, and its result, every root's batches.tests/common::run_roots(executor test support only): one run of any DAG's roots in either scope.Algorithm changes
How the runtime runs a node
SeriesBinary), never let the slow reader move, so the run hung. Examples: y16, b17 and g10 hung; withjoin_all, y16 completed.Producerper (node, evaluation time) and one queue.Pendingonce the queue holdsmax_buffered_batches.runtime::start,Input::poll_next):reach): for each (node, evaluation time), validate its run context, count its consumers and record its inputs' context (ReachedNode). One private rule,run_as(node, query, consumers) -> NodeRun, decides how each node runs. At query time a computedLifetime::Nonenode with more than one consumer is anError::Invalid, for examplenode 1 (Identity) at evaluation time 100 has 2 consumers but Lifetime::None; a shared query-time output must be OneEvaluation (planner bug).Lifetime::NoneorAcrossEvaluations(rows a store or the operator keeps):None(a raw data scan the plan does not materialize, or a constant, which holds its own rows) orAcrossEvaluations(a view slot, or raw data kept at ingestion): one producer per consumer, so one read each, and nothing is held. Stage 3 never gives a query-timeNoneresult more than one consumer; a view slot, or a leaf in a hand-built DAG, may have several. The reads agree with each other because of theDataStorecontract: a run's store handle presents one consistent state to every read of that run, so a refresh that lands during the run is seen by the next run, not by one read of this one.OneEvaluation(a computed node or a raw data scan) with more than one consumer: the producer runs once, to the end, before any consumer reads it, as the planner charges one run whose output is held. Likewise a computedAcrossEvaluationsnode with more than one consumer, which only a hand-built DAG has at query time (a compiled plan reads it through its view slot). Example: in y03 the three queries' shared scan (node 0,OneEvaluation) is one store read, not three. With one consumer the node streams to it instead (nothing is held), since no sibling can wait on that consumer. Every consumer then reads the finished batches from its own queue. The batches stay charged to the byte budget until every consumer has read them, and going over givesMemoryLimit. Nothing spills. A held node that fails partway (an upstream error orMemoryLimit) clears its consumers' queues, so each gets only the error, never a prefix that looks like success.Nonecomputed node: one consumer; the stream passes through.execute_ingestiondrives all roots withtry_join_all.Producer::waitersstays, while reader positions and the shared queue are gone. Two readers of one held or ingestion fan-out producer can both see the upstream returnPending, and the upstream wakes only the reader that polled it last. So a reader registers inwaiterswhen it getsPending, and any push (for a streamed node) or the end wakes it.startdoes not end the node for a consumer that subscribes after it.Cancelled. Cancelling sets the failure, clears the unread queues and wakes the readers, but keeps each stream: the producer that saw the cancellation may be borrowed up the poll stack, and an input inside another producer's stream would borrow it again on drop. Each stream is dropped with its node's last input.Pendingit registers its waker so that any reader's progress, or the end, wakes it. The queues are unbounded, so the only remaining limit is the byte budget: a run completes or fails withMemoryLimit, never hangs. Tests:roots_reading_one_leaf_drain_in_orderone_root_with_two_branches_over_one_leafstored_leaf_is_read_per_consumeringestion_fan_out_reads_the_source_oncequery_time_fan_out_of_a_not_materialized_node_is_an_errorone_evaluation_output_over_budget_is_a_memory_limitcancel_inside_a_held_node_ends_every_root,cancel_inside_an_ingestion_fan_out_ends_the_runmain's 1.25 MB (8.6×); b11 was 5.9×, y03 3.9×, p10 1.9×, y09 1.5×. Query time was unchanged on all five. The only bound is the byte budget.OneEvaluationscan with k consumers is read once, as onmain. A leaf withLifetime::NoneorAcrossEvaluationswith k consumers is read k times, where before it was read once. So a stored view is read once per consumer, and its reads and bytes read grow with its consumers: on the review workloads view reads went from 1 to 2 (y14, n02, n14, y16), 1 to 3 (g04) and 2 to 3 (b14), and y16's view bytes from 46,656 to 93,312 B. With the in-memory store this adds about 25–40 µs on small queries (n02 +10%, y14 +9%) and nothing measurable elsewhere, while peak memory falls (y16 89,576 → 46,424 B). Against a remote or disk-backed store, k consumers are k remote reads.plan_costhas no separate store-read charge for a view; the view node itself is costed at its own (ingestion) rate, and each consumer is costed for processing the view's rows at its own rate (crates/cost/src/plan/execution_demand.rs,consumers: a walk stops at a materialized node and counts each consumer separately). A query-timeNonescan is charged once per consumer (its multiplier is its consumer count), and anAcrossEvaluationsraw scan is charged its retained rows and no held bytes. So the per-consumer reads match the cost model; no issue filed.execute_ingestionpolling every root, that is a few batches. An operator that reads one input to its end first holds that input's whole output.execute, before any operator starts, not incompile. A node read at two evaluation times (under an@anchor) is two producers with one consumer each, andcompilehas no evaluation time to tell them apart.Lifetime::Noneleaf is read per consumer.Decisions and open questions
Needs reviewer decision
None. Items 1 to 4 below were approved and are kept for reference.
Where the fan-out error is raised. It is raised at
execute(andexecute_ingestion, where fan-out is allowed), not incompile. Reason: under an@anchor, one Planner node can be read at two evaluation times with one consumer each, andcompilecannot see evaluation times. Example:x @ 100 + xsharesxacross two times, and that must not be an error.New public type alias
IngestionOutputs<V>, which shortens theexecute_ingestionfuture's type:CompiledPhysicalDAG::from_operatorskeepsmain's signature. Every node runs asLifetime::None, as every caller (tests, andpromql_values::compile_scalar) passed. So at query time a computed node may have at most one consumer; a shared computed node needs aPhysicalDAGbuilt withPhysicalDAG::add(…, lifetime). No caller has one today.Planner charge vs executor memory for
OneEvaluationoutputs. They do not agree. The executor holds 2.4 to 8 times what the planner charges, at the same row count. The planner charges state or value bytes per row. The executor counts Arrow buffers, including label and identity columns. Measured with the review harness at 12 series; the planner figure is its 1,000-series charge scaled to 12 rows:The held time the planner charges is about 0.001–0.02 ms, so the cost effect is small. Decided: follow-up Planner charges less memory for a held OneEvaluation output than the executor holds #1301.
Decided
Lifetime) replaces the byte-budget patch: no shared streaming fan-out at query time, and no auto-materialization.Lifetime::NoneorAcrossEvaluations(raw data, a constant or a view slot) is one read per consumer; aOneEvaluationleaf with several consumers is one read held for all of them. The "None with more than one consumer" error applies to computed nodes only. (Round 2: anAcrossEvaluationsraw scan with several consumers was held before; it is now read per consumer, which is what the planner charges.)execute_ingestion(roots, context)returns one future with every root's batches, and the caller writes them.executerejects the Ingestion scope. NoDataStorewrite method is added.SeriesBinarykeeps collecting its left input before its right; under this change that is safe.addtakesplanner_types::stages::Lifetime. No new lifetime type.Review status
NodeRun) instead of three predicates; one pre-run walk; the stale LIMIT test rewritten at ingestion;execute/execute_ingestiondocs state the rejections; the copied test scope dispatch moved intotests/common::run_roots.OneEvaluationscan is read once and held, and a materialized node is held only with more than one consumer (b945cc2); query time is back tomain's.AcrossEvaluationsraw scan with several consumers is no longer held, which the planner never charged. View slots carry their plan lifetime; the view-slot constant is deleted. Test first:stored_leaf_is_read_per_consumerand theAcrossEvaluationscase ofroots_reading_one_leaf_drain_in_orderfailed before the fix.CompiledPhysicalDAG::lifetimes;RawDataSlot::lifetime,compiled::Node::Operator's lifetime and the loweringaddargument deleted;from_operatorsback to pairs. The publicPhysicalDAG::addkeepslifetime(Design scope).NodeRunand the README; "the planner charges aNoneread per consumer" replaced by "Stage 3 never gives a query-timeNoneresult more than one consumer"; a lifetime column in the design doc's slot table; the executing-a-plan tutorial namesexecute_ingestionandexecute; forwarder docs fixed.OneEvaluation, leaves included. This broke 3kll_tumbling_window_executiontests ("an ingestion-time output cannot be materialized for one evaluation"):views_atmoved the now-OneEvaluationshared scan to ingestion time. Fixed inviews_at, which now makes such a nodeLifetime::None, as Stage 3 would; the tests are unchanged and pass.cancelandfinishshare oneend(A6);IngestionOutputsis the batches, used insideResult(A7);query_time_leaves_are_read_per_consumerrenamednot_materialized_leaf_is_read_per_consumerand test comments say what the DAG is (A8);ReachedNode,Producer::consumers, theplan::Node::lifetimeexample (F9); b17 node numbers and theLimitsdiff (F5, F8); this Design scope section (F4).cancel_inside_a_held_node_ends_every_root(query time,OneEvaluation) andcancel_inside_an_ingestion_fan_out_ends_the_run(execute_ingestion,None) both panicked before the fix.execute_ingestion(maintenance.roots(), …).from_operatorsdoc and Decision 3 say every node runs asLifetime::None, so a shared computed node needsPhysicalDAG::add(…, lifetime).lifetimesis built only from the compiled (reached) nodes; its doc and the Design scope say so.hold_until_finisheddoc says "a [NodeRun::Held] node". Abstraction N2: theviews_atdoc gives the reason, the DAG's validation rejectingOneEvaluationat ingestion.Producerdoc say a leaf read per consumer runs once per consumer; a held computedAcrossEvaluationsnode is described as occurring only in a hand-built DAG; constants and other source leaves are described as holding their own rows, not as store-kept. Algorithm steps 2–3 and the Decided list follow. The tutorial now saysScope::Ingestion/Scope::Queryconsistently.canceldoc says operator state inside a kept stream is freed when the caller drops the run's inputs.Status: rounds 1–3 complete. Ready to merge.
Verification
main59365cecargo test -p asap-executor --features test-support(lib,executorsuite, memory-limit suites, README doc test)cargo test -p asap-types(lib, suite, doc)physical_asap_dag_at_query_time_with_node_idschangecargo test -p asap-planner(lib,plannersuite, doc tests)check_compiles_reports_the_executors_error,identical_ungrouped_queries_share_their_producers) fail onmain59365ce; the tutorials have no code blocks, so 0 doc testscargo test -p asap-integration-tests --features asap-executor/test-supportjoint_search_equivalence2,materialized_views1,plan_sharing2,promql_to_post_asap2) aremain's 7; the 3kll_tumbling_window_executiontests the helper change broke were fixed inviews_atand passcargo clippy -p asap-executor -p asap-types -p asap-planner -p asap-integration-tests --all-targets --features asap-executor/test-support -- -D warnings, and-p asap-executor --all-targetswithout the featurecargo fmt --all --checkNew tests (
crates/executor/src/runtime/tests.rs). Each hang-prone one runs on its own thread with a 10 s timeout, so a hang fails instead of blocking:roots_reading_one_leaf_drain_in_order: two roots read one leaf that emits 20 batches; draining them in turn equalsjoin_all. Runs with the leafLifetime::NoneandAcrossEvaluations(2 store reads per run) andOneEvaluation(1 read, held). Hangs onmain(timeout).one_root_with_two_branches_over_one_leaf: one root,LeftThenRight(collects left, then right, asSeriesBinarydoes) over two readers of one leaf. Hangs onmain.stored_leaf_is_read_per_consumer: anAcrossEvaluationsleaf read by two roots is read twice, and only the batch being read is charged (8 B, not the 32 B a held read would keep).not_materialized_leaf_is_read_per_consumerandmaterialized_node_runs_once_and_is_held: replace the old backpressure and diamond tests.ingestion_fan_out_reads_the_source_once: one ingestion read feeds three consumers (one throughLeftThenRight). The source starts once and every root gets all batches. Themain-API form of the same run hangs.query_time_fan_out_of_a_not_materialized_node_is_an_error: aNonenode with two consumers is an error naming node 1, and no source starts. Read at two evaluation times, the same node runs.one_evaluation_output_over_budget_is_a_memory_limit: a held output over a 40-byte budget ends inMemoryLimit, and nothing stays reserved.execute_rejects_ingestion_scope_and_vice_versa: each entry point rejects the other scope with a message naming both calls.limit_consumer_of_an_ingestion_fan_out_drops_its_read(physical_semantics, replaceslimit_branch_finishes_without_blocking_shared_sibling): at ingestion, a LIMIT consumer of a fan-out node drops its queue after its row; the run's peak stays at its 101 output batches (it is 40,000 B instead of 20,200 B when the drop is disabled).an_input_dropped_at_start_leaves_the_node_to_later_consumers: a held node read by an operator that drops its input instartand by anIdentity; theIdentitystill gets all four batches.cancellation_releases_held_batches_of_unpolled_roots: after one root reads a batch and the run is cancelled, polling that root releases everything the held node kept for the other root (retained bytes 0), and the other root ends withCancelled.cancel_inside_a_held_node_ends_every_rootandcancel_inside_an_ingestion_fan_out_ends_the_run: an operator cancels the run inside its own poll after 3 batches; every root ends withCancelledand nothing stays reserved.broadcast_error_and_cancel(changed): a held node whose source fails at its second batch gives each consumer only the error, not the first batch.one_evaluation_node_with_one_consumer_streams: aOneEvaluationnode with one consumer gives its first batch after one source batch, with 8 B peak, instead of after the whole output.End to end: the Stage 3 review harness (#1236's, not committed) planned and executed 103 workloads at cacb7a0: the round-7 set plus b17, g10, n02 and n14. Every root was drained in turn, not with
join_all. The round-2 functional review reran all 103 at cacb7a0 with the same result.p34: oldfixed_intervalworkload format), as in round 7y16, which needed
join_allin round 7, now completes with roots drained in turn and the same answers.Query time and reads after the round-2 fixes
The same harness, 15 workloads and settings as the round-1 table below (12 series, 2 h, minimum of 7 query runs, arms interleaved per workload, roots drained in turn): the round-2 head rebased onto 59365ce (b945cc2) against this push (built before a final amend that changes only a doc comment).
AcrossEvaluationsraw scan with several consumers, so the stored-slot rule changes no read here; its effect is covered bystored_leaf_is_read_per_consumer.Query time and memory after the leaf-lifetime fix (round 1)
The review harness at 12 series and 2 h of data, on the 15 workloads where the branch before that fix read the store more than
main. Each time is the minimum of 7 query runs. The arms were interleaved per workload:main4de4075 (roots joined), the branch before the fix (48819b4, roots in turn) and after it (cacb7a0, roots in turn).OneEvaluation, so it is one read, and query time is back tomain's, within noise.main's, because each read streams.At 200 series (53 MB per scan), query time in ms, main / before / after: b07 6.8 / 22.2 / 7.0, g14 22.5 / 34.9 / 23.0, p10 16.8 / 30.1 / 17.1, y03 22.9 / 39.6 / 23.7, y15 25.4 / 59.7 / 25.5. Reads and peak are the same as
main.At 50 series and 12 h, with raw data stored in 10,000-row batches (88 MB per scan, about 430 batches):
The peak in this case is the held scan's filtered output. That is the memory the plan charges for a
OneEvaluationscan, in exchange for one 88 MB read instead of two or three.Lifetime flips on real plans
The review harness can override node lifetimes before compiling (
RV_FLIP, a local change only). Results at cacb7a0:SeriesLabels, 2 consumers) set toNone: rejected withnode 3 (SeriesLabels) at evaluation time 1700002800000 has 2 consumers but Lifetime::None; a shared query-time output must be OneEvaluation (planner bug).TimeRange) or node 8 (ScopeTimestamp) set toNone: rejected the same way.None: the same answers, which equal the reference within 1e-9. It takes 2 store reads instead of 1.None, or nodes 2, 4 and 11 set toOneEvaluation: the same answers, which equal the reference.OneEvaluation: the same answers, which equal the reference.Follow-ups
OneEvaluationworking-memory bytes are 2.4–8 times below what the executor holds.Limits::max_buffered_batchesis gone; a caller that set it must drop the field.Closes #1290