Skip to content

fix(executor): run each node by its plan Lifetime, so roots drain in any order - #1300

Merged
zzylol merged 7 commits into
mainfrom
fix/1290-executor-shared-producer
Nov 16, 2026
Merged

zzylol merged 7 commits into
mainfrom
fix/1290-executor-shared-producer

Conversation

@zzylol

@zzylol zzylol commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: every node read by several consumers ran as one shared producer, with one queue capped at max_buffered_batches (8). A reader that got ahead waited for the others, so a run whose consumers do not all advance together hung with no error. For sum(sum_over_time(m[5m] offset 5m)) / sum(sum_over_time(m[5m])) (b17), the query run never returned: the harness killed it after 900 s.
  • Change: the runtime now runs each node by the plan's Lifetime. At query time no output is streamed to several consumers, so no reader waits on another. A leaf whose rows the store keeps (Lifetime::None or AcrossEvaluations: a raw data scan or a view slot) is read once per consumer. A OneEvaluation output with several consumers, whether a computed node or a raw data Scan, runs once and is held for all of them. Ingestion runs, the only place left with fan-out, are driven by the runtime itself through the new execute_ingestion.
  • Effect: b17 now completes with every root drained one after another, and its answer, 1.017169267494475, equals the exact reference. The other workloads that hung (y16, g10, n02, n14) also complete. Of the 70 review workloads that answered in round 7, 68 give identical answers; the other 2 are KLL quantiles that stay within their rank target.

Review guide

Rebase onto main 9c752ddc6

Rebased from 59365ce80 onto main 9c752ddc6 (after #1242, #1309, #1339 and #1330). The previous head d2d5f6077 is kept as backup/fix/1290-executor-shared-producer-d2d5f6077. One new commit.

File Conflict Resolution
crates/executor/src/physical_planner/mod.rs (commits 1 and 3) #1242 lowers an exact top-k to one top_k_sort node; this PR's sort and limit pair with a Lifetime argument, which a later commit replaces by the lifetimes map main's single top_k_sort node, with no Lifetime argument once the map takes over
crates/executor/tests/stored_tumbling_window_memory_limit.rs main's Operator::sort(…, None) (new fetch argument); this PR's Lifetime::None both

New commit 88c83a059: the tests #1242 added (blocking_resources.rs, and the grouped-limit, exact top-k and chain tests in physical_semantics.rs) build a PhysicalDAG with three-argument add. Each node they add has one consumer, so it now gets Lifetime::None.

#1242's lowering does not need the Lifetime run rule or the lifetimes map. Its new helper nodes are the without projection, the top-k / count_values helper and the summary-merge union. Each has one consumer, so it runs as Lifetime::None, the default for ids the map does not hold. The Planner node itself keeps node.lifetime through the map.

Checked on 88c83a059, by test name against main 9c752ddc6: cargo check --workspace --all-targets, cargo fmt --all -- --check and cargo clippy -p asap-executor -p asap-integration-tests -p asap-planner -p asap-types -p asap-cost --all-targets -- -D warnings (with #1330's wildcard lint) are clean. cargo test --no-fail-fast on asap-executor, asap-integration-tests, asap-planner and asap-types: 814 passed, 8 failed, 6 ignored. All 8 fail on main 9c752ddc6 too (joint_search_equivalence ×2, materialized_views_equal_the_stored_outputs_main_listed, plan_sharing ×2, promql_to_post_asap ×2, identical_ungrouped_queries_share_their_producers). Nothing newly fails.

Checked against main 3e434c760

After the Stage 3 execution-contexts refactor (#1237) merged, this head (d2d5f6077) was rebased locally onto main 3e434c760 with no conflicts and built there; it needs no new push. cargo check --workspace --all-targets, cargo fmt --all -- --check and cargo clippy -p asap-executor -p asap-integration-tests -p asap-planner -p asap-types --all-targets -- -D warnings are clean. cargo test on those four crates: 750 passed, 9 failed, 6 ignored; all 9 fail on main 3e434c760 too, by name. Nothing newly fails.

Behavior: before → after

The workload is b17. One query is evaluated every 60 s, raw data is not kept, and there are 1,000 gauge series at one sample per second:

sum(sum_over_time(m[5m] offset 5m)) / sum(sum_over_time(m[5m]))

Stage 3 selects one stored ingestion-time chain (node 1, exact Sum, 15 s windows). Both sides of the division read it at query time. A TimeShift without @ compiles to a Project at the same evaluation time, so both branches read the same producer key (1, T).

Before After
Runtime node 1, the view slot, was one shared producer with two readers. The root SeriesBinary collects its left input to the end before its right. The left branch pulled 8 window batches while the right reader stayed at 0. The queue was full, the left reader returned Pending, and nothing woke it. node 1 is an AcrossEvaluations leaf, so a store read per consumer. Each consumer opens its own read (read_materialized_view), so neither waits on the other.
Harness (execute ids=[11]) never returned; killed at 900 s returns 1.017169267494475, equal to the exact reference
y16 (two queries, roots drained in order) hung on root 8 both roots answer; same as join_all

This serves correctness: a plan Stage 3 selects must run whatever order the caller drains its roots in.

Interfaces

crates/executor/src/plan/mod.rs

 impl<'a, V: 'a, S: Clone + PartialEq + Debug + 'a> PhysicalDAG<'a, V, S> {
+    /// Add node `id`, reading `inputs`, run by its plan `lifetime`.
     pub fn add(
         &mut self,
         id: NodeId,
         inputs: Vec<NodeId>,
         operator: impl PhysicalOperator<V, S> + 'a,
+        lifetime: Lifetime,
     ) -> Result<(), Error>;
+    /// [`Self::add`] with an already boxed operator.
     pub fn add_boxed(
         &mut self,
         id: NodeId,
         inputs: Vec<NodeId>,
         operator: Box<dyn PhysicalOperator<V, S> + 'a>,
+        lifetime: Lifetime,
     ) -> Result<(), Error>;
     pub fn validate(&self, roots: &[NodeId]) -> Result<(), Error>;
     pub fn properties(&self, roots: &[NodeId]) -> Result<BTreeMap<NodeId, PlanProperties>, Error>;
+    /// A query run: one output stream per root, which the caller may drain
+    /// in any order. Rejects an ingestion scope (run it with
+    /// [`Self::execute_ingestion`]), and a computed node with
+    /// `Lifetime::None` and more than one consumer at one evaluation time (a
+    /// planner bug), before any operator starts.
     pub fn execute<'r>(
         &'r self,
         roots: &[NodeId],
         context: RunContext,
     ) -> Result<Vec<Input<'r, V>>, Error>
     where
         'a: 'r;
+    /// An ingestion run: one future that resolves to every root's batches,
+    /// in `roots` order, for the caller to write. Rejects a query scope (run
+    /// it with [`Self::execute`]).
+    pub fn execute_ingestion<'r>(
+        &'r self,
+        roots: &[NodeId],
+        context: RunContext,
+    ) -> Result<impl std::future::Future<Output = Result<IngestionOutputs<V>, Error>> + 'r, Error>
+    where
+        'a: 'r;
 }
 pub(crate) struct Node<'a, V, S> {
     pub(crate) inputs: Vec<NodeId>,
     pub(crate) operator: Box<dyn PhysicalOperator<V, S> + 'a>,
+    /// How long its output is kept, from the plan: what the runtime runs it
+    /// by (see the crate README). Example: `OneEvaluation`, computed once
+    /// and held for its consumers at query time.
+    pub(crate) lifetime: Lifetime,
 }

crates/executor/src/runtime/mod.rs

+/// Every root's batches of an ingestion run, in the order of its roots.
+pub type IngestionOutputs<V> = Vec<Vec<SharedValue<V>>>;

crates/executor/src/runtime/context.rs

 #[derive(Clone, Debug)]
 pub struct Limits {
-    pub max_buffered_batches: usize,
     pub max_bytes: usize,
 }

max_buffered_batches is removed because no reader waits on another any more. Each reader of an ingestion fan-out has its own unbounded queue: the byte budget is the only bound, so a lagging consumer holds what the others have read, and over the budget the run fails with MemoryLimit instead of hanging.

crates/executor/src/physical_planner/compiled.rs

 pub struct CompiledPhysicalDAG {
     nodes: BTreeMap<NodeId, Node>,
     roots: Vec<NodeId>,
+    /// The plan lifetime of each compiled Planner node, which the runtime
+    /// runs it by (see `NodeRun` in the runtime). A helper node the lowering
+    /// adds is not in it and has `Lifetime::None`. Example: node 1, a stored view,
+    /// maps to `AcrossEvaluations(Memory)`, and node 3, a query-time output
+    /// two queries read, to `OneEvaluation(Memory)`.
+    lifetimes: BTreeMap<NodeId, Lifetime>,
 }

The field is private; it is shown because it is where the compiled DAG keeps the plan's lifetimes. from_operators keeps main's signature: its nodes are not in the map, so every node runs as Lifetime::None, which every caller passed. So at query time a computed node built with from_operators may have at most one consumer; a shared one needs a PhysicalDAG built with add(…, lifetime). Its doc says so.

Usage: the b17 query run as the review harness runs it, roots drained one after another.

// plan: Stage 3's PhysicalASAPDAG for b17. Node 1 is the stored chain
// (AcrossEvaluations); nodes 3, 5 and 10 are OneEvaluation (one consumer
// each, so they stream); every other node is None. Root 11.
let views = plan.materialized_views()?; // [view of node 1]
// compiled: node 1 becomes a materialized view slot (a leaf); the compiled
// DAG keeps every node's lifetime in one map.
let compiled = compile(&plan, &views, &[11])?;
// instance: slots bound to the store, which holds node 1's windows; each
// runtime node carries its lifetime from that map.
let instance = compiled.instantiate(Arc::new(store))?;
// context: the query run at T, with a 16 GiB byte budget.
let context = RunContext::new(Scope::Query { evaluation_time_ms: t }, Limits { max_bytes: 1 << 34 })?;
// outputs: one stream per root. Node 1 has two consumers, node 2 TimeRange
// and node 6 (the shifted branch); it is an AcrossEvaluations leaf, so each
// opens its own store read.
let outputs = instance.execute(compiled.roots(), context)?;
for mut output in outputs {
    // Drained in turn: SeriesBinary collects its left branch, then its
    // right, and neither waits on the other.
    while let Some(batch) = output.next().await { /* 1 row: the ratio */ }
}
// An ingestion refresh of node 1 for one 15 s window compiles node 1 on its
// own, with no view at it, so it computes its windows instead of reading
// the stored ones:
//   let maintenance = compile(&plan, &[], &[1])?;
//   let refresh = maintenance.instantiate(Arc::new(store))?;
//   refresh.execute_ingestion(maintenance.roots(), ingestion_context)?.await?
// The result is node 1's batches for the caller to append to the store.

Design scope

Every new or changed item, where it sits in the engine, and why it is general:

  • PhysicalDAG::add / add_boxed take lifetime, stored on plan::Node::lifetime (executor runtime DAG). PhysicalDAG is the generic runtime DAG, and a hand-built one (tests, the README example, any embedder) has no plan to read lifetimes from, so the public add keeps the argument. Every runtime node has a lifetime; it reuses planner_types::stages::Lifetime, no new lifetime type.
  • CompiledPhysicalDAG::lifetimes: BTreeMap<NodeId, Lifetime> (executor lowering, private), filled once in compile_internal from the plan nodes the roots reach (the nodes that are compiled), read by unbound and instantiate through the private lifetime(id). It is "the plan lifetime of each compiled Planner node", which every slot and operator has. It replaces RawDataSlot::lifetime, the lifetime on compiled::Node::Operator, the view-slot lifetime constant and the extra argument of the crate-private lowering add, which are deleted. from_operators is back to main's pairs.
  • NodeRun { ReadPerConsumer, Held, Streamed } and run_as (executor runtime, private): how any node runs, from the run's scope, its lifetime, whether it is a leaf and its consumer count. The one statement of the read rule; the README describes it for callers, and other docs link to them.
  • ReachedNode { consumers, input_context } and Reached (executor runtime, private): what the pre-run walk records for each (node, evaluation time) any run reaches.
  • RunProducers (executor runtime, private): every producer of one run, so a cancellation seen by one stops the others.
  • Producer::consumers (renamed from expected_readers) and Producer::end (private): the consumer count, as NodeRun and the error text call it; end is the end of a node's output shared by finish (which drops the stream) and cancel (which keeps it).
  • execute_ingestion and IngestionOutputs<V> (executor runtime API): an ingestion run of any DAG, and its result, every root's batches.
  • tests/common::run_roots (executor test support only): one run of any DAG's roots in either scope.

Algorithm changes

How the runtime runs a node

  • Problem: with one bounded queue per shared node, a reader that got ahead waited for the slowest one. A caller draining roots in turn, or an operator reading one input to its end (SeriesBinary), never let the slow reader move, so the run hung. Examples: y16, b17 and g10 hung; with join_all, y16 completed.
  • Before:
    1. One Producer per (node, evaluation time) and one queue.
    2. A batch stays until the slowest reader reads it.
    3. A reader at the head returns Pending once the queue holds max_buffered_batches.
    4. Only another reader's progress wakes it.
  • After (runtime::start, Input::poll_next):
    1. Walk the DAG once before anything starts (reach): for each (node, evaluation time), validate its run context, count its consumers and record its inputs' context (ReachedNode). One private rule, run_as(node, query, consumers) -> NodeRun, decides how each node runs. At query time a computed Lifetime::None node with more than one consumer is an Error::Invalid, for example node 1 (Identity) at evaluation time 100 has 2 consumers but Lifetime::None; a shared query-time output must be OneEvaluation (planner bug).
    2. Query time, a leaf with Lifetime::None or AcrossEvaluations (rows a store or the operator keeps): None (a raw data scan the plan does not materialize, or a constant, which holds its own rows) or AcrossEvaluations (a view slot, or raw data kept at ingestion): one producer per consumer, so one read each, and nothing is held. Stage 3 never gives a query-time None result more than one consumer; a view slot, or a leaf in a hand-built DAG, may have several. The reads agree with each other because of the DataStore contract: a run's store handle presents one consistent state to every read of that run, so a refresh that lands during the run is seen by the next run, not by one read of this one.
    3. Query time, OneEvaluation (a computed node or a raw data scan) with more than one consumer: the producer runs once, to the end, before any consumer reads it, as the planner charges one run whose output is held. Likewise a computed AcrossEvaluations node with more than one consumer, which only a hand-built DAG has at query time (a compiled plan reads it through its view slot). Example: in y03 the three queries' shared scan (node 0, OneEvaluation) is one store read, not three. With one consumer the node streams to it instead (nothing is held), since no sibling can wait on that consumer. Every consumer then reads the finished batches from its own queue. The batches stay charged to the byte budget until every consumer has read them, and going over gives MemoryLimit. Nothing spills. A held node that fails partway (an upstream error or MemoryLimit) clears its consumers' queues, so each gets only the error, never a prefix that looks like success.
    4. Query time, None computed node: one consumer; the stream passes through.
    5. Ingestion time: every node streams, whatever its lifetime (nothing is held). A node with several consumers runs once: every batch is pushed to each consumer's own queue, and any reader may poll the upstream. A consumer that finishes early (a LIMIT) drops its queue. execute_ingestion drives all roots with try_join_all.
    6. Producer::waiters stays, while reader positions and the shared queue are gone. Two readers of one held or ingestion fan-out producer can both see the upstream return Pending, and the upstream wakes only the reader that polled it last. So a reader registers in waiters when it gets Pending, and any push (for a streamed node) or the end wakes it.
    7. A producer's stream is dropped early (its consumers have all left, for example after a LIMIT) only once every consumer the walk counted has subscribed; an operator that drops its input in start does not end the node for a consumer that subscribes after it.
    8. Cancellation: the first producer that sees it cancels every producer of the run (a list of weak references kept per run), so batches a held node keeps for a root nobody polls are released at once; every root then ends with Cancelled. Cancelling sets the failure, clears the unread queues and wakes the readers, but keeps each stream: the producer that saw the cancellation may be borrowed up the poll stack, and an input inside another producer's stream would borrow it again on drop. Each stream is dropped with its node's last input.
  • Correctness: no reader ever waits on another reader's progress. A reader with an empty queue polls the upstream itself, and when it gets Pending it registers its waker so that any reader's progress, or the end, wakes it. The queues are unbounded, so the only remaining limit is the byte budget: a run completes or fails with MemoryLimit, never hangs. Tests:
    • roots_reading_one_leaf_drain_in_order
    • one_root_with_two_branches_over_one_leaf
    • stored_leaf_is_read_per_consumer
    • ingestion_fan_out_reads_the_source_once
    • query_time_fan_out_of_a_not_materialized_node_is_an_error
    • one_evaluation_output_over_budget_is_a_memory_limit
    • cancel_inside_a_held_node_ends_every_root, cancel_inside_an_ingestion_fan_out_ends_the_run
    • the Stage 3 review harness (Verification)
  • Complexity:
    • Time is unchanged per batch: one push per consumer and one pop.
    • Memory at query time: a materialized node with several consumers holds its whole output until its last consumer reads it, where before at most 8 batches were held. The planner already charges this as working memory, though below what the executor holds (Planner charges less memory for a held OneEvaluation output than the executor holds #1301). A materialized node with one consumer streams. Peak memory grows with the input: with 50 series, 12 h of history and 10,000-row raw chunks, the worst case measured was y02 at 10.8 MB against main's 1.25 MB (8.6×); b11 was 5.9×, y03 3.9×, p10 1.9×, y09 1.5×. Query time was unchanged on all five. The only bound is the byte budget.
    • A query-time OneEvaluation scan with k consumers is read once, as on main. A leaf with Lifetime::None or AcrossEvaluations with k consumers is read k times, where before it was read once. So a stored view is read once per consumer, and its reads and bytes read grow with its consumers: on the review workloads view reads went from 1 to 2 (y14, n02, n14, y16), 1 to 3 (g04) and 2 to 3 (b14), and y16's view bytes from 46,656 to 93,312 B. With the in-memory store this adds about 25–40 µs on small queries (n02 +10%, y14 +9%) and nothing measurable elsewhere, while peak memory falls (y16 89,576 → 46,424 B). Against a remote or disk-backed store, k consumers are k remote reads.
    • What Stage 3 charges for these reads: per consumer, never once in total. plan_cost has no separate store-read charge for a view; the view node itself is costed at its own (ingestion) rate, and each consumer is costed for processing the view's rows at its own rate (crates/cost/src/plan/execution_demand.rs, consumers: a walk stops at a materialized node and counts each consumer separately). A query-time None scan is charged once per consumer (its multiplier is its consumer count), and an AcrossEvaluations raw scan is charged its retained rows and no held bytes. So the per-consumer reads match the cost model; no issue filed.
    • Ingestion fan-out holds what the slowest consumer has not yet read. With execute_ingestion polling every root, that is a few batches. An operator that reads one input to its end first holds that input's whole output.
  • Assumptions and limits:
    • The fan-out check runs at execute, before any operator starts, not in compile. A node read at two evaluation times (under an @ anchor) is two producers with one consumer each, and compile has no evaluation time to tell them apart.
    • Leaves are exempt from the check: a Lifetime::None leaf is read per consumer.

Decisions and open questions

Needs reviewer decision

None. Items 1 to 4 below were approved and are kept for reference.

  1. Where the fan-out error is raised. It is raised at execute (and execute_ingestion, where fan-out is allowed), not in compile. Reason: under an @ anchor, one Planner node can be read at two evaluation times with one consumer each, and compile cannot see evaluation times. Example: x @ 100 + x shares x across two times, and that must not be an error.

  2. New public type alias IngestionOutputs<V>, which shortens the execute_ingestion future's type:

    /// Every root's batches of an ingestion run, in the order of its roots.
    pub type IngestionOutputs<V> = Vec<Vec<SharedValue<V>>>;
    // let views: IngestionOutputs<Batch> = dag.execute_ingestion(&[1], ctx)?.await?;
  3. CompiledPhysicalDAG::from_operators keeps main's signature. Every node runs as Lifetime::None, as every caller (tests, and promql_values::compile_scalar) passed. So at query time a computed node may have at most one consumer; a shared computed node needs a PhysicalDAG built with PhysicalDAG::add(…, lifetime). No caller has one today.

  4. Planner charge vs executor memory for OneEvaluation outputs. They do not agree. The executor holds 2.4 to 8 times what the planner charges, at the same row count. The planner charges state or value bytes per row. The executor counts Arrow buffers, including label and identity columns. Measured with the review harness at 12 series; the planner figure is its 1,000-series charge scaled to 12 rows:

    Workload Node Planner charge Executor held
    y16 5, Finalize 32 B/row → 384 B 1,464 B
    b17 3, SummaryMerge 56 B/row → 672 B 3,736 B
    b17 5, Aggregate (1 row) 24 B 200 B
    b17 10, Aggregate (1 row) 24 B 200 B
    g10 6, Aggregate (1 row) 24 B 200 B
    g10 11, Aggregate (1 row) 24 B 200 B
    n02 7, TimeRange over 5 windows 1,680 B 9,320 B
    n02 5, 6, 10, 11, Project (6 rows) 144 B each 696 B each
    n14 3, KLL merge 1,256 B/row → 15,072 B 35,608 B
    n14 7, KLL merge 1,256 B/row → 15,072 B 60,184 B

    The held time the planner charges is about 0.001–0.02 ms, so the cost effect is small. Decided: follow-up Planner charges less memory for a held OneEvaluation output than the executor holds #1301.

Decided

  • Option C (run by Lifetime) replaces the byte-budget patch: no shared streaming fan-out at query time, and no auto-materialization.
  • At query time a leaf runs by its plan lifetime: a leaf with Lifetime::None or AcrossEvaluations (raw data, a constant or a view slot) is one read per consumer; a OneEvaluation leaf with several consumers is one read held for all of them. The "None with more than one consumer" error applies to computed nodes only. (Round 2: an AcrossEvaluations raw scan with several consumers was held before; it is now read per consumer, which is what the planner charges.)
  • A materialized node is held only when it has more than one consumer at its evaluation time.
  • execute_ingestion(roots, context) returns one future with every root's batches, and the caller writes them. execute rejects the Ingestion scope. No DataStore write method is added.
  • SeriesBinary keeps collecting its left input before its right; under this change that is safe.
  • add takes planner_types::stages::Lifetime. No new lifetime type.

Review status

  • Round 1 (quality, functional, efficiency at 3ca7753), all fixed:
    • Quality: holding at ingestion removed (query time only); one run rule (NodeRun) instead of three predicates; one pre-run walk; the stale LIMIT test rewritten at ingestion; execute / execute_ingestion docs state the rejections; the copied test scope dispatch moved into tests/common::run_roots.
    • Functional: L1 (later consumer panicked with "unfinished producer"), L2 (cancel released only the polled producer's batches), L3 (a held node failing partway gave a prefix) fixed in d83cdda, each with a test.
    • Efficiency: a shared OneEvaluation scan is read once and held, and a materialized node is held only with more than one consumer (b945cc2); query time is back to main's.
  • Round 2 (abstraction, layering, functional at cacb7a0), all fixed:
    • One stored-slot rule (layering F1, abstraction A2): a leaf whose rows the store keeps is read per consumer, so an AcrossEvaluations raw scan with several consumers is no longer held, which the planner never charged. View slots carry their plan lifetime; the view-slot constant is deleted. Test first: stored_leaf_is_read_per_consumer and the AcrossEvaluations case of roots_reading_one_leaf_drain_in_order failed before the fix.
    • One lifetime map (A1): CompiledPhysicalDAG::lifetimes; RawDataSlot::lifetime, compiled::Node::Operator's lifetime and the lowering add argument deleted; from_operators back to pairs. The public PhysicalDAG::add keeps lifetime (Design scope).
    • Docs (A3, F2, F7, F3, F6): the rule lives in NodeRun and the README; "the planner charges a None read per consumer" replaced by "Stage 3 never gives a query-time None result more than one consumer"; a lifetime column in the design doc's slot table; the executing-a-plan tutorial names execute_ingestion and execute; forwarder docs fixed.
    • Test helper (A4): every shared node is OneEvaluation, leaves included. This broke 3 kll_tumbling_window_execution tests ("an ingestion-time output cannot be materialized for one evaluation"): views_at moved the now-OneEvaluation shared scan to ingestion time. Fixed in views_at, which now makes such a node Lifetime::None, as Stage 3 would; the tests are unchanged and pass.
    • Small fixes: the planner-bug check states its condition directly (A5) and computes query time from the node's own key (F10); cancel and finish share one end (A6); IngestionOutputs is the batches, used inside Result (A7); query_time_leaves_are_read_per_consumer renamed not_materialized_leaf_is_read_per_consumer and test comments say what the DAG is (A8); ReachedNode, Producer::consumers, the plan::Node::lifetime example (F9); b17 node numbers and the Limits diff (F5, F8); this Design scope section (F4).
    • Functional F1: an operator that cancelled the run inside its own poll, while another consumer of its node was off the poll stack, panicked with "RefCell already borrowed": cancelling the other producers dropped their streams, whose inputs borrowed the producer being polled. Cancelling now keeps each stream until its node's last input drops. Test first: cancel_inside_a_held_node_ends_every_root (query time, OneEvaluation) and cancel_inside_an_ingestion_fan_out_ends_the_run (execute_ingestion, None) both panicked before the fix.
  • Round 3 (abstraction, layering, functional, efficiency at 4c64ca3): all four approved, with no blocker, high or medium finding. The remaining low findings and nits are fixed in this push:
    • Layering L1: the Usage example's ingestion refresh now compiles node 1 on its own and runs execute_ingestion(maintenance.roots(), …).
    • Layering L2: the from_operators doc and Decision 3 say every node runs as Lifetime::None, so a shared computed node needs PhysicalDAG::add(…, lifetime).
    • Abstraction N3, layering N1, efficiency E3: lifetimes is built only from the compiled (reached) nodes; its doc and the Design scope say so.
    • Abstraction N1: the hold_until_finished doc says "a [NodeRun::Held] node". Abstraction N2: the views_at doc gives the reason, the DAG's validation rejecting OneEvaluation at ingestion.
    • Layering N2–N4: the README and the Producer doc say a leaf read per consumer runs once per consumer; a held computed AcrossEvaluations node is described as occurring only in a hand-built DAG; constants and other source leaves are described as holding their own rows, not as store-kept. Algorithm steps 2–3 and the Decided list follow. The tutorial now says Scope::Ingestion / Scope::Query consistently.
    • Functional N1: the cancel doc says operator state inside a kept stream is freed when the caller drops the run's inputs.
    • Efficiency E1: the Complexity section states that a stored view is read once per consumer, with the measured cost, and what Stage 3 charges for it (per consumer; no mismatch). E2: the 8.6× worst-case peak and Planner charges less memory for a held OneEvaluation output than the executor holds #1301 are in "Memory at query time".
    • Functional N2 (three base tests removed) needed no change; the new tests listed in Verification cover them.

Status: rounds 1–3 complete. Ready to merge.

Verification

Scope Passed / failed / ignored Newly failing vs main 59365ce Notes
cargo test -p asap-executor --features test-support (lib, executor suite, memory-limit suites, README doc test) 295 / 0 / 1 none includes the 3 new round-2 tests; rerun at d2d5f60 with the same result
cargo test -p asap-types (lib, suite, doc) 242 / 0 / 0 none includes the physical_asap_dag_at_query_time_with_node_ids change
cargo test -p asap-planner (lib, planner suite, doc tests) 31 / 2 / 1 none the 2 (check_compiles_reports_the_executors_error, identical_ungrouped_queries_share_their_producers) fail on main 59365ce; the tutorials have no code blocks, so 0 doc tests
cargo test -p asap-integration-tests --features asap-executor/test-support 163 / 7 / 4 none (rerun at d2d5f60, same 7) the 7 (joint_search_equivalence 2, materialized_views 1, plan_sharing 2, promql_to_post_asap 2) are main's 7; the 3 kll_tumbling_window_execution tests the helper change broke were fixed in views_at and pass
cargo clippy -p asap-executor -p asap-types -p asap-planner -p asap-integration-tests --all-targets --features asap-executor/test-support -- -D warnings, and -p asap-executor --all-targets without the feature clean
cargo fmt --all --check clean

New tests (crates/executor/src/runtime/tests.rs). Each hang-prone one runs on its own thread with a 10 s timeout, so a hang fails instead of blocking:

  • roots_reading_one_leaf_drain_in_order: two roots read one leaf that emits 20 batches; draining them in turn equals join_all. Runs with the leaf Lifetime::None and AcrossEvaluations (2 store reads per run) and OneEvaluation (1 read, held). Hangs on main (timeout).
  • one_root_with_two_branches_over_one_leaf: one root, LeftThenRight (collects left, then right, as SeriesBinary does) over two readers of one leaf. Hangs on main.
  • stored_leaf_is_read_per_consumer: an AcrossEvaluations leaf read by two roots is read twice, and only the batch being read is charged (8 B, not the 32 B a held read would keep).
  • not_materialized_leaf_is_read_per_consumer and materialized_node_runs_once_and_is_held: replace the old backpressure and diamond tests.
  • ingestion_fan_out_reads_the_source_once: one ingestion read feeds three consumers (one through LeftThenRight). The source starts once and every root gets all batches. The main-API form of the same run hangs.
  • query_time_fan_out_of_a_not_materialized_node_is_an_error: a None node with two consumers is an error naming node 1, and no source starts. Read at two evaluation times, the same node runs.
  • one_evaluation_output_over_budget_is_a_memory_limit: a held output over a 40-byte budget ends in MemoryLimit, and nothing stays reserved.
  • execute_rejects_ingestion_scope_and_vice_versa: each entry point rejects the other scope with a message naming both calls.
  • limit_consumer_of_an_ingestion_fan_out_drops_its_read (physical_semantics, replaces limit_branch_finishes_without_blocking_shared_sibling): at ingestion, a LIMIT consumer of a fan-out node drops its queue after its row; the run's peak stays at its 101 output batches (it is 40,000 B instead of 20,200 B when the drop is disabled).
  • an_input_dropped_at_start_leaves_the_node_to_later_consumers: a held node read by an operator that drops its input in start and by an Identity; the Identity still gets all four batches.
  • cancellation_releases_held_batches_of_unpolled_roots: after one root reads a batch and the run is cancelled, polling that root releases everything the held node kept for the other root (retained bytes 0), and the other root ends with Cancelled.
  • cancel_inside_a_held_node_ends_every_root and cancel_inside_an_ingestion_fan_out_ends_the_run: an operator cancels the run inside its own poll after 3 batches; every root ends with Cancelled and nothing stays reserved.
  • broadcast_error_and_cancel (changed): a held node whose source fails at its second batch gives each consumer only the error, not the first batch.
  • one_evaluation_node_with_one_consumer_streams: a OneEvaluation node with one consumer gives its first batch after one source batch, with 8 B peak, instead of after the whole output.

End to end: the Stage 3 review harness (#1236's, not committed) planned and executed 103 workloads at cacb7a0: the round-7 set plus b17, g10, n02 and n14. Every root was drained in turn, not with join_all. The round-2 functional review reran all 103 at cacb7a0 with the same result.

Outcome Workloads
Exact answers equal the exact reference and the round-7 run 58
Exact answers equal the reference; hung or not run before (b17, g10, n02) 3
Approximate answers within target, same as round 7 10
Approximate answers within target, differ from round 7 (b13, y09: KLL compaction is random; two runs of the same binary differ) 2
Approximate, new (n14) 1
Planned, but no answers, as in round 7 (missing view window: z02/z03/z05/b19/g06; ...) 6
Plan or install errors, as in round 7 22
Harness input error (p34: old fixed_interval workload format), as in round 7 1

y16, which needed join_all in round 7, now completes with roots drained in turn and the same answers.

Query time and reads after the round-2 fixes

The same harness, 15 workloads and settings as the round-1 table below (12 series, 2 h, minimum of 7 query runs, arms interleaved per workload, roots drained in turn): the round-2 head rebased onto 59365ce (b945cc2) against this push (built before a final amend that changes only a doc comment).

workload store reads: before / after query time (µs): before / after query peak
b07 raw 1 / 1 468 / 468 6.6 MB, same
b11 raw 1 / 1 1629 / 1657 6.6 MB, same
b14 view 3 / 3 451 / 457 17.8 KB, same
c05 raw 1 / 1 590 / 589 7.7 MB, same
g04 view 3 / 3 1953 / 1954 53.7 KB, same
g14 raw 1 / 1 1381 / 1370 6.6 MB, same
n02 view 2 / 2 247 / 252 14.5 KB, same
n14 view 2 / 2 1475 / 1485 87.2 KB, same
p10 raw 1 / 1 1172 / 1148 6.6 MB, same
y02 raw 1 / 1 3075 / 3119 6.6 MB, same
y03 raw 1 / 1 1522 / 1521 6.6 MB, same
y09 raw 1 / 1 1350 / 1352 6.6 MB, same
y14 view 2 / 2 208 / 202 11.3 KB, same
y15 raw 1 / 1 1685 / 1664 6.6 MB, same
y16 view 2 / 2 536 / 561 46.4 KB, same
  • Store reads and query peak are identical on every workload, and query time is within noise (the largest relative difference is y16, 25 µs or 4.7%); ingestion time and peak are unchanged too.
  • Answers are identical except n14 and y09, the KLL workloads whose compaction is random.
  • None of these plans has an AcrossEvaluations raw scan with several consumers, so the stored-slot rule changes no read here; its effect is covered by stored_leaf_is_read_per_consumer.

Query time and memory after the leaf-lifetime fix (round 1)

The review harness at 12 series and 2 h of data, on the 15 workloads where the branch before that fix read the store more than main. Each time is the minimum of 7 query runs. The arms were interleaved per workload: main 4de4075 (roots joined), the branch before the fix (48819b4, roots in turn) and after it (cacb7a0, roots in turn).

workload store reads: main / before / after query time (µs): main / before / after query peak bytes: main / before / after
b07 raw 1 / 2 / 1 469 / 824 / 467 6.6 MB, all three
b11 raw 1 / 2 / 1 1636 / 2267 / 1612 6.6 MB, all three
c05 raw 1 / 2 / 1 601 / 935 / 596 7.7 MB, all three
g14 raw 1 / 2 / 1 1366 / 1964 / 1359 6.6 MB, all three
p10 raw 1 / 3 / 1 1148 / 1906 / 1161 6.6 MB, all three
y02 raw 1 / 2 / 1 3076 / 3471 / 3074 6.6 MB, all three
y03 raw 1 / 3 / 1 1525 / 2541 / 1600 6.6 MB, all three
y09 raw 1 / 2 / 1 1375 / 1727 / 1357 6.6 MB, all three
y15 raw 1 / 2 / 1 1647 / 2042 / 1682 6.6 MB, all three
b14 view 2 / 3 / 3 461 / 464 / 451 42.0 / 17.8 / 17.8 KB
g04 view 1 / 3 / 3 1877 / 1931 / 1925 135.5 / 53.7 / 53.7 KB
n02 view 1 / 2 / 2 245 / 250 / 244 24.4 / 14.5 / 14.5 KB
n14 view 1 / 2 / 2 1484 / 1477 / 1479 87.2 KB, all three
y14 view 1 / 2 / 2 201 / 207 / 203 15.4 / 11.3 / 11.3 KB
y16 view 1 / 2 / 2 532 / 549 / 549 89.6 / 46.4 / 46.4 KB
  • Every shared raw data scan in these plans is OneEvaluation, so it is one read, and query time is back to main's, within noise.
  • View slots stay one read per consumer. That costs nothing measurable, and the peak is lower than main's, because each read streams.
  • Answers are identical on all three arms, except n14 and y09: these are KLL, and their compaction is random.

At 200 series (53 MB per scan), query time in ms, main / before / after: b07 6.8 / 22.2 / 7.0, g14 22.5 / 34.9 / 23.0, p10 16.8 / 30.1 / 17.1, y03 22.9 / 39.6 / 23.7, y15 25.4 / 59.7 / 25.5. Reads and peak are the same as main.

At 50 series and 12 h, with raw data stored in 10,000-row batches (88 MB per scan, about 430 batches):

workload reads: main / before / after query time (ms): main / before / after query peak: main / before / after
b07 1 / 2 / 1 11.5 / 20.8 / 11.4 1.89 / 1.35 / 1.52 MB
p10 1 / 3 / 1 14.0 / 32.8 / 14.5 0.91 / 0.83 / 1.76 MB
y03 1 / 3 / 1 17.5 / 72.6 / 16.4 0.99 / 3.82 / 3.82 MB

The peak in this case is the held scan's filtered output. That is the memory the plan charges for a OneEvaluation scan, in exchange for one 88 MB read instead of two or three.

Lifetime flips on real plans

The review harness can override node lifetimes before compiling (RV_FLIP, a local change only). Results at cacb7a0:

  • y11, node 3 (SeriesLabels, 2 consumers) set to None: rejected with node 3 (SeriesLabels) at evaluation time 1700002800000 has 2 consumers but Lifetime::None; a shared query-time output must be OneEvaluation (planner bug).
  • y15, node 5 (TimeRange) or node 8 (ScopeTimestamp) set to None: rejected the same way.
  • y15, node 0 (the shared scan) set to None: the same answers, which equal the reference within 1e-9. It takes 2 store reads instead of 1.
  • b17, nodes 3, 5 and 10 set to None, or nodes 2, 4 and 11 set to OneEvaluation: the same answers, which equal the reference.
  • y16, nodes 2, 3 and 8 set to OneEvaluation: the same answers, which equal the reference.

Follow-ups

Closes #1290

@zzylol zzylol added the bug Something isn't working label Nov 14, 2026
@zzylol
zzylol force-pushed the fix/1290-executor-shared-producer branch from 35f5e57 to 48819b4 Compare November 14, 2026 20:00
@zzylol
zzylol force-pushed the fix/1290-executor-shared-producer branch from cacb7a0 to 4c64ca3 Compare November 15, 2026 07:02
…any order

A shared producer kept one queue bounded by `max_buffered_batches`; a
reader ahead of the others waited for them, so draining root streams one
after another (y16), or a SeriesBinary collecting its left branch before
its right over one stored chain (b17, g10), hung forever.

The runtime now runs each node by the plan's `Lifetime` and the run's
scope:
- query time, a leaf (raw data or view slot): one store read per consumer;
- query time, OneEvaluation or AcrossEvaluations: computed to the end
  once, then read by every consumer, charged to the byte budget;
- query time, None: one consumer; more is an error naming the node;
- ingestion time: a node with several consumers runs once and each batch
  reaches every consumer; `execute_ingestion` polls every root and
  returns one future with each root's batches.

`PhysicalDAG::add` takes the node's `Lifetime`; compiling a
PhysicalASAPDAG passes each node's own. `Limits::max_buffered_batches`,
reader positions and the batch-count backpressure are removed.

Closes #1290
Address the quality review of the Lifetime runtime:
- `run_as` returns `NodeRun { ReadPerConsumer, Held, Streamed }`, the one
  rule for per-consumer leaf reads, held outputs and the query-time fan-out
  check; holding now happens at query time only, so ingestion streams.
- `start` walks the DAG once (`reach`): validates each node's run context,
  counts its consumers and records its inputs' context per evaluation time.
- Error messages name the call for a wrong scope, and the fan-out error
  gives the evaluation time and says the node should be OneEvaluation.
- The LIMIT test runs at ingestion and checks that the LIMIT consumer of a
  fan-out node drops its read; scope rejections get their own test.
- Docs on `add`, `add_boxed`, `execute`, `execute_ingestion` and the README
  state the rejections; test helper and doc nits.
… partial held output

Address the functional review:
- An operator may drop its input in `start` before the node's next
  consumer subscribes; the stream is now dropped early only once every
  expected consumer has subscribed and left, instead of panicking later.
- A cancellation seen by one producer cancels every producer of the run,
  releasing batches a held node keeps for consumers nobody polls.
- A held node that fails partway clears its consumers' queues, so each
  gets only the error rather than a prefix that looks like success.
Address the efficiency review:
- A raw data slot carries its Scan node's plan lifetime. At query time a
  OneEvaluation scan with several consumers is read once and held for
  all of them, as the planner charges one read whose output is held; only
  a Lifetime::None scan (and a view slot, whose states the store keeps)
  is read once per consumer.
- A materialized node is held only when it has more than one consumer at
  its evaluation time; with one consumer it streams, since no sibling can
  wait on that consumer.
- README and runtime docs say which leaves are read per consumer, and
  that ingestion queues are bounded only by the byte budget.
… drop on cancel

- A query-time leaf whose rows the store keeps (`None` or
  `AcrossEvaluations`) is read per consumer; a `OneEvaluation` node, or a
  computed `AcrossEvaluations` node, with more than one consumer is held.
  View slots carry their plan lifetime; the view-slot lifetime constant is
  gone. An `AcrossEvaluations` raw scan with several consumers is no
  longer held, which the planner never charged.
- `CompiledPhysicalDAG` keeps every Planner node's lifetime in one map;
  `RawDataSlot::lifetime`, the operator node's lifetime and the extra
  lowering argument are gone, and `from_operators` takes main's pairs.
- Cancelling the other producers of a run no longer drops their streams
  while the producer that saw the cancellation is borrowed, which
  panicked with "already borrowed" when an operator cancelled inside its
  own poll.
- The fan-out check states its condition directly; `ReachedNode` names
  the walk's per-node facts; `Producer::consumers`; `IngestionOutputs` is
  the batches, used inside `Result`.
- The query-time test helper marks every shared node `OneEvaluation`,
  leaves included; `views_at` makes a `OneEvaluation` node it moves to
  ingestion time `None`.
- Docs: the read rule lives in `NodeRun` and the README; the design doc's
  slot table has a lifetime column; the executing-a-plan tutorial names
  `execute_ingestion` and `execute`.
- Build CompiledPhysicalDAG::lifetimes from the nodes the roots reach.
- from_operators doc: every node runs as Lifetime::None, so a shared
  computed node needs PhysicalDAG::add with a lifetime.
- NodeRun, Producer, README: a leaf read per consumer runs once per
  consumer; constants hold their own rows; a held computed
  AcrossEvaluations node occurs only in a hand-built DAG.
- cancel doc: operator state in a kept stream is freed when the caller
  drops the run's inputs.
- views_at doc gives the validation reason; tutorial uses
  Scope::Ingestion / Scope::Query consistently.
The grouped-limit, exact top-k and chain tests added on main build a
PhysicalDAG with three-argument add calls; each node they add is read by
one consumer, so it runs with Lifetime::None.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Executor deadlocks when two query roots that share a producer are drained one after another

1 participant