Skip to content

refactor(stage2): physical expansion (decision-tree levels 3–4), physical edges, pruned alternatives; Count-Min weights via column_signs (#1207 PR 5) - #1235

Merged
zzylol merged 6 commits into
mainfrom
refactor/1207-stage2
Nov 12, 2026
Merged

zzylol merged 6 commits into
mainfrom
refactor/1207-stage2

Conversation

@zzylol

@zzylol zzylol commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: Stage 2 was a todo! stub: Pass 2's logical candidates got no timing or lifetime, no physical edges and no pruned list, and Count-Min weights were "proven" non-negative by a per-update WeightDomain proof that Pass 1 set and the executor checked.
  • Change: Stage 2 expands each logical candidate Pass 2 kept into one physical candidate per offered (τ, ℓ), each logical edge into physical edges with α^log and α^phys, and records every pruned alternative with its reason (six reasons). The Count-Min check reads the build's state group's column_signs; WeightDomain and its plumbing are deleted.
  • Effect: for quantile_over_time(0.95, latency[1m]) every 60 s (query A), Stage 2 emits 5 alternatives for the scan, 4 for the chain and 3 for each time-dependent candidate; across 88 review workloads it emits 12,166 physical candidates, 52,956 edges (50,632 admissible) and 9,584 pruned alternatives, with 0 invariant violations except one unservable workload.

Review guide

Behavior: before → after

Query A, quantile_over_time(0.95, latency[1m]) evaluated every 60 s at phase 0, ingesting data, everything retained (test offers_each_candidate_its_timings_and_lifetimes). Before this PR, Stage 2 was todo!().

Group Candidate After: offered (τ, ℓ)
Scan latency (time-independent, no grid) scan (QT, None), (QT, One), (QT, Across), (IT, None), (IT, Across)
latency[1m] range (QT, None), (QT, One), (QT, Across)
KLL state build over the minute; merge over the chain's windows (QT, None), (QT, One), (QT, Across) each
Chain, 60 s windows (time-independent) chain build (QT, None), (QT, One), (QT, Across), (IT, Across); (IT, None) is not offered to a build
TimeRange 1m over the chain windows (QT, None), (QT, One), (QT, Across)
p95 readout (root) readout (QT, None), (QT, One), (QT, Across)

With the default deployment (query_time_results_retained: false) every (QT, Across) is pruned QueryTimeResultsNotRetained. With raw_data_retained: false the scan keeps only (IT, None), the build over the minute is pruned InputHasNoBuildableProducer, and only the ingestion-time chain serves the query.

The two review fixes, on max_over_time(rate(c[1m])[5m:1m] @ 90) (dump hook, before = a82bf38, after = this head):

  • every 60 s at phase 0 (x18): before, the rate, its range, the scan, the chain and its windows below the @ 90 subquery were offered (·, AcrossEvaluations) with unbounded retention; after, none of them is.
  • fixed interval without a phase (p34, no grid): before, the Max state over the pinned window got only (QT, None) and (QT, One), so a constant was recomputed every evaluation; after, it is offered (QT, Across) and (IT, Across), and the ingestion-time subtree under it is reached.

Interfaces

crates/stage2-physical-optimization/src/lib.rs

 /// Stage 2: expand every logical candidate that survived Pass 2 into its
 /// physical alternatives, every logical edge into its physical edges with
 /// α^log and α^phys, and record every rejected alternative with its reason.
 pub fn generate_physical_asap_memo_candidates(
     logical: &LogicalASAPMemoCandidates,
     data: &DataWorkload,
     capabilities: &DeploymentCapabilities,
 ) -> PhysicalASAPMemoCandidates;

The signature is unchanged; the body replaces todo!().

crates/types/src/stages/physical/edge.rs

+impl PhysicalEdge {
+    /// Whether a plan may use the edge: α^log = α^phys = 1. Read by Stage 2's
+    /// pruning, the admissible-edge indexes and Stage 3.
+    pub fn is_logically_and_physically_admissible(&self) -> bool;
+}

crates/types/src/stages/physical/memo_candidates/mod.rs

 impl PhysicalASAPMemoCandidates {
     /// Stage 2's output over `physical_candidates` and `edges` (grouped by
     /// (consumer, input)), with the admissible-edge indexes built once.
     pub fn new(
         logical_candidates: LogicalASAPMemoCandidates,
         physical_candidates: Vec<PhysicalCandidate>,
         edges: Vec<PhysicalEdge>,
         pruned_candidates: Vec<PrunedCandidate<PhysicalCandidate>>,
     ) -> Self;
     /// P^adm_{c,i}: the admissible edges into input `input_index_of_consumer` of `consumer`.
     pub fn admissible_edges_into(&self, consumer: PhysicalCandidateId, input_index_of_consumer: usize) -> &[PhysicalEdgeId];
     /// The admissible edges out of `producer`: every consumer that may share it.
     pub fn admissible_edges_out_of(&self, producer: PhysicalCandidateId) -> &[PhysicalEdgeId];
 }

Signatures unchanged; bodies replace todo!.

crates/types/src/stages/pruning.rs

 /// Why an alternative was pruned (stays).
 pub enum PruneReason {
     NoIngestionPipeline,
-    /// Stage 2: it scans raw data at query time, and the deployment does
-    /// not retain raw data, so no window of it is there to scan.
+    /// Stage 2: it scans raw data at query time, or retains the raw rows it
+    /// scans across evaluations, and the deployment does not retain raw
+    /// data, so no window of it is there to scan or keep.
     RawDataNotRetained,
-    /// Stage 2: it is a query-time result materialized (`OneEvaluation` or
-    /// `AcrossEvaluations`), and the deployment retains no query-time
-    /// results.
+    /// Stage 2: it is a query-time result materialized across evaluations
+    /// (`AcrossEvaluations`), and the deployment retains no query-time
+    /// result across evaluations
+    /// (`DeploymentCapabilities::query_time_results_retained`).
     QueryTimeResultsNotRetained,
     InputHasNoBuildableProducer,
     ReachesNoQueryOutput,
     CountMinWeightsNotProvenNonNegative,
     UsageNotCostable,
 }

(Only the doc lines of these two variants change.)

crates/types/src/summary_type/update/mod.rs (moved from summary_type/statistic.rs; re-exported from asap_types::summary_type as before)

 /// One summary-state update (stays).
 pub struct SummaryUpdate {
     pub item: Option<SummaryInputExpr>,
     pub weight: SummaryInputExpr,
-    #[serde(default)]
-    pub weight_domain: WeightDomain,
 }
-pub enum WeightDomain {
-    UnknownOrSigned,
-    NonNegative { proof: NonNegativeWeightProof },
-    RawDataSamplesOfMetric { metric: String },
-    CounterDerivativeOfMetric { metric: String },
-}
-pub enum NonNegativeWeightProof {
-    UnitCount,
-    ResetAwareCounterDerivative,
-    CounterRawDataSamples,
-    DeclaredNonNegativeRawDataSamples,
-    PositiveConstant,
-}
-pub fn raw_data_samples_of_metric<I: OperatorInput>(input: &I) -> Option<String>;

crates/types/src/stages/logical/group/mod.rs

-pub fn raw_data_samples_of_metric_of_group(memo: &Memo, group: MemoGroupId) -> Option<String>;

Removed: the weight sign is the state group's column_signs, which Stage 2 reads; a second, per-update derivation could disagree with it. Old JSON with weight_domain still reads (the field is ignored).

Executor: validate_hydra_cms, unit_frequency_column and create_planner_accumulator no longer read a weight proof; a negative Count-Min weight is still refused at update time (WeightedFrequencyUpdater::update → check_weight).

Usage (end to end over query A; the CLI cannot print Stage 2 at this PR, so this is the unit test's flow):

// Pass 2's output for query A: scan, its last minute, the KLL state (build
// over the minute + merge over the chain's windows), the 60 s chain, the
// windows, the p95 readout; grids 60 s for time-dependent groups.
let example = example();
// Ingesting data at 1000 rows/s, no out-of-order bound; raw data and
// query-time results retained.
let physical = generate_physical_asap_memo_candidates(
    &example.logical, &ingesting(None), &retaining_everything());
// The chain's alternatives: (QT, None), (QT, One), (QT, Across), (IT, Across).
let chain: Vec<_> = physical.physical_candidates.iter()
    .filter(|c| c.logical_candidate_id == example.chain).collect();
// Edges into the chain: 4 chain alternatives × 5 scan alternatives = 20;
// Stage 3 reads only the admissible ones, e.g. into the IT chain only the IT scan.
let ingestion_time_chain = physical.physical_candidates.iter()
    .position(|c| c.logical_candidate_id == example.chain
        && c.physical_properties.timing() == ExecutionTiming::IngestionTime)
    .map(PhysicalCandidateId).unwrap();
let into_ingestion_time_chain = physical.admissible_edges_into(ingestion_time_chain, 0);
// Nothing is pruned here; with DeploymentCapabilities::default() every
// (QT, Across) would be in physical.pruned_candidates as QueryTimeResultsNotRetained.

Algorithm changes

Stage 2 expansion and pruning (new)

  • Problem: generate_physical_asap_memo_candidates was todo!().
  • After:
    1. Per group, once (group.rs): ingestion_time_offered_per_group = time-independent groups and, transitively, the inputs of their kept candidates; across_evaluations_offered_per_group = chain, known grid, or time-independent, and not read under an absolute @ T (below a TimeShift anchored at a timestamp through any operator except another @).
    2. Per kept logical candidate (offered_physical_properties): every valid (τ, ℓ) in {QT, IT} × {None, One, Across} the group offers, IT only for a counter state over ordered rows (PromQL samples or a declared out-of-order bound), no (IT, None) for a summary build; each with the first prune reason that applies: CountMinWeightsNotProvenNonNegative, NoIngestionPipeline, RawDataNotRetained, QueryTimeResultsNotRetained.
    3. expand_edges: each logical edge × the producer's and consumer's alternatives, α^log copied, α^phys = producer_satisfies_physical_requirements.
    4. prune_reasons: least fixed point of "every input has a buildable admissible producer" (InputHasNoBuildableProducer), then reach from query outputs over admissible edges (ReachesNoQueryOutput).
    5. Renumber survivors, keep edges between survivors, sort by (consumer, input), index admissible edges.
  • Correctness: every alternative is either emitted or recorded with a reason; (logical candidate, τ, ℓ) is unique; every emitted candidate has an admissible producer on each input and is reached from a query output. Unit tests in src/tests/; an independent checker over 88 workloads (Verification).
  • Complexity: O(Σ alternatives + Σ physical edges) for steps 1–3 and 5; step 4's fixed point is O(depth · Σ admissible edges) (a counter worklist would be linear; not needed at current sizes). Physical edges are at most 5 × 5 per logical edge.
  • Assumptions and limits: no sizing, no costs; alignment with each execution context is Stage 3's delivers_on; retention $ is PR 7.

Decisions and open questions

Needs reviewer decision: None.

Decided:

  • 2026-11-07: Count-Min prune via the state group's column_signs; time indexing (any grid-evaluated group may be AcrossEvaluations); no executor-support check anywhere.
  • 2026-11-09: WeightDomain deleted (rejected: a per-update proof set by Pass 1 and checked by the executor). PhysicalEdge::is_logically_and_physically_admissible (rejected: is_admissible).
  • QueryTimeResultsNotRetained prunes only (QT, AcrossEvaluations) (rejected: OneEvaluation too, which with default capabilities removes all within-evaluation sharing).
  • Ingestion time is offered through inputs transitively (rejected: one level, which could build nothing below an @ T).
  • A query reads its output as a query-time consumer: (IT, None) of a root is ReachesNoQueryOutput.
  • 2026-11-09: no summary build (SummaryAgg, chain, MaintainPopulation) is offered (IT, None): fed at ingestion it keeps its state.
  • RawDataNotRetained also rejects a scan materialized across evaluations at ingestion time (it retains raw rows).
  • 2026-11-12 (review R1): no AcrossEvaluations for any group below an absolute @ T, through any operator other than another @; a group also read on a relative path counts as below it. Rejected: only the anchor's direct child (left the rate, range, scan and chain of max_over_time(rate(c[1m])[5m:1m] @ 90) with unbounded retention); leaving it all to PR 7's $.
  • 2026-11-12 (review R2): a time-independent group with no evaluation grid is offered AcrossEvaluations, following the concepts doc's alignment table; the doc's decision-tree row now matches.
  • The executor no longer refuses rate-weighted updates at bind time; typed binding of window-weighted heaps is PR 6.

Verification

Scope Passed / failed / ignored Newly failing vs main 70b0abc Notes
asap-stage2-physical-optimization 15 / 0 / 0 0 main: 0 tests
asap-types 149 / 0 / 0 0 main: 148
asap-stage1-pass1-per-computation-candidates 37 / 0 / 0 0
asap-cost lib 31 / 1 / 0 0 the failure is main's PR 6 todo!
asap-executor (lib + tests) 203 / 61 / 1 0 all failures also fail on main (PR 6 todo!s)
cargo test --workspace --no-fail-fast 1231 / 183 / 9 (after the rebase over #1263: 1232 / 183 / 9, same failing names) 0 main: 1216 / 184 / 9; failing-name diff: none new; hydra_cms_execution::unknown_sign_weight_is_rejected is deleted with the bind-time sign check
cargo fmt --check, cargo clippy -D warnings on touched crates clean

New tests:

  • every_group_below_an_absolute_anchor_is_not_materialized_across_evaluations: (x[1m]) @ T; neither the range nor the scan has an AcrossEvaluations alternative (failed before the R1 fix).
  • a_time_independent_group_without_a_grid_is_materialized_across_evaluations: a Max state over a pinned minute with no grid is offered (QT, Across) and (IT, Across), and the pinned minute's (IT, None) is kept (failed before the R2 fix).
  • memo_candidates::tests::{new_indexes_only_admissible_edges, an_edge_is_admissible_when_both_requirements_hold}: the indexes list exactly the admissible edges.
  • Stage 2's other 11 tests cover each offered set and prune reason, edges and pruning (src/tests/{candidates,edges,pruning}.rs).

End to end (uncommitted dump hook in asap_planner, 90 workloads: 71 PromQL, 19 SQL; 88 reach Stage 2, 2 are refused earlier as capability contradictions): an independent checker recomputes the offered sets and reasons and checks edges, indexes, buildability and reach.

before (a82bf38) after (this head)
groups below an absolute @ T with an AcrossEvaluations alternative (R1) 12 0
time-independent groups without a grid and without AcrossEvaluations (R2) 6 0
workloads with violations under the updated rules 5 1 (p32: no phase and no raw data, no plan can serve it)
physical candidates / edges / admissible / pruned 12,164 / 52,946 / 50,628 / 9,586 12,166 / 52,956 / 50,632 / 9,584

Snapshots: weight_domain removed from 13 corpus and 10 sketch-plan snapshot files (the field no longer exists; JSON otherwise identical, checked by parsing). No other snapshot changes.

Follow-ups

@zzylol zzylol added optimizer Workload optimization and plan selection refactor labels Nov 7, 2026
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch 2 times, most recently from bdf0888 to bcae9e5 Compare November 9, 2026 07:11
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from d11f12e to 616f765 Compare November 9, 2026 07:43
@zzylol zzylol changed the title refactor(stage2): physical expansion (decision-tree levels 4–5), physical edges, pruned_candidates (#1207 PR 5) refactor(stage2): physical expansion (decision-tree levels 3–4), physical edges, pruned alternatives; Count-Min weights via column_sign (#1207 PR 5) Nov 9, 2026
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from 616f765 to f20177e Compare November 9, 2026 11:53
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from bcae9e5 to 8107319 Compare November 9, 2026 17:11
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from f20177e to e05016c Compare November 9, 2026 17:44
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from 8107319 to 5d3f6de Compare November 9, 2026 20:52
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from e05016c to 513a1e7 Compare November 9, 2026 21:28
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from 5d3f6de to dea088e Compare November 10, 2026 01:26
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from 513a1e7 to 4c77ec6 Compare November 10, 2026 01:26
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from dea088e to e98e80b Compare November 10, 2026 13:37
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch from 4c77ec6 to 8876072 Compare November 10, 2026 13:37
@zzylol
zzylol force-pushed the refactor/1207-pass2 branch from e98e80b to 5ac57fc Compare November 11, 2026 18:33
@zzylol
zzylol force-pushed the refactor/1207-stage2 branch 2 times, most recently from a82bf38 to 07678ec Compare November 12, 2026 19:19
@zzylol
zzylol changed the base branch from refactor/1207-pass2 to main November 12, 2026 19:19
@zzylol zzylol changed the title refactor(stage2): physical expansion (decision-tree levels 3–4), physical edges, pruned alternatives; Count-Min weights via column_sign (#1207 PR 5) refactor(stage2): physical expansion (decision-tree levels 3–4), physical edges, pruned alternatives; Count-Min weights via column_signs (#1207 PR 5) Nov 12, 2026
…natives; Count-Min weights via column_signs (#1207 PR 5)

Stage 2 expands each logical candidate Pass 2 kept into its offered
(timing, lifetime): ingestion time for time-independent groups and their
inputs (counter states only over ordered rows), AcrossEvaluations for
chains and grid-evaluated groups not read under an absolute @ T, and no
(IngestionTime, None) for a summary build. Prune reasons:
NoIngestionPipeline, RawDataNotRetained (query-time scans and scans kept
across evaluations), QueryTimeResultsNotRetained (query time across
evaluations), CountMinWeightsNotProvenNonNegative, InputHasNoBuildableProducer,
ReachesNoQueryOutput. Physical edges carry alpha^log and alpha^phys;
PhysicalASAPMemoCandidates indexes the admissible ones
(PhysicalEdge::is_logically_and_physically_admissible). Pass 2's output,
exact_only_groups_of_query included, is carried whole.

Crate layout: lib.rs (entry, ingestion-time offer), candidates.rs
(decision-tree levels 3 and 4), pruning.rs (buildable and reached fixed
points), tests.rs over the shared memo builders.

Deletes WeightDomain, NonNegativeWeightProof, SummaryUpdate::weight_domain,
raw_data_samples_of_metric(_of_group) and Pass 1's
with_raw_data_samples_requirement (the state group's column_signs
replaces them), and the executor's weight-domain checks (CMS contract
check, factory rate-weight refusal, Hydra unknown-sign check) with their
tests.
… modules; named offered physical properties; tests split by step over shared fixtures (#1207 PR 5)

lib.rs keeps the entry point and the renumbering. candidate.rs (was
candidates.rs) holds levels 3 and 4: offered_physical_properties returns
OfferedPhysicalProperties { physical_properties, prune_reason } instead
of an anonymous pair, and one summary_build helper serves the counter
order and Count-Min rules. group.rs computes the per-group offers once
(ingestion time, across evaluations). edge.rs expands the logical edges.

Tests: tests.rs (module list and output checkers), test_fixtures.rs
(query A and Pass 2's output by hand, over memo_fixtures range, state,
readout and shift), tests/{candidates,edges,pruning}.rs. The tests of
PhysicalASAPMemoCandidates::new and its admissible-edge indexes move to
the types crate's memo_candidates/tests.rs, whose fixture now goes
through new. No dates, review tags or issue numbers in comments.
… update types in their own module (#1207 PR 5)

Inline test modules of the files this PR touches move to sibling files
(state_bytes, capability, summary_kernels::factory, ir::schema and
ir::schema::topk). SummaryUpdate, SummaryInputExpr and EntityIdentity
move from summary_type::statistic to summary_type::update; statistic
keeps SketchStatistic. Re-exports are unchanged.
…ossEvaluations for a time-independent group without a grid (#1207 PR 5)

The rule that a group read under an absolute @ T is not materialized
across evaluations looked only at the anchor's direct child (through
offset-only time shifts). In max_over_time(rate(c[1m])[5m:1m] @ 90) the
rate, its range, the scan and the chain below the subquery were still
offered AcrossEvaluations with unbounded retention. The rule now holds
for every group below the anchor, through any operator other than
another @, computed once per group over the candidates Pass 2 kept.

A time-independent group with no evaluation grid (a result under @ T on
a fixed interval without a phase) is now offered AcrossEvaluations: its
one result is readable at any time, as the alignment table says. Before,
such a state was rebuilt every evaluation and the ingestion-time
alternatives under it were pruned as reaching no query output.

Tests: every_group_below_an_absolute_anchor_is_not_materialized_across_evaluations
and a_time_independent_group_without_a_grid_is_materialized_across_evaluations
fail before this change; the query A, Count-Min, raw-data and @ T tests
now expect the scan and states without a grid to be offered
AcrossEvaluations.
…gical candidate, all six prune reasons, its modules and its one crate edge (#1207 PR 5)

The contracts doc's Stage 2 section described materialization options for
summary builds only and P<n>-m<k> candidate names; it now states what is
offered per logical candidate (ingestion time, across evaluations, the
@ T rule) and lists all six prune reasons. planner-layering prunes per
physical alternative, not per operator node. The architecture docs no
longer say Stage 2 does physical operator implementation, and name its
modules. extend-asap-aware-mapping points at candidate.rs and
offered_physical_properties.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Workload optimization and plan selection refactor

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant