Repository navigation
refactor(stage2): physical expansion (decision-tree levels 3–4), physical edges, pruned alternatives; Count-Min weights via column_signs (#1207 PR 5) - #1235
Merged
Conversation
zzylol
force-pushed
the
refactor/1207-pass2
branch
2 times, most recently
from
November 9, 2026 07:11
bdf0888 to
bcae9e5
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 9, 2026 07:43
d11f12e to
616f765
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 9, 2026 11:53
616f765 to
f20177e
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 9, 2026 17:11
bcae9e5 to
8107319
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 9, 2026 17:44
f20177e to
e05016c
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 9, 2026 20:52
8107319 to
5d3f6de
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 9, 2026 21:28
e05016c to
513a1e7
Compare
This was referenced Nov 10, 2026
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 10, 2026 01:26
5d3f6de to
dea088e
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 10, 2026 01:26
513a1e7 to
4c77ec6
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 10, 2026 13:37
dea088e to
e98e80b
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 10, 2026 13:37
4c77ec6 to
8876072
Compare
zzylol
force-pushed
the
refactor/1207-pass2
branch
from
November 11, 2026 18:33
e98e80b to
5ac57fc
Compare
zzylol
force-pushed
the
refactor/1207-stage2
branch
2 times, most recently
from
November 12, 2026 19:19
a82bf38 to
07678ec
Compare
…natives; Count-Min weights via column_signs (#1207 PR 5) Stage 2 expands each logical candidate Pass 2 kept into its offered (timing, lifetime): ingestion time for time-independent groups and their inputs (counter states only over ordered rows), AcrossEvaluations for chains and grid-evaluated groups not read under an absolute @ T, and no (IngestionTime, None) for a summary build. Prune reasons: NoIngestionPipeline, RawDataNotRetained (query-time scans and scans kept across evaluations), QueryTimeResultsNotRetained (query time across evaluations), CountMinWeightsNotProvenNonNegative, InputHasNoBuildableProducer, ReachesNoQueryOutput. Physical edges carry alpha^log and alpha^phys; PhysicalASAPMemoCandidates indexes the admissible ones (PhysicalEdge::is_logically_and_physically_admissible). Pass 2's output, exact_only_groups_of_query included, is carried whole. Crate layout: lib.rs (entry, ingestion-time offer), candidates.rs (decision-tree levels 3 and 4), pruning.rs (buildable and reached fixed points), tests.rs over the shared memo builders. Deletes WeightDomain, NonNegativeWeightProof, SummaryUpdate::weight_domain, raw_data_samples_of_metric(_of_group) and Pass 1's with_raw_data_samples_requirement (the state group's column_signs replaces them), and the executor's weight-domain checks (CMS contract check, factory rate-weight refusal, Hydra unknown-sign check) with their tests.
… modules; named offered physical properties; tests split by step over shared fixtures (#1207 PR 5) lib.rs keeps the entry point and the renumbering. candidate.rs (was candidates.rs) holds levels 3 and 4: offered_physical_properties returns OfferedPhysicalProperties { physical_properties, prune_reason } instead of an anonymous pair, and one summary_build helper serves the counter order and Count-Min rules. group.rs computes the per-group offers once (ingestion time, across evaluations). edge.rs expands the logical edges. Tests: tests.rs (module list and output checkers), test_fixtures.rs (query A and Pass 2's output by hand, over memo_fixtures range, state, readout and shift), tests/{candidates,edges,pruning}.rs. The tests of PhysicalASAPMemoCandidates::new and its admissible-edge indexes move to the types crate's memo_candidates/tests.rs, whose fixture now goes through new. No dates, review tags or issue numbers in comments.
… update types in their own module (#1207 PR 5) Inline test modules of the files this PR touches move to sibling files (state_bytes, capability, summary_kernels::factory, ir::schema and ir::schema::topk). SummaryUpdate, SummaryInputExpr and EntityIdentity move from summary_type::statistic to summary_type::update; statistic keeps SketchStatistic. Re-exports are unchanged.
…weight_domain field (#1207 PR 5)
…ossEvaluations for a time-independent group without a grid (#1207 PR 5) The rule that a group read under an absolute @ T is not materialized across evaluations looked only at the anchor's direct child (through offset-only time shifts). In max_over_time(rate(c[1m])[5m:1m] @ 90) the rate, its range, the scan and the chain below the subquery were still offered AcrossEvaluations with unbounded retention. The rule now holds for every group below the anchor, through any operator other than another @, computed once per group over the candidates Pass 2 kept. A time-independent group with no evaluation grid (a result under @ T on a fixed interval without a phase) is now offered AcrossEvaluations: its one result is readable at any time, as the alignment table says. Before, such a state was rebuilt every evaluation and the ingestion-time alternatives under it were pruned as reaching no query output. Tests: every_group_below_an_absolute_anchor_is_not_materialized_across_evaluations and a_time_independent_group_without_a_grid_is_materialized_across_evaluations fail before this change; the query A, Count-Min, raw-data and @ T tests now expect the scan and states without a grid to be offered AcrossEvaluations.
…gical candidate, all six prune reasons, its modules and its one crate edge (#1207 PR 5) The contracts doc's Stage 2 section described materialization options for summary builds only and P<n>-m<k> candidate names; it now states what is offered per logical candidate (ingestion time, across evaluations, the @ T rule) and lists all six prune reasons. planner-layering prunes per physical alternative, not per operator node. The architecture docs no longer say Stage 2 does physical operator implementation, and name its modules. extend-asap-aware-mapping points at candidate.rs and offered_physical_properties.
zzylol
force-pushed
the
refactor/1207-stage2
branch
from
November 12, 2026 20:01
07678ec to
89bf901
Compare
zzylol
marked this pull request as ready for review
November 12, 2026 20:02
This was referenced Nov 16, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
todo!stub: Pass 2's logical candidates got no timing or lifetime, no physical edges and no pruned list, and Count-Min weights were "proven" non-negative by a per-updateWeightDomainproof that Pass 1 set and the executor checked.column_signs;WeightDomainand its plumbing are deleted.quantile_over_time(0.95, latency[1m])every 60 s (query A), Stage 2 emits 5 alternatives for the scan, 4 for the chain and 3 for each time-dependent candidate; across 88 review workloads it emits 12,166 physical candidates, 52,956 edges (50,632 admissible) and 9,584 pruned alternatives, with 0 invariant violations except one unservable workload.Review guide
main(PR 5 of the Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 stack; PR 4 merged as 70b0abc; rebased over fix(types)!: reject fixed_interval without an evaluation phase (#1260) #1263, merged as 4e0fa25). PR 6 and later (refactor(stage3): baseline selection MILP and its PhysicalASAPDAG; executor θ, native chains and a time-indexed store (#1207 PR 6) #1236 …) still sit on the old PR 5 commit and must be restacked onto this layout (theirsrc/catalog/module goes away).crates/stage2-physical-optimization/src/:lib.rs(entry and renumbering),candidate.rs(levels 3 and 4, prune rules),group.rs(per-group offers: ingestion time, across evaluations, the@ Trule),edge.rs,pruning.rs(buildable and reached fixed points). Thentypes/src/stages/physical/memo_candidates/mod.rs(admissible-edge indexes).WeightDomaindeletion rebased onto main; (2) layout by scope and test split; (3) mechanical: inline unit tests moved to sibling files,summary_type::updatesplit fromstatistic; (4) mechanical:weight_domainremoved from committed snapshot JSON; (5) review fix R1/R2 with failing-first tests; (6) docs.group.rs(R1, R2), the prune-reason precedence incandidate.rs,pruning.rs.asap_planner --run-until stage2panics on PR 9todo!s in its output writers (stage2_pruned_candidates_json,planning_stages_text). The behaviour below comes from the unit tests and an uncommitted dump hook.Behavior: before → after
Query A,
quantile_over_time(0.95, latency[1m])evaluated every 60 s at phase 0, ingesting data, everything retained (testoffers_each_candidate_its_timings_and_lifetimes). Before this PR, Stage 2 wastodo!().latency(time-independent, no grid)latency[1m]TimeRange 1mover the chainWith the default deployment (
query_time_results_retained: false) every (QT, Across) is prunedQueryTimeResultsNotRetained. Withraw_data_retained: falsethe scan keeps only (IT, None), the build over the minute is prunedInputHasNoBuildableProducer, and only the ingestion-time chain serves the query.The two review fixes, on
max_over_time(rate(c[1m])[5m:1m] @ 90)(dump hook, before = a82bf38, after = this head):x18): before, therate, its range, the scan, the chain and its windows below the@ 90subquery were offered (·, AcrossEvaluations) with unbounded retention; after, none of them is.p34, no grid): before, theMaxstate over the pinned window got only (QT, None) and (QT, One), so a constant was recomputed every evaluation; after, it is offered (QT, Across) and (IT, Across), and the ingestion-time subtree under it is reached.Interfaces
crates/stage2-physical-optimization/src/lib.rs/// Stage 2: expand every logical candidate that survived Pass 2 into its /// physical alternatives, every logical edge into its physical edges with /// α^log and α^phys, and record every rejected alternative with its reason. pub fn generate_physical_asap_memo_candidates( logical: &LogicalASAPMemoCandidates, data: &DataWorkload, capabilities: &DeploymentCapabilities, ) -> PhysicalASAPMemoCandidates;The signature is unchanged; the body replaces
todo!().crates/types/src/stages/physical/edge.rscrates/types/src/stages/physical/memo_candidates/mod.rsimpl PhysicalASAPMemoCandidates { /// Stage 2's output over `physical_candidates` and `edges` (grouped by /// (consumer, input)), with the admissible-edge indexes built once. pub fn new( logical_candidates: LogicalASAPMemoCandidates, physical_candidates: Vec<PhysicalCandidate>, edges: Vec<PhysicalEdge>, pruned_candidates: Vec<PrunedCandidate<PhysicalCandidate>>, ) -> Self; /// P^adm_{c,i}: the admissible edges into input `input_index_of_consumer` of `consumer`. pub fn admissible_edges_into(&self, consumer: PhysicalCandidateId, input_index_of_consumer: usize) -> &[PhysicalEdgeId]; /// The admissible edges out of `producer`: every consumer that may share it. pub fn admissible_edges_out_of(&self, producer: PhysicalCandidateId) -> &[PhysicalEdgeId]; }Signatures unchanged; bodies replace
todo!.crates/types/src/stages/pruning.rs/// Why an alternative was pruned (stays). pub enum PruneReason { NoIngestionPipeline, - /// Stage 2: it scans raw data at query time, and the deployment does - /// not retain raw data, so no window of it is there to scan. + /// Stage 2: it scans raw data at query time, or retains the raw rows it + /// scans across evaluations, and the deployment does not retain raw + /// data, so no window of it is there to scan or keep. RawDataNotRetained, - /// Stage 2: it is a query-time result materialized (`OneEvaluation` or - /// `AcrossEvaluations`), and the deployment retains no query-time - /// results. + /// Stage 2: it is a query-time result materialized across evaluations + /// (`AcrossEvaluations`), and the deployment retains no query-time + /// result across evaluations + /// (`DeploymentCapabilities::query_time_results_retained`). QueryTimeResultsNotRetained, InputHasNoBuildableProducer, ReachesNoQueryOutput, CountMinWeightsNotProvenNonNegative, UsageNotCostable, }(Only the doc lines of these two variants change.)
crates/types/src/summary_type/update/mod.rs(moved fromsummary_type/statistic.rs; re-exported fromasap_types::summary_typeas before)/// One summary-state update (stays). pub struct SummaryUpdate { pub item: Option<SummaryInputExpr>, pub weight: SummaryInputExpr, - #[serde(default)] - pub weight_domain: WeightDomain, } -pub enum WeightDomain { - UnknownOrSigned, - NonNegative { proof: NonNegativeWeightProof }, - RawDataSamplesOfMetric { metric: String }, - CounterDerivativeOfMetric { metric: String }, -} -pub enum NonNegativeWeightProof { - UnitCount, - ResetAwareCounterDerivative, - CounterRawDataSamples, - DeclaredNonNegativeRawDataSamples, - PositiveConstant, -} -pub fn raw_data_samples_of_metric<I: OperatorInput>(input: &I) -> Option<String>;crates/types/src/stages/logical/group/mod.rs-pub fn raw_data_samples_of_metric_of_group(memo: &Memo, group: MemoGroupId) -> Option<String>;Removed: the weight sign is the state group's
column_signs, which Stage 2 reads; a second, per-update derivation could disagree with it. Old JSON withweight_domainstill reads (the field is ignored).Executor:
validate_hydra_cms,unit_frequency_columnandcreate_planner_accumulatorno longer read a weight proof; a negative Count-Min weight is still refused at update time (WeightedFrequencyUpdater::update→check_weight).Usage (end to end over query A; the CLI cannot print Stage 2 at this PR, so this is the unit test's flow):
Algorithm changes
Stage 2 expansion and pruning (new)
generate_physical_asap_memo_candidateswastodo!().group.rs):ingestion_time_offered_per_group= time-independent groups and, transitively, the inputs of their kept candidates;across_evaluations_offered_per_group= chain, known grid, or time-independent, and not read under an absolute@ T(below aTimeShiftanchored at a timestamp through any operator except another@).offered_physical_properties): every valid (τ, ℓ) in {QT, IT} × {None, One, Across} the group offers, IT only for a counter state over ordered rows (PromQL samples or a declared out-of-order bound), no (IT, None) for a summary build; each with the first prune reason that applies:CountMinWeightsNotProvenNonNegative,NoIngestionPipeline,RawDataNotRetained,QueryTimeResultsNotRetained.expand_edges: each logical edge × the producer's and consumer's alternatives, α^log copied, α^phys =producer_satisfies_physical_requirements.prune_reasons: least fixed point of "every input has a buildable admissible producer" (InputHasNoBuildableProducer), then reach from query outputs over admissible edges (ReachesNoQueryOutput).src/tests/; an independent checker over 88 workloads (Verification).delivers_on; retention $ is PR 7.Decisions and open questions
Needs reviewer decision: None.
Decided:
column_signs; time indexing (any grid-evaluated group may beAcrossEvaluations); no executor-support check anywhere.WeightDomaindeleted (rejected: a per-update proof set by Pass 1 and checked by the executor).PhysicalEdge::is_logically_and_physically_admissible(rejected:is_admissible).QueryTimeResultsNotRetainedprunes only (QT, AcrossEvaluations) (rejected: OneEvaluation too, which with default capabilities removes all within-evaluation sharing).@ T).ReachesNoQueryOutput.SummaryAgg, chain,MaintainPopulation) is offered (IT, None): fed at ingestion it keeps its state.RawDataNotRetainedalso rejects a scan materialized across evaluations at ingestion time (it retains raw rows).AcrossEvaluationsfor any group below an absolute@ T, through any operator other than another@; a group also read on a relative path counts as below it. Rejected: only the anchor's direct child (left therate, range, scan and chain ofmax_over_time(rate(c[1m])[5m:1m] @ 90)with unbounded retention); leaving it all to PR 7's $.AcrossEvaluations, following the concepts doc's alignment table; the doc's decision-tree row now matches.Verification
asap-stage2-physical-optimizationasap-typesasap-stage1-pass1-per-computation-candidatesasap-costlibtodo!asap-executor(lib + tests)todo!s)cargo test --workspace --no-fail-fasthydra_cms_execution::unknown_sign_weight_is_rejectedis deleted with the bind-time sign checkcargo fmt --check,cargo clippy -D warningson touched cratesNew tests:
every_group_below_an_absolute_anchor_is_not_materialized_across_evaluations:(x[1m]) @ T; neither the range nor the scan has anAcrossEvaluationsalternative (failed before the R1 fix).a_time_independent_group_without_a_grid_is_materialized_across_evaluations: aMaxstate over a pinned minute with no grid is offered (QT, Across) and (IT, Across), and the pinned minute's (IT, None) is kept (failed before the R2 fix).memo_candidates::tests::{new_indexes_only_admissible_edges, an_edge_is_admissible_when_both_requirements_hold}: the indexes list exactly the admissible edges.src/tests/{candidates,edges,pruning}.rs).End to end (uncommitted dump hook in
asap_planner, 90 workloads: 71 PromQL, 19 SQL; 88 reach Stage 2, 2 are refused earlier as capability contradictions): an independent checker recomputes the offered sets and reasons and checks edges, indexes, buildability and reach.@ Twith anAcrossEvaluationsalternative (R1)AcrossEvaluations(R2)p32: no phase and no raw data, no plan can serve it)Snapshots:
weight_domainremoved from 13 corpus and 10 sketch-plan snapshot files (the field no longer exists; JSON otherwise identical, checked by parsing). No other snapshot changes.Follow-ups
column_signs.src/catalog/becomescandidate.rs,group.rs,edge.rs,pruning.rs; their local memo fixtures switch toasap_types::test_support.