Repository navigation
refactor(types): properties and requirements from #1207; chains as time-indexed state (#1207 PR 1) - #1227
Merged
Conversation
zzylol
force-pushed
the
refactor/1207-properties
branch
4 times, most recently
from
November 6, 2026 15:37
5ef1ef3 to
d9d64df
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 15:47
d9d64df to
2a62109
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 16:30
2a62109 to
b87b7c0
Compare
zzylol
changed the base branch from
refactor/1207-interfaces
to
refactor/1207-crates-per-stage
November 6, 2026 16:30
zzylol
force-pushed
the
refactor/1207-properties
branch
3 times, most recently
from
November 6, 2026 17:12
460e9c7 to
ba60db8
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 6, 2026 18:56
4d55373 to
ae4843e
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 18:56
ba60db8 to
60a16ee
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 6, 2026 19:10
ae4843e to
8d020b0
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 19:12
60a16ee to
107a77b
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 6, 2026 21:29
8d020b0 to
d1feaad
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 21:31
107a77b to
c7a0bc9
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 6, 2026 21:38
d1feaad to
9a3bb19
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 21:38
c7a0bc9 to
9e5ce62
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 6, 2026 21:57
9a3bb19 to
345b5df
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 22:11
9e5ce62 to
96d3457
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 6, 2026 22:46
cbd0359 to
4445d45
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 6, 2026 22:46
96d3457 to
b250852
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 7, 2026 17:47
4445d45 to
84cf80d
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 7, 2026 18:04
b250852 to
7ceb7c3
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 7, 2026 20:51
84cf80d to
69fed95
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 7, 2026 20:55
7ceb7c3 to
12a4b96
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 8, 2026 15:06
69fed95 to
dfd6a5b
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 8, 2026 15:10
12a4b96 to
cf93b37
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 8, 2026 21:29
dfd6a5b to
b768e8d
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 8, 2026 21:42
cf93b37 to
1ca33f1
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 9, 2026 03:33
b768e8d to
0e0f75a
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 9, 2026 03:45
1ca33f1 to
3c28259
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 9, 2026 04:31
0e0f75a to
058ea42
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
2 times, most recently
from
November 10, 2026 01:26
8030d3f to
c2e717b
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 10, 2026 13:37
058ea42 to
a4290ee
Compare
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 10, 2026 13:37
c2e717b to
13063ab
Compare
zzylol
force-pushed
the
refactor/1207-crates-per-stage
branch
from
November 10, 2026 15:01
a4290ee to
cadf5d5
Compare
- LogicalProperties::new(expression, parameters): a sketch build (a chain included) needs θ from parameter_list(algorithm) of its FieldDataType::Sketch(algorithm); an exact operator has none. - parameter_list(algorithm): structures only. Count-Min lists each width with depths 5, 7, 10 and 12, Count Sketch with 47, 81, 117 and 155 (its median-of-rows δ levels 1e-2 … 1e-5); heap variants each of those with the 1-2-5 heap sizes; UnivMon 40 entries; KLL, DDSketch, HLL, KMV one entry per ε level. A test keeps the lists equal to today's sizing, which is test-only. - PhysicalProperties::new rejects (IngestionTime, OneEvaluation); producer_satisfies_physical_requirements(producer, consumer_timing). - PhysicalCandidate::delivers_on(candidates, context): None in every context, OneEvaluation only in Evaluation contexts, a chain with AcrossEvaluations when its width divides the grid's period and phase (or the refresh window's width and phase), any other AcrossEvaluations result everywhere when its group does not depend on the evaluation time, else when its group's evaluation grid holds the context's times; Materialized(v) contexts stay todo!() for PR 7. - Chains: one schema (window_end at column 0, keys, state), TimeRange and TimeShift over a chain select windows (an instant TimeRange is rejected), a SummaryMerge over them finalizes and rolls up as the chain's build over that range (the two PR 0 todos). - summary_type_of_operator; QueryRecurrence::evaluation_grid_ms is None for a fixed interval without a phase.
zzylol
force-pushed
the
refactor/1207-properties
branch
from
November 10, 2026 15:41
13063ab to
b65c396
Compare
zzylol
marked this pull request as ready for review
November 10, 2026 15:43
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
todo!(): logical properties and the θ lists, physical properties, α^phys,delivers_onover execution contexts; a chain had no time index andTimeRange/TimeShiftrejected state.parameter_list), contexts are evaluation grids, and aSummaryTumblingWindowChainis a time-indexed state (window_end, keys…, state).sum_over_time(m[5m])read every 60 s and every 120 s (grid 60 s @ 0),delivers_onnow answers true for 60 s @ 0, 120 s @ 0 and 120 s @ 60 s and false for 30 s @ 0 and 60 s @ 15 s, where it panicked before; KLL gets 7 listed θ (k = 26 … 2863), Count-Min 28, UnivMon 40.Review guide
6bcf3984b); feat(stage0, types): memo hash-consing, group properties (column_sign, depends_on_evaluation_time, aggregate_coverage), @ end()/@ start() above their subtree (#1207 PR 2) #1230 (PR 2, memo) builds on it. One commit, headb65c3964c. Diff vs base: 22 files, +1,532 / −243. The stack's PRs merge into main one by one after review.LogicalProperties::newinstages/logical/candidate.rs,parameter_listand the θconstlists insummary_type/parameter_lists.rs,delivers_onandproducer_satisfies_physical_requirementsinstages/physical/candidate.rs,PhysicalProperties::newinstages/physical/properties.rs,QueryRecurrence::evaluation_grid_msinworkload/query.rs, the estimator changes inaccuracy/src/guarantee/. Previous head13063ab21(backupbackup/1207-layout-properties-13063ab21).types/src/stages/logical/candidate.rs:LogicalProperties::new;types/src/summary_type/parameter_lists.rs:parameter_listand theconstlists.types/src/stages/physical/candidate.rs:delivers_on,producer_satisfies_physical_requirements(their tests instages/physical/memo_candidates.rs, which can build aPhysicalASAPMemoCandidateswith its private fields);types/src/stages/physical/properties.rs:PhysicalProperties::new.types/src/ir/operator/asap.rs(chain_schema,summary_build,selected_from,summary_type_of_operator) andtypes/tests/tumbling_window_chain.rs.types/src/workload/query.rs:QueryRecurrence::evaluation_grid_ms.constθ tables (KLL_PARAMETERS…UNIVMON_PARAMETERS, built bywith_depths,with_heap_sizes,univmon_parameters);asap-accuracysizing made#[cfg(test)];propagation.rsreads merged windows throughsummary_build.delivers_ontable (chain width vs context period/phase; stored results on their group's grid); the chain schema and the rule that aSummaryMergeover a time-indexed state reads exactly oneTimeRange; the Count Sketch depths (median-of-rows model).Behavior: before → after
No workload reaches these functions end to end yet (Stage 0, Pass 1, Pass 2 and Stage 2 are still
todo!()in PRs 2–5); the example is the concepts doc's section 5 flow, Q_A =quantile_over_time(0.99, latency[1m])every 10 s at phase 0 (FixedIntervalAt), checked by the unit tests in Verification. It serves the design need that every later stage reads one set of property checks.parameter_list(SketchAlgorithm::Kll)todo!()[k=26, 52, 132, 269, 547, 1404, 2863]LogicalProperties::new(chain[KLL, w=10 s], Some(Kll{k:547}))todo!()Ok;k = 270→ParametersNotInList; no θ →SketchBuildWithoutParametersPhysicalProperties::new(IngestionTime, OneEvaluation)todo!()Err; every other pairOkproducer_satisfies_physical_requirements(IT AcrossEvaluations chain, QueryTime)todo!()truechain(w=10 s).delivers_on(grid 10 s @ 0 / 30 s @ 0 / 10 s @ 5 s)todo!()SummaryTumblingWindowChainTimeRange/TimeShiftover a chainTimeRangeover a chain rejectedInterfaces
crates/types/src/stages/logical/candidate.rs·crates/types/src/summary_type/parameter_lists.rsimpl Memo { /// The candidate `id` (body filled in; `delivers_on` reads a chain's width through it). pub fn candidate(&self, id: CandidateForMemoGroupId) -> &CandidateForMemoGroup; } /// p^log = θ_v; the summary type and a chain's width are operator arguments. The type stays. pub struct LogicalProperties { /// θ_v from the algorithm's list; None for an exact operator. parameters: Option<SketchParams>, } impl LogicalProperties { /// Rejects θ missing on a sketch build (a chain included), present on an operator that builds no /// sketch, or not in `parameter_list(algorithm)` (body filled in; signature unchanged). pub fn new( expression: &CandidateExpression, parameters: Option<SketchParams>, ) -> Result<Self, LogicalPropertiesError>; /// θ_v. pub fn parameters(&self) -> Option<&SketchParams>; } /// Why `LogicalProperties::new` rejects; bad input from a candidate generator (a planner bug). The type stays. pub enum LogicalPropertiesError { /// θ on an operator that builds no sketch. ParametersOnExactOperator, /// A sketch build (a chain included) without θ. SketchBuildWithoutParameters, /// θ not in its algorithm's list (also another algorithm's θ). ParametersNotInList, } /// The algorithm's predefined θ list, structures only, smallest first; empty for an algorithm /// without an error model (body filled in; signature unchanged). pub fn parameter_list(algorithm: SketchAlgorithm) -> &'static [SketchParams];crates/types/src/stages/physical/candidate.rs·crates/types/src/stages/physical/properties.rsimpl PhysicalCandidate { /// Whether a consumer running in `context` can read this producer: None everywhere, OneEvaluation /// only in `Evaluation` contexts, an AcrossEvaluations chain when its width divides the context's /// period and phase, any other stored result on its group's grid (body filled in; signature PR 0's). pub fn delivers_on( &self, candidates: &PhysicalASAPMemoCandidates, context: &ExecutionContext, ) -> bool; } /// α^phys: the timing × lifetime table only (body filled in). pub fn producer_satisfies_physical_requirements( producer: &PhysicalCandidate, consumer_timing: ExecutionTiming, ) -> bool; impl PhysicalProperties { /// (τ, ℓ), unless (IngestionTime, OneEvaluation) (body filled in). pub fn new( timing: ExecutionTiming, lifetime: Lifetime, ) -> Result<Self, PhysicalPropertiesError>; }crates/types/src/ir/operator/asap.rs·crates/types/src/ir/operator/tumbling_window_chain.rsmerged_chain: replaced bysummary_build/selected_fromthrough the window selection.crates/types/src/workload/query.rscrates/accuracy/src/sizing.rs·crates/accuracy/src/test_support.rsNo caller remains: the lists replace sizing; the sizing helpers only the list test reads are
#[cfg(test)](kll_k,kmv_k,count_sketch_depth, UnivMon'sdistinct_heapand its constants).Usage, end to end (the flow PRs 2–7 run for the doc's section 5 example with the calls this PR implements; each step is checked by a unit test named in Verification at the linked commit):
The planner's text output does not change in this PR: planning still stops at Stage 0's
todo!().Algorithm changes
None in planning results: no stage that would reach these checks runs yet. The new checks:
θ lists (
parameter_list)Problem: PR 0 left the lists
todo!(); θ is structure only (2026-11-08), so the lists hold no sized-for fields.After:
constarrays built by twoconst fns:with_depths(widths, depths, shape)crosses 7 widths with 4 depths (CMS_DEPTHSorCOUNT_SKETCH_DEPTHS),with_heap_sizes(shapes)crosses those 28 with 10 heap sizes;univmon_parameters()builds the 40 UnivMon shapes. Thematchlists everySketchAlgorithm(no_), so a new sketchlib algorithm fails to compile until it gets a list.distinct_failure_probability)Correctness: the widths,
ks, precisions and heap sizes are today's sizing over ε = 0.1 … 0.001 (smallest first);asap-accuracy'sparameter_list_is_the_sizing_over_the_error_gridkeeps them equal to it (that sizing is test-only; PR 4 replaces the test with a per-entry guarantee test).Complexity:
&'staticslices, no allocation.Assumptions and limits: no parameter floors (
floor_for_cap,strongest,weakest,dominates,combine): a readout's input carries aRequiredAccuracy, checked against the producer's guarantee (PR 4).delivers_onover execution contextsProblem: PR 0's
todo!(); it readQueryRecurrence/RangeEvaluationfrom the context before.After: match ℓ; for AcrossEvaluations read the candidate's operator from
candidates.logical_candidates.memo. A chain needs itstumbling_window_width_msw to divide the context's (period, phase) (the grid's, or the refresh window's (width, phase)), so w divides every time phase + k·period. Any other result delivers everywhere when its group'sdepends_on_evaluation_timeis false; else every context time must lie onevaluation_grid_of_group[group]: its period divides the context's and the phases agree modulo it.Evaluation { grid }IngestionPipeline(RefreshWindow)Materialized(v)period_msandphase_mswidth_msandphase_mstodo!()(PR 7: v's own schedule)depends_on_evaluation_timeevaluation_grid_of_group[group]period_msandphase_ms≡ g's phase (mod g's period)width_mstodo!()(PR 7)None)todo!()(PR 7)Correctness:
delivers_on_by_lifetime_without_windows,delivers_on_a_chain_when_its_width_divides_the_context_times,delivers_on_a_stored_result_on_its_group_grid.Complexity: O(1) per call (one memo lookup).
Assumptions and limits:
delivers_ondoes not shift the context's times by aTimeShifton the reading path:evaluation_grid_of_groupis the grid of the times the group itself is evaluated at, so Pass 2 (PR 4) must fold the queries' shifts into it.α^phys (
producer_satisfies_physical_requirements)Chains as time-indexed state (schema derivation)
todo!().chain_schemaderives the build's schema, removes its time column if it has one, inserts a computedwindow_end(the input time column's type) at column 0 as the time index, and sets the unique key to (window_end, plain columns).TimeRangeandTimeShiftaccept a time-indexed state and keep its schema (rejected over any other state; anInstantTimeRangeover a chain is a validation error). ASummaryMergeover a time-indexed state reads exactly oneTimeRangeover a chain; its schema issummary_agg_schema(chain input, summary type, reduction), the chain's build over that range, so it joins the original state group.FinalizeExactAccumulatorand a rollup over merged windows derive as over that build.types/tests/tumbling_window_chain.rs(five tests, below).Decisions and open questions
Needs reviewer decision: None (P1–P3 approved 2026-11-09).
Decided:
distinct_failure_probability)floor_for_cap/strongest/weakest/dominates/combine;RequiredAccuracyinsteadFieldDataType::Sketch(SketchAlgorithm);summary_type_of_operator(wasfamily_of_operator);sketch_algorithm_ofdeletedproducer_satisfies_physical_requirements(producer, consumer_timing)delivers_onEvaluationcontextsdelivers_onforIngestionPipeline(RefreshWindow)/Materialized(v)Materialized(v)staystodo!()for PR 7delivers_on(&PhysicalASAPMemoCandidates, context)(P2)GroupProperties::depends_on_evaluation_time,LogicalASAPMemoCandidates::evaluation_grid_of_group(P3)delivers_onreads them hereTimeRangeover a chain rejectedSummaryMergeover a time-indexed stateTimeRangeselected_from/summary_buildsee throughTimeRange/TimeShiftmerged_chainevaluation_grid_msNoneforFixedIntervalwithout a phase; theQueryWorkloadEntrywrapper deletedasap-accuracyDeleted besides the interfaces above: this PR's earlier
strongest,weakest,dominates,combineand their tests (strongest_and_weakest_combine_one_family_per_parameter,a_listed_univmon_dominates_an_l2_and_a_distinct_floor,hydra_dominance_compares_every_size_of_one_kind); its earlierproducer_satisfies_logical_requirements(properties, requirement),LogicalRequirements::newandlogical_requirement_new_accepts_any_floor(replaced byasap_accuracy::producer_satisfies_logical_requirements(memo, producer, consumer, input_index, evidence), PR 0 interface, PR 4 body, andLogicalRequirements { required_accuracy });a_configuration_satisfying_a_listed_floor_is_no_less_accurate; KLL, HLL and KMV list entries' sized-for fields and UnivMon'sdistinct_failure_probability; PR 0'snew_rejects_one_evaluation_at_ingestion,new_accepts_each_valid_pair(replaced bynew_over_every_timing_and_lifetime).Verification
Restack check (2026-11-10, head
b65c3964cvs previous head13063ab21):cargo test -p asap-types -p asap-accuracy --all-features --no-fail-fast: 225 passed / 0 failed / 2 ignored on both, the same tests by name (the two ignored are thedelivers_ontests below);cargo check --workspace --testswith no warnings,cargo clippy --workspace --all-targets -- -D warningsandcargo fmt --checkclean. PR 1 touches onlyasap-typesandasap-accuracy, so the other crates were not rerun; the table below is from the earlier head.cargo test -p <crate> --all-features --no-fail-fastat8030d3fe6;cargo fmt --all -- --checkandcargo clippy --workspace --all-targets --all-features -- -D warningsclean;crate_graph: 6 passed. By test name against the base PR 0b (058ea4234): no test that passed fails; 12 new tests pass; 0 newly failing. Two new tests (delivers_on_a_chain_when_its_width_divides_the_context_times,delivers_on_a_stored_result_on_its_group_grid) build their memo fixture withfind_or_add_group, which istodo!()until PR 2, so they are#[ignore = "filled in by #1207 PR 2 (#1230)"]here; PR 2 removes the attribute and they pass there. At the headc2e717bf9(only change: those two attributes)cargo test -p asap-types --no-fail-fast: 164 passed, 0 failed, 2 ignored (the two above); PR 0's twoPhysicalProperties::newtests (failing at itstodo!()) are replaced. Every failure panics at a later PR'stodo!()or is the stale fixture (PR 9).delivers_ontests, filled in by PR 2retention_ms=Memo::find_or_add_group1Memo::find_or_add_group: 18output.status.success(): 2retention_ms: 3; old-wire fixture ("invalid type: sequence, expected string or map"): 1Against the previous head
3c2825903: only the 36 tests PR 0 now deletes withpass1/composition.rs(P11) are gone (35 failed, 1 passed there); no test changed result.New tests:
stages::logical::candidate::tests::logical_properties_new_over_every_operator_and_parameters: 5 operators (KLL build,Sketch(HydraCms)build, exact Sum, KLL chain, readout) × 5 θ (none, KLL 269, CMS, Hydra CMS, unlisted KLL 270).stages::logical::candidate::tests::summary_type_of_operator_reads_the_builds_summary_type(it builds aCandidateExpression);summary_type::parameter_lists::tests::parameter_list_holds_its_algorithm_without_repeats(one type per list, empty exactly for algorithms without an error model, the depth sets, sizes 28 / 280 / 40).stages::physical::memo_candidates::tests:delivers_on_by_lifetime_without_windows(None in every context; OneEvaluation only inEvaluation);delivers_on_a_chain_when_its_width_divides_the_context_times(grids 10 s @ 0, @ 5 s, @ 2 s, 7 s @ 0; refresh windows 5 s @ 0, 10 s @ 5 s, 10 s @ 2.5 s; widths 5 s and 10 s);delivers_on_a_stored_result_on_its_group_grid(X = the scan's last 5 minutes, grid 60 s @ 0, against 60 s @ 0, 120 s @ 0, 120 s @ 60 s, 30 s @ 0, 60 s @ 15 s, refresh 60 s @ 0, and no grid);producer_satisfies_physical_requirements_over_every_valid_timing_and_lifetime.stages::physical::properties::tests::new_over_every_timing_and_lifetime.types/tests/tumbling_window_chain.rs:a_chain_is_a_state_with_a_time_index,an_instant_time_range_over_a_chain_is_rejected,time_range_and_time_shift_select_a_chains_windows,merged_windows_are_the_build_over_the_selected_range,a_merge_over_a_chain_reads_one_time_range.workload::query::tests::range_evaluation_is_optional_carried_and_validated:FixedIntervalwithout a phase has no grid;FixedIntervalAtkeeps its grids.asap-accuracyguarantee::tests::parameter_list_is_the_sizing_over_the_error_grid: every list is today's sizing over the ε grid at δ = 0.01, in order (UnivMon has 28 more: the combined shapes).End-to-end evidence: not applicable (planning stops at Stage 0's
todo!()until PR 2).Self-review: no over-abstraction (
delivers_onreads the two new fields directly; the(period, phase)pair is local; the twoconst fns exist because the lists areconst, replacing hand-written 28- and 280-entry arrays). Duplication removed: onewith_depthstakes the depth array; the chain and stored-result branches share the context-times match;sketch_algorithm_of,family_of_operatorand the sizing wrappers are gone. The lists duplicate today's sizing as constants by design, kept equal by one test.QueryRecurrence::evaluation_grid_mshas no caller in this PR (Pass 2's chain widths and Stage 3's grids use it in PRs 4 and 7).Follow-ups
delivers_onforMaterialized(v)with ℓ = AcrossEvaluations: PR 7 (refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237).evaluation_grid_of_groupmust fold the queries'TimeShifts: PR 4 (refactor(stage1-pass2): rollup from the finest common group, readout requirements, dead-candidate elimination, chain groups (#1207 PR 4) #1234).