Repository navigation
Conversation
zzylol
force-pushed
the
refactor/1207-contexts
branch
from
November 15, 2026 15:54
796d435 to
44a223d
Compare
zzylol
force-pushed
the
test/installable-selection-and-optimality
branch
from
November 15, 2026 17:16
7ccf511 to
e17fc55
Compare
…ty contract (tests only) An oracle enumerates every plan of Stage 2's physical memo without Stage 3's coefficients or MILP, costs each with plan_cost and checks the planner's install rules. The tests check that Stage 3 returns a plan only when one installs, that the plan installs, and that it is the cheapest installable plan, over 12 small corpus workloads and the regression workloads (y13, the SQL whole-table view, the subquery shape, the scan feeding two chain widths, negative offsets). Adds PruneReason::NotInstallable and a todo!() stub of InstallableDagError::node_id; no implementation.
zzylol
force-pushed
the
test/installable-selection-and-optimality
branch
from
November 16, 2026 01:45
e17fc55 to
11c2d25
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SELECT approx_percentile_cont(bytes, 0.9) FROM flowsevery 10 s returns an ingestion-timeProjectview over the whole table, which failsViewReadsUnwindowedSource. Separately, "objective =plan_costof the selected plan" cannot see a cost coefficient that is wrong only on candidates the MILP did not select.asap_cost::plan_costand checks each againstmaterialized_views()andraw_data_retention(). The tests then check three things: Stage 3 returns a plan exactly when some plan installs, that plan installs, and that plan is the cheapest installable one.plan_costequals the oracle's minimum (largest relative gap 7.6e-11, s01).Review guide
Base:
mainat3e434c760(the earlier base,refactor/1207-contexts, merged as refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237). Read first:plan_enumeration/mod.rs: the oracle and its plan-validity rules (module doc);stage3_installable_optimal_selection.rs: the corpus and the tests.The only non-test changes:
PruneReason::NotInstallable(approved) and atodo!()stubInstallableDagError::node_id.Needs careful review: the oracle's validity rules and its retention rule (provisional answers A6 and A8 below). If they are wrong, the contract proves the wrong thing.
Retarget to main
3e434c760The earlier base branch,
refactor/1207-contexts, merged intomainas #1237 (3e434c760). This PR now targetsmain, and its one commit was rebased onto3e434c760with no conflicts, so the diff shows only this PR's tests.Checked on the rebased head, by test name against
main:cargo check --workspace --all-targets,cargo fmt --all -- --checkandcargo clippy -p asap-integration-tests -p asap-types --all-targets -- -D warningsare clean.cargo test -p asap-integration-tests -p asap-types: 449 passed, 12 failed, 4 ignored. 7 of the 12 fail onmaintoo (joint_search_equivalence×2,materialized_views×1,plan_sharing×2,promql_to_post_asap×2). The other 5 are this PR's expected failures (the missing Fix 3), unchanged from the old base:installable_dag_error::node_id_names_the_node_of_a_subtree_local_rule_onlyandstage3_installable_optimal_selection::{a_view_over_a_whole_table_is_reported_not_installable, cheapest_sql_percentile_whole_table, cheapest_sql_percentile_whole_table_rows_held, every_selected_plan_passes_the_install_rules}. 28 of the 32 contract tests pass, as before.Behavior: before → after
Tests only; no planner behavior changes. Example: the
flowstable declaresrows_held1,000,000, and the workload runsSELECT approx_percentile_cont(bytes, 0.9) FROM flowsevery 10 s at rank 0.01.select_planreturnsScan → Aggregate → Project (IngestionTime, AcrossEvaluations), which failsViewReadsUnwindowedSource { node_id: 2 }.cheapest_sql_percentile_whole_table_rows_heldfails on this, while 204 of the oracle's 372 plans install.NotInstallableand returns the cheapest installable plan.Interfaces
crates/types/src/stages/pruning.rs/// Why a stage removed a candidate (stays). pub enum PruneReason { NoIngestionPipeline, RawDataNotRetained, QueryTimeResultsNotRetained, InputHasNoBuildableProducer, ReachesNoQueryOutput, CountMinWeightsNotProvenNonNegative, DominatedBy(PhysicalCandidateId), UsageNotCostable, + /// Stage 3: a selected plan with it was refused by a planner-owned install + /// rule over its subtree; the MILP was solved again without it. + NotInstallable, }crates/types/src/plan/materialized_view/mod.rsUsage: after implementation,
select_planbuilds the plan withplan_ofand callsplan.materialized_views()andplan.raw_data_retention(). On a subtree-local error it readserror.node_id(), addsPrunedCandidate { candidate, reason: NotInstallable }and solves again. On any other error it returns the error.a_view_over_a_whole_table_is_reported_not_installablechecks this end to end.Design scope
crates/integration-tests/tests/plan_enumeration/: test support at the Stage 3 / plan boundary. It holds every plan of a physical memo under the concepts doc's execution-context rules. It is general: any memo without@anchors.crates/types/tests/installable_dag_error.rs: tests of the installer's error type intypes::plan.NotInstallableis a general Stage 3 prune reason, not tied to one rule.Algorithm
The oracle runs a depth-first enumeration:
delivers_on). A materialized producer runs inMaterialized(v), other ingestion producers inEVERY_ARRIVING_ROW, and the inputs of an ingestion view on its refresh window's pipeline.PhysicalASAPDAG. Each node gets the least retention its plan's readers need (read_back_ms), and the plan is costed withplan_cost.@anchors are refused, and accuracy (K) and latency bounds are not modeled (Stage 3 has neither yet).Decisions and open questions
Needs reviewer decision. These are the provisional answers the tests encode. The full list is in
/mydata/notes/test-plans/fix3-and-optimality.md, section 1.5.node_id()isSomeonly for the subtree-local rules:UnsupportedRawDataWindow(itsTimeRange),ViewReadsUnwindowedSource,ViewWithoutCoverageandIngestionBuildReadsWindowsOfDifferentOffsets. It isNonefor the 4 retention rules andInvalidDag. A name likesubtree_local_node_idwould be clearer.UnsupportedRawDataWindow. So this is not tested.Erris accepted. Recommendation: a newErrorSelectingPlan::NotInstallable(InstallableDagError).select_plan. They are covered by the classification test and bystage3_retains_at_least_what_its_plans_readers_need.NotInstallabletest assumes the cheapest plan is unique: no ties.raw_data_retention()as well asmaterialized_views(). Executor compile is not checked.Findings for the implementation
rows_held, the SQL whole-table query has no installable plan. Its query-time plans cannot be costed (MissingSourceFact { flows: rows_held }), so Fix 3 must returnErrthere.max_over_time(sum(rate(c[5m]))[1h:1m]): all 7,200 of its plans install;[6h:1m]: the selected plan installs.Verification
cargo test -p asap-integration-tests --test integration stage3_installable_optimal_selection::cargo test -p asap-types --test types installable_dag_error::todo!()cargo fmt --all -- --check,cargo clippy -p asap-types -p asap-integration-tests --all-targetscheapest_<workload>×19 (p01–p11, s01, y13, r5, subquery 1 h, b08, b09, SQL whole table ±rows_held)plan_cost≤ (1+1e-6) × the oracle's minimum installable;Errexactly when none installsevery_selected_plan_passes_the_install_rulesa_view_over_a_whole_table_is_reported_not_installabletodo!)NotInstallable; the plan installsnode_id_names_the_node_of_a_subtree_local_rule_onlytodo!)the_oracle_space_holds_every_selected_planstage3_retains_at_least_what_its_plans_readers_needevery_selected_edge_is_admissible,no_unmaterialized_query_time_node_has_two_readers,no_query_reads_a_sketch_of_a_group_it_must_read_exactly,no_candidate_reported_not_installable_is_selectedthe_objective_is_the_plan_cost_over_the_corpus,the_selected_cost_does_not_depend_on_query_ordery13_selects_an_installable_plan_and_no_shift_view_over_arriving_rows,a_negative_offset_has_no_installable_plan_and_stage3_returns_an_error,a_ratio_subquery_over_raw_ranges_selects_an_installable_planOptimality on the current head: no violations. Selected cost = oracle minimum on all 17 workloads that have an installable plan.
Follow-ups
Next: implementation after review. That is Fix 3 in
select_planandInstallableDagError::node_id, on this branch's tests. Also:@anchors and to (K) once it lands;UnsupportedRawDataWindow(A2).