Skip to content

test: installable selection and plan_cost optimality contract (tests only) - #1334

Draft
zzylol wants to merge 1 commit into
mainfrom
test/installable-selection-and-optimality
Draft

zzylol wants to merge 1 commit into
mainfrom
test/installable-selection-and-optimality

Conversation

@zzylol

@zzylol zzylol commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem. Stage 3 can return a plan that the planner's own install rules refuse. Example: SELECT approx_percentile_cont(bytes, 0.9) FROM flows every 10 s returns an ingestion-time Project view over the whole table, which fails ViewReadsUnwindowedSource. Separately, "objective = plan_cost of the selected plan" cannot see a cost coefficient that is wrong only on candidates the MILP did not select.
  • Change (tests only). An oracle enumerates every plan of Stage 2's physical memo. It uses neither Stage 3's coefficients nor its MILP. It costs each plan with asap_cost::plan_cost and checks each against materialized_views() and raw_data_retention(). The tests then check three things: Stage 3 returns a plan exactly when some plan installs, that plan installs, and that plan is the cheapest installable one.
  • Effect on this head. 28 of 32 integration tests pass. The 4 failures and the 1 types failure are all the expected missing Fix 3. No optimality violation: on all 17 workloads with an installable plan, the selected plan_cost equals the oracle's minimum (largest relative gap 7.6e-11, s01).

Review guide

Retarget to main 3e434c760

The earlier base branch, refactor/1207-contexts, merged into main as #1237 (3e434c760). This PR now targets main, and its one commit was rebased onto 3e434c760 with no conflicts, so the diff shows only this PR's tests.

Checked on the rebased head, by test name against main: cargo check --workspace --all-targets, cargo fmt --all -- --check and cargo clippy -p asap-integration-tests -p asap-types --all-targets -- -D warnings are clean. cargo test -p asap-integration-tests -p asap-types: 449 passed, 12 failed, 4 ignored. 7 of the 12 fail on main too (joint_search_equivalence ×2, materialized_views ×1, plan_sharing ×2, promql_to_post_asap ×2). The other 5 are this PR's expected failures (the missing Fix 3), unchanged from the old base: installable_dag_error::node_id_names_the_node_of_a_subtree_local_rule_only and stage3_installable_optimal_selection::{a_view_over_a_whole_table_is_reported_not_installable, cheapest_sql_percentile_whole_table, cheapest_sql_percentile_whole_table_rows_held, every_selected_plan_passes_the_install_rules}. 28 of the 32 contract tests pass, as before.

Behavior: before → after

Tests only; no planner behavior changes. Example: the flows table declares rows_held 1,000,000, and the workload runs SELECT approx_percentile_cont(bytes, 0.9) FROM flows every 10 s at rank 0.01.

  • Today: select_plan returns Scan → Aggregate → Project (IngestionTime, AcrossEvaluations), which fails ViewReadsUnwindowedSource { node_id: 2 }. cheapest_sql_percentile_whole_table_rows_held fails on this, while 204 of the oracle's 372 plans install.
  • After Fix 3: Stage 3 reports that view's candidate as NotInstallable and returns the cheapest installable plan.

Interfaces

crates/types/src/stages/pruning.rs

 /// Why a stage removed a candidate (stays).
 pub enum PruneReason {
     NoIngestionPipeline,
     RawDataNotRetained,
     QueryTimeResultsNotRetained,
     InputHasNoBuildableProducer,
     ReachesNoQueryOutput,
     CountMinWeightsNotProvenNonNegative,
     DominatedBy(PhysicalCandidateId),
     UsageNotCostable,
+    /// Stage 3: a selected plan with it was refused by a planner-owned install
+    /// rule over its subtree; the MILP was solved again without it.
+    NotInstallable,
 }

crates/types/src/plan/materialized_view/mod.rs

+impl InstallableDagError {
+    /// The node whose subtree a subtree-local rule refuses (Stage 3 selects again
+    /// without its candidate); None for a retention rule or an invalid DAG. todo!() here.
+    pub fn node_id(&self) -> Option<PhysicalASAPNodeId>;
+}

Usage: after implementation, select_plan builds the plan with plan_of and calls plan.materialized_views() and plan.raw_data_retention(). On a subtree-local error it reads error.node_id(), adds PrunedCandidate { candidate, reason: NotInstallable } and solves again. On any other error it returns the error. a_view_over_a_whole_table_is_reported_not_installable checks this end to end.

Design scope

  • crates/integration-tests/tests/plan_enumeration/: test support at the Stage 3 / plan boundary. It holds every plan of a physical memo under the concepts doc's execution-context rules. It is general: any memo without @ anchors.
  • crates/types/tests/installable_dag_error.rs: tests of the installer's error type in types::plan.
  • NotInstallable is a general Stage 3 prune reason, not tied to one rule.

Algorithm

The oracle runs a depth-first enumeration:

  • Each query output picks an execution of its group, in its evaluation context.
  • Each input of each new execution picks a producer over an admissible edge, where the producer delivers on the context (delivers_on). A materialized producer runs in Materialized(v), other ingestion producers in EVERY_ARRIVING_ROW, and the inputs of an ingestion view on its refresh window's pipeline.
  • A non-materialized query-time result has one reader.
  • Each complete choice is checked for acyclicity and (X), then built into a PhysicalASAPDAG. Each node gets the least retention its plan's readers need (read_back_ms), and the plan is costed with plan_cost.
  • Cost is exponential in the alternatives per input. The cap is 400,000 choices, and the largest corpus memo has 10,000. A whole debug run takes 97 s on 40 threads.
  • Limits: @ anchors are refused, and accuracy (K) and latency bounds are not modeled (Stage 3 has neither yet).

Decisions and open questions

Needs reviewer decision. These are the provisional answers the tests encode. The full list is in /mydata/notes/test-plans/fix3-and-optimality.md, section 1.5.

  • A1. node_id() is Some only for the subtree-local rules: UnsupportedRawDataWindow (its TimeRange), ViewReadsUnwindowedSource, ViewWithoutCoverage and IngestionBuildReadsWindowsOfDifferentOffsets. It is None for the 4 retention rules and InvalidDag. A name like subtree_local_node_id would be clearer.
  • A2. Exclusion is global, as approved. A candidate refused under one reader may install under another, but no workload on this head reaches UnsupportedRawDataWindow. So this is not tested.
  • A3. When nothing installs, any Err is accepted. Recommendation: a new ErrorSelectingPlan::NotInstallable(InstallableDagError).
  • A4. Reader-dependent errors cannot be reached through select_plan. They are covered by the classification test and by stage3_retains_at_least_what_its_plans_readers_need.
  • A5. The NotInstallable test assumes the cheapest plan is unique: no ties.
  • A6. An enumerated plan's retention is the least its own readers need (what the installer accepts).
  • A7. Tolerance is a relative 1e-6.
  • A8. The one-consumer rule is a plan-validity rule.
  • A10. Fix 3 checks raw_data_retention() as well as materialized_views(). Executor compile is not checked.

Findings for the implementation

Verification

Scope Passed Failed Notes
cargo test -p asap-integration-tests --test integration stage3_installable_optimal_selection:: 28 4 all 4 expected before Fix 3 (table below)
cargo test -p asap-types --test types installable_dag_error:: 0 1 expected: todo!()
cargo fmt --all -- --check, cargo clippy -p asap-types -p asap-integration-tests --all-targets clean
Test Result Purpose
cheapest_<workload> ×19 (p01–p11, s01, y13, r5, subquery 1 h, b08, b09, SQL whole table ±rows_held) 17 pass, 2 fail (both SQL whole table: a refused plan is returned) selected plan_cost ≤ (1+1e-6) × the oracle's minimum installable; Err exactly when none installs
every_selected_plan_passes_the_install_rules fail (SQL whole table) I1 over the corpus
a_view_over_a_whole_table_is_reported_not_installable fail (todo!) the refused candidate is reported NotInstallable; the plan installs
node_id_names_the_node_of_a_subtree_local_rule_only fail (todo!) A1 classification
the_oracle_space_holds_every_selected_plan pass the oracle builds Stage 3's plan, so the contract is not vacuous
stage3_retains_at_least_what_its_plans_readers_need pass Stage 3's retention ≥ the least need
every_selected_edge_is_admissible, no_unmaterialized_query_time_node_has_two_readers, no_query_reads_a_sketch_of_a_group_it_must_read_exactly, no_candidate_reported_not_installable_is_selected pass feasibility kept through the re-solve
the_objective_is_the_plan_cost_over_the_corpus, the_selected_cost_does_not_depend_on_query_order pass I4 and I7
y13_selects_an_installable_plan_and_no_shift_view_over_arriving_rows, a_negative_offset_has_no_installable_plan_and_stage3_returns_an_error, a_ratio_subquery_over_raw_ranges_selects_an_installable_plan pass regressions

Optimality on the current head: no violations. Selected cost = oracle minimum on all 17 workloads that have an installable plan.

Follow-ups

Next: implementation after review. That is Fix 3 in select_plan and InstallableDagError::node_id, on this branch's tests. Also:

  • extend the oracle to @ anchors and to (K) once it lands;
  • find an input that reaches UnsupportedRawDataWindow (A2).

@zzylol zzylol added optimizer Workload optimization and plan selection testing Corpus, regression, differential, or benchmark tests labels Nov 15, 2026
@zzylol
zzylol force-pushed the refactor/1207-contexts branch from 796d435 to 44a223d Compare November 15, 2026 15:54
@zzylol
zzylol force-pushed the test/installable-selection-and-optimality branch from 7ccf511 to e17fc55 Compare November 15, 2026 17:16
…ty contract (tests only)

An oracle enumerates every plan of Stage 2's physical memo without
Stage 3's coefficients or MILP, costs each with plan_cost and checks the
planner's install rules. The tests check that Stage 3 returns a plan only
when one installs, that the plan installs, and that it is the cheapest
installable plan, over 12 small corpus workloads and the regression
workloads (y13, the SQL whole-table view, the subquery shape, the scan
feeding two chain widths, negative offsets).

Adds PruneReason::NotInstallable and a todo!() stub of
InstallableDagError::node_id; no implementation.
@zzylol
zzylol force-pushed the test/installable-selection-and-optimality branch from e17fc55 to 11c2d25 Compare November 16, 2026 01:45
@zzylol
zzylol changed the base branch from refactor/1207-contexts to main November 16, 2026 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

optimizer Workload optimization and plan selection testing Corpus, regression, differential, or benchmark tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant