Skip to content

test: update six tests made stale by the Stage 3 refactor and measured unit costs - #1389

Open
zzylol wants to merge 1 commit into
mainfrom
fix/stale-tests-after-measured-costs
Open

zzylol wants to merge 1 commit into
mainfrom
fix/stale-tests-after-measured-costs

Conversation

@zzylol

@zzylol zzylol commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Review guide

  • Base: main (ca4be93f0). Not stacked.
  • Read first: crates/integration-tests/tests/joint_search_equivalence.rs (two tests rewritten, one split out and ignored), then plan_sharing.rs (nested_exact_sum_reuses_newest_tumbling_windows).
  • Mechanical: the fixture deletion in materialized_views.rs and the helper in crates/integration-tests/src/lib.rs.
  • No production code changes.

Behavior: before → after

One justification per test. Every test failed at 9ad2c5422 (#1236), 6b4847c9a, 14308fd2c (#1355), 4cb4110c1 (#1363) and ca4be93f0 (main), with the same assertion.

Test Why the expectation is stale Before → after
joint_search_equivalence::rootless_request_overhead_respects_capacity_and_latency It expects 1 every 1 ms at 2 ms per request to be refused for max_latency_ms: 1 and for a 1-core cluster. Stage 3 has no latency rows and supports cost model 1 only. Both were decided: latency (L) and resource budgets are out of scope for the selection MILP (#1204 item 5, 2026-11-06), and cost models 2 and 3 return UnsupportedCostModel. #1316 tracks bounding latency. Today the capacity half passes only because cost model 3 is refused, and the latency half returns a plan. Fails → split in two: rootless_request_overhead_is_charged (new, passes) checks that the selected cost is exactly the request overhead, 2,000/s for 1,000 requests/s x 2,000,000 operations, with no operator node; the old test is #[ignore]d on #1316 and now asserts a refusal that is not UnsupportedCostModel.
joint_search_equivalence::unsupported_exact_operation_never_returns_an_unvalidated_plan It expects stddev_over_time(m[5m]) (exact) to find no plan. Exact plans of any operator are planned from raw rows; the selected plan is Scan → TimeRange(5m) → Aggregate { PerEntity, [StdDev] }, which validates, is exact, and the executor lowers (AggIntent::StdDev in crates/executor/src/physical_planner/mod.rs). Executor support does not restrict planner design. Fails → renamed exact_operation_without_a_summary_plans_from_raw_rows: asserts a valid exact plan with a StdDev aggregate.
materialized_views::materialized_views_equal_the_stored_outputs_main_listed Its fixture (stored_outputs_before_view_coverage.json) holds DAGs in the IR wire format from before #1226: SummaryAgg has family: {"Sketch": [{algorithm, params}, "PerSubpopulationInstance"]}, which no longer deserializes (invalid type: sequence, expected string or map). Since #1236 a view's coverage is the node's view_coverage, which the planner sets and the fixture lacks, so converting the DAGs cannot make it pass either. It compared the derived views with the stored_outputs field that was deleted when views were derived from the DAG. Live plans' views are checked by crates/planner/tests/runtime_install.rs, sql_table_scans.rs, columnar_batches.rs and crates/planner/src/installable/tests.rs. Fails → test and fixture deleted; the module keeps the wire-version test.
plan_sharing::nested_exact_sum_reuses_newest_tumbling_windows It expects sum(sum_over_time(m_gauge[1m])) added to the 2m sum (both every 60 s, 10,000 series) to cost at most 0.1% more, by reading the 2m chain's newest window. With raw data kept, both are answered at query time: 0.0857/s together, 0.0590/s for the 2m sum alone. The plan that shares one ingestion-time chain costs 2.159/s under the measured unit costs (#1355, #1363), 25 times more, so no cheaper plan exists and the 45% is each query's own scan window and sum. Fails → the reuse is checked where a chain is needed: with raw_data_retained: false, the plan has one SummaryTumblingWindowChain and both roots read it.
promql_to_post_asap::promql_quantile_of_rate_binds_kll_over_rate_accumulator logical_candidates returns the plan Stage 3 selects, not every candidate. Under the default unit costs, the selected plan for this query on 1,000 series is the exact Aggregate { Quantile } over the finalized Rate (0.0060/s), so there is no KLL to inspect. Other shape tests already cost summaries' CPU free for this reason (summary_cpu_free, #1355). Fails → candidates_using uses the new logical_candidates_with_summary_cpu_free (accuracy still enforced). The selected plan is the pinned shape: SummaryEstimate ← SummaryAgg(Kll) ← FinalizeExactAccumulator ← SummaryAgg(Rate).
asap-planner summary_sharing::identical_ungrouped_queries_share_their_producers It expects two shared summary states for sum(rate(x[5m])) twice ("rate, then sum"). The plan has one: the per-series Rate accumulator, with the sum over the finalized rates as a plain exact Aggregate. Whether the sum is a state or an aggregate is a cost choice between two exact candidates (the same shape the quantile test above shows). The sharing it tests holds: both queries read the same states. Fails (left: 1, right: 2) → asserts the shared states and that the plan deploys exactly one query's states.

Interfaces

Test-only. crates/integration-tests/src/lib.rs

 pub mod planning {
+    /// [`logical_candidates`] with summaries' CPU costed free
+    /// ([`summary_cpu_free`]) and accuracy enforced: the selected plan reads a
+    /// summary candidate wherever Stage 1 offers one that meets `accuracy`.
+    /// For tests of a summary plan's shape.
+    pub fn logical_candidates_with_summary_cpu_free(
+        root: Rc<OperatorNode>,
+        accuracy: AccuracyTarget,
+    ) -> Vec<Rc<OperatorNode>>;
 }

Usage: promql_quantile_of_rate_binds_kll_over_rate_accumulator lowers quantile(0.99, rate(http_requests_total[5m])) at max_error(0.01), plans it over 1,000 series at 1,000 rows/s with this helper, and gets the KLL plan above (cost 0.00113/s with summaries' CPU free), whose nodes the test then checks one by one.

Algorithm changes

None.

Decisions and open questions

Verification

Scope Command Result
Changed integration tests cargo test -p asap-integration-tests --test integration -- joint_search_equivalence materialized_views plan_sharing promql_to_post_asap 25 passed, 2 ignored (#1316, #1142), 1 failed: slower_repeat_reuses_faster_sum (#1385, unchanged)
Changed planner test cargo test -p asap-planner --test planner -- summary_sharing 3 passed, 1 ignored (unchanged)
Format cargo fmt --all -- --check clean

New test: rootless_request_overhead_is_charged checks that a query with no operator node still pays its request overhead.

Follow-ups

…asured costs made stale

- rootless request overhead: check its charge (2,000/s for `1` every
  1 ms at 2 ms per request); the latency and capacity half is ignored on
  #1316 (latency (L) and budgets are outside the selection MILP).
- stddev_over_time exact: planned from raw rows as a per-series StdDev
  aggregate, not refused.
- materialized views vs stored_outputs: fixture DAGs predate the current
  IR wire format and carry no view_coverage; test and fixture deleted.
- nested exact sum: with raw data kept, query-time plans are 25x cheaper
  than a shared chain; the chain reuse is checked with no raw data.
- quantile of rate: the KLL shape is inspected with summaries' CPU free.
- identical sum(rate): sharing holds; the sum is an exact aggregate, so
  one shared state, not two.
@zzylol zzylol added cost-model Cost formulas, statistics, ranking, and selection optimizer Workload optimization and plan selection testing Corpus, regression, differential, or benchmark tests labels Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cost-model Cost formulas, statistics, ranking, and selection optimizer Workload optimization and plan selection testing Corpus, regression, differential, or benchmark tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant