Repository navigation
Conversation
…asured costs made stale - rootless request overhead: check its charge (2,000/s for `1` every 1 ms at 2 ms per request); the latency and capacity half is ignored on #1316 (latency (L) and budgets are outside the selection MILP). - stddev_over_time exact: planned from raw rows as a per-series StdDev aggregate, not refused. - materialized views vs stored_outputs: fixture DAGs predate the current IR wire format and carry no view_coverage; test and fixture deleted. - nested exact sum: with raw data kept, query-time plans are 25x cheaper than a shared chain; the chain reuse is checked with no raw data. - quantile of rate: the KLL shape is inspected with summaries' CPU free. - identical sum(rate): sharing holds; the sum is an exact aggregate, so one shared state, not two.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
promql_quantile_of_rate_binds_kll_over_rate_accumulatorlooks for a KLL in the plan Stage 3 selects forquantile(0.99, rate(http_requests_total[5m])), but on 1,000 series the exact sort is cheaper (0.0060/s) and is selected.asap-integration-testsgoes from 6 failures to 1 (slower_repeat_reuses_faster_sum, a planner bug filed as Stage 3 does not share a result between queries on nested evaluation grids: the same exact sum every 60 s and every 120 s runs twice #1385), andasap-planner's tests go from 1 failure to none.Review guide
main(ca4be93f0). Not stacked.crates/integration-tests/tests/joint_search_equivalence.rs(two tests rewritten, one split out and ignored), thenplan_sharing.rs(nested_exact_sum_reuses_newest_tumbling_windows).materialized_views.rsand the helper incrates/integration-tests/src/lib.rs.Behavior: before → after
One justification per test. Every test failed at
9ad2c5422(#1236),6b4847c9a,14308fd2c(#1355),4cb4110c1(#1363) andca4be93f0(main), with the same assertion.joint_search_equivalence::rootless_request_overhead_respects_capacity_and_latency1every 1 ms at 2 ms per request to be refused formax_latency_ms: 1and for a 1-core cluster. Stage 3 has no latency rows and supports cost model 1 only. Both were decided: latency (L) and resource budgets are out of scope for the selection MILP (#1204 item 5, 2026-11-06), and cost models 2 and 3 returnUnsupportedCostModel. #1316 tracks bounding latency. Today the capacity half passes only because cost model 3 is refused, and the latency half returns a plan.rootless_request_overhead_is_charged(new, passes) checks that the selected cost is exactly the request overhead, 2,000/s for 1,000 requests/s x 2,000,000 operations, with no operator node; the old test is#[ignore]d on #1316 and now asserts a refusal that is notUnsupportedCostModel.joint_search_equivalence::unsupported_exact_operation_never_returns_an_unvalidated_planstddev_over_time(m[5m])(exact) to find no plan. Exact plans of any operator are planned from raw rows; the selected plan isScan → TimeRange(5m) → Aggregate { PerEntity, [StdDev] }, which validates, is exact, and the executor lowers (AggIntent::StdDevincrates/executor/src/physical_planner/mod.rs). Executor support does not restrict planner design.exact_operation_without_a_summary_plans_from_raw_rows: asserts a valid exact plan with aStdDevaggregate.materialized_views::materialized_views_equal_the_stored_outputs_main_listedstored_outputs_before_view_coverage.json) holds DAGs in the IR wire format from before #1226:SummaryAgghasfamily: {"Sketch": [{algorithm, params}, "PerSubpopulationInstance"]}, which no longer deserializes (invalid type: sequence, expected string or map). Since #1236 a view's coverage is the node'sview_coverage, which the planner sets and the fixture lacks, so converting the DAGs cannot make it pass either. It compared the derived views with thestored_outputsfield that was deleted when views were derived from the DAG. Live plans' views are checked bycrates/planner/tests/runtime_install.rs,sql_table_scans.rs,columnar_batches.rsandcrates/planner/src/installable/tests.rs.plan_sharing::nested_exact_sum_reuses_newest_tumbling_windowssum(sum_over_time(m_gauge[1m]))added to the 2m sum (both every 60 s, 10,000 series) to cost at most 0.1% more, by reading the 2m chain's newest window. With raw data kept, both are answered at query time: 0.0857/s together, 0.0590/s for the 2m sum alone. The plan that shares one ingestion-time chain costs 2.159/s under the measured unit costs (#1355, #1363), 25 times more, so no cheaper plan exists and the 45% is each query's own scan window and sum.raw_data_retained: false, the plan has oneSummaryTumblingWindowChainand both roots read it.promql_to_post_asap::promql_quantile_of_rate_binds_kll_over_rate_accumulatorlogical_candidatesreturns the plan Stage 3 selects, not every candidate. Under the default unit costs, the selected plan for this query on 1,000 series is the exactAggregate { Quantile }over the finalized Rate (0.0060/s), so there is no KLL to inspect. Other shape tests already cost summaries' CPU free for this reason (summary_cpu_free, #1355).candidates_usinguses the newlogical_candidates_with_summary_cpu_free(accuracy still enforced). The selected plan is the pinned shape:SummaryEstimate ← SummaryAgg(Kll) ← FinalizeExactAccumulator ← SummaryAgg(Rate).asap-plannersummary_sharing::identical_ungrouped_queries_share_their_producerssum(rate(x[5m]))twice ("rate, then sum"). The plan has one: the per-series Rate accumulator, with the sum over the finalized rates as a plain exactAggregate. Whether the sum is a state or an aggregate is a cost choice between two exact candidates (the same shape the quantile test above shows). The sharing it tests holds: both queries read the same states.left: 1, right: 2) → asserts the shared states and that the plan deploys exactly one query's states.Interfaces
Test-only.
crates/integration-tests/src/lib.rspub mod planning { + /// [`logical_candidates`] with summaries' CPU costed free + /// ([`summary_cpu_free`]) and accuracy enforced: the selected plan reads a + /// summary candidate wherever Stage 1 offers one that meets `accuracy`. + /// For tests of a summary plan's shape. + pub fn logical_candidates_with_summary_cpu_free( + root: Rc<OperatorNode>, + accuracy: AccuracyTarget, + ) -> Vec<Rc<OperatorNode>>; }Usage:
promql_quantile_of_rate_binds_kll_over_rate_accumulatorlowersquantile(0.99, rate(http_requests_total[5m]))atmax_error(0.01), plans it over 1,000 series at 1,000 rows/s with this helper, and gets the KLL plan above (cost 0.00113/s with summaries' CPU free), whose nodes the test then checks one by one.Algorithm changes
None.
Decisions and open questions
plan_sharing::slower_repeat_reuses_faster_sumstill fails. It is a planner bug (Stage 3 does not share a result between queries on nested evaluation grids: the same exact sum every 60 s and every 120 s runs twice #1385: execution contexts are one per evaluation grid, so the 120 s copy cannot read the 60 s copy's result at the instants both evaluate; a witness plan costs 0.0590/s against the selected 0.0885/s). This PR does not touch it. Should it be#[ignore]d on Stage 3 does not share a result between queries on nested evaluation grids: the same exact sum every 60 s and every 120 s runs twice #1385, asexact_coarse_sum_reuses_fine_sumis on Stage 3 keeps a MatAtQuery build whose NotMaterialized sibling avoids a second scan (#923 workload) #1142?Verification
cargo test -p asap-integration-tests --test integration -- joint_search_equivalence materialized_views plan_sharing promql_to_post_asapslower_repeat_reuses_faster_sum(#1385, unchanged)cargo test -p asap-planner --test planner -- summary_sharingcargo fmt --all -- --checkNew test:
rootless_request_overhead_is_chargedchecks that a query with no operator node still pays its request overhead.Follow-ups
slower_repeat_reuses_faster_sumpass).rootless_request_overhead_respects_capacity_and_latency).