Skip to content

test(integration): hand-computed cost and execution oracles (#1253 T4) - #1259

Draft
zzylol wants to merge 1 commit into
test/1253-t2-contractsfrom
test/1253-t4-oracles
Draft

zzylol wants to merge 1 commit into
test/1253-t2-contractsfrom
test/1253-t4-oracles

Conversation

@zzylol

@zzylol zzylol commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: no test computed a cost or an answer independently of the planner: o11y q21's plan (Tracking: restructure ASAPPlanner tests around layers and independent oracles #1253 Example 1) was costed with one window where its TimeRange over a TimeShift over a chain selects five (0.8939/s instead of 0.9020/s), and every test passed, because Stage 3 and plan_cost share node_resource_usage_and_cost (the T2 contracts also pass on that code). No PromQL plan built on a sketch had its answers checked by the executor.
  • Change: two oracles in crates/integration-tests: cost_oracle.rs computes the expected cost of five canonical shapes (query-time Scan, ingestion-time chain, TimeRange over a chain, TimeRange over a TimeShift over a chain, SummaryMerge) from counts stated in each test, and compares each node's plan_cost; execution_oracle.rs runs seven selected plans through the executor on synthetic data (200 series, 30 min, fixed seed, values drifting each minute so a wrong window moves an answer outside ε) and compares their answers with an exact evaluation written in the test.
  • Effect: on the stack head every oracle holds (no bug found); with the refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237 (D8) hunk reverted locally, the q21 test fails: node 3 (TimeRange): plan_cost says 1016.6666666666666, the oracle 5083.333333333333, while the contracts still pass.

Review guide

Behavior: before → after

cargo test -p asap-integration-tests --test integration oracle against the q21 shape, sum(rate(m[5m] offset 1h)) as a 1 h range query (step 60 s) every 60 s over 1,000 counter series sampled every second:

Before the #1237 fix (D8 hunk reverted locally) After (this stack)
Plan Scan → chain (60 s, exact rate) → TimeShift 1h → TimeRange 5m → SummaryMerge → finalize → sum same
TimeRange CPU per second, plan_cost 1,016.67 (one window: 1,000 states × 61/60) 5,083.33
Oracle: 5 windows × 1,000 states × 61/60 5,083.33 5,083.33
a_time_range_over_a_shifted_chain_merges_every_selected_window fails passes
T2 contracts (MILP objective = plan_cost, …) on o11y pass pass

It serves #1253's acceptance items "hand-computed cost tests cover the canonical shapes, including Example 1, which fails on the pre-fix code" and "the execution oracle runs on synthetic data for exact and approximate queries".

Interfaces

None outside the integration test binary. Inside it: physical_common::execute(plan, store, scope, limits: Limits) gains its limits argument; cost_oracle::plan_one_promql_query(query, accuracy, source, range_evaluation, capabilities) -> PlannedWorkload and cost_oracle::operator_kinds(&PhysicalASAPDAG) -> Vec<&'static str> are the two oracles' shared helpers (pub(crate)).

Algorithm changes

None in the planner. The oracles' method:

  • Cost oracle. Plan with the default cost inputs (the plan Stage 3 selects; the test asserts its operator sequence), then cost that plan with plan_cost under cost inputs that charge 1 per CPU operation only, or 1 per retained byte-second only. Expected per node = executions per second ρ_κ × operations per execution, or retained windows × states × 56 bytes. The counts are the oracle's (e.g. q21: ρ = (3,600/60 + 1)/60 = 61/60 for query-time nodes, 5 windows of 1,000 states selected, retention (1 h offset + 5 min range + 1 h span)/60 s = 125 windows plus the one being built); the per-operation work (one operation per row, state or accumulator; an exact accumulator build 1 + depth 1; a time shift 0) and the 56-byte exact state are the cost model's, stated in the module doc. Relative tolerance 1e-9.
  • Execution oracle. Each sample's value is uniform in [0, 100) plus 10 × its minute, so each 60 s window has its own range and reading a wrong window moves a quantile far outside ε (reading only the newest of the 5 windows puts the 0.9 quantile at rank ≈ 0.98). The plan's materialized views (chains) are refreshed per window their retention keeps, newest ending at T = 30 min, each in an ingestion run bound the samples of that window and its neighbours (time-sorted samples, sliced by binary search; each series' identity encoded once); the query run at T then reads them, bound raw data only when the deployment keeps it. Answers are keyed by instance and read from the value column the test names (sum for an aggregate, value per series); a second answer for one series fails. Exact: equal to 1e-9 relative. Approximate (rank ε 0.05 / relative ε 0.05 at δ 0.01 per value, 10,000,000 series declared so a sketch is cheaper): at most δ × 200 = 2 of the 200 values may be off.

Decisions and open questions

Needs reviewer decision

  • Isolating cost inputs. The cost oracle checks CPU counts and the chain's retained bytes under single-resource cost inputs, not each node's full default cost: the default cost adds working memory held for ops × ms_per_cpu_op per execution, which would make the oracle restate the memory model term by term. Alternative: also hand-compute the default-cost total for q21 (0.9004/s on this workload).
  • Pre-fix evidence. Produced by reverting the D8 hunk in a local tree (the two TimeShift arms in asap_cost::usage that carry tumbling_windows and chain_tumbling_window_width_ms), not by running backup/1207-contexts-08164285c: that backup is PR 7's head after D8; the pre-D8 head is backup/1207-contexts-a1493df05. Not pushed.

Decided

  • 2026-11-10 (Tracking: restructure ASAPPlanner tests around layers and independent oracles #1253): synthetic data for the execution oracle; a known executor gap is ignored with its issue rather than reshaping the test (none was needed for these seven queries).
  • 2026-11-10, simplifications from the review: one charging_only(unit_costs) cost-inputs helper (with CPU_OPERATION and RETAINED_BYTE_SECOND); one shared one-query planning helper (plan_one_promql_query, with an optional range_evaluation for q21); one operator_kinds helper; ViewState { batch, start_ms, end_ms }; run takes one node id.
  • 2026-11-10, kept after the review (finding 10): the cost tests assert the selected plan's operator sequence. The oracle's counts are for that shape only; if Stage 3 selects another plan the test fails with "the selected plan is not the canonical shape", which names the cause, rather than with a wrong count. The module doc says so.

Independent review, round 1: findings fixed

  1. Duplicate answer rows silently dropped: answers panics on a second answer for one series.
  2. Same data in every window: values drift 10 per minute (above), so a plan that reads the wrong windows fails the approximate tests too.
  3. Query run given raw data on deployments that keep none: the query run is bound raw data only when raw_data_retained; the no-raw-data tests pass, so their query runs read no raw data.
  4. Value read as "the last float column": read from the column the test names.
  5. Matrix not updated: testing.md updates the PromQL min/max, quantile and offset cells, the operator × time shape × accuracy cells (repeating exact sum, max, offset; repeating ε quantile), and gaps 2–4.
  6. Third copy of the executor helper: physical_common::execute takes limits, and the oracle reuses it and raw_data_store.
  7. A forbidden term in a doc comment: now "consumer".
  8. "+1 window being built" undocumented: the module doc lists it as a cost-model term, and the tests name it window_being_built.
  9. Unnamed numbers: GROUPING_KEYS and ACCUMULATORS, used at every aggregate.
  • Efficiency: each series' identity encoded once; samples sorted by time once and sliced per window (the oracle tests now take about 1.3 s, 7.4 s before).

Skipped

  • Planning the chain workload once for the three chain cost tests (OnceLock): small cost, and separate tests keep separate names.

Verification

Scope Passed / failed / ignored Newly failing vs base Notes
asap-integration-tests --test integration 173 / 0 / 6 0 base (T2) 161 / 0 / 6; +5 cost oracle, +7 execution oracle
... cost_oracle:: execution_oracle:: kll_tumbling_window_execution:: 15 / 0 / 0 0 both oracles about 1.3 s; kll_tumbling_window_execution is the other physical_common::execute caller
... cost_oracle:: contracts:: execution_oracle::, D8 hunk reverted locally (not pushed) 19 / 1 / 1 (expected) node 3 (TimeRange): plan_cost says 1016.6666666666666, the oracle 5083.333333333333; every contracts test still passes on that tree
cargo fmt --check; cargo clippy -p asap-devtools -p asap-integration-tests --tests --bins -D warnings clean

New tests:

  • a_query_time_scan_costs_the_rows_its_lookback_reads: sum(m): Scan, instant TimeRange, Aggregate at 1,000 rows each, 1/60 per second.
  • an_ingestion_time_chain_costs_its_arriving_rows_and_retains_its_windows: 1,000 + 2,000 operations per second; 6 × 1,000 × 56 bytes retained.
  • a_time_range_over_a_chain_selects_the_windows_its_range_covers: 5 windows × 1,000 states per evaluation.
  • a_summary_merge_reads_every_selected_window_state: 5,000 states merged; plan total 3,200 operations per second.
  • a_time_range_over_a_shifted_chain_merges_every_selected_window: Tracking: restructure ASAPPlanner tests around layers and independent oracles #1253 Example 1 (fails without refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237's D8 fix).
  • an_exact_instant_sum_equals_the_sum_of_the_latest_samples, an_exact_range_sum_per_series_equals_the_window_sums, an_exact_sum_over_chain_windows_equals_the_sum_of_the_window, an_exact_sum_over_shifted_chain_windows_equals_the_shifted_window_sum, an_exact_max_over_chain_windows_equals_the_window_maximum: exact answers equal.
  • an_approximate_quantile_is_within_its_{rank,relative}_error_for_all_but_delta_of_the_series: at most 2 of 200 values off by more than ε.

Follow-ups

  • No bug found, so no issue filed.
  • Not covered yet: range-query execution (the executor is run at one evaluation time), rate/increase (needs Prometheus's extrapolation in the exact evaluator), top-k heaps, SQL, MetricSQL; and the exhaustive-search oracle (T5).

@zzylol zzylol added the testing Corpus, regression, differential, or benchmark tests label Nov 10, 2026
@zzylol
zzylol force-pushed the test/1253-t2-contracts branch from 5752dbe to fc6cec8 Compare November 10, 2026 02:43
@zzylol
zzylol force-pushed the test/1253-t4-oracles branch from 35f8977 to fd3b8b6 Compare November 10, 2026 02:43
… shapes (q21's TimeRange over a shifted chain fails without the #1237 fix) and execution of selected plans on synthetic data against exact evaluation (#1253 T4)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing Corpus, regression, differential, or benchmark tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant