Repository navigation
Conversation
zzylol
force-pushed
the
test/1253-t2-contracts
branch
from
November 10, 2026 02:43
5752dbe to
fc6cec8
Compare
zzylol
force-pushed
the
test/1253-t4-oracles
branch
from
November 10, 2026 02:43
35f8977 to
fd3b8b6
Compare
zzylol
force-pushed
the
test/1253-t2-contracts
branch
from
November 10, 2026 13:37
fc6cec8 to
1bb8557
Compare
zzylol
force-pushed
the
test/1253-t4-oracles
branch
from
November 10, 2026 13:37
fd3b8b6 to
4295564
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
TimeRangeover aTimeShiftover a chain selects five (0.8939/s instead of 0.9020/s), and every test passed, because Stage 3 andplan_costsharenode_resource_usage_and_cost(the T2 contracts also pass on that code). No PromQL plan built on a sketch had its answers checked by the executor.crates/integration-tests:cost_oracle.rscomputes the expected cost of five canonical shapes (query-timeScan, ingestion-time chain,TimeRangeover a chain,TimeRangeover aTimeShiftover a chain,SummaryMerge) from counts stated in each test, and compares each node'splan_cost;execution_oracle.rsruns seven selected plans through the executor on synthetic data (200 series, 30 min, fixed seed, values drifting each minute so a wrong window moves an answer outside ε) and compares their answers with an exact evaluation written in the test.node 3 (TimeRange): plan_cost says 1016.6666666666666, the oracle 5083.333333333333, while the contracts still pass.Review guide
test/1253-t2-contracts), stacked on test(devtools): one corpus registry and production-path workload builders (#1253 T1) #1256, docs(testing): testing strategy, tiers and a measured coverage matrix (#1253 T0) #1254 and refactor: tools and snapshots over the #1207 memo, edges and contexts (#1207 PR 9) #1239. One commit, headfd3b8b63b. Top of the test stack.cost_oracle.rsmodule doc (what is independent and what is the cost model's), thena_time_range_over_a_shifted_chain_merges_every_selected_window;execution_oracle.rs:answers(the stand-in runtime), then the two approximate tests' off-by-ε rules.main.rsadds the two modules;physical_common::executetakes aLimits(its three callers inkll_tumbling_window_execution.rspassLimits::default()), so the execution oracle reuses it andraw_data_storeinstead of its own copy;testing.mdnames the files, describes the data and the cost inputs, and updates the coverage matrix cells and gaps 2–4 this PR fills.Behavior: before → after
cargo test -p asap-integration-tests --test integration oracleagainst the q21 shape,sum(rate(m[5m] offset 1h))as a 1 h range query (step 60 s) every 60 s over 1,000 counter series sampled every second:TimeRangeCPU per second,plan_costa_time_range_over_a_shifted_chain_merges_every_selected_windowplan_cost, …) on o11yIt serves #1253's acceptance items "hand-computed cost tests cover the canonical shapes, including Example 1, which fails on the pre-fix code" and "the execution oracle runs on synthetic data for exact and approximate queries".
Interfaces
None outside the
integrationtest binary. Inside it:physical_common::execute(plan, store, scope, limits: Limits)gains itslimitsargument;cost_oracle::plan_one_promql_query(query, accuracy, source, range_evaluation, capabilities) -> PlannedWorkloadandcost_oracle::operator_kinds(&PhysicalASAPDAG) -> Vec<&'static str>are the two oracles' shared helpers (pub(crate)).Algorithm changes
None in the planner. The oracles' method:
plan_costunder cost inputs that charge 1 per CPU operation only, or 1 per retained byte-second only. Expected per node = executions per second ρ_κ × operations per execution, or retained windows × states × 56 bytes. The counts are the oracle's (e.g. q21: ρ = (3,600/60 + 1)/60 = 61/60 for query-time nodes, 5 windows of 1,000 states selected, retention (1 h offset + 5 min range + 1 h span)/60 s = 125 windows plus the one being built); the per-operation work (one operation per row, state or accumulator; an exact accumulator build 1 + depth 1; a time shift 0) and the 56-byte exact state are the cost model's, stated in the module doc. Relative tolerance 1e-9.instanceand read from the value column the test names (sumfor an aggregate,valueper series); a second answer for one series fails. Exact: equal to 1e-9 relative. Approximate (rank ε 0.05 / relative ε 0.05 at δ 0.01 per value, 10,000,000 series declared so a sketch is cheaper): at most δ × 200 = 2 of the 200 values may be off.Decisions and open questions
Needs reviewer decision
ops × ms_per_cpu_opper execution, which would make the oracle restate the memory model term by term. Alternative: also hand-compute the default-cost total for q21 (0.9004/s on this workload).TimeShiftarms inasap_cost::usagethat carrytumbling_windowsandchain_tumbling_window_width_ms), not by runningbackup/1207-contexts-08164285c: that backup is PR 7's head after D8; the pre-D8 head isbackup/1207-contexts-a1493df05. Not pushed.Decided
charging_only(unit_costs)cost-inputs helper (withCPU_OPERATIONandRETAINED_BYTE_SECOND); one shared one-query planning helper (plan_one_promql_query, with an optionalrange_evaluationfor q21); oneoperator_kindshelper;ViewState { batch, start_ms, end_ms };runtakes one node id.Independent review, round 1: findings fixed
answerspanics on a second answer for one series.raw_data_retained; the no-raw-data tests pass, so their query runs read no raw data.testing.mdupdates the PromQL min/max, quantile and offset cells, the operator × time shape × accuracy cells (repeating exact sum, max, offset; repeating ε quantile), and gaps 2–4.physical_common::executetakeslimits, and the oracle reuses it andraw_data_store.window_being_built.GROUPING_KEYSandACCUMULATORS, used at every aggregate.Skipped
OnceLock): small cost, and separate tests keep separate names.Verification
asap-integration-tests --test integration... cost_oracle:: execution_oracle:: kll_tumbling_window_execution::kll_tumbling_window_executionis the otherphysical_common::executecaller... cost_oracle:: contracts:: execution_oracle::, D8 hunk reverted locally (not pushed)node 3 (TimeRange): plan_cost says 1016.6666666666666, the oracle 5083.333333333333; every contracts test still passes on that treecargo fmt --check;cargo clippy -p asap-devtools -p asap-integration-tests --tests --bins -D warningsNew tests:
a_query_time_scan_costs_the_rows_its_lookback_reads:sum(m): Scan, instant TimeRange, Aggregate at 1,000 rows each, 1/60 per second.an_ingestion_time_chain_costs_its_arriving_rows_and_retains_its_windows: 1,000 + 2,000 operations per second; 6 × 1,000 × 56 bytes retained.a_time_range_over_a_chain_selects_the_windows_its_range_covers: 5 windows × 1,000 states per evaluation.a_summary_merge_reads_every_selected_window_state: 5,000 states merged; plan total 3,200 operations per second.a_time_range_over_a_shifted_chain_merges_every_selected_window: Tracking: restructure ASAPPlanner tests around layers and independent oracles #1253 Example 1 (fails without refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237's D8 fix).an_exact_instant_sum_equals_the_sum_of_the_latest_samples,an_exact_range_sum_per_series_equals_the_window_sums,an_exact_sum_over_chain_windows_equals_the_sum_of_the_window,an_exact_sum_over_shifted_chain_windows_equals_the_shifted_window_sum,an_exact_max_over_chain_windows_equals_the_window_maximum: exact answers equal.an_approximate_quantile_is_within_its_{rank,relative}_error_for_all_but_delta_of_the_series: at most 2 of 200 values off by more than ε.Follow-ups
rate/increase(needs Prometheus's extrapolation in the exact evaluator), top-k heaps, SQL, MetricSQL; and the exhaustive-search oracle (T5).