Skip to content

test(integration): regression guard for each query meeting its own accuracy target over a shared readout group - #1384

Open
zzylol wants to merge 1 commit into
mainfrom
test/per-query-accuracy-targets
Open

zzylol wants to merge 1 commit into
mainfrom
test/per-query-accuracy-targets

Conversation

@zzylol

@zzylol zzylol commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Review guide

  • Position: base main (ca4be93). Not stacked.
  • Read crates/integration-tests/tests/per_query_accuracy_targets.rs first: query_meets_its_target is the independent check, and the four tests follow it.
  • crates/integration-tests/tests/main.rs only registers the module.

Behavior: before → after

No planner behavior changes. What the tests pin, for one series of latency at 10 Hz, every 60 s, no raw data retained:

Workload Before #1238 (79d9932) Main (and this PR)
p99 at rank 0.05 + p99 at rank 0.001 one KLL, k = 52 (0.001 missed) one KLL, k = 2,863
l2_over_time at relative 0.1 + 0.05 UnivMon, 624 columns (0.05 missed) UnivMon, 2,367 columns
p99 at rank 0.05 + sort(p99) at rank 0.001 not checked each meets its target

This serves correctness: a plan must meet every query's own target.

Interfaces

None (tests only).

Algorithm changes

None.

Decisions and open questions

None.

Verification

Scope Passed / failed / ignored Newly failing vs base Notes
cargo test -p asap-integration-tests --test integration per_query_accuracy_targets:: 4 / 0 / 0 none (new tests) on ca4be93
cargo clippy -p asap-integration-tests --tests, cargo fmt clean n/a

New tests:

  • one_query_text_at_two_rank_targets_reads_a_kll_meeting_the_stricter: the p99 at rank 0.05 and 0.001, in both orders. Each query meets its own target, and the plan holds a KLL with k ≥ 2,863.
  • one_query_text_at_two_relative_targets_reads_a_univmon_meeting_the_stricter: l2_over_time(latency[24h]) at relative 0.1 and 0.05, in both orders.
  • two_query_texts_sharing_a_readout_group_each_meet_their_target: p99 and sort(p99), which root at different groups over one readout group, with the targets swapped between runs.
  • every_query_meets_its_own_target_on_the_selected_plan: a property test over 16 fixed-seed sets of 2–3 queries drawn from 9 (query, target) pairs (p99 rank and relative, p50, sort(p99), L2). Every query's own check holds on the selected plan.

The check is independent of the planner. From each query's root it walks to every SummaryEstimate, takes the θ of the state that readout reads, and requires asap_accuracy::theoretical_guarantee at the target's δ to be in an allowed metric and within ε.

Probe evidence. The probe is a metamorphic relation: adding a query never lowers the optimal cost, so {q at ε, q at ε/5} must cost at least {q at ε/5} alone. For every query in the sketch-heavy workload set, it plans the pair in both orders, both as one text and with the looser query wrapped in max(...), and compares the cost with the strict query alone. All plans go through the production path:

Commit Pairs checked Violations Errors
79d9932 (before #1238; (K) not built) 330 190 (KLL, UnivMon, KMV, CMS+heap) 0
2f3664d (#1238 merged) 330 0 0

Follow-ups

…ies share a readout group at different targets

Regression tests through the production stages: one p99 text at rank 0.05
and 0.001 (KLL), one L2 text at relative 0.1 and 0.05 (UnivMon), two texts
sharing the p99 readout group, and a fixed-seed property over query sets
checking each query's readouts against its own target on the selected plan.
@zzylol zzylol added accuracy Accuracy requirements, guarantees, and propagation testing Corpus, regression, differential, or benchmark tests labels Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accuracy Accuracy requirements, guarantees, and propagation testing Corpus, regression, differential, or benchmark tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant