Skip to content

test(integration): contracts suite over every corpus registry workload (#1253 T2) - #1258

Draft
zzylol wants to merge 1 commit into
test/1253-t1-corpus-buildersfrom
test/1253-t2-contracts
Draft

zzylol wants to merge 1 commit into
test/1253-t1-corpus-buildersfrom
test/1253-t2-contracts

Conversation

@zzylol

@zzylol zzylol commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: the invariants every plan must keep were checked in one place each, on one or two hand-picked workloads: MILP objective = plan_cost only in milp_objective.rs (two workloads, to 1e-6), no check that the plan's edges are admissible, that every query is served by its own group, or that the selected plan meets (K) when recomputed from the plan rather than read back from the MILP.
  • Change: crates/integration-tests/tests/contracts.rs: contract_violations(&PlannedWorkload, &CostInputs) checks ids resolve (memo, Pass 2 edges and pruned list, Stage 2 candidates and edges, plan nodes, roots), every surviving candidate's inputs and fused groups have a surviving candidate and every Stage 2 candidate's inputs an admissible edge, every plan edge is a Stage 2 edge with α^log = α^phys = 1, MILP objective = plan_cost to a relative 1e-9 (a selected plan without a cost per second is a violation), every node cost ≥ 0, every query root is a node of its query's group, and (K) recomputed over the plan as the MILP bounds it; one test per registry source, the 30 random mixed workloads in the full tier, planned in parallel. tier_markers.rs checks every full tier: / nightly tier: ignore sits in its tier module.
  • Effect: all 68 registry workloads meet every contract on the Tracking: implement the optimization formulation (memo candidates, properties, edges y_{u,c,i}, MILP) as named code per stage #1207 stack head (38 in the fast tier in about 7 s of the integration binary, 30 in the full tier in 11 s on 56 cores, 221 s before in series); no bug found. Tampering tests show the contracts are not vacuous (a tightened ε, a rank metric, δ = 1e-12, an inadmissible selected edge, a 0.1 % objective drift and swapped roots are each caught).

Review guide

Behavior: before → after

cargo test -p asap-integration-tests --test integration contracts:: on the #1207 stack head:

Before After
Workloads with MILP objective = plan_cost checked 2 (milp_objective.rs, relative 1e-6) 68 registry workloads + those 2, relative 1e-9
Plan edges admissible, ids resolve, roots served, node cost ≥ 0, every candidate input has a producer not checked on any planned workload checked on all 68
(K) on the selected plan accuracy_at_query_roots.rs checks θ of one example recomputed from readout requirements and error contracts on all 68; at least one sketch plan workload composes a nonzero error
A violation — one line per violation and workload, in the form <workload id>: (K) holds: query <q>'s root node <n> has error <e> in <metric>, its target allows <ε>

It serves #1253's acceptance item "every corpus workload passes the contracts".

Interfaces

None outside the test binary. contracts::contract_violations(planned: &PlannedWorkload, cost_inputs: &CostInputs) -> Vec<String> is pub inside the integration test binary only, for regression tests in that binary (milp_objective.rs uses it); cost_inputs are the ones the workload was planned with (T1 dropped PlannedWorkload.cost_inputs).

Algorithm changes

(K) recomputed over the selected plan (test only)

  • Problem: the only (K) evidence was the MILP's own accepted_accuracy_per_query, which is read from the MILP solution: a wrong encoding would agree with itself.
  • Steps (constraint_k_holds), concepts doc §4, "Query composition and root checks":
    1. For each plan node (children first, memoized), its logical candidate c: own error = the first input requirement's (metric, max_error) and δ, as the MILP reads it (only a readout's input carries one); a second input with a required accuracy is a violation.
    2. For each input i with error e > 0 in metric m: LogicalRequirements::error_propagation(memo, c.operator, m); None is a violation (metric rule on the selected path); an error above validity_cap_per_input[i] is a violation; else add factor_per_input[i] × e in m. Failure probabilities add (union bound). A node read on two paths contributes on both, as the MILP's worst case.
    3. At query q's root, as the MILP bounds it: at an operator root, every metric's error ≤ ε_q when the target names no metric, else the target metric's ≤ ε_q and every other metric's = 0; at a scalar root no error in any metric. δ ≤ failure_probability_per_value(n_q) only when the target sets a δ and the root is an operator root (no limit without a δ, or for a whole-result δ with n_q unknown). Slack 1e-9 relative.
  • Correctness: the same linear bound the MILP encodes per execution; it reuses error_propagation (the operators' contracts) but not the MILP's E/F/D/H variables or accepted_accuracy. It is a consistency check (testing doc).
  • Complexity: linear in plan nodes and metrics.
  • Limits: trusts each readout's requirement and the edge's α^log (checked by the admissibility contract) for the readout's own error; does not re-derive a sketch's guarantee from θ.

Decisions and open questions

Needs reviewer decision

  • "Every candidate's inputs exist" is read as: after Pass 2, every candidate not in pruned reads input groups and fuses groups that each keep a candidate not in pruned, and no logical edge joins a pruned candidate; every Stage 2 candidate expands a surviving logical candidate and each of its inputs has an admissible Stage 2 edge (holds on all 68 workloads). Alternative: only that input group ids resolve (weaker).

Decided

  • 2026-11-10: MILP objective = plan_cost at relative 1e-9 (task brief); it holds on all 68 workloads.
  • 2026-11-10: the random mixed workloads are in the full tier (mod full_tier, #[ignore = "full tier: …"]), planned in parallel with std::thread::scope inside the test.
  • 2026-11-10, kept as is after the review: violations stay strings matched by prefix (no ContractViolation enum); ComposedError keeps its Vec<(AccuracyMetric, f64)>; the source prefixes come from T1's corpus_registry::SOURCES.

Independent review, round 1: findings fixed

  1. δ check failed correct plans without a target δ: the δ limit is failure_probability.and(failure_probability_per_value(n)), skipped when None. Regression a_target_without_a_failure_probability_bounds_no_failure_probability (the ratio-of-quantiles sketch plan workload with δ removed composes δ = 2) fails with the old check ("fails with probability 2 per value, its target allows 1") and passes now.
  2. (K) looser than the MILP at scalar roots: a scalar root allows 0 error in every metric and has no δ limit, as the MILP.
  3. Weak non-vacuity: the sketch plan source asserts at least one targeted query composes a nonzero error; three tamper cases added (rank metric, δ = 1e-12, the selected edge marked not logically admissible via PhysicalASAPMemoCandidates::new). The o11y source does not assert it: its plans are exact (no query composes an error), as the snapshots doc says.
  4. Silent skip without a cost per second: a selected plan whose plan_cost has none is a violation.
  5. Partial "inputs exist": added fused groups keep a surviving candidate, no logical edge joins a pruned candidate, every Stage 2 candidate input has an admissible edge.
  6. Panic on damaged input: the contract returns after the memo and Stage 2 id checks when either found a violation, before anything looks candidates up by those ids.
  7. Own error summed over inputs: the first required accuracy is the own error, as the MILP; a second one is a violation.
  8. Misnamed helper costing twice: plan_meeting_contracts returns the MILP objective, which the contract checked equals plan_cost.
  9. Unwrapped doc line: rewrapped.
  • Tier markers (T0 decision): tier_markers.rs::every_tier_ignore_sits_in_its_tier_module.

Skipped

Verification

Scope Passed / failed / ignored Newly failing vs base Notes
asap-integration-tests --test integration 161 / 0 / 6 0 base (T1) 151 / 0 / 5; +8 contracts tests, +2 tier_markers tests, +1 ignored (full tier)
... -- --ignored full_tier 1 / 0 / 0 0 30 random mixed workloads in 11 s (debug, 56 cores; 221 s in series before)
cargo fmt --check; cargo clippy -p asap-devtools -p asap-integration-tests --tests --bins -D warnings clean

New tests:

  • every_{o11y_bench_workload,sketch_plan_workload,planner_layering_example,memo_example,fuzz_seed}_meets_the_contracts, full_tier::every_random_mixed_workload_meets_the_contracts: every registry workload of that source meets every contract.
  • every_registry_source_has_a_contracts_test: the sources tested here equal corpus_registry::SOURCES.
  • a_target_without_a_failure_probability_bounds_no_failure_probability: a ratio of two DDSketch quantiles with ε and no δ meets the contracts (regression of review finding 1).
  • a_tampered_target_edge_objective_or_root_violates_the_contracts: ε tightened to 1e-9, the rank metric and δ = 1e-12 each violate (K); an inadmissible selected edge violates admissibility and "inputs exist"; an objective 0.1 % above plan_cost violates the equality; swapping example 1's two roots leaves its queries unserved.
  • tier_markers::every_tier_ignore_sits_in_its_tier_module (and a unit test of its module reading): every full tier: / nightly tier: ignore reason in crates/ sits in a full_tier / nightly_tier module.

Follow-ups

  • T4 (oracles) stacks on this PR. T3 replaces other per-test copies of these checks while classifying the integration tests.
  • The fast tier holds 38 workloads; the random mixed sets could join it if CI has enough cores (11 s on 56).

@zzylol zzylol added the testing Corpus, regression, differential, or benchmark tests label Nov 10, 2026
@zzylol
zzylol force-pushed the test/1253-t1-corpus-builders branch from 9ada1ac to 083e2e3 Compare November 10, 2026 02:43
@zzylol
zzylol force-pushed the test/1253-t2-contracts branch from 5752dbe to fc6cec8 Compare November 10, 2026 02:43
… resolve, admissible plan edges, MILP objective = plan_cost (1e-9), non-negative node costs, served query roots, (K) recomputed over the plan (#1253 T2)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing Corpus, regression, differential, or benchmark tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant