Skip to content

feat(accuracy, cost, devtools): measured sketch accuracy table; delete the cost table's provenance and legacy format - #1372

Open
zzylol wants to merge 3 commits into
mainfrom
test/measured-sketch-accuracy
Open

zzylol wants to merge 3 commits into
mainfrom
test/measured-sketch-accuracy

Conversation

@zzylol

@zzylol zzylol commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: Pass 2 can only read a sketch's theoretical accuracy. A deployment cannot supply accuracy measured on its own executor, so KLL k = 547 is pruned under a per-value rank target of 0.0025 at δ = 0.01 (theory: 0.004998), even when a calibration run measured 0.0024. Also, the cost table (MeasuredOperationCosts) still loads a provenance field and the legacy array format, so a stale file loads with fields silently dropped.
  • Change: the measured accuracy table MeasuredSketchAccuracy, which Pass 2 reads through the existing AccuracyEvidence trait; and the cost table loads only its object form (deny_unknown_fields), with provenance, the legacy array format and their types deleted. convert_rqe_atomic_costs now writes OperatorCost entries itself. This PR takes over the library tests of test(devtools): asap_run test suite (tests and API skeleton only) #1332 (draft), with their hand-computed expected values unchanged.
  • Effect: with the table, the k = 547 edge is kept (0.0024 ≤ 0.0025 at δ′ = 0.01). A provenance field or a legacy array is refused when loaded, and the error names the field.

Review guide

  • Base: main at 1760a69f5. Not stacked. Three commits: the approved tests, the implementation, then the final review fixes.
  • Read first: crates/accuracy/src/measured.rs (lookup and validation), then crates/cost/src/measured.rs (deletions only).
  • Careful review:
    • has_deterministic_guarantee (rule (d)): since main now conditions DDSketch's guarantee on the sign of the state's values, the table is skipped for any θ whose guarantee is deterministic under its required data (evaluated with one-sign values). Without this, a table entry would bypass DDSketch's sign condition.
    • The doc of MeasuredAccuracyEntry: each recorded error is the worst case over the merge counts measured.
  • Test changes since the approved tests, both forced by main moving (no expected value changed):
    • closed_form_applies_below_the_measured_range passes None, None for the two arguments theoretical_guarantee gained (state sign and distinct bound); KLL reads neither.
    • The Pass 2 fixture p99_with_per_value_target declares m's values non-negative (with_non_negative_values("m").with_series_count(1)). Without it, DDSketch's guarantee on main has an unknown failure probability, so α = 0.01 was pruned as NoGuarantee instead of BelowRequiredAccuracy, and test 15 failed before reaching the table.
    • full_table() in crates/cost/tests/measured_operation_costs.rs drops ..MeasuredOperationCosts::default(), a no-op once provenance is gone (clippy).
  • Mechanical: re-exports in asap-cost and asap-planner, the CostInputs doc and the library-api.md row.

Behavior: before → after

The workload is quantile_over_time(0.99, m[5m]) over one series, with a per-value rank target of 0.0025 at δ = 0.01. The deployment supplies this accuracy.json (the example in #1317):

{"entries": [{"sketch": {"Kll": {"k": 547}}, "statistic": {"Quantile": {"q": 0.99}},
  "accuracy_metric": "rank", "trials": 1000,
  "error_at": [{"failure_probability": 0.003, "error": 0.0031},
               {"failure_probability": 0.01, "error": 0.0024},
               {"failure_probability": 0.05, "error": 0.0016}]}]}
Before After
KLL k = 547 build → readout edge pruned, BelowRequiredAccuracy (theory 0.004998) kept (measured 0.0024 at δ′ = 0.01)
DDSketch α = 0.01 with a table claiming 0.004 relative pruned still pruned: a deterministic guarantee is never replaced
{"operators": [], "ns_per_evaluation": 0, "provenance": {..}} loads; provenance is recorded but never read refused: unknown field `provenance`
legacy [{"sketch": .., "operation": "update", ..}] loads, converted refused
convert_rqe_atomic_costs on a KLL k = 200 row wrote 3 legacy entries writes 4 OperatorCost entries (SummaryAgg, SummaryTumblingWindowChain, SummaryMerge, SummaryEstimate), phase: null

Interfaces

crates/accuracy/src/measured.rs (new)

+/// Errors measured on the executor's own sketches. Default: empty, every
+/// sketch keeps its theoretical guarantee.
+#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MeasuredSketchAccuracy {
+    pub entries: Vec<MeasuredAccuracyEntry>,
+}
+/// One sketch configuration, read for one statistic, in one metric. Each
+/// recorded error is the worst case over the merge counts the calibration
+/// measured: a state built over one window and a state merged from many
+/// windows of the same θ share the entry, so its error must hold for
+/// every one of them. Example: KLL k = 547 measured unmerged and merged
+/// from 5 and 60 windows, with rank errors 0.0021, 0.0024 and 0.0023 at
+/// δ = 0.01, records 0.0024 at 0.01.
+#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct MeasuredAccuracyEntry {
+    /// The exact parameters measured; only these match.
+    pub sketch: SketchParams,
+    /// The readout, e.g. `Quantile { q: 0.99 }`; only the same statistic matches.
+    pub statistic: SketchStatistic,
+    /// The metric every recorded error is in, e.g. `Rank` for KLL.
+    pub accuracy_metric: AccuracyMetric,
+    /// Independent trials; every recorded failure probability is at least 3 / trials.
+    pub trials: u32,
+    /// Ascending failure probability; the error does not increase as it grows.
+    pub error_at: Vec<ErrorAtFailureProbability>,
+}
+/// One measured point: the error not exceeded except with probability `failure_probability`.
+#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct ErrorAtFailureProbability {
+    pub failure_probability: f64,
+    /// The (1 − δ) quantile of the per-value errors, the worst over the
+    /// measured merge counts, in the entry's `accuracy_metric`.
+    pub error: f64,
+}
+/// Why a measured accuracy table is not valid; always bad input.
+#[derive(Debug, Clone, PartialEq)]
+#[non_exhaustive]
+pub enum MeasuredSketchAccuracyError {
+    FailureProbabilityOutOfRange { entry: usize, failure_probability: f64 },
+    /// The entry records no point, so it can answer no request.
+    NoPoints { entry: usize },
+    InvalidError { entry: usize, error: f64 },
+    NotMonotone { entry: usize },
+    DuplicateEntry { first: usize, second: usize },
+}
+impl MeasuredSketchAccuracy {
+    /// At least one point, failure probabilities in (0, 1] and at least 3 / trials, errors finite
+    /// and non-negative, ascending failure probability with non-increasing
+    /// error, no two entries for one (sketch, statistic, metric).
+    pub fn validate(&self) -> Result<(), MeasuredSketchAccuracyError>;
+}
+/// The entry for exactly these parameters and statistic, in the readout's
+/// own metric, at its largest recorded failure probability at most the
+/// request; the guarantee states that recorded failure probability. `None`
+/// below the smallest (never extrapolated), at a failure probability that is
+/// not positive, and for a θ whose theoretical guarantee can be deterministic.
+impl AccuracyEvidence for MeasuredSketchAccuracy {
+    fn measured_guarantee(&self, summary_type: &FieldDataType, parameters: &SketchParams,
+        statistic: &SketchStatistic, failure_probability: f64) -> Option<AccuracyGuarantee>;
+}

crates/accuracy/src/lib.rs

+pub use measured::{
+    ErrorAtFailureProbability, MeasuredAccuracyEntry, MeasuredSketchAccuracy,
+    MeasuredSketchAccuracyError,
+};

Kept although no loader reads it yet: MeasuredSketchAccuracy and its entry types are the library's input type (a deployment builds or deserializes one and passes it as &dyn AccuracyEvidence); the loader that finds accuracy.json follows (see Follow-ups).

crates/cost/src/measured.rs, crates/cost/src/lib.rs, crates/planner/src/lib.rs

-#[derive(Debug, Clone, Default, PartialEq, Serialize)]
+#[derive(Debug, Clone, Default, PartialEq, Serialize, Deserialize)]
+#[serde(default, deny_unknown_fields)]
 pub struct MeasuredOperationCosts {
     pub operators: Vec<OperatorCost>,
     pub ns_per_evaluation: f64,
-    pub provenance: Option<CostTableProvenance>,
 }
-pub struct CostTableProvenance { pub executor_version: String, pub machine: String }
-pub enum SummaryOperation { Update, Merge, Estimate }
-pub struct MeasuredOperationCost { pub sketch: SketchParams, pub operation: SummaryOperation, pub cpu_ns_per_operation: f64 }
-impl From<Vec<MeasuredOperationCost>> for MeasuredOperationCosts;
-impl<'de> Deserialize<'de> for MeasuredOperationCosts;   // array or object
-struct TableFields;
 // re-exports (asap-cost, asap-planner)
-    CostTableProvenance, MeasuredOperationCost, SummaryOperation,

crates/devtools/src/rqe_atomic_costs.rs

 pub struct ConvertedCosts {
-    pub measured_operation_costs: Vec<MeasuredOperationCost>,
+    /// The table in its object form: per row, `SummaryAgg` and
+    /// `SummaryTumblingWindowChain` (update), `SummaryMerge`, `SummaryEstimate`, phase `None`.
+    pub measured_operation_costs: MeasuredOperationCosts,
     pub skipped_rows: Vec<SkippedRow>,
 }

crates/stage1-pass2-reuse-candidates/src/test_fixtures.rs (test fixture)

+/// `pass2` with the deployment's measured accuracy `evidence`.
+pub fn pass2_with_evidence(memo: Memo, entries: &[QueryWorkloadEntry],
+    evidence: &dyn AccuracyEvidence) -> LogicalASAPMemoCandidates;

Usage (from test 14): the deployment's table reaches Pass 2 through the existing PlanningModels::with_evidence.

// accuracy.json above, loaded; checked before planning.
let table: MeasuredSketchAccuracy = serde_json::from_str(ACCURACY_JSON)?;
table.validate()?;
// Pass 2 reads it before theory: KLL k = 547 → measured 0.0024 at δ′ = 0.01,
// so its edge into the p99 readout satisfies the 0.0025 target.
let logical = generate_logical_asap_memo_candidates(pass1_memo, &entries, &table)?;

Design scope

New type or file Scope Why it is general
crates/accuracy/src/measured.rs: MeasuredSketchAccuracy, MeasuredAccuracyEntry, ErrorAtFailureProbability, MeasuredSketchAccuracyError the deployment's accuracy evidence, beside evidence.rs, in asap-accuracy keyed by any sketch's θ, statistic and metric; the data form of the existing AccuracyEvidence trait, so no stage changes
crates/accuracy/tests/measured_sketch_accuracy.rs, crates/cost/tests/measured_operation_costs.rs crate integration tests #1332's files
pass2_with_evidence Pass 2 test fixture any Pass 2 test that needs evidence

Algorithm changes

The lookup (measured_guarantee):

  • (a) exact parameters and statistic;
  • (b) the readout's own metric only (readout_error_metric, e.g. Rank for a KLL quantile);
  • (c) the largest recorded δ′ ≤ δ, never extrapolated; the returned guarantee states δ′, so Pass 2's check that it holds at δ passes by construction;
  • (d) a θ whose theoretical guarantee can be deterministic (failure probability zero under its required data: DDSketch's relative α over one-sign values) never reads the table.

Decisions and open questions

Verification

cargo nextest is not installed on the build machine, so the changed crates' tests ran with cargo test. cargo fmt --all --check is clean, and so is cargo clippy -p asap-accuracy -p asap-cost -p asap-stage1-pass2-reuse-candidates -p asap-devtools -p asap-planner --all-targets.

Commands, run on this branch and on main (1760a69f5), compared by test name: every test that exists on main passes on both; this PR only adds tests (26 names), and all pass.

Command main This PR
cargo test -p asap-accuracy 46 passed (lib) 46 passed (lib) + 13 passed (measured_sketch_accuracy)
cargo test -p asap-cost 83 passed (lib) 85 passed (lib: + node_priced_by_hand, aggregate_needs_every_measure) + 9 passed (measured_operation_costs)
cargo test -p asap-stage1-pass2-reuse-candidates --lib 28 passed 30 passed (+ the two measured_accuracy_* edge tests)
cargo test -p asap-devtools --lib rqe_atomic_costs 2 passed the same 2 passed (a_small_sample_converts_and_round_trips rewritten, malformed_rows_are_errors unchanged)

The final-review commit reran cargo test -p asap-accuracy (46 lib + 13 measured_sketch_accuracy), cargo clippy -p asap-accuracy --all-targets and cargo fmt --all --check; it changes only measured.rs (one helper) and docs.

No plan changes: with no table (the default), NoMeasuredGuarantees and the empty cost table behave as before, and the cost table's loaded fields are unchanged.

Final review fixes

Item Status
1. One same-entry check: MeasuredAccuracyEntry::measures(sketch, statistic, metric), used by the lookup and the duplicate check; measures_the_same_as deleted done
2. Body: "kept: library input type; loader follows" line; MeasuredSketchAccuracyError::NoPoints in the Interfaces diff; Interfaces relinked to 7bfebfccb; library-api.md conflict with #1374 in the merge-order notes done
3. Docs: accuracy-models.md evidence scope (per θ, population-independent, worst over the measured merge counts, the deployment's responsibility); the deterministic-guarantee (DDSketch) exception in accuracy-models.md and asap-aware-mapping-contracts.md; measured.rs in the accuracy-models.md crate tree and the asap-aware-mapping-architecture.md accuracy row; library-api.md names asap_accuracy::MeasuredSketchAccuracy done

Follow-ups

  • Where the planner finds accuracy.json, the staleness warning, measuring the table on the executor's kernels, merge counts and trials.
  • Not changed: Pass 2 still requires MergedError::Exact or AsIfBuiltOverUnion for a merged state, whatever the table says.

@zzylol zzylol added accuracy Accuracy requirements, guarantees, and propagation cost-model Cost formulas, statistics, ranking, and selection dev-tools Developer tools, diagnostics, and demos testing Corpus, regression, differential, or benchmark tests labels Nov 16, 2026
@zzylol
zzylol force-pushed the test/measured-sketch-accuracy branch from a8f01e5 to b15a348 Compare November 16, 2026 22:41
@zzylol zzylol changed the title test(accuracy, cost, devtools): measured sketch accuracy table; refuse the cost table's provenance and legacy format (tests only) feat(accuracy, cost, devtools): measured sketch accuracy table; delete the cost table's provenance and legacy format Nov 16, 2026
@zzylol
zzylol marked this pull request as ready for review November 16, 2026 22:59
…moved cost-table formats (tests only)

Tests for MeasuredSketchAccuracy as Pass 2's AccuracyEvidence (exact
parameters, the readout's own metric, largest recorded failure probability
at most the request, deterministic guarantees never replaced), for
refusing the cost table's provenance field and legacy array format, and
for convert_rqe_atomic_costs writing the object form. Takes over the
library tests of the asap_run draft. Stubs: measured.rs (todo!), and the
converter ends in todo! after validating rows.
…idence; delete the cost table's provenance and legacy array format

MeasuredSketchAccuracy returns, for exactly the requested sketch parameters
and statistic in the readout's own metric, the error at the largest
recorded failure probability at most the request, stating that recorded
failure probability; never extrapolated, and never for a θ whose
theoretical guarantee can be deterministic (DDSketch). Each recorded error
is the worst case over the merge counts measured.

MeasuredOperationCosts derives Deserialize with deny_unknown_fields:
provenance, MeasuredOperationCost, SummaryOperation and the array format
are deleted. convert_rqe_atomic_costs writes OperatorCost entries itself.

Test updates for main: the closed-form test passes the sign and distinct
bound arguments theoretical_guarantee now takes; the Pass 2 fixture
declares non-negative values, which DDSketch's guarantee now requires.
…eck; evidence scope and DDSketch exception in the docs

MeasuredAccuracyEntry::measures is the one (sketch, statistic, metric)
check, used by the lookup and the duplicate check. The docs state that the
table is keyed per θ, independent of the population, holds the worst error
over the measured merge counts, and that its coverage is the deployment's
responsibility; that a deterministic guarantee (DDSketch) is never replaced
by a measurement; and list measured.rs in the crate layout.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accuracy Accuracy requirements, guarantees, and propagation cost-model Cost formulas, statistics, ranking, and selection dev-tools Developer tools, diagnostics, and demos enhancement New feature or request testing Corpus, regression, differential, or benchmark tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant