Repository navigation
Conversation
zzylol
force-pushed
the
test/measured-sketch-accuracy
branch
from
November 16, 2026 22:41
a8f01e5 to
b15a348
Compare
zzylol
marked this pull request as ready for review
November 16, 2026 22:59
…moved cost-table formats (tests only) Tests for MeasuredSketchAccuracy as Pass 2's AccuracyEvidence (exact parameters, the readout's own metric, largest recorded failure probability at most the request, deterministic guarantees never replaced), for refusing the cost table's provenance field and legacy array format, and for convert_rqe_atomic_costs writing the object form. Takes over the library tests of the asap_run draft. Stubs: measured.rs (todo!), and the converter ends in todo! after validating rows.
…idence; delete the cost table's provenance and legacy array format MeasuredSketchAccuracy returns, for exactly the requested sketch parameters and statistic in the readout's own metric, the error at the largest recorded failure probability at most the request, stating that recorded failure probability; never extrapolated, and never for a θ whose theoretical guarantee can be deterministic (DDSketch). Each recorded error is the worst case over the merge counts measured. MeasuredOperationCosts derives Deserialize with deny_unknown_fields: provenance, MeasuredOperationCost, SummaryOperation and the array format are deleted. convert_rqe_atomic_costs writes OperatorCost entries itself. Test updates for main: the closed-form test passes the sign and distinct bound arguments theoretical_guarantee now takes; the Pass 2 fixture declares non-negative values, which DDSketch's guarantee now requires.
zzylol
force-pushed
the
test/measured-sketch-accuracy
branch
from
November 17, 2026 00:45
b15a348 to
a11b9a9
Compare
…eck; evidence scope and DDSketch exception in the docs MeasuredAccuracyEntry::measures is the one (sketch, statistic, metric) check, used by the lookup and the duplicate check. The docs state that the table is keyed per θ, independent of the population, holds the worst error over the measured merge counts, and that its coverage is the deployment's responsibility; that a deterministic guarantee (DDSketch) is never replaced by a measurement; and list measured.rs in the crate layout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MeasuredOperationCosts) still loads aprovenancefield and the legacy array format, so a stale file loads with fields silently dropped.MeasuredSketchAccuracy, which Pass 2 reads through the existingAccuracyEvidencetrait; and the cost table loads only its object form (deny_unknown_fields), withprovenance, the legacy array format and their types deleted.convert_rqe_atomic_costsnow writesOperatorCostentries itself. This PR takes over the library tests of test(devtools): asap_run test suite (tests and API skeleton only) #1332 (draft), with their hand-computed expected values unchanged.provenancefield or a legacy array is refused when loaded, and the error names the field.Review guide
mainat1760a69f5. Not stacked. Three commits: the approved tests, the implementation, then the final review fixes.crates/accuracy/src/measured.rs(lookup and validation), thencrates/cost/src/measured.rs(deletions only).has_deterministic_guarantee(rule (d)): sincemainnow conditions DDSketch's guarantee on the sign of the state's values, the table is skipped for any θ whose guarantee is deterministic under its required data (evaluated with one-sign values). Without this, a table entry would bypass DDSketch's sign condition.MeasuredAccuracyEntry: each recorded error is the worst case over the merge counts measured.mainmoving (no expected value changed):closed_form_applies_below_the_measured_rangepassesNone, Nonefor the two argumentstheoretical_guaranteegained (state sign and distinct bound); KLL reads neither.p99_with_per_value_targetdeclaresm's values non-negative (with_non_negative_values("m").with_series_count(1)). Without it, DDSketch's guarantee onmainhas an unknown failure probability, so α = 0.01 was pruned asNoGuaranteeinstead ofBelowRequiredAccuracy, and test 15 failed before reaching the table.full_table()incrates/cost/tests/measured_operation_costs.rsdrops..MeasuredOperationCosts::default(), a no-op onceprovenanceis gone (clippy).asap-costandasap-planner, theCostInputsdoc and thelibrary-api.mdrow.Behavior: before → after
The workload is
quantile_over_time(0.99, m[5m])over one series, with a per-value rank target of 0.0025 at δ = 0.01. The deployment supplies thisaccuracy.json(the example in #1317):{"entries": [{"sketch": {"Kll": {"k": 547}}, "statistic": {"Quantile": {"q": 0.99}}, "accuracy_metric": "rank", "trials": 1000, "error_at": [{"failure_probability": 0.003, "error": 0.0031}, {"failure_probability": 0.01, "error": 0.0024}, {"failure_probability": 0.05, "error": 0.0016}]}]}BelowRequiredAccuracy(theory 0.004998){"operators": [], "ns_per_evaluation": 0, "provenance": {..}}provenanceis recorded but never readunknown field `provenance`[{"sketch": .., "operation": "update", ..}]convert_rqe_atomic_costson a KLL k = 200 rowOperatorCostentries (SummaryAgg,SummaryTumblingWindowChain,SummaryMerge,SummaryEstimate),phase: nullInterfaces
crates/accuracy/src/measured.rs(new)crates/accuracy/src/lib.rsKept although no loader reads it yet:
MeasuredSketchAccuracyand its entry types are the library's input type (a deployment builds or deserializes one and passes it as&dyn AccuracyEvidence); the loader that findsaccuracy.jsonfollows (see Follow-ups).crates/cost/src/measured.rs,crates/cost/src/lib.rs,crates/planner/src/lib.rscrates/devtools/src/rqe_atomic_costs.rspub struct ConvertedCosts { - pub measured_operation_costs: Vec<MeasuredOperationCost>, + /// The table in its object form: per row, `SummaryAgg` and + /// `SummaryTumblingWindowChain` (update), `SummaryMerge`, `SummaryEstimate`, phase `None`. + pub measured_operation_costs: MeasuredOperationCosts, pub skipped_rows: Vec<SkippedRow>, }crates/stage1-pass2-reuse-candidates/src/test_fixtures.rs(test fixture)Usage (from test 14): the deployment's table reaches Pass 2 through the existing
PlanningModels::with_evidence.Design scope
crates/accuracy/src/measured.rs:MeasuredSketchAccuracy,MeasuredAccuracyEntry,ErrorAtFailureProbability,MeasuredSketchAccuracyErrorevidence.rs, inasap-accuracyAccuracyEvidencetrait, so no stage changescrates/accuracy/tests/measured_sketch_accuracy.rs,crates/cost/tests/measured_operation_costs.rspass2_with_evidenceAlgorithm changes
The lookup (
measured_guarantee):readout_error_metric, e.g. Rank for a KLL quantile);Decisions and open questions
convert_rqe_atomic_costs; the tool itself is deleted later, once calibration writes tables. A measured entry may make a readout without a known theoretical failure probability plannable (HLL, test 12). The wire name ofAccuracyMetricis"rank".validatealso refuses an entry with no points, and loaders must call it before handing a table to the planner (stated on theAccuracyEvidenceimpl). A measured HLL entry currently replaces the HLL certificate whatever the source's declared bound; that interaction is tracked in a follow-up issue.crates/cost/src/node/usage/tests.rs. The other PR merges first; this one is rebased after, keeping both blocks.main+ feat(cost): node_cost_terms splits a node's formula CPU into named per-unit constants #1371 (5f080872b) + this PR (7bfebfccb) merged together, with that conflict resolved by keeping both blocks, passescargo fmt --all --checkandcargo test -p asap-cost -p asap-accuracy -p asap-stage1-pass2-reuse-candidatesandcargo test -p asap-devtools --lib rqe_atomic_costs.docs/develop_docs/library-api.md(this PR thePlanningModels::new(evidence)row, feat(types, stage2, stage3): deployment exclusions: one Stage 2 filter for a target's capabilities and the caller's exclusions #1374 theDeploymentCapabilitiesrows); whichever merges second resolves that text conflict, keeping both edits.Verification
cargo nextestis not installed on the build machine, so the changed crates' tests ran withcargo test.cargo fmt --all --checkis clean, and so iscargo clippy -p asap-accuracy -p asap-cost -p asap-stage1-pass2-reuse-candidates -p asap-devtools -p asap-planner --all-targets.Commands, run on this branch and on
main(1760a69f5), compared by test name: every test that exists onmainpasses on both; this PR only adds tests (26 names), and all pass.maincargo test -p asap-accuracymeasured_sketch_accuracy)cargo test -p asap-costnode_priced_by_hand,aggregate_needs_every_measure) + 9 passed (measured_operation_costs)cargo test -p asap-stage1-pass2-reuse-candidates --libmeasured_accuracy_*edge tests)cargo test -p asap-devtools --lib rqe_atomic_costsa_small_sample_converts_and_round_tripsrewritten,malformed_rows_are_errorsunchanged)The final-review commit reran
cargo test -p asap-accuracy(46 lib + 13measured_sketch_accuracy),cargo clippy -p asap-accuracy --all-targetsandcargo fmt --all --check; it changes onlymeasured.rs(one helper) and docs.No plan changes: with no table (the default),
NoMeasuredGuaranteesand the empty cost table behave as before, and the cost table's loaded fields are unchanged.Final review fixes
MeasuredAccuracyEntry::measures(sketch, statistic, metric), used by the lookup and the duplicate check;measures_the_same_asdeletedMeasuredSketchAccuracyError::NoPointsin the Interfaces diff; Interfaces relinked to7bfebfccb;library-api.mdconflict with #1374 in the merge-order notesaccuracy-models.mdevidence scope (per θ, population-independent, worst over the measured merge counts, the deployment's responsibility); the deterministic-guarantee (DDSketch) exception inaccuracy-models.mdandasap-aware-mapping-contracts.md;measured.rsin theaccuracy-models.mdcrate tree and theasap-aware-mapping-architecture.mdaccuracy row;library-api.mdnamesasap_accuracy::MeasuredSketchAccuracyFollow-ups
accuracy.json, the staleness warning, measuring the table on the executor's kernels, merge counts and trials.MergedError::ExactorAsIfBuiltOverUnionfor a merged state, whatever the table says.