Skip to content

test(devtools): asap_run test suite (tests and API skeleton only) - #1332

Draft
zzylol wants to merge 2 commits into
mainfrom
test/asap-run
Draft

zzylol wants to merge 2 commits into
mainfrom
test/asap-run

Conversation

@zzylol

@zzylol zzylol commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: asap_run ([Design] asap_run: one binary to execute plans, calibrate measured unit costs per machine, and bench predicted vs measured #1317) needs tests before code: today nothing checks a measured cost table end to end, nothing reads MeasuredSketchAccuracy, and nothing compares the plan Stage 3 selects with every alternative.
  • Change: the full test suite of the asap_run test plan (102 tests in 4 crates) plus the API skeleton they compile against. Every new non-test body is todo!("asap_run: implement after test approval"). No feature is implemented.
  • Effect today: 17 tests pass (tests of existing code), 4 fail, 84 cannot run yet (they reach a todo!). Example of a passing hand check: a query-time exact-sum SummaryAgg over 600 rows at 10 ns/row + 500 ns/invocation is priced 6,500 ns = 0.0065 ms per execution, from the table.

Review guide

Test plan summary

The plan specifies the four subcommands (execute, calibrate, bench, facts) and the MeasuredSketchAccuracy lookup, with invariants I1–I10:

  • I1: answers within target;
  • I2: the fit reproduces the measurements;
  • I3: costs.json round-trips;
  • I4: the accuracy table is monotone;
  • I5: the gate;
  • I6: the facts margins;
  • I7: the fallback to the formulas;
  • I8: the machine key carries no hostname and no memory size;
  • I9: determinism under a seed;
  • I10: lookup monotonicity.

The tests are in four tiers:

  • T0: unit;
  • T1: small integration on the executor;
  • T2: timing, #[ignore = "timing: run on a dedicated host"];
  • T3: milestone scale, #[ignore = "milestone scale"].

Expected values are never computed with the planner's own optimization or cost code. They are hand-computed, or they come from a fake clock or kernel of known cost, from sketch bounds, from the executor's own kernel charge, or from independent enumeration.

Provisional answers

The user had not yet answered the plan's questions Q1–Q23, so these tests use provisional answers, to be reviewed. Each test that depends on one is marked // provisional answer N, so grep -rn "provisional answer N" crates/ finds every test to change.

# Provisional answer Tests
1 Gate cost: cost model 1 recomputed from measured usage at the planner's prices; p50 latency and CPU reported beside it gate, regression
3 Error vs target for δ > 0: the fraction of values above ε is ≤ δ plus a one-sided 99% binomial slack; deterministic sketches use the maximum; a whole-result target holds when every value is within ε in ≥ (1−δ) of the evaluations gate, oracle
4 Quantile reference: the nearest-rank bracket execution
5 series_count and label counts are margined suggestions when a query has a whole-result target, otherwise written directly source_facts
6 column_magnitudes written with min_abs/2 and max_abs×2 source_facts
7 The lookup returns only entries in the readout's own metric; the hook is unchanged accuracy tests, measured_accuracy
8 DDSketch keeps its deterministic α accuracy tests
9 Memory per state only for fixed-size states cost_accounting
10 Ingestion ns_per_invocation converted per second; cost model unchanged cost_accounting
11 Relative-weighted least squares, ≥ 3 sizes over ≥ 2 decades, ±10%; a negative intercept clamped to 0, refitted and flagged cost_fit
13 Merge counts 1, 6, 60, 360; uniform and Zipf 1.1; 3,000 trials measured_accuracy
14 (1−δ) quantile by nearest rank, ⌈(1−δ)·n⌉ measured_accuracy
15 Repo root from git; --costs wins over --cost-inputs' table with a warning; a corrupt file is an error machine_and_tables
16 Machine key = CPU architecture + CPU model + logical cores + memory type (EDAC dimm_mem_type; unknown-memory without EDAC, overridable by --memory-type / ASAP_MEMORY_TYPE); no memory size (decided 2026-11-16, user) machine_and_tables
17 Oracle: a test-support enumerator over PhysicalASAPMemoCandidates oracle, enumerator
18 --data is JSONL {labels, ts_ms, value}; an evaluation before warm-up is an error execution, source_facts
19 sample_interval is the median of the per-series medians; distinct values and norms are taken over the longest window, per group; label sets only for by sets of ≥ 2 labels source_facts
20 Each data setting is its own gate row; noraw is compared with exact over raw data regression
21 Exact-match tolerance 1e-9 relative common
22 provenance deleted; old tables fail to load cost tests
23 Median of 5 runs; the noise band added to the 1.05 limit gate, oracle

Answers 2 and 12 have no test that depends on them alone.

Findings (the 4 failures)

  1. kll_sized_by_closed_form_without_measured_table: a real finding in existing code.
    • The test: quantile_over_time(0.99, m[5m]) over 8 series, with a per-value rank target of 0.0025 at δ = 0.01.
    • Expected: KLL k = 1404, whose error is 0.00200 at δ = 0.01.
    • Actual: k = 2863.
    • Cause: readout_caps/mod.rs:187 divides δ by max(values_covered, readout_values). For a per-value target, values_covered is 1, but the readout's 8 values still split δ to 0.00125. At that δ, k = 1404 has error 0.002505 > 0.0025.
    • Question: is splitting δ under a per-value target meant as a conservative union bound, or is it a bug? The planned measured_accuracy_shrinks_sketch depends on the answer.
  2. joint_sharing_plan_space_is_enumerable: the plan space is too large for the oracle as planned.
    • The workload: §2.11's joint workload, the sum and quantile families plus count_over_time(m[5m]) over 8 series.
    • Stage 2 gives it 452 physical candidates in 23 groups (groups of up to 160 candidates), and more than 1,000,000 one-candidate-per-group plans.
    • The exact family has 27 candidates and 624 plans.
    • So the oracle cannot measure every plan of the joint workload. It needs a restricted space, for example per-family products, or dominance-pruned candidates.
  3. costs_json_rejects_removed_fields: expected to fail. It tests the deletion of provenance and the legacy array format (Q22), which is part of the implementation.
  4. asap_planner_cli_uses_same_resolution: expected to fail. asap-planner does not resolve the calibration tables yet, which is part of the implementation.

Also checked against existing code, and passing:

  • the 8 lookup and serde tests;
  • the two node-pricing unit tests;
  • planner_prices_from_table (Δ = rows × 9 ns on each DDSketch build);
  • shared_node_priced_once;
  • invocation_cost_ranks_chain_widths;
  • the closed-form KLL value 0.005997;
  • the seeded aqpbm-datagen stream;
  • the two oracle-enumerator checks. In both, the plan Stage 3 selects is one of the enumerated plans, for the exact and sketch families.

Tests

Test Tier Would catch Result
accuracy::accuracy_json_round_trips T0 the table's JSON format (issue example, rank metric) not loading passes
accuracy::closed_form_applies_below_the_measured_range T0 wrong KLL closed form at δ < 0.01 (expected 0.005997) passes
accuracy::empty_table_measures_nothing T0 an empty table returning a guarantee cannot run yet (todo!)
accuracy::lookup_error_does_not_grow_with_delta T0 lookup monotonicity (I10) cannot run yet (todo!)
accuracy::lookup_exact_params_statistic_metric T0 matching by algorithm only, or another metric hiding the closed form cannot run yet (todo!)
accuracy::lookup_no_extrapolation_below_smallest T0 extrapolating 0.0031 below δ = 0.003 cannot run yet (todo!)
accuracy::lookup_takes_largest_recorded_delta_at_most_request T0 taking the nearest δ' above (0.05 for 0.02) cannot run yet (todo!)
accuracy::measured_entry_never_replaces_a_deterministic_guarantee T0 a measured entry replacing DDSketch's deterministic α cannot run yet (todo!)
accuracy::table_validation T0 accepting δ < 3/trials, increasing error, unsorted δ, δ ∉ (0,1], bad errors, duplicates cannot run yet (todo!)
cost::aggregate_needs_every_measure T0 an Aggregate priced from a partial set of measure entries passes
cost::costs_json_rejects_removed_fields T0 provenance / legacy array still loading fails: provenance and the array format still load (the deletion is part of the implementation)
cost::costs_json_round_trips T0 lossy costs.json serde (I3) passes
cost::lookup_exact_params_win T0 interpolating despite an exact match passes
cost::lookup_interpolates_in_log_size T0 linear-in-size interpolation (166.7 instead of 200) passes
cost::lookup_memory_needs_both_neighbours T0 memory interpolated with a missing neighbour passes
cost::lookup_no_extrapolation T0 extrapolation outside the measured sizes passes
cost::lookup_other_algorithm_misses T0 KLL entries pricing a DDSketch node passes
cost::lookup_phase_entries_shadow_both T0 mixing phase-specific and both-phase entries passes
cost::node_priced_by_hand T0 units or ns scale wrong (600·10 + 500 = 6,500 ns) passes
cost_accounting::calibrated_table_covers_showcase_plans T1 an operator left to the formulas cannot run yet (todo!)
cost_accounting::fixed_size_state_bytes T1 calibrated bytes ≠ the kernel's charge for CMS 272×5 cannot run yet (todo!)
cost_accounting::invocation_cost_ranks_chain_widths T1 a 1 s chain chosen over 10 s (3,600 vs 360 merged states) passes
cost_accounting::memory_per_state_hand T1 view bytes ≠ 6 windows × 2 groups × B_sum (1.5×) cannot run yet (todo!)
cost_accounting::planner_prices_from_table T1 the table not changing a node's price by rows × 9 ns passes
cost_accounting::shared_node_measured_once T2 a shared chain measured twice cannot run yet (todo!)
cost_accounting::shared_node_priced_once T1 a shared chain built or priced twice passes
cost_fit::fit_five_sizes_tightens T0 a fit using two points only cannot run yet (todo!)
cost_fit::fit_flags_nonlinear T0 a quadratic kernel accepted silently cannot run yet (todo!)
cost_fit::fit_is_relative_weighted T0 plain OLS (intercept 2,783) cannot run yet (todo!)
cost_fit::fit_negative_intercept_is_clamped T0 a negative intercept written (fails validate) cannot run yet (todo!)
cost_fit::fit_recovers_exact_line T0 swapped slope / intercept cannot run yet (todo!)
cost_fit::fit_recovers_fake_clock_kernel T0 timing per batch, or wrong unit count, in the measurement loop cannot run yet (todo!)
cost_fit::fit_recovers_spin_kernel T2 wall clock instead of thread CPU; warm-up counted cannot run yet (todo!)
cost_fit::units_match_cost_model T1 microbenchmark units ≠ the planner's units (merge per window vs per state) cannot run yet (todo!)
end_to_end::calibrate_accuracy_seeded T1 unseeded trials cannot run yet (todo!)
end_to_end::datagen_seeded_bit_identical T1 non-deterministic generator passes
end_to_end::e2e_full_showcase T3 milestone criteria cannot run yet (todo!)
end_to_end::e2e_small_showcase T2 any showcase row failing the gate cannot run yet (todo!)
end_to_end::e2e_tiny_hand_answers T1 the four subcommands not composing (answers 180 / 420) cannot run yet (todo!)
end_to_end::facts_seeded_bit_identical T1 non-deterministic facts JSON cannot run yet (todo!)
end_to_end::generated_samples_seeded_bit_identical T1 facts → datagen not deterministic cannot run yet (todo!)
end_to_end::plan_given_tables_identical T1 plan.txt differing under the same tables cannot run yet (todo!)
end_to_end::plan_stable_under_recalibration T3 recalibration noise changing plans that then fail cannot run yet (todo!)
execution::execute_answers_bit_identical T1 answers depending on run or drain order cannot run yet (todo!)
execution::execute_before_warmup T1 a silent partial answer before warm-up cannot run yet (todo!)
execution::execute_drain_order T1 shared-producer deadlock or order-dependent answers cannot run yet (todo!)
execution::execute_ingestion_equals_query_time T1 a chain missing a window cannot run yet (todo!)
execution::execute_on_first_read_view T1 OnFirstRead views refused or refreshed per evaluation cannot run yet (todo!)
execution::execute_quantile_reference T1 sketch vs interpolated quantile reported as an error cannot run yet (todo!)
execution::execute_rate_linear_counter T1 rate extrapolation or counter bugs cannot run yet (todo!)
execution::execute_retention_evicts T1 unbounded view growth cannot run yet (todo!)
execution::execute_several_eval_times T1 stale windows reused (180 / 225 / 315) cannot run yet (todo!)
execution::execute_shared_ingestion_scan_feeds_every_consumer T1 an ingestion-time scan shared by two chain builds whose batches reach only one of them cannot run yet (todo!)
execution::execute_sum_over_time_hand T1 a window closed on both ends (183 / 427) cannot run yet (todo!)
execution::execute_window_boundary T1 off-by-one at t − range cannot run yet (todo!)
gate::gate_counterexample_passes_by_exact T2 the 1m-window counterexample failing cannot run yet (todo!)
gate::gate_fails_on_lying_accuracy_table T1 an accuracy violation passing the gate cannot run yet (todo!)
gate::gate_fails_on_lying_cost_table T2 a cost violation passing the gate cannot run yet (todo!)
gate::gate_output_files T1 the run directory contract cannot run yet (todo!)
gate::gate_rule_pure T0 threshold arithmetic; missing / extra / failed / NaN rows skipped cannot run yet (todo!)
machine_and_tables::asap_planner_cli_uses_same_resolution T1 asap-planner and asap_run resolving different tables fails: asap-planner prints no missing-table warning yet (part of the implementation)
machine_and_tables::corrupt_costs_is_an_error T1 a corrupt table silently ignored cannot run yet (todo!)
machine_and_tables::costs_flag_wins_over_cost_inputs_table T0 --cost-inputs' table winning over --costs cannot run yet (todo!)
machine_and_tables::machine_key_from_cpu_cores_memory T0 key format: architecture, normalized model, cores, memory type cannot run yet (todo!)
machine_and_tables::machine_key_ignores_hostname T0 the hostname in the key cannot run yet (todo!)
machine_and_tables::machine_key_slug_charset T0 characters outside [a-z0-9-] cannot run yet (todo!)
machine_and_tables::machine_key_ignores_memory_size T0 memory size (64 vs 256 GiB, jitter) changing the key cannot run yet (todo!)
machine_and_tables::machine_key_without_edac_is_unknown_memory T0 a VM key without a memory type, or without a warning naming it cannot run yet (todo!)
machine_and_tables::machine_key_memory_type_override T0 --memory-type / ASAP_MEMORY_TYPE ignored cannot run yet (todo!)
machine_and_tables::machine_key_mixed_dimm_types T0 mixed DIMM types not joined sorted and unique cannot run yet (todo!)
machine_and_tables::missing_accuracy_warns_separately T1 one missing table masking the other cannot run yet (todo!)
machine_and_tables::missing_costs_warns_and_falls_back T1 no warning, or a fallback plan ≠ the default plan cannot run yet (todo!)
machine_and_tables::path_precedence T0 wrong order of --costs / env / repo / fallback cannot run yet (todo!)
measured_accuracy::accuracy_table_is_monotone T1 a quantile at the wrong order statistic (I4) cannot run yet (todo!)
measured_accuracy::empty_table_equals_no_measured T1 an empty table changing the plan (I7) cannot run yet (todo!)
measured_accuracy::kll_sized_by_closed_form_without_measured_table T1 KLL sized by the wrong closed form fails: smallest admissible k is 2863, not 1404 (see Findings)
measured_accuracy::measured_accuracy_shrinks_sketch T1 the hook not wired or not read cannot run yet (todo!)
measured_accuracy::measured_within_known_bounds T2 a measurement bug, or a quantile-definition mismatch cannot run yet (todo!)
measured_accuracy::quantile_of_errors_nearest_rank T0 a non-conservative off-by-one cannot run yet (todo!)
measured_accuracy::worst_over_merge_counts_is_recorded T0 mean or last instead of worst cannot run yet (todo!)
measured_accuracy::wrong_metric_entry_does_not_prune T1 an entry in another metric pruning k = 1404 cannot run yet (todo!)
oracle::enumerated_plans_are_closed_under_edges T1 an enumerator bug (open inputs, two candidates per group) passes
oracle::enumerated_plans_contain_the_selected_plan T1 Stage 3 selecting an invalid plan passes
oracle::joint_sharing_plan_space_is_enumerable T1 an oracle that cannot measure every plan fails: more than 1,000,000 plans (see Findings)
oracle::oracle_exact_family T2 double charging, a formula-priced operator, wrong merge units cannot run yet (todo!)
oracle::oracle_joint_sharing T2 sharing priced or measured twice cannot run yet (todo!)
oracle::oracle_sketch_family T2 a plan missing its target cannot run yet (todo!)
regression::regression_quantile_half_speed T2 DDSketch at query time selected at 0.50× cannot run yet (todo!)
regression::regression_quantile_half_speed_full T3 the same at 1,000 series cannot run yet (todo!)
regression::regression_ranking_rate T3 ranking below 90%; measured / predicted > 1.5× cannot run yet (todo!)
regression::regression_rate_1hz_every_300s T2 query-time SummaryAgg selected over a 45× faster chain cannot run yet (todo!)
regression::regression_rate_1hz_every_300s_full T3 the same at 1,000 series cannot run yet (todo!)
source_facts::facts_column_magnitudes_exclude_zero T0 min_abs 0 cannot run yet (todo!)
source_facts::facts_cost_only_written_directly T0 a ×2 margin on cost-only facts cannot run yet (todo!)
source_facts::facts_counter_reset T0 a reset taken as a negative value cannot run yet (todo!)
source_facts::facts_datagen_round_trip T1 generator / facts disagreement cannot run yet (todo!)
source_facts::facts_distinct_per_label T0 a missing label not counted cannot run yet (todo!)
source_facts::facts_guarantee_margin T0 guarantee facts written by default, or ×1 cannot run yet (todo!)
source_facts::facts_label_set_from_workload_by T0 the product 6, or a single-label set cannot run yet (todo!)
source_facts::facts_margined_set_stays_valid T0 a set count above its labels' product cannot run yet (todo!)
source_facts::facts_reads_jsonl_samples T0 the JSONL schema cannot run yet (todo!)
source_facts::facts_sample_interval_median_gap T0 the mean gap, or the pooled median cannot run yet (todo!)
source_facts::facts_series_count T0 counting samples or label values cannot run yet (todo!)
source_facts::facts_sign_suggestion_sign_bit T0 v >= 0.0 accepting −0.0 cannot run yet (todo!)
source_facts::facts_single_sample_series T0 a divide by zero cannot run yet (todo!)

Interfaces

All added. No existing interface changes.

crates/accuracy/src/measured.rs: the approved types of #1317, data only.

+/// Errors measured on the executor's own sketches. Default: empty, every sketch keeps its theoretical guarantee.
+pub struct MeasuredSketchAccuracy { pub entries: Vec<MeasuredAccuracyEntry> }
+/// One sketch configuration, read for one statistic, in one metric.
+pub struct MeasuredAccuracyEntry {
+    /// The exact parameters measured; only these match.
+    pub sketch: SketchParams,
+    /// The readout, e.g. Quantile { q: 0.99 }; only the same statistic matches.
+    pub statistic: SketchStatistic,
+    /// The metric every recorded error is in.
+    pub accuracy_metric: AccuracyMetric,
+    /// Independent trials; every recorded δ is at least 3 / trials.
+    pub trials: u32,
+    /// Ascending δ; the error does not increase as δ grows.
+    pub error_at: Vec<ErrorAtFailureProbability>,
+}
+/// One measured point: the (1 − δ) quantile of the per-value errors, the worst over the merge counts.
+pub struct ErrorAtFailureProbability { pub failure_probability: f64, pub error: f64 }
+/// Why a table is not valid (bad input).
+pub enum MeasuredSketchAccuracyError {
+    FailureProbabilityOutOfRange { entry: usize, failure_probability: f64 },
+    InvalidError { entry: usize, error: f64 },
+    NotMonotone { entry: usize },
+    DuplicateEntry { first: usize, second: usize },
+}
+impl MeasuredSketchAccuracy { pub fn validate(&self) -> Result<(), MeasuredSketchAccuracyError>; /* todo! */ }
+/// Pass 2 reads the table through the existing hook.
+impl AccuracyEvidence for MeasuredSketchAccuracy { fn measured_guarantee(..) -> Option<AccuracyGuarantee>; /* todo! */ }

crates/devtools/src/measurement/: the asap_run driver module. Every body is todo!.

+// samples.rs: the data a run ingests
+pub type SeriesLabels = BTreeMap<String, String>;
+pub struct Sample { pub labels: SeriesLabels, pub ts_ms: i64, pub value: f64 }
+pub fn read_samples_jsonl(text: &str) -> Result<Vec<Sample>, SampleReadError>;
+pub struct GeneratedDataSettings { pub seed: u64, pub start_ms: i64, pub duration_ms: u64 }
+pub fn time_series_description_of(metric: &str, facts: &SourceFacts, settings: &GeneratedDataSettings) -> Result<TimeSeriesDescription, String>;
+pub fn generate_samples(data: &DataWorkload, settings: &GeneratedDataSettings) -> Result<Vec<Sample>, String>;
+pub fn samples_of_generated(metric: &str, generated: &GeneratedTimeSeries) -> Result<Vec<Sample>, String>;
+// execution.rs: the reference driver (asap_run execute)
+pub enum DataInput { GeneratedFromFacts(GeneratedDataSettings), Samples(Vec<Sample>) }
+pub enum RootDrainOrder { WorkloadOrder, Reversed, Concurrent }
+pub struct ExecuteRequest<'a> { pub workload: &'a PlanningWorkload, pub data: DataInput, pub evaluation_times_ms: Vec<i64>,
+    pub deployment_capabilities: DeploymentCapabilities, pub cost_inputs: CostInputs, pub accuracy: MeasuredSketchAccuracy, pub root_drain_order: RootDrainOrder }
+pub struct ExecutionReport { pub plan: InstallablePlan, pub predicted: SelectedPlanReport, pub evaluations: Vec<EvaluationAnswers>, pub views: Vec<ViewActivity> }
+pub struct EvaluationAnswers { pub at_ms: i64, pub per_root: Vec<RootAnswers> }
+pub struct RootAnswers { pub values: BTreeMap<SeriesLabels, f64> }
+pub struct ViewActivity { pub node_id: PhysicalASAPNodeId, pub refresh: MaterializedViewRefresh, pub refresh_count: u64,
+    pub states_after_evaluation: Vec<u64>, pub bytes_after_evaluation: Vec<u64> }
+pub enum ExecuteError { Plan(String), Data(String), BeforeWarmUp { at_ms: i64, first_covered_ms: i64 }, Executor(String) }
+pub fn execute(request: &ExecuteRequest<'_>) -> Result<ExecutionReport, ExecuteError>;
+// cost_calibration.rs: the cost microbenchmarks and fit
+pub struct CostSample { pub units: f64, pub cpu_ns: f64 }
+pub struct FittedOperatorCost { pub ns_per_unit: f64, pub ns_per_invocation: f64, pub flags: Vec<FitFlag> }
+pub enum FitFlag { NegativeInterceptClamped { raw_intercept_ns: f64 }, ResidualAboveTolerance { units: f64, relative_residual: f64 }, TooFewSizes { sizes: usize } }
+pub fn fit_operator_cost(samples: &[CostSample]) -> FittedOperatorCost;
+pub trait CpuClock { fn thread_cpu_ns(&self) -> u64; }
+pub struct ThreadCpuClock;
+pub trait CalibrationKernel { fn run(&mut self, units: u64); }
+pub fn measure_kernel(kernel: &mut dyn CalibrationKernel, clock: &dyn CpuClock, sizes: &[u64], repetitions: u32) -> Vec<CostSample>;
+pub enum CalibratedFamily { None, Sketch(SketchParams), Exact(ExactKind), Measure(String) }
+pub struct CalibrationCase { pub operator: OperatorKind, pub phase: ExecutionTiming, pub family: CalibratedFamily }
+pub enum CalibrationFamilies { All, Exact, Only(Vec<String>) }
+pub struct CalibrationDag { pub dag: PhysicalASAPDAG, pub entries: Vec<QueryWorkloadEntry>, pub data: DataWorkload, pub measured_node: PhysicalASAPNodeId, pub units: u64 }
+pub fn calibration_cases(families: &CalibrationFamilies) -> Vec<CalibrationCase>;
+impl CalibrationCase { pub fn dag_at(&self, units: u64) -> CalibrationDag; pub fn operator_cost(&self, fit: &FittedOperatorCost, memory_bytes_per_state: Option<f64>) -> OperatorCost; }
+pub struct CostFitRecord { pub case: CalibrationCase, pub samples: Vec<CostSample>, pub fit: FittedOperatorCost, pub memory_bytes_per_state: Option<f64> }
+pub fn cost_table_of(records: &[CostFitRecord], ns_per_evaluation: f64) -> MeasuredOperationCosts;
+// accuracy_calibration.rs: the accuracy trials and their aggregation
+pub fn error_quantile(errors: &[f64], failure_probability: f64) -> f64;
+pub struct MergeCountErrors { pub sketch: SketchParams, pub statistic: SketchStatistic, pub accuracy_metric: AccuracyMetric,
+    pub merge_count: u32, pub trials: u32, pub error_at: Vec<ErrorAtFailureProbability> }
+pub fn worst_over_merge_counts(results: &[MergeCountErrors]) -> Vec<MeasuredAccuracyEntry>;
+// machine.rs: the machine key
+pub struct MachineDescription { pub cpu_architecture: String, pub cpu_model: String, pub logical_cores: u32, pub memory_type: String, pub memory_kib: u64 /* information only, not in the key */, pub os: String }
+impl MachineDescription { pub fn from_proc(cpu_architecture: &str, cpuinfo: &str, meminfo: &str, dimm_memory_types: &[&str], memory_type_override: Option<&str>, os: &str) -> Result<Self, String>; pub fn detect(memory_type_override: Option<&str>) -> Result<Self, String>; pub fn machine_key(&self) -> String; pub fn warning(&self) -> Option<String>; }
+// calibration_tables.rs: running a calibration, and finding and loading the tables
+pub const CALIBRATION_DIR_ENV: &str = "ASAP_CALIBRATION_DIR";
+pub struct CalibrateRequest { pub families: CalibrationFamilies, pub small: bool, pub seed: u64 }
+pub struct Calibration { pub machine: MachineDescription, pub costs: MeasuredOperationCosts, pub accuracy: MeasuredSketchAccuracy,
+    pub cost_fits: Vec<CostFitRecord>, pub accuracy_by_merge_count: Vec<MergeCountErrors> }
+pub fn calibrate(request: &CalibrateRequest) -> Result<Calibration, String>;
+pub fn write_calibration(dir: &Path, calibration: &Calibration) -> Result<PathBuf, String>;
+pub struct TableOverrides { pub costs: Option<PathBuf>, pub accuracy: Option<PathBuf>, pub cost_inputs: Option<PathBuf> }
+pub enum TablePath { Given(PathBuf), Calibrated(PathBuf), Missing { looked_at: PathBuf } }
+pub struct ResolvedTablePaths { pub costs: TablePath, pub accuracy: TablePath }
+pub fn resolve_table_paths(overrides: &TableOverrides, env_dir: Option<&Path>, repo_root: Option<&Path>, machine_key: &str) -> ResolvedTablePaths;
+pub struct PlanningTables { pub cost_inputs: CostInputs, pub accuracy: MeasuredSketchAccuracy, pub warnings: Vec<String> }
+pub struct TableLoadError { pub path: PathBuf, pub reason: String }
+pub fn load_planning_tables(resolved: &ResolvedTablePaths, overrides: &TableOverrides, machine_key: &str) -> Result<PlanningTables, TableLoadError>;
+// benchmark.rs: the arms, the gate and the showcase
+pub enum BenchArm { Exact, Planned, NoRaw }
+pub struct DataSetting { pub sample_interval_ms: u64, pub evaluation_interval_ms: u64, pub raw_data_retained: bool }
+pub struct BenchRequest<'a> { pub workload: &'a PlanningWorkload, pub arms: Vec<BenchArm>, pub settings: Vec<DataSetting>, pub series_count: u64,
+    pub duration_ms: u64, pub repetitions: u32, pub tables: PlanningTables, pub seed: u64, pub cost_gate: bool, pub output_root: Option<PathBuf> }
+pub struct MeasuredUsage { pub p50_query_ms: f64, pub p95_query_ms: f64, pub ingestion_cpu_seconds_per_second: f64, pub query_cpu_seconds_per_second: f64,
+    pub average_memory_bytes: f64, pub peak_memory_bytes: u64, pub scanned_bytes_per_second: f64 }
+pub struct ArmMeasurement { pub arm: BenchArm, pub setting: DataSetting, pub plan: InstallablePlan, pub predicted: SelectedPlanReport,
+    pub measured: Result<MeasuredUsage, String>, pub measured_cost_per_second: Option<f64>, pub repetition_costs_per_second: Vec<f64>, pub measured_error: Option<f64> }
+pub struct GateRow { pub workload: String, pub setting: DataSetting, pub exact_cost_per_second: Option<f64>,
+    pub selected_cost_per_second: Result<f64, String>, pub measured_error: f64, pub target_error: f64 }
+pub enum GateFailure { CostAboveLimit {..}, ErrorAboveTarget {..}, ErrorNotANumber {..}, ExactArmMissing {..}, SelectedArmFailed {..}, MissingRow {..}, ExtraRow {..} }
+pub enum GateVerdict { Pass, Fail(Vec<GateFailure>) }
+pub fn gate_verdict(expected: &[(String, DataSetting)], rows: &[GateRow], cost_gate: bool) -> GateVerdict;
+pub struct BenchReport { pub arms: Vec<ArmMeasurement>, pub rows: Vec<GateRow>, pub verdict: GateVerdict, pub run_dir: Option<PathBuf> }
+pub fn bench(request: &BenchRequest<'_>) -> Result<BenchReport, String>;
+pub fn showcase_suite(small: bool) -> Vec<(String, PlanningWorkload, Vec<DataSetting>)>;
+pub fn bench_showcase(small: bool, tables: &PlanningTables, output_root: Option<PathBuf>) -> Result<BenchReport, String>;
+pub fn bench_plan(plan: &InstallablePlan, workload: &PlanningWorkload, setting: &DataSetting, request: &BenchRequest<'_>) -> Result<ArmMeasurement, String>;
+pub fn installable_plan_of_selection(physical: &PhysicalASAPMemoCandidates, entries: &[QueryWorkloadEntry],
+    candidates: &BTreeSet<PhysicalCandidateId>, edges: &BTreeSet<PhysicalEdgeId>) -> Result<InstallablePlan, String>;
+// source_facts.rs: asap_run facts
+pub struct ObservedMagnitude { pub min_abs: f64, pub max_abs: f64 }
+pub struct ObservedSourceStatistics { pub series_count: u64, pub sample_interval_ms: Option<u64>, pub distinct_values_per_label: BTreeMap<String, u64>,
+    pub distinct_values_per_label_set: Vec<LabelSetDistinctValues>, pub column_magnitudes: BTreeMap<String, ObservedMagnitude>,
+    pub column_signs: BTreeMap<String, ValueSign>, pub max_distinct_per_evaluation: Option<u64>, pub stream_norms: Option<StreamNorms> }
+pub fn observe_source_statistics(samples: &[Sample], workload: &PlanningWorkload) -> BTreeMap<String, ObservedSourceStatistics>;
+pub enum SuggestedFact { ColumnSign { column: String, sign: ValueSign }, SeriesCount(u64), DistinctValuesPerLabel { label: String, distinct_values: u64 },
+    DistinctValuesPerLabelSet(LabelSetDistinctValues), MaxDistinctPerEvaluation(u64), StreamNorms(StreamNorms) }
+pub struct GuaranteeFactSuggestion { pub source: String, pub fact: SuggestedFact }
+pub struct FactsOptions { pub trust_observed_bounds: bool }
+pub struct SourceFactsReport { pub observed: BTreeMap<String, ObservedSourceStatistics>, pub data_workload: DataWorkload,
+    pub column_magnitudes: BTreeMap<String, BTreeMap<String, ObservedMagnitude>>, pub suggestions: Vec<GuaranteeFactSuggestion> }
+pub fn source_facts_of_samples(samples: &[Sample], workload: &PlanningWorkload, options: FactsOptions) -> Result<SourceFactsReport, String>;

crates/types/src/test_support/physical_plan_enumeration.rs: test-support only, and implemented.

+/// One plan: the candidate picked per memo group and the edge feeding each input of each picked candidate.
+pub struct PhysicalPlanChoice { pub candidate_of_group: BTreeMap<MemoGroupId, PhysicalCandidateId>, pub edge_into_input: BTreeMap<(PhysicalCandidateId, usize), PhysicalEdgeId> }
+impl PhysicalPlanChoice { pub fn candidates(&self) -> BTreeSet<PhysicalCandidateId>; pub fn edges(&self) -> BTreeSet<PhysicalEdgeId>; }
+/// The enumeration stopped: more than `limit` plans.
+pub struct TooManyPlans { pub limit: usize }
+pub fn every_physical_plan(physical: &PhysicalASAPMemoCandidates, limit: usize) -> Result<Vec<PhysicalPlanChoice>, TooManyPlans>;

crates/devtools/src/bin/asap_run.rs: main is todo!. The planned CLI, asap_run execute | calibrate | bench | facts, is in its module doc.

Usage: none yet. Every entry point is todo!. The planned flow is e2e_tiny_hand_answers: facts on a hand-made JSONL, then calibrate --small --families exact, then execute --at 600000, which prints 180 / 420, then bench --no-cost-gate, which writes runs/.

Design scope

New type / file Scope Why it is general
MeasuredSketchAccuracy, MeasuredAccuracyEntry, ErrorAtFailureProbability, MeasuredSketchAccuracyError (accuracy/src/measured.rs) accuracy rules shared by the stages, the deployment's evidence that Pass 2 reads approved in #1317; one table for any sketch, statistic or metric, read through the existing AccuracyEvidence hook
devtools::measurement, with execution, samples, cost_calibration, accuracy_calibration, machine, calibration_tables, benchmark and source_facts devtools: running selected plans on the executor and measuring them each submodule is one concern of any deployment runtime (drive, data, calibrate, locate tables, benchmark, analyze). None is named after the binary or after a workload
asap_run bin devtools CLI the one binary of #1317
PhysicalPlanChoice, every_physical_plan (types/src/test_support/physical_plan_enumeration.rs) Stage 2 output, test support enumerates any PhysicalASAPMemoCandidates, shares no code with Stage 3
aqpbm-datagen git dependency (rev cf8f867) devtools only per #1317; the planner library and the executor do not depend on it

No existing production code path changes. The accuracy crate gains a serde dependency for the new types.

Decisions and open questions

Needs reviewer decision:

Decided:

  • Q16, the machine key (2026-11-16, user): CPU architecture + CPU model + logical cores + memory type, and not memory size; never the hostname. The memory type is read without root from EDAC (/sys/devices/system/edac/mc/mc*/dimm*/dimm_mem_type): the single value if all DIMMs agree, otherwise the sorted unique values joined. Without EDAC the type is unknown-memory, overridable by --memory-type / ASAP_MEMORY_TYPE, and a warning names it. Slug [a-z0-9-]+ with no double dashes; the model drops (R), (TM), CPU and @ <freq>. Example: x86_64-intel-xeon-e5-2683-v3-56c-registered-ddr4. Test changes: machine_key_from_cpu_cores_memory is now a fixture (arch, model, cores, memory type) with the expected slug; machine_key_stable_under_memtotal_jitter became machine_key_ignores_memory_size; added machine_key_without_edac_is_unknown_memory, machine_key_memory_type_override and machine_key_mixed_dimm_types; machine_key_ignores_hostname and machine_key_slug_charset stay. MachineDescription gains cpu_architecture and memory_type; memory_kib stays for machine.json only; from_proc takes the architecture, the EDAC DIMM types and the override; detect takes the override; warning is new.

New counts after this change: 17 pass, 4 fail, 84 cannot run yet (the 3 new tests reach a todo!).

Verification

All tests ran locally, ignored tests included (--include-ignored).

Scope Command Passed Failed on existing code Cannot run (todo!)
asap-cost cargo test -p asap-cost --test measured_operation_costs, and --lib -- node_priced_by_hand aggregate_needs_every_measure 9 1 (expected: Q22) 0
asap-accuracy cargo test -p asap-accuracy --test measured_sketch_accuracy 2 0 7
asap-devtools cargo test -p asap-devtools --test devtools -- measurement:: --include-ignored 6 3 77
Total 17 4 84

cargo fmt is clean.

Retarget to main 863153e3c: #1300 was squash-merged, so this branch was rebased with --onto origin/main, dropping #1300's six pre-squash commits. The previous head d6ea9616d is kept as backup/test/asap-run-d6ea9616d. One conflict, in crates/cost/src/node/usage/tests.rs: main (#1324) and this PR both append tests at the end of the file, so both were kept. This PR's rows_of helper sets main's new ProducerOutput::chain_tumbling_window_width_ms to None. cargo check --workspace --all-targets and cargo fmt are clean. Each count is unchanged:

Scope Passed Failed on existing code Cannot run (todo!)
asap-cost (measured_operation_costs, plus the two --lib tests) 9 1 0
asap-accuracy (measured_sketch_accuracy) 2 0 7
asap-devtools (measurement::, --include-ignored) 6 3 77
Total 17 4 84

Restack onto #1300 at d2d5f6077, with aqpbm-datagen at cf8f867: the counts are unchanged, 17 / 4 / 81, and so is each test's outcome. The same 17 tests pass, the same 4 fail with the same values (k = 2863; more than 1,000,000 plans; provenance still loads; no missing-table warning), and the other 81 reach a todo!. Changes made for the restack:

  • aqpbm-datagen 74ef8d0 → cf8f867: LabelSpec is SeriesLabelSpec, value is value_distribution, and GeneratedTimeSeries::samples() yields Result items (a counter past 2^53 is an error). datagen_seeded_bit_identical unwraps each item and still passes. samples_of_generated now returns Result<Vec<Sample>, String>, so the error reaches the caller; facts_datagen_round_trip unwraps it and still reaches its todo!.
  • fix(executor): run each node by its plan Lifetime, so roots drain in any order #1300's executor rules: at ingestion time a node with several consumers runs once and each batch reaches every consumer's queue; queries run through execute, ingestion through execute_ingestion. The driver's module doc now names both. execute_shared_ingestion_stream_cleared_after_producer is renamed execute_shared_ingestion_scan_feeds_every_consumer, and its doc states that rule; its assertions (180 / 420, 120 / 120) are unchanged. No test or skeleton used RawDataSlot::lifetime or the executor's lifetime map, so nothing else changed. No other suite was run, and GitHub CI was not awaited (per the speed rule).

Next: implementation after review

  1. Settle the provisional answers, the three open questions and the two findings. Adjust the marked tests.
  2. Implement in the order the tests unlock: the fit, the machine key and the paths (T0), then the accuracy lookup and its validate, then facts, then the execute driver, then calibrate, then bench and the gate.
  3. In the calibrate PR, delete provenance, the legacy array format and rqe_atomic_costs, per [Design] asap_run: one binary to execute plans, calibrate measured unit costs per machine, and bench predicted vs measured #1317.
  4. Run the T2 tests on a dedicated host.

Part of #1317.

@zzylol zzylol added accuracy Accuracy requirements, guarantees, and propagation cost-model Cost formulas, statistics, ranking, and selection dev-tools Developer tools, diagnostics, and demos testing Corpus, regression, differential, or benchmark tests labels Nov 15, 2026
@zzylol
zzylol force-pushed the fix/1290-executor-shared-producer branch from cacb7a0 to 4c64ca3 Compare November 15, 2026 07:02
@zzylol
zzylol force-pushed the fix/1290-executor-shared-producer branch from d2d5f60 to 88c83a0 Compare November 16, 2026 04:11
The test suite of the asap_run test plan: cost fit, measured cost
lookup, measured accuracy lookup and its Stage 1 hook, machine key and
table resolution, source facts, the execution driver, cost accounting,
the bench gate, regressions, the exhaustive oracle, end to end and
determinism. T2 tests are ignored as timing tests, T3 as milestone
scale.

Non-test code is the API skeleton only: the asap_run bin and the
devtools measurement module, every body todo!; the MeasuredSketchAccuracy
data types in asap-accuracy, their validate and AccuracyEvidence impl
todo!; the aqpbm-datagen git dependency in devtools. The oracle's plan
enumerator over Stage 2's candidates is test support in asap-types,
written independently of Stage 3.
…ical cores + memory type

Memory size and hostname are not part of the key. Memory type comes from EDAC, else unknown-memory, overridable by --memory-type / ASAP_MEMORY_TYPE. Tests and API skeleton only.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accuracy Accuracy requirements, guarantees, and propagation cost-model Cost formulas, statistics, ranking, and selection dev-tools Developer tools, diagnostics, and demos testing Corpus, regression, differential, or benchmark tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant