Repository navigation
Conversation
zzylol
force-pushed
the
fix/1290-executor-shared-producer
branch
from
November 15, 2026 07:02
cacb7a0 to
4c64ca3
Compare
zzylol
force-pushed
the
test/asap-run
branch
from
November 15, 2026 15:24
484a9e3 to
ec63991
Compare
zzylol
force-pushed
the
fix/1290-executor-shared-producer
branch
from
November 16, 2026 04:11
d2d5f60 to
88c83a0
Compare
The test suite of the asap_run test plan: cost fit, measured cost lookup, measured accuracy lookup and its Stage 1 hook, machine key and table resolution, source facts, the execution driver, cost accounting, the bench gate, regressions, the exhaustive oracle, end to end and determinism. T2 tests are ignored as timing tests, T3 as milestone scale. Non-test code is the API skeleton only: the asap_run bin and the devtools measurement module, every body todo!; the MeasuredSketchAccuracy data types in asap-accuracy, their validate and AccuracyEvidence impl todo!; the aqpbm-datagen git dependency in devtools. The oracle's plan enumerator over Stage 2's candidates is test support in asap-types, written independently of Stage 3.
…ical cores + memory type Memory size and hostname are not part of the key. Memory type comes from EDAC, else unknown-memory, overridable by --memory-type / ASAP_MEMORY_TYPE. Tests and API skeleton only.
zzylol
force-pushed
the
test/asap-run
branch
from
November 16, 2026 04:38
d6ea961 to
3b31953
Compare
zzylol
changed the base branch from
fix/1290-executor-shared-producer
to
main
November 16, 2026 04:38
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
asap_run([Design] asap_run: one binary to execute plans, calibrate measured unit costs per machine, and bench predicted vs measured #1317) needs tests before code: today nothing checks a measured cost table end to end, nothing readsMeasuredSketchAccuracy, and nothing compares the plan Stage 3 selects with every alternative.asap_runtest plan (102 tests in 4 crates) plus the API skeleton they compile against. Every new non-test body istodo!("asap_run: implement after test approval"). No feature is implemented.todo!). Example of a passing hand check: a query-time exact-sumSummaryAggover 600 rows at 10 ns/row + 500 ns/invocation is priced 6,500 ns = 0.0065 ms per execution, from the table.Review guide
mainat863153e3c(fix(executor): run each node by its plan Lifetime, so roots drain in any order #1300 merged). Head:3b3195300. Draft. Do not merge.crates/devtools/tests/measurement/mod.rsand the skeletoncrates/devtools/src/measurement/.Cargo.lock(theaqpbm-datagengit dependency and its transitive crates).Test plan summary
The plan specifies the four subcommands (
execute,calibrate,bench,facts) and theMeasuredSketchAccuracylookup, with invariants I1–I10:costs.jsonround-trips;The tests are in four tiers:
#[ignore = "timing: run on a dedicated host"];#[ignore = "milestone scale"].Expected values are never computed with the planner's own optimization or cost code. They are hand-computed, or they come from a fake clock or kernel of known cost, from sketch bounds, from the executor's own kernel charge, or from independent enumeration.
Provisional answers
The user had not yet answered the plan's questions Q1–Q23, so these tests use provisional answers, to be reviewed. Each test that depends on one is marked
// provisional answer N, sogrep -rn "provisional answer N" crates/finds every test to change.series_countand label counts are margined suggestions when a query has a whole-result target, otherwise written directlycolumn_magnitudeswritten with min_abs/2 and max_abs×2ns_per_invocationconverted per second; cost model unchanged--costswins over--cost-inputs' table with a warning; a corrupt file is an errordimm_mem_type;unknown-memorywithout EDAC, overridable by--memory-type/ASAP_MEMORY_TYPE); no memory size (decided 2026-11-16, user)PhysicalASAPMemoCandidates--datais JSONL{labels, ts_ms, value}; an evaluation before warm-up is an errorsample_intervalis the median of the per-series medians; distinct values and norms are taken over the longest window, per group; label sets only forbysets of ≥ 2 labelsprovenancedeleted; old tables fail to loadAnswers 2 and 12 have no test that depends on them alone.
Findings (the 4 failures)
kll_sized_by_closed_form_without_measured_table: a real finding in existing code.quantile_over_time(0.99, m[5m])over 8 series, with a per-value rank target of 0.0025 at δ = 0.01.readout_caps/mod.rs:187divides δ bymax(values_covered, readout_values). For a per-value target,values_coveredis 1, but the readout's 8 values still split δ to 0.00125. At that δ, k = 1404 has error 0.002505 > 0.0025.measured_accuracy_shrinks_sketchdepends on the answer.joint_sharing_plan_space_is_enumerable: the plan space is too large for the oracle as planned.count_over_time(m[5m])over 8 series.costs_json_rejects_removed_fields: expected to fail. It tests the deletion ofprovenanceand the legacy array format (Q22), which is part of the implementation.asap_planner_cli_uses_same_resolution: expected to fail.asap-plannerdoes not resolve the calibration tables yet, which is part of the implementation.Also checked against existing code, and passing:
lookupand serde tests;planner_prices_from_table(Δ = rows × 9 ns on each DDSketch build);shared_node_priced_once;invocation_cost_ranks_chain_widths;aqpbm-datagenstream;Tests
accuracy::accuracy_json_round_tripsrankmetric) not loadingaccuracy::closed_form_applies_below_the_measured_rangeaccuracy::empty_table_measures_nothingtodo!)accuracy::lookup_error_does_not_grow_with_deltatodo!)accuracy::lookup_exact_params_statistic_metrictodo!)accuracy::lookup_no_extrapolation_below_smallesttodo!)accuracy::lookup_takes_largest_recorded_delta_at_most_requesttodo!)accuracy::measured_entry_never_replaces_a_deterministic_guaranteetodo!)accuracy::table_validationtodo!)cost::aggregate_needs_every_measureAggregatepriced from a partial set of measure entriescost::costs_json_rejects_removed_fieldsprovenance/ legacy array still loadingprovenanceand the array format still load (the deletion is part of the implementation)cost::costs_json_round_tripscosts.jsonserde (I3)cost::lookup_exact_params_wincost::lookup_interpolates_in_log_sizecost::lookup_memory_needs_both_neighbourscost::lookup_no_extrapolationcost::lookup_other_algorithm_missescost::lookup_phase_entries_shadow_bothcost::node_priced_by_handcost_accounting::calibrated_table_covers_showcase_planstodo!)cost_accounting::fixed_size_state_bytestodo!)cost_accounting::invocation_cost_ranks_chain_widthscost_accounting::memory_per_state_handtodo!)cost_accounting::planner_prices_from_tablecost_accounting::shared_node_measured_oncetodo!)cost_accounting::shared_node_priced_oncecost_fit::fit_five_sizes_tightenstodo!)cost_fit::fit_flags_nonlineartodo!)cost_fit::fit_is_relative_weightedtodo!)cost_fit::fit_negative_intercept_is_clampedvalidate)todo!)cost_fit::fit_recovers_exact_linetodo!)cost_fit::fit_recovers_fake_clock_kerneltodo!)cost_fit::fit_recovers_spin_kerneltodo!)cost_fit::units_match_cost_modeltodo!)end_to_end::calibrate_accuracy_seededtodo!)end_to_end::datagen_seeded_bit_identicalend_to_end::e2e_full_showcasetodo!)end_to_end::e2e_small_showcasetodo!)end_to_end::e2e_tiny_hand_answerstodo!)end_to_end::facts_seeded_bit_identicaltodo!)end_to_end::generated_samples_seeded_bit_identicaltodo!)end_to_end::plan_given_tables_identicaltodo!)end_to_end::plan_stable_under_recalibrationtodo!)execution::execute_answers_bit_identicaltodo!)execution::execute_before_warmuptodo!)execution::execute_drain_ordertodo!)execution::execute_ingestion_equals_query_timetodo!)execution::execute_on_first_read_viewOnFirstReadviews refused or refreshed per evaluationtodo!)execution::execute_quantile_referencetodo!)execution::execute_rate_linear_countertodo!)execution::execute_retention_evictstodo!)execution::execute_several_eval_timestodo!)execution::execute_shared_ingestion_scan_feeds_every_consumertodo!)execution::execute_sum_over_time_handtodo!)execution::execute_window_boundarytodo!)gate::gate_counterexample_passes_by_exacttodo!)gate::gate_fails_on_lying_accuracy_tabletodo!)gate::gate_fails_on_lying_cost_tabletodo!)gate::gate_output_filestodo!)gate::gate_rule_puretodo!)machine_and_tables::asap_planner_cli_uses_same_resolutionasap-plannerandasap_runresolving different tablesasap-plannerprints no missing-table warning yet (part of the implementation)machine_and_tables::corrupt_costs_is_an_errortodo!)machine_and_tables::costs_flag_wins_over_cost_inputs_table--cost-inputs' table winning over--coststodo!)machine_and_tables::machine_key_from_cpu_cores_memorytodo!)machine_and_tables::machine_key_ignores_hostnametodo!)machine_and_tables::machine_key_slug_charset[a-z0-9-]todo!)machine_and_tables::machine_key_ignores_memory_sizetodo!)machine_and_tables::machine_key_without_edac_is_unknown_memorytodo!)machine_and_tables::machine_key_memory_type_override--memory-type/ASAP_MEMORY_TYPEignoredtodo!)machine_and_tables::machine_key_mixed_dimm_typestodo!)machine_and_tables::missing_accuracy_warns_separatelytodo!)machine_and_tables::missing_costs_warns_and_falls_backtodo!)machine_and_tables::path_precedence--costs/ env / repo / fallbacktodo!)measured_accuracy::accuracy_table_is_monotonetodo!)measured_accuracy::empty_table_equals_no_measuredtodo!)measured_accuracy::kll_sized_by_closed_form_without_measured_tablemeasured_accuracy::measured_accuracy_shrinks_sketchtodo!)measured_accuracy::measured_within_known_boundstodo!)measured_accuracy::quantile_of_errors_nearest_ranktodo!)measured_accuracy::worst_over_merge_counts_is_recordedtodo!)measured_accuracy::wrong_metric_entry_does_not_prunetodo!)oracle::enumerated_plans_are_closed_under_edgesoracle::enumerated_plans_contain_the_selected_planoracle::joint_sharing_plan_space_is_enumerableoracle::oracle_exact_familytodo!)oracle::oracle_joint_sharingtodo!)oracle::oracle_sketch_familytodo!)regression::regression_quantile_half_speedtodo!)regression::regression_quantile_half_speed_fulltodo!)regression::regression_ranking_ratetodo!)regression::regression_rate_1hz_every_300stodo!)regression::regression_rate_1hz_every_300s_fulltodo!)source_facts::facts_column_magnitudes_exclude_zerotodo!)source_facts::facts_cost_only_written_directlytodo!)source_facts::facts_counter_resettodo!)source_facts::facts_datagen_round_triptodo!)source_facts::facts_distinct_per_labeltodo!)source_facts::facts_guarantee_margintodo!)source_facts::facts_label_set_from_workload_bytodo!)source_facts::facts_margined_set_stays_validtodo!)source_facts::facts_reads_jsonl_samplestodo!)source_facts::facts_sample_interval_median_gaptodo!)source_facts::facts_series_counttodo!)source_facts::facts_sign_suggestion_sign_bitv >= 0.0accepting −0.0todo!)source_facts::facts_single_sample_seriestodo!)Interfaces
All added. No existing interface changes.
crates/accuracy/src/measured.rs: the approved types of #1317, data only.crates/devtools/src/measurement/: theasap_rundriver module. Every body istodo!.crates/types/src/test_support/physical_plan_enumeration.rs:test-supportonly, and implemented.crates/devtools/src/bin/asap_run.rs:mainistodo!. The planned CLI,asap_run execute | calibrate | bench | facts, is in its module doc.Usage: none yet. Every entry point is
todo!. The planned flow ise2e_tiny_hand_answers:factson a hand-made JSONL, thencalibrate --small --families exact, thenexecute --at 600000, which prints 180 / 420, thenbench --no-cost-gate, which writesruns/.Design scope
MeasuredSketchAccuracy,MeasuredAccuracyEntry,ErrorAtFailureProbability,MeasuredSketchAccuracyError(accuracy/src/measured.rs)AccuracyEvidencehookdevtools::measurement, withexecution,samples,cost_calibration,accuracy_calibration,machine,calibration_tables,benchmarkandsource_factsasap_runbinPhysicalPlanChoice,every_physical_plan(types/src/test_support/physical_plan_enumeration.rs)PhysicalASAPMemoCandidates, shares no code with Stage 3aqpbm-datagengit dependency (revcf8f867)No existing production code path changes. The accuracy crate gains a
serdedependency for the new types.Decisions and open questions
Needs reviewer decision:
invocation_cost_ranks_chain_widths: the planned version compares invocation costs at equal units. The cost model charges ingestionns_per_invocationper second of arriving rows, so chains of different widths differ only in the states merged at query time (360 against 3,600). The test checks that the 10 s chain is kept.regression_ranking_rate: the plan's 85-case grid is not in the repo, so the test runs over the full showcase instead.accuracy_json_round_trips:AccuracyMetricserializes as"rank", but [Design] asap_run: one binary to execute plans, calibrate measured unit costs per machine, and bench predicted vs measured #1317's example JSON writes"Rank". The test uses"rank".factsmagnitudes:SourceFacts::column_magnitudes(feat(types, cost): cost DDSketch state from declared column_magnitudes #1324) is not on this base, so the written magnitudes are checked onSourceFactsReport::column_magnitudes.Decided:
/sys/devices/system/edac/mc/mc*/dimm*/dimm_mem_type): the single value if all DIMMs agree, otherwise the sorted unique values joined. Without EDAC the type isunknown-memory, overridable by--memory-type/ASAP_MEMORY_TYPE, and a warning names it. Slug[a-z0-9-]+with no double dashes; the model drops(R),(TM),CPUand@ <freq>. Example:x86_64-intel-xeon-e5-2683-v3-56c-registered-ddr4. Test changes:machine_key_from_cpu_cores_memoryis now a fixture (arch, model, cores, memory type) with the expected slug;machine_key_stable_under_memtotal_jitterbecamemachine_key_ignores_memory_size; addedmachine_key_without_edac_is_unknown_memory,machine_key_memory_type_overrideandmachine_key_mixed_dimm_types;machine_key_ignores_hostnameandmachine_key_slug_charsetstay.MachineDescriptiongainscpu_architectureandmemory_type;memory_kibstays formachine.jsononly;from_proctakes the architecture, the EDAC DIMM types and the override;detecttakes the override;warningis new.New counts after this change: 17 pass, 4 fail, 84 cannot run yet (the 3 new tests reach a
todo!).Verification
All tests ran locally, ignored tests included (
--include-ignored).todo!)cargo test -p asap-cost --test measured_operation_costs, and--lib -- node_priced_by_hand aggregate_needs_every_measurecargo test -p asap-accuracy --test measured_sketch_accuracycargo test -p asap-devtools --test devtools -- measurement:: --include-ignoredcargo fmtis clean.Retarget to
main863153e3c: #1300 was squash-merged, so this branch was rebased with--onto origin/main, dropping #1300's six pre-squash commits. The previous headd6ea9616dis kept asbackup/test/asap-run-d6ea9616d. One conflict, incrates/cost/src/node/usage/tests.rs: main (#1324) and this PR both append tests at the end of the file, so both were kept. This PR'srows_ofhelper sets main's newProducerOutput::chain_tumbling_window_width_mstoNone.cargo check --workspace --all-targetsandcargo fmtare clean. Each count is unchanged:todo!)measured_operation_costs, plus the two--libtests)measured_sketch_accuracy)measurement::,--include-ignored)Restack onto #1300 at
d2d5f6077, withaqpbm-datagenatcf8f867: the counts are unchanged, 17 / 4 / 81, and so is each test's outcome. The same 17 tests pass, the same 4 fail with the same values (k = 2863; more than 1,000,000 plans;provenancestill loads; no missing-table warning), and the other 81 reach atodo!. Changes made for the restack:aqpbm-datagen74ef8d0→cf8f867:LabelSpecisSeriesLabelSpec,valueisvalue_distribution, andGeneratedTimeSeries::samples()yieldsResultitems (a counter past 2^53 is an error).datagen_seeded_bit_identicalunwraps each item and still passes.samples_of_generatednow returnsResult<Vec<Sample>, String>, so the error reaches the caller;facts_datagen_round_tripunwraps it and still reaches itstodo!.execute, ingestion throughexecute_ingestion. The driver's module doc now names both.execute_shared_ingestion_stream_cleared_after_produceris renamedexecute_shared_ingestion_scan_feeds_every_consumer, and its doc states that rule; its assertions (180 / 420, 120 / 120) are unchanged. No test or skeleton usedRawDataSlot::lifetimeor the executor's lifetime map, so nothing else changed. No other suite was run, and GitHub CI was not awaited (per the speed rule).Next: implementation after review
validate, thenfacts, then theexecutedriver, thencalibrate, thenbenchand the gate.calibratePR, deleteprovenance, the legacy array format andrqe_atomic_costs, per [Design] asap_run: one binary to execute plans, calibrate measured unit costs per machine, and bench predicted vs measured #1317.Part of #1317.