Skip to content

test(devtools): one corpus registry and production-path workload builders (#1253 T1) - #1256

Draft
zzylol wants to merge 1 commit into
test/1253-t0-strategy-docfrom
test/1253-t1-corpus-builders
Draft

zzylol wants to merge 1 commit into
test/1253-t0-strategy-docfrom
test/1253-t1-corpus-builders

Conversation

@zzylol

@zzylol zzylol commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: in-repo workloads live in six places (o11y corpus, sketch plan manifest, planner-layering examples, viewer memo examples, random mixed sets, fuzz seeds), each read by its own harness; tests build workloads by hand (integration-tests' fixtures::lower_promql lowers without the series identity column the production path adds); and the PromQL-or-SQL lowering and the planning-model setup were copied in asap_planner and planner_fuzz.
  • Change: asap_devtools::corpus_registry: one list of 68 workloads, each with its deployment (capabilities, cost inputs, accuracy enforced or not), and builders that plan only through the production path (plan_workload, repeating_promql_workload). asap_planner, planner_fuzz and the registry now share one lowering (planning_stages::lower_workload) and one model setup (with_executor_summaries + planning_models); planner_fuzz's tests read their seeds from the registry; the snapshot tests and milp_objective.rs read the registry.
  • Effect: T2's contracts and T4's oracles iterate one list planned exactly as the CLI plans; a workload added to any source is planned by every suite. Snapshots are unchanged (both snapshot tests pass in release on this head).

Review guide

  • Position: Tracking: restructure ASAPPlanner tests around layers and independent oracles #1253 T1. Base docs(testing): testing strategy, tiers and a measured coverage matrix (#1253 T0) #1254 (T0, test/1253-t0-strategy-doc), stacked on refactor: tools and snapshots over the #1207 memo, edges and contexts (#1207 PR 9) #1239. One commit, head 083e2e31e. T2 stacks on it.
  • Read first: crates/devtools/src/corpus_registry.rs (CorpusWorkload, plan_workload, corpus_workloads, one private function per source), then planning_stages.rs (lower_workload, with_executor_summaries, planning_models).
  • Mechanical: asap_planner and planner_fuzz::Planner call the shared lowering and model helpers instead of their copies; FuzzWorkload::set_arrival_from_sources calls arrival_from_sources; corpus_snapshots.rs and sketch_plan_snapshots.rs loop over corpus_registry::o11y_bench() / sketch_plans(); corpus.json loses its always-empty asap_planner_args; planner_fuzz.rs's seed(name) reads fuzz_seeds/<name> from the registry.
  • Careful: plan_workload must plan as asap_planner does. It now shares the binary's lowering and model setup (one copy), and runs Stage 3 through the facade's run_planning (run_search_over with inspect), as planner_fuzz does. A MetricsQL workload is now an "unsupported query language" error in all three, not a SQL parse error.

Behavior: before → after

A contracts test wants every in-repo workload planned for its deployment:

Before After
Listing them read corpus.json, manifest.json (with its deployment argument lists), index.json (sets, cost model names), two directory scans, and the fuzz seed file parser corpus_registry::corpus_workloads()?: 68 CorpusWorkloads, ids such as o11y_bench/q21, sketch_plans/ratio-of-quantiles-1h-mvp, random_mixed/random-mixed-seed-862/workload-01
Planning one copy asap_planner's lowering and model setup, or run the binary workload.plan(RunUntil::Stage3)?: PlannedWorkload { entries, data, stages }, with the binary's own lowering and models; o11y q21 plans in 0.29 s (debug) at objective 0.9020/s, as asap_planner prints
A small rule or oracle workload hand-written JSON or IR repeating_promql_workload(&["sum(rate(m[5m]))"], 60_000, AccuracyTarget::exact(), &[("m", SourceFacts { series_count: Some(10), sample_interval: Some(DurationMs(1000)), ..SourceFacts::default() })]), then plan_workload(.., RunUntil::Pass1)

It serves #1253's goal 2 (one corpus registry read by contracts, snapshots and fuzz) and the rule that test inputs go through the production lowering.

Interfaces

crates/devtools/src/corpus_registry.rs

+/// Every source's id prefix, in `corpus_workloads` order: a workload's id is `<source>/<name>`.
+pub const SOURCES: [&str; 6] = ["o11y_bench", "sketch_plans", "planner_layering", "memo_examples", "random_mixed", "fuzz_seeds"];
+/// One registry workload with the deployment it is planned for: what `asap_planner` is given for it.
+pub struct CorpusWorkload {
+    /// `<source>/<name>`, unique in the registry, e.g. `o11y_bench/q21`.
+    pub id: String,
+    /// The workload as a user writes it (`asap_planner --workload`).
+    pub workload: PlanningWorkload,
+    /// A SQL workload's tables, each an `--sql-catalog-table` JSON object; empty for PromQL.
+    pub sql_tables: Vec<Value>,
+    /// `--deployment-capabilities`; `supported_summaries: None` plans with the executor's families.
+    pub deployment_capabilities: DeploymentCapabilities,
+    /// `--cost-inputs`.
+    pub cost_inputs: CostInputs,
+    /// `--ignore-accuracy-guarantees`.
+    pub ignore_accuracy_guarantees: bool,
+}
+/// A workload planned through the production path; costing it takes the cost inputs it was planned with.
+pub struct PlannedWorkload {
+    /// One entry per query, in workload order.
+    pub entries: Vec<QueryWorkloadEntry>,
+    /// The data workload (empty when the workload declares none).
+    pub data: DataWorkload,
+    /// Every stage's output up to the one planned until.
+    pub stages: PlanningStagesRun,
+}
+/// `workload` planned as `asap_planner --run-until <until>` plans it (its lowering and models);
+/// to Stage 3 through the facade's `run_planning`, every stage's output kept.
+pub fn plan_workload(
+    workload: &PlanningWorkload,
+    sql_tables: &[Value],
+    deployment_capabilities: &DeploymentCapabilities,
+    cost_inputs: &CostInputs,
+    ignore_accuracy_guarantees: bool,
+    until: RunUntil,
+) -> Result<PlannedWorkload, String>;
+impl CorpusWorkload {
+    /// This workload planned for its deployment up to `until`.
+    pub fn plan(&self, until: RunUntil) -> Result<PlannedWorkload, String>;
+    /// The `asap_planner` arguments for this workload, its input files written to `directory`.
+    pub fn asap_planner_args(&self, directory: &Path) -> Result<Vec<String>, String>;
+}
+/// Every registry workload, in source order.
+pub fn corpus_workloads() -> Result<Vec<CorpusWorkload>, String>;
+/// The two sources the snapshot tests iterate; the other four are private (filter by id prefix).
+pub fn o11y_bench() -> Result<Vec<CorpusWorkload>, String>;
+pub fn sketch_plans() -> Result<Vec<CorpusWorkload>, String>;
+/// The `planner_fuzz` seed workloads as the tester reads them, with their ids `fuzz_seeds/<file stem>`.
+pub fn fuzz_seed_workloads() -> Result<Vec<(String, FuzzWorkload)>, String>;
+/// The ingesting arrival at the sum of the sources' own rates, when every source declares one.
+pub fn arrival_from_sources(data: &DataWorkload) -> Option<DataArrival>;
+/// A PromQL workload of `queries`, each every `interval_ms` at phase 0 at `accuracy`,
+/// over `sources` (metric → facts), arriving at `arrival_from_sources`.
+pub fn repeating_promql_workload(
+    queries: &[&str],
+    interval_ms: u64,
+    accuracy: AccuracyTarget,
+    sources: &[(&str, SourceFacts)],
+) -> PlanningWorkload;

crates/devtools/src/planning_stages.rs

+/// `workload` lowered by its language's frontend (SQL over `catalog`), PromQL series identity
+/// as a column; MetricsQL and Elastic DSL are an "unsupported query language" error.
+/// `asap_planner`, `planner_fuzz` and the registry all lower with it.
+pub fn lower_workload(workload: &PlanningWorkload, catalog: &SqlCatalog) -> Result<Vec<QueryRoot>, String>;
+/// `capabilities` as `asap_planner` plans with them: the executor's summary families when none declared.
+pub fn with_executor_summaries(capabilities: DeploymentCapabilities) -> DeploymentCapabilities;
+/// The models `asap_planner` plans with: built-in evidence, these cost inputs and capabilities, accuracy enforced or not.
+pub fn planning_models<'a>(
+    capabilities: &'a DeploymentCapabilities,
+    cost_inputs: &'a CostInputs,
+    ignore_accuracy_guarantees: bool,
+) -> PlanningModels<'a>;

crates/devtools/tests/corpus_snapshots/o11y_bench/corpus.json: the field asap_planner_args (always []; the registry rejected anything else) is removed; every o11y query is planned on the default deployment.

Usage (as T2 runs it, checked by planning_until_a_stage_leaves_the_later_ones_empty and the contracts):

// Every in-repo workload with its deployment: 68 at this head.
let workloads = asap_devtools::corpus_registry::corpus_workloads()?;
// o11y q21: sum(rate(http_requests_total{job="order-service"}[5m] offset 1h)), a 1 h range query every 60 s.
let q21 = workloads.iter().find(|w| w.id == "o11y_bench/q21").unwrap();
// Frontend (lower_workload) → Stage 0 → Pass 1 → Pass 2 → Stage 2 → Stage 3 (run_planning), as asap_planner plans it.
let planned = q21.plan(RunUntil::Stage3)?;
// The plan Stage 3 selected and its report: objective 0.9020/s.
let (plan, selection) = planned.stages.run.selection.as_ref().unwrap();
// Its cost, with the workload's own cost inputs: 0.9020/s, Scan → chain (60 s) → TimeShift 1h → TimeRange 5m → SummaryMerge → finalize → sum.
let (cost, _) = asap_cost::plan_cost(plan, &planned.entries, &planned.data, &q21.cost_inputs).unwrap();

Algorithm changes

None.

Decisions and open questions

Needs reviewer decision

  • The new pub test-support API above (corpus_registry, in asap-devtools, which both devtools and integration-tests tests already depend on). Example: corpus_workloads()?.iter().filter(|w| !w.id.starts_with("random_mixed/")) is T2's fast-tier list. Alternative: a test-only crate; rejected for now as one more crate for one module.
  • What the registry includes. The six sources in the module doc; not the integration tests' sharing_issue_fixtures (they declare their SQL table in the test), not asap_planner CLI tests' inline workloads, not tools/dag-viewer/test-fixtures/ (a viewer fixture, not a planning input). No ASAPWorkloads workload files exist in-repo (only planner_fuzz generator patterns). The memo examples are planned on the default deployment (raw data retained, accuracy enforced), as the viewer opens them; two of them are also in sketch_plans/ under the MVP deployment.
  • plan_workload plans with the executor's summary families when the deployment declares none, as asap_planner and plan_installable do; milp_objective.rs used every family before. Both its tests still pass.

Decided

Independent review, round 1: findings fixed

  • F1, plan_workload was a copy of the binary's setup: one copy now; asap_planner and the registry call planning_stages::lower_workload, with_executor_summaries and planning_models.
  • F2, duplicate helpers: planner_fuzz::Planner::lower calls lower_workload (and planning_models); FuzzWorkload::set_arrival_from_sources calls the registry's arrival_from_sources; the registry's own lower_workload copy is gone.
  • F3, circular source check: every_source_loads_with_unique_ids asserts each source's count (o11y from its committed snapshot files, a listing the registry does not read; sketch_plans 10, planner_layering 4, memo_examples 7, random_mixed 30, fuzz_seeds 3) and five named ids.
  • F4, vacuous Pass 1 test: asserts a Scan is reached, selection is Err(NotRun), and physical_asap_candidates is None.
  • F5, corpus_snapshots.rs plans only o11y_bench: new fast test the_corpus_directory_holds_only_the_o11y_benchmark.
  • F6, dead asap_planner_args: removed from corpus.json, the registry check and planner-snapshots.md.
  • F7, missing keys read the repository root: every manifest and index key is read with an error naming the entry and the key.
  • F8, manifest test hard-coded today's deployments: compares each workload's capabilities and cost inputs with the files its manifest deployment names, parsed in the test.
  • F9, Stage 3 through a devtools copy: plan_workload(.., Stage3) runs run_search_over(.., PlanSearchSettings { inspect: true }), the facade's run_planning.
  • F10, MetricsQL went to the SQL lowering: lower_workload matches SQL(_) | DataFusion and returns "unsupported query language" otherwise (a_language_without_a_frontend_here_is_reported_as_unsupported).
  • Goal 2: planner_fuzz's tests read their seeds from the registry (fuzz_seed_workloads).

Skipped

  • fast_corpus_workloads() / a slow field: the tier split stays in T2 (one place, by source).
  • lower_sql_roots building a tokio runtime per call: once per workload, negligible at 68 workloads.

Verification

Scope Passed / failed / ignored Newly failing vs base Notes
asap-devtools lib 16 / 0 / 0 0 base 12; +4 registry tests
asap-devtools --test devtools 69 / 2 / 1 0 the 2 failures also fail on the base (workload_examples_select_valid_plans_with_linked_materialized_views, the_nine_query_workload_lowers_and_installs); planner_fuzz tests read the registry's seeds
asap-devtools --test corpus_snapshots (debug) 7 / 0 / 2 0 +1 the_corpus_directory_holds_only_the_o11y_benchmark
asap-devtools --test corpus_snapshots --release -- --include-ignored 9 / 0 / 0 0 both snapshot tests pass through the registry and the shared lowering: no snapshot changed
asap-integration-tests --test integration 151 / 0 / 5 0 same as base; milp_objective plans through the facade
cargo fmt --check; cargo clippy -p asap-devtools -p asap-integration-tests --tests --bins -D warnings clean

New tests:

  • every_source_loads_with_unique_ids: every source loads with its expected count, ids are unique, named workloads are present.
  • a_repeating_promql_workload_arrives_at_its_sources_rates: 1,000 series per 1 s plus 100 per 10 s arrive at 1,010 rows/s.
  • planning_until_a_stage_leaves_the_later_ones_empty: planning until Pass 1 keeps the PromQL series identity on every scan (at least one) and leaves Pass 2 to Stage 3 empty (NotRun).
  • a_language_without_a_frontend_here_is_reported_as_unsupported: a MetricsQL workload fails as an unsupported language.
  • manifest_entries_load_with_their_deployments (replaces manifest_entries_build_their_arguments): each entry loads with the capabilities and cost inputs of its deployment's files and its accuracy setting.
  • the_corpus_directory_holds_only_the_o11y_benchmark: a second corpus directory fails until it is added to the registry and the snapshot test.

Planning time per registry workload (debug, to Stage 3): 0.02–7 s; each random mixed workload about 7 s.

Follow-ups

  • T2 (contracts over every registry workload) and T4 (oracles) stack on this PR.
  • T3 deletes the integration tests' own lowering helpers (fixtures::lower_promql, which drops the series identity, and planning::run_all_stages), moving their callers to these builders.
  • T5: a memo builder for rule unit tests, when the first rule test needs one.
  • asap_planner --run-until below Stage 3 still runs run_stages_until, a copy of run_planning's stage order; Move asap_planner's stop-after-stage into run_planning #1169 moves the stop point into run_planning.

@zzylol zzylol added the testing Corpus, regression, differential, or benchmark tests label Nov 10, 2026
@zzylol
zzylol force-pushed the test/1253-t0-strategy-doc branch from 399034d to 2eb3209 Compare November 10, 2026 02:43
@zzylol
zzylol force-pushed the test/1253-t1-corpus-builders branch from 9ada1ac to 083e2e3 Compare November 10, 2026 02:43
…lders that plan through the production frontends and Stage 0; snapshots read the registry (#1253 T1)
@zzylol
zzylol force-pushed the test/1253-t0-strategy-doc branch from 2eb3209 to 525bf2d Compare November 10, 2026 13:37
@zzylol
zzylol force-pushed the test/1253-t1-corpus-builders branch from 083e2e3 to dcc5f2f Compare November 10, 2026 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

testing Corpus, regression, differential, or benchmark tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant