Skip to content

test(devtools): sketch-heavy workload set, five family snapshots, and a rollup test that selects a KLL - #1388

Draft
zzylol wants to merge 2 commits into
mainfrom
test/sketch-heavy-workloads
Draft

zzylol wants to merge 2 commits into
mainfrom
test/sketch-heavy-workloads

Conversation

@zzylol

@zzylol zzylol commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem. No checked-in workload shows the planner selecting UnivMon, a Count-Min or Count Sketch heap, or KMV. A family can drop out of plans without any test noticing. And the sketch-rollup fuzz test no longer tests a rollup: its workload now plans an exact quantile, so the test fails with "q's result is not a summary estimate".
  • Change. This adds the 75-workload sketch-heavy set to the DAG viewer, adds five MVP sketch_plan_snapshots entries (one per family, plus a mixed dashboard), and moves the rollup test to a grouped quantile that selects a KLL.
  • Effect. For example, l2_over_time(dns_query_client[6h]) at relative 0.05 over 4 streams is now pinned to an ingestion-time UnivMon chain (5 × 2,367). sketch_rollup_witness_costs_for_a_grouped_quantile goes from failing to passing.

Review guide

  • Base: main (1760a69f5, where refactor: tools and snapshots over the #1207 memo, edges and contexts (#1207 PR 9) #1239 landed). The two commits were rebased from refactor/1207-tools without conflicts.
  • Read first:
    • crates/devtools/src/planner_fuzz/relations/mod.rs: column_projections_folded;
    • crates/devtools/tests/planner_fuzz.rs: sketch_rollup_witness_costs_for_a_grouped_quantile;
    • tools/dag-viewer/workloads/sketch-heavy/gen.py.
  • Mechanical:
    • the 75 workload files and tables.json, all written by gen.py;
    • the index.json entry, also written by gen.py;
    • the five regenerated snapshot files (see Snapshot changes);
    • the workload count in test_viewer.py (30 → 105).

Behavior: before → after

The sketch-heavy set. A user opens Workload sets in the DAG viewer and picks Sketch-heavy (75). The 75 workloads are in five subsets, named by file prefix:

Subset Count What it holds
heavy-hitters 22 topk(10, sum by (url) (rate(http_requests_total[W]))) at L1- or L2-normalized targets
frequency-moments 13 l2_over_time
distinct 12 SQL COUNT(DISTINCT user_id) and PromQL distinct_over_time
mixed 21 queries that could share one sketch
blocked 7 shapes no sketch serves yet; see #1377, #1378, #1379

All names are synthetic. The set is planned with the default deployment and guarantees enforced. The two bug-* reproducers from the study are left out.

Snapshots (MVP, Fargate cost inputs), checked under sketch_plans_match_snapshots:

id covers
sketch-heavy-univmon-l2-mvp ingestion UnivMon chain, 5 rows × 2,367 columns
sketch-heavy-cmsheap-topk10-mvp query-time CMS heap (depth 10, width 272) over the exact rate chain
sketch-heavy-csheap-topk10-mvp query-time Count Sketch heap (depth 117, width 1,200)
sketch-heavy-sql-distinct-by-region-mvp KMV chain per region, k = 26,656
sketch-heavy-edge-dashboard-mvp CMS heap, exact rate, UnivMon chain (624 columns) and KMV chain in one plan

The rollup test.

  • Before: the test planned a generated PromQL quantile by (job, instance) (m_gauge). That reads only 40 latest samples per group, so the plan is exact and there is no KLL to roll up.
  • After: it plans SQL p90 per (city, host) over 1 h at 100,000 rows/s (4 cities × 5 hosts, ε 0.05, δ 0.01). That selects a KLL tumbling-window chain (k = 132), and asserts the SketchRollup { labels: ["city"] } witness builds and costs.
  • Shape search: p90 per (city, host) or per city over 5 min at A filtered query (city = 'NYC') doesn't read its group from a grouped summary another query builds #902's 666.67 rows/s still plans exact (Aggregate quantile, 2.59 /s), so the test does not use it.

Interfaces

None public. The one private helper is in devtools (relations/mod.rs):

+/// `root` with a chain of `Project`s of plain column references folded
+/// into one: SQL puts a grouped estimate under two (`Project [city, v]`
+/// over `Project [v, city]` over the `SummaryEstimate`). A root that is
+/// not such a chain is returned as it is.
+fn column_projections_folded(root: &Rc<OperatorNode>) -> Rc<OperatorNode>;

root_estimate and with_root_estimate call it first. Before this change, no SQL grouped estimate could give a SketchRollup, QuantileRank or SmallerTopK witness, because its root has two column-only Projects.

Usage. In the DAG viewer, the user picks Sketch-heavy → frequency-moments-univmon-l2__r4-10hz-6h-every60s-releps0.05-d0.05. server.py runs asap_planner --workload … --run-until stage3, and the viewer shows the selected UnivMon plan.

The snapshot check is:

cargo test --release -p asap-devtools --test corpus_snapshots -- --ignored sketch_plans_match_snapshots

It plans each manifest entry and compares it with <id>.json.

Algorithm changes

None in the planner. The witness change above is test tooling.

Decisions and open questions

Decided:

Snapshot changes

After the rebase onto main, #1239's float round-trip printing changed the last digits of four cost_per_second values in three of the new snapshots (sketch-heavy-univmon-l2-mvp, sketch-heavy-csheap-topk10-mvp, sketch-heavy-edge-dashboard-mvp), for example 2.6236000343500004e-12 → 2.62360003435e-12. They were regenerated with UPDATE_SNAPSHOTS=1. No plan changed.

Note: the exact COUNT state reads the bytes column

In the SQL plans of this set, the exact Count state's update input names a column, for example weight: Column(Named("bytes")) for SELECT url, COUNT(*) … GROUP BY url. This is not a sum of bytes:

  • The executor's Count state adds 1 for each row and ignores the value (ScalarState::update in crates/executor/src/summary_kernels/exact.rs). The column only tells it which rows to skip: a row whose column is NULL is not counted.
  • So Pass 1 picks a column known to be non-NULL on every counted row (counted_column in crates/stage1-pass1-per-computation-candidates/src/summary_realization.rs): the column a COUNT(col) filter proves non-NULL, else the first non-nullable Float64 column, else any non-nullable column. In tables.json, bytes is the only non-nullable Float64 column of clicks.

Checked end to end on the executor: SELECT g, COUNT(*) AS c, SUM(y) AS s … GROUP BY g with y a non-nullable Float64 column and the rows a: 100, 250, 4000 and b: 7 plans Count with weight: Column(y) and returns a 3 4350.0, b 1 7.0, the exact answer. With y nullable, the Count reads ts instead and returns the same result.

Verification

All results are at the PR head 62a6591a2 (rebased onto main 1760a69f5).

Scope Result Notes
cargo test --release -p asap-devtools --test corpus_snapshots -- --ignored sketch_plans_match_snapshots 1 passed 15 entries, all optimal; three new snapshots regenerated for the number printing (see Snapshot changes), the older ones unchanged
cargo test -p asap-devtools --test corpus_snapshots -- manifest_entries_build_their_arguments passed
cargo test --release -p asap-devtools --test devtools -- planner_fuzz 14 passed, 2 failed The 2 failures, issue_891_filtered_rate_reads_the_grouped_rate and finds_issue_902_…, are the missing filter-the-result candidate, #891, and also failed on #1239's head. sketch_rollup_witness_costs_for_a_grouped_quantile passes.
cargo test -p asap-devtools --lib planner_fuzz 9 passed
cargo test -p asap-devtools --test devtools -- workload_set_index_and_files_validate passed Every file validates as a PlanningWorkload.
python3 -m unittest test_viewer test_server (tools/dag-viewer, planner built) 81 passed
node --test *.test.mjs (Node 22) 29 passed
cargo fmt --all --check clean
cargo clippy -p asap-devtools --all-targets -- -D warnings clean

Follow-ups

…family snapshots

75 synthetic PlanningWorkloads (PromQL and SQL) where UnivMon, Count-Min or
Count Sketch with a heap, or KMV is the cheapest plan that meets the target,
with controls and the shapes no sketch serves yet (blocked-*). gen.py writes
the files, tables.json and the index entry. Five MVP entries in
sketch_plan_snapshots pin one plan per family.
…lects a KLL

The generated PromQL grouped quantile now plans exact, so there was no
sketch to roll up. Use SQL p90 per (city, host) over 1 h at 100,000 rows/s,
which reads a KLL chain, and fold nested column-only projections so the
witness finds SQL's estimate under two Projects.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dag-viewer DAG export and visualization tooling dev-tools Developer tools, diagnostics, and demos testing Corpus, regression, differential, or benchmark tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant