Skip to content

fix(cost): charge KMV builds and merges their measured time (build 110 + 0.62, merge 2.6 + 0.43 per compared hash) - #1375

Open
zzylol wants to merge 27 commits into
fix/delta-split-over-dependent-readoutsfrom
fix/kmv-build-cost-refit
Open

zzylol wants to merge 27 commits into
fix/delta-split-over-dependent-readoutsfrom
fix/kmv-build-cost-refit

Conversation

@zzylol

@zzylol zzylol commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Closes #1373. Stacked on the δ-split PR (base fix/delta-split-over-dependent-readouts), whose k = 4,323 sizing exposes these charges.

Summary.

  • Problem:
    • The KMV build was charged 0.61–0.85 of its measured time. At the viewer rollup's shape (k = 4,323, 60,000 rows per state), it was charged 553 ns per row and measures 712.
    • The KMV merge was charged 0.9x to 4.9x its measured time, and more the more states a group merges. It took every merged hash as distinct, while merged states share hashes.
  • Change:
    • The build is refit on 132 executor builds: 110 per row plus 0.62 per compared hash (was 65 and 0.5).
    • The merge is refit on 45 executor merges: 2.6 per hash plus 0.43 per compared hash (was 65 and 0.5), with the states sharing the smallest hashes of the column's values. StateInputRows now carries the folded column's distinct values for that.
  • Effect:

Review guide.

  • Read first: crates/cost/src/node/operations_per_unit.rs. It holds the four KMV constants, update_operations_per_row and merge_operations_per_state.
  • Then StateInputRows::distinct_values in crates/cost/src/node/state_bytes/mod.rs, and where node_resource_usage_and_cost sets it (crates/cost/src/node/usage/mod.rs).
  • Then the tests:
    • crates/cost/src/node/usage/tests.rs:
      • a_kmv_build_charges_its_measured_time_per_row (new);
      • a_kmv_build_costs_per_retained_hash_it_compares (re-measured);
      • a_kmv_merge_charges_its_measured_time_per_state (new);
      • a_kmv_rollup_merge_costs_per_hash_it_inserts (re-measured).
    • crates/integration-tests/tests/rollup_controlled.rs.
  • Commits:
    • 3ae2dc498 tests only; the build tests fail on the base.
    • 68a1efa0f the build fix.
    • 310f03f5a test only; the merge test fails on 68a1efa0f.
    • 8632711b3 the merge fix.

Why

The planner should pick the plan that measures faster. In the viewer's distinct rollup (rollupheavy-t11: COUNT(DISTINCT user_id) by service and by (service, endpoint), 5 min every 60 s, 1,000,000 users), measured whole plans flip with the rows per state:

rows per state exact KMV (k = 4,323)
60,000 609.6 s 483.6 s (KMV 1.26x faster)
12,000 127.9 s 178.4 s (exact 1.39x faster)

Each plan has to be charged its measured cost for the choice to follow that crossover. The KMV build was the term furthest off. Once it is fixed, 56 grid plans move work into merges and windows, so the merge has to be charged its measured cost too.

About the 12,000-rows-per-state case. It was reported as picking KMV on the base. That report came from a probe that changed only arrival.rows_per_second; the planner prices a scan from series_count × sample_interval, so the probe still planned 600,000,000 rows. Planned at 12,000 rows per state (10,000 series sampled every 25 ms), the base already picks exact. The new test production_distinct_rollup_counts_exactly_at_12k_rows_per_state therefore passes on the base too.

What

  • The KMV build's per-row and per-compared-hash constants are refit.
  • The KMV merge's per-hash and per-compared-hash constants are refit, and its compared hashes count the hashes its states share.
  • StateInputRows gains distinct_values: the distinct values of the column the state folds. The build sets it from the column's declared distinct values. The build charge now reads it there instead of computing it a second time. Merging keeps the largest.

How

The executor's KMV is asap_sketchlib::Kmv. Its update_with_hash walks every retained hash to find a duplicate, unless the state is full and the hash is above its largest. So an update is O(k) while the state fills and for each hash that enters it (about k·ln(n/k) of them once full). A merge updates the group's state with every hash the input retains, so it walks the same way. This is a library defect: a membership check could be O(1) or O(log k), and a merge could be an O(k) sorted merge instead of O(k²). That fix belongs in asap_sketchlib as its own PR, pinned here afterwards, and the KMV constants must be measured again after it. This PR describes the executor as it is today.

kmv_compared_hashes_per_row already counts that walk: k × the fill fraction while filling, then the walks of the repeats and of the hashes that enter. The build fit keeps it as the size term.

The merge passes it the hashes the states share:

  • A state retains the smallest hashes of the values it folded.
  • So all states of a group draw their hashes from the same smallest hashes of the column's values.
  • Of V values, a state that folded D distinct ones and retains h hashes draws them from the V·h/D smallest.
  • A shared hash is walked until it is found, not added again.

Example: 1-minute states of (service, endpoint) at k = 4,323, each of 12,000 rows of 1,000,000 users. Each retains 4,323 hashes of about 11,930 users, drawn from the 362,000 smallest.

Key code interfaces added / modified

The private constants in crates/cost/src/node/operations_per_unit.rs:

-const KMV_UPDATE_OPERATIONS_PER_ROW: f64 = 65.0;
+const KMV_UPDATE_OPERATIONS_PER_ROW: f64 = 110.0;
-const KMV_OPERATIONS_PER_COMPARED_HASH: f64 = 0.5;
+const KMV_UPDATE_OPERATIONS_PER_COMPARED_HASH: f64 = 0.62;
+/// Per hash a KMV merge reads from an input state, beyond the retained hashes it compares it with.
+const KMV_MERGE_OPERATIONS_PER_HASH: f64 = 2.6;
+/// Per retained hash a KMV merge compares an inserted hash with.
+const KMV_MERGE_OPERATIONS_PER_COMPARED_HASH: f64 = 0.43;

StateInputRows (public, crates/cost/src/node/state_bytes/mod.rs) and the crate-private merge charge:

 pub struct StateInputRows {
     pub rows_per_state: Option<u64>,
     pub non_negative: bool,
     pub value_magnitude_range: Option<ValueMagnitudeRange>,
+    /// The distinct values the folded column takes, each folded row a draw from them; `None` when
+    /// not declared, which takes every row's value as new. E.g. `Some(1_000_000)` for `user_id`.
+    pub distinct_values: Option<u64>,
 }

 pub(crate) fn merge_operations_per_state(
     family: StateFamily<'_>,
     state_bytes: u64,
     merged_state_bytes: u64,
-    rows_per_input_state: Option<u64>,
+    input_rows: Option<StateInputRows>,
     input_states_per_group: u64,
 ) -> f64

Usage, end to end:

// The viewer's rollup workload: two COUNT(DISTINCT user_id) queries, 5% relative error per
// value, over 10,000 series sampled every 5 ms (600,000,000 rows per evaluation).
let (workload, roots) = fixture();
// Stage 1 offers KMV at k = 4,323 (the δ split). Stage 3 charges each candidate through
// asap_cost:
// - A KMV build's update_operations_per_row is
//   110 + 0.62 × kmv_compared_hashes_per_row(4_323, Some(60_000), Some(1_000_000)) = 715 ns per row
//   (553 before; 712 measured at 400 states, 757 at 1,000).
// - Its 1-minute window states (12,000 rows each, distinct_values Some(1_000_000)) merge 5 per
//   group at 4,323 × (2.6 + 0.43 × compared) = 3.40 ms per state (3.12 measured; 4.22 charged before).
let run = asap_planner::run_planning(roots, &entries, data, models, PlanSearchSettings::default())?;
// Selected: both queries read one KMV (k = 4,323) by (service, endpoint), built in 1-minute tumbling
// windows at ingestion and merged over the 5 the range covers. Charged 401 s per evaluation; the
// query-time build is charged 481 s and the exact plan 590 s.

Major algorithm changes

Cost model: the KMV build.

  • Problem: at 400 states × 60,000 rows of 1,000,000 values at k = 4,323, the build was charged 553 ns per row and measures 712. Every one of the 132 grid builds was under-charged, by 1.17–1.63x.

  • Before: 65 + 0.5 × compared per row, fitted on 28 builds of 300,000 rows.

  • After: 110 + 0.62 × compared per row.

  • How it was fitted: same method as the exact distinct refit.

    • Measured on the executor's KMV SummaryAgg by (service, endpoint): the build's own CPU (its time minus its input's), median of 3 runs on one pinned core, 10 rows per group per batch.
    • Grid: 10, 100, 400, 1,000, 2,000 and 10,000 states × 1,200, 12,000 and 60,000 rows per state × 10,000, 100,000 and 1,000,000 distinct values × k = 1,081, 4,323, 26,656 and 106,263. That is 132 builds, capped at 24,000,000 rows.
    • The host was shared, and two settings measured up to 1.8x slow in one pass (1,000 × 1,200 and 100 × 12,000). Both were measured three or four times, and the median per point is kept.
    • Weighted least squares on relative error. Forms were compared by leave-one-setting-out error, where a setting is (states, rows per state, distinct values); errors are RMS of the log ratio.
    form leave one setting out worst point leave each k out
    per row + per compared hash (chosen) 0.058 0.25 0.057
    + per inserted hash, or + per inserted hash × log2 k 0.058 0.25
    + per compared hash × cache-miss fraction (10 MB of retained hashes) 0.049 0.22
    constants before this PR 0.280 (whole grid)
    • The heap's insert term adds nothing. The cache term fits a machine-specific cache size for a 0.009 gain, so it is left out.
    • Fitted 108.8 and 0.620, charged 110 and 0.62.
    • Leaving out each states count gives 0.065, each rows-per-state 0.061, each distinct count 0.057.
  • Held-out shapes, not in the grid, each the median of 3 passes:

    states × rows per state, distinct values, k measured ns/row new charge ÷ old charge ÷
    1 × 300,000, 100,000, 4,323 329 305 1.08 222 1.48
    4 × 75,000, 100,000, 26,656 8,357 8,679 0.96 6,975 1.20
    300 × 30,000, 300,000, 8,000 2,499 2,509 1.00 1,999 1.25
    3 × 200,000, 50,000, 2,000 234 183 1.28 134 1.75
    5,000 × 3,000, 2,000,000, 4,323 1,062 1,041 1.02 816 1.30
    30 × 100,000, 3,000,000, 50,000 20,090 18,519 1.08 14,911 1.35
    1,000 × 6,000, 1,000,000, 13,000 1,962 1,963 1.00 1,560 1.26
    200 × 40,000, 1,000,000, 4,323 882 899 0.98 728 1.21
    50 × 20,000, 20,000, 500 133 143 0.93 92 1.45

    RMS log error is 0.094 (was 0.322). The first two rows are the shapes in a_kmv_build_costs_per_retained_hash_it_compares. Their earlier measurements (191 and 7,410) do not reproduce on today's executor (329 and 8,357), so the test now quotes these.

Cost model: the KMV merge.

  • Problem: the merge was charged 65 per hash plus 0.5 per compared hash, taking every merged hash as distinct. Measured, merges cost 0.21–1.10 of that charge on the grid and 0.33–1.58 on held-out rollups. The error grows with the states per group, and it reverses when states share many values.

  • Before: retained × (65 + 0.5 × kmv_compared_hashes_per_row(k, retained × states, None)) per input state.

  • After: retained × (2.6 + 0.43 × kmv_compared_hashes_per_row(k, retained × states, Some(shared))) per input state:

    • retained is the hashes a state keeps, at most k: the distinct values among its rows.
    • shared is the V·h/D smallest hashes the states draw theirs from (see How).
  • How it was measured:

    • The executor's SummaryMerge over KMV states stored as a materialized view: a 1-minute tumbling-window chain by (service, endpoint) over 5 minutes, read through its time range.
    • Two measurements:
      • the window merge: 5 states per group;
      • a rollup that reads the stored windows by service: 5 × (endpoints per service) states per group, 10 to 200.
    • Each is the merge node's own CPU (its time less its input's, which reads the stored states), median of 3 runs, one pinned core, sibling hyperthread idle. Load average before and after each run was under 3.
    • The window states hold the k smallest hashes of their rows' user_ids, hashed as the executor hashes them, inserted in random order.
    • Check against states built by real ingestion runs, at k = 4,323 and 26,656: the window merge measures 3.09 and 79.9 ms per state either way, and the rollup 2.06 against 2.01 and 81.7 against 80.3.
  • Grid: k = 1,081, 4,323, 26,656 and 106,263 × window states of 1,200, 12,000 and 60,000 rows of 1,000,000 users × 5, 10, 50 and 200 states per group. That is 45 merges; states per group times state size was capped at a few minutes of CPU.

  • Forms compared by leave-one-setting-out error, where a setting is (k, rows per state, states per group):

    form leave one setting out worst point leave each k out leave each states-per-group out held-out rollups
    per hash + per compared hash, states share the smallest hashes (chosen) 0.050 0.10 0.122 0.061 0.091
    the same, every hash distinct (the form before this PR) 0.104 0.39 0.106 0.101 0.318
    the same, every hash distinct, each group's first state passed through 0.120 0.38 0.128 0.126 0.329
    chosen + per state, or + per hash × log2 k 0.046 0.10–0.11 0.111–0.134 0.064–0.065 0.080
    constants before this PR 0.524 (whole grid) 0.436
    • The extra terms gain 0.004 and fit a negative per-hash constant, so they are left out.
    • Fitted 2.61 and 0.428, charged 2.6 and 0.43.
  • Held-out rollups, not in the grid: states merged from 5 windows (6,000 to 300,000 rows each), then rolled up 10 or 100 per group. Each rollup is its node's time less the window merge below it, median of 3 runs. ns per input state, in ms:

    k states per group × hashes each (rows each) measured new charge ÷ old charge ÷
    1,081 10 × 1,081 (6,000) 0.150 0.144 1.04 0.234 0.64
    1,081 100 × 1,081 (6,000) 0.034 0.030 1.14 0.101 0.33
    1,081 10 × 1,081 (60,000 of 20,000 users) 0.224 0.238 0.94 0.234 0.96
    1,081 100 × 1,081 (300,000) 0.083 0.077 1.07 0.101 0.82
    4,323 10 × 4,323 (60,000) 2.006 2.261 0.89 2.902 0.69
    4,323 100 × 4,323 (6,000) 0.370 0.441 0.84 0.780 0.47
    4,323 100 × 4,323 (300,000) 1.235 1.202 1.03 0.780 1.58
    4,323 10 × 4,323 (60,000 of 20,000 users) 3.331 3.779 0.88 2.902 1.15
    26,656 10 × 26,656 (60,000) 80.25 85.61 0.94 101.40 0.79
    26,656 100 × 26,656 (60,000) 19.72 19.73 1.00 20.72 0.95
    106,263 10 × 11,929 (12,000) 298.8 281.3 1.06 348.3 0.86
    106,263 10 × 58,237 (60,000) 1,089.4 1,058.1 1.03 1,246.8 0.87

    This is 12 of the 24. Over all 24, measured ÷ charged is 0.84–1.14 and the RMS log error is 0.091 (was 0.33–1.58 and 0.436).

    • The rows of 300,000 per state, and of 20,000 users, are where states share the most hashes. Taking every hash as distinct (refit to this grid) charges them 0.36–0.68 of their time.
    • The k = 106,263, 10 × 11,929 row is the rollup in a_kmv_rollup_merge_costs_per_hash_it_inserts. It now quotes 299 s for its 1,000 states (median of 3 on an idle core), not the earlier 352 s (a plan's time less its build's).
  • One state per group. A merge whose groups each read one state passes it through. Measured that way, it costs 2–10 µs per state at k = 1,081 and 106,263 (1,000 states). The cost model sends a merge that keeps its input's groups down the pass-through path. A rollup whose estimates tie its input's (10,000 (service, endpoint) states into 10,000 services) still reaches this charge with one state per group. It is then charged as one state filled from empty: 3.9 ms per state at k = 4,323, the most any number of states per group costs. That rule predates this PR and is unchanged (see Limitations).

  • Correctness: the charge stays O(1) per node (the existing 64-slice average). No search or MILP change.

Before this PR

On the base, the viewer rollup at 600,000,000 rows per evaluation is charged 394.0 s per evaluation for a query-time KMV build (331.9 s of it the build), and 589.8 s for exact. The build measures about 1.3x its charge.

After this PR

The planner selects the KMV built in 1-minute tumbling windows at ingestion, then merged over the 5 windows at each evaluation. The three plans are charged:

plan build merges total (s per evaluation)
KMV in 1-minute windows, merged (selected) 191.0 169.8 (windows) + 38.8 (rollup to service) 400.8
query-time KMV build 429.2 38.8 (rollup) 480.5
exact 589.8

At the build-only refit (68a1efa0f) they were charged 453, 491 and 590.

Measured end to end at a scaled shape with the same rows per state:

  • 1,000 series, each sampled every 5 ms: 60,000 rows per (service, endpoint) state per evaluation, 12,000 per 1-minute window.
  • One pinned core, sibling hyperthread idle, load average under 3 before and after.
  • Per evaluation:
    • the window plan is one ingestion run (one new window) plus the query;
    • the other two plans are the query.
  • Two data mappings:
    • one endpoint per service, which the planner assumes (10,000 series, no per-label counts, so 10,000 services);
    • 10 services × 100 endpoints.
plan charged at this shape measured, one endpoint per service measured, 100 endpoints per service
KMV in 1-minute windows, merged (selected) 40.1 s 31.3 s (ingestion 15.8 + query 15.5) 31.7 s (15.6 + 16.1)
query-time KMV build 48.1 s 39.5 s 39.4 s
exact 59.6 s 62.9 s 57.1 s
  • Per node, one endpoint per service:
    • the window build measures 15.8 s against 19.1 s charged;
    • the window merge 15.5 s against 17.0 s;
    • the query-time build 38.8 s against 42.9 s;
    • the rollup to service passes each state through, about 0 s against 3.9 s charged (see Limitations).
  • With 100 endpoints per service, the rollup does merge: 0.5 s in the window plan and 0.8 s in the query-time plan.
  • The exact plan's one-endpoint run ended as another job started on the host (load average 27 after). Its 3 query runs came before that, and its by-(service, endpoint) aggregate matches the clean 100-endpoint run (32.6 s against 32.2 s).

The planner's choice is the measured fastest. Measured, the window plan is 1.26x faster than the query-time build and 1.8–2.0x faster than exact, with either data mapping. Scaled ×10 to the planned 10,000 series, that is about 313 s, 395 s and 571–629 s per evaluation, against 401, 481 and 590 charged.

Evidence

Plan survey, base (#1360 head) against this PR, through run_default_search_over with the default and the per-resource (UsageOverTime) cost inputs:

  • fix(cost): charge sketch states and merges, raw batch visits, grouping keys, distinct counts and KMV at their measured costs, so the measured-best plan is selected #1363's sweep (202 workloads): the grid of 120, showcase 24, held-out 48, 2,000-series 10. All 202 plans are identical node for node, and no cost changes. None of them contains a distinct count.

  • Distinct-count grid (220 plans):

    • Generated from the rollup workload and the SQL production shape (production_rollup_workloads), plus an ungrouped COUNT(DISTINCT user_id).
    • Swept over 100, 1,000 and 10,000 series × sample intervals of 5, 25, 250 and 2,500 ms × 1%, 5% and 10% error, plus rollupheavy-t11 and the planner-layering example 2.
    • Against the base, 96 plans change. These are the build refit's 63 operator or k changes, its 22 Project-order changes, and the merge refit's changes below.
  • The merge refit, against the build refit alone (68a1efa0f): 29 plans change.

    group build refit only with the merge refit count
    rollup, 250 ms, every series count and error, both cost inputs query-time KMV build, then a rollup merge query-time KMV built in 1-minute windows, then the window merge and the rollup merge 18
    SQL production shape and rollup, 100 series, 2,500 ms KMV at k = 26,656 or 106,263 KMV at k = 4,323 3
    exact plans, 25 ms Project held for the evaluation Project streamed (same operators, same cost) 8
    • The 56 plans the build refit moved into merges or windows all keep them.
    • At 250 ms a state folds 1,200 rows per evaluation. Built in 5 windows of 240 rows, each state's walk is shorter, and merging the 5 window states costs less than the rows' build. The merge measures that cheap (the grid's 1,200-row states cost 0.21–1.47 ms per state).

Verification

PR is 8632711b3. Base counts are from 5e17d5727; the #1360 head 50cf1976a adds only an assertion to default_cost_plan_ranking.rs and renames a test.

scope command base PR newly failing
asap-cost cargo test --release -p asap-cost 83 pass 85 pass none
asap-cost, merge test commit only (310f03f5a) cargo test --release -p asap-cost --lib a_kmv_merge_charges n/a 1 fail (charged 4,223,195 ns per state, measured 3,115,412) expected
asap-stage3-plan-selection lib cargo test --release -p asap-stage3-plan-selection --lib 18 pass 18 pass none
asap-integration-tests cargo test --release -p asap-integration-tests --no-fail-fast 206 pass / 6 fail / 5 ignored 207 / 6 / 5 none: the same 6 fail on both
fmt cargo fmt --all --check clean
clippy cargo clippy -p asap-cost --all-targets no warnings in asap-cost
  • The 6 failures on both: joint_search_equivalence ×2, plan_sharing ×2, promql_to_post_asap::promql_quantile_of_rate_binds_kll_over_rate_accumulator, materialized_views::materialized_views_equal_the_stored_outputs_main_listed.
  • With -D warnings, clippy stops at an unused StorageMedium import in asap-types, a file this PR does not touch.

New and changed tests:

  • a_kmv_build_charges_its_measured_time_per_row (new): six measured builds (10 and 1,000 states, 120,000–12,000,000 rows, k = 1,081–106,263) are each charged within 0.8–1.25 of their time. It fails on the base (109 charged vs 176 measured at k = 1,081).
  • a_kmv_build_costs_per_retained_hash_it_compares: now quotes the re-measured 329 and 8,357 ns per row. It fails on the base (222 vs 329).
  • a_kmv_merge_charges_its_measured_time_per_state (new): six measured merges (5 to 200 states per group, k = 1,081–106,263, one of 20,000 users) are each charged within 0.8–1.25 of their time. It fails on the build refit (4,223,195 charged vs 3,115,412 measured for 5 window states at k = 4,323; 88,151 vs 18,109 for 200 states at k = 1,081).
  • a_kmv_rollup_merge_costs_per_hash_it_inserts: now quotes the re-measured 299 s, within 0.8–1.25 (was 352 s, within 0.75–1.33).
  • production_distinct_rollup_counts_exactly_at_12k_rows_per_state (new): at a sample every 25 ms, both queries count exactly. It passes on base and PR (see Why).
  • production_distinct_rollup_builds_kmv_at_60k_rows_per_state (changed): the shape is now set by sample_interval, and a KMV built in tumbling windows also counts as a KMV build.

Design scope

  • StateInputRows::distinct_values (cost, the description of what one summary state folds). It is the folded column's declared distinct values. Any summary whose cost depends on repeated values can read it, not only KMV.

Architectural decisions

  • The merge's compared hashes count shared hashes. Taking every hash as distinct fits the grid at 0.104 and the held-out rollups at 0.318. Counting the hashes states share fits them at 0.050 and 0.091.
  • The cache term is left out of the build fit (see the build fit table).
  • No library fix here, as decided. The constants describe today's executor.

Limitations and follow-up

  • asap_sketchlib: O(1)/O(log k) duplicate check and O(k) sorted merge (KMV insert and merge are O(k) and O(k²) in asap_sketchlib; unusable at the k the planner sizes #709). After it is pinned, measure the KMV build and merge again.
  • A rollup whose group estimates tie its input's is charged as one state per group filled from empty. A real rollup at k = 4,323 measures 2.0 ms per state at 10 states per group and 0.44 ms at 100, against that 3.9 ms. With no per-label counts, the planner cannot know the states per group. Both KMV plans of the headline rollup carry this charge (38.8 s), and measured with one endpoint per service it costs nothing.
  • distinct_values is the column's declared distinct values. When a workload declares none, the merge takes every hash as distinct. That charges a merge of states that share many values at about 0.36–0.68 of its time.
  • The build grid ran on a shared host. Two settings needed repeated passes; the rest are single passes of median-of-3. The merge grid and the end-to-end runs ran on an idle host.

Human review — do not complete with an agent

  • The MVP boundary is correct.
  • New conceptual layers or public interfaces are necessary.
  • The before/after description matches the intended product behavior.
  • Human reviewer:
  • Decision and rationale:

…e covered output values depend on

Cases: per-value and whole-result targets without a reduction, a global
max, a grouped sum per value and over the whole result, nested
reductions, `or` per value and over the whole result, unknown readout
values, a record-share scope and a per-value ratio. Seven fail on the
current split, which divides by max(values covered, readout values).
…ues depend on

A query's failure probability was split per readout value by
max(values covered, readout values per evaluation): a per-value target on
`quantile_over_time(0.99, m[5m])` over 8 series got δ / 8 although each
output value is one readout value, doubling KLL's k (1404 -> 2863).

The union bound needs only the readout values the output values one
statement covers depend on: min(R, V * fan_in), with R the readout values
per evaluation, V the values the scope covers and fan_in the most readout
values one output value depends on (1 at a readout, carried value by value
through filters, projections, binary operators and `or`, all of the input's
values through a reduction). One function states it,
`AccuracyTarget::failure_probability_per_readout_value` over
`ReadoutValuesRead`; the readout walk computes both counts. An unknown
term drops out of the min, so a per-value target over series of unknown
count keeps δ, and a whole-result target with unknown n_q splits over R.
…ncatenations and redundant levels

A readout group that also holds a reduction over other readouts, a top-k
readout, and a concatenation each split δ over the readout values their
output values depend on; a level the same listed builds meet as a
stricter one is dropped.
…ver all values, drop redundant levels

- A readout group takes the larger of its own readout counts and its other
  candidates' (a reduction over readouts below it), field by field.
- A top-k readout's output value depends on every value it ranks.
- A concatenation takes each value from one input, like `or`.
- A level the same listed builds meet as a stricter level of the group is
  dropped: a copy at it accepts the same producers.
- ReadoutValuesRead is renamed ReadoutValueCounts, with larger and plus.
…the declared groups of rows

A whole-result quantile grouped by date_trunc('minute', ts) over a table
declaring 6 groups of rows returns about 16 values per evaluation, so its
readout must not split δ over 6. Grouped by src alone, the split over the
6 groups of rows still holds.
…en it is keyed by source labels

Memo::max_values_per_evaluation bounded every readout by the series (groups
of rows) scanned below it. A readout grouped by a computed key or a time,
such as date_trunc('minute', ts), returns one value per bucket, which that
count does not bound: a whole-result target with an unknown value count
split δ over 6 for about 16 values. The bound now applies only when every
key of the readout's state (or aggregate) is a source label or column
copied unchanged and not a time; otherwise R is unknown and, with n_q
unknown too, no per-value failure probability follows, as on the base.
… clamped to the rows read

- `flows JOIN hosts ... GROUP BY f.src, h.region` splits δ over 6 × 6 = 36
  readout values, or over the rows the join reads when fewer.
- `GROUP BY bytes`, a measure key with no declared count, splits over the
  rows read (96 for 6 groups of rows sampled every minute over 15
  minutes), and gets no level when they are unknown.
- A declared SQL key uses its declared distinct values.
- A self-join's readout keyed by both sides' `job` returns up to 20 × 20.
…t values per scan, clamped to the rows read

As SQL optimizers estimate a GROUP BY's groups (the product of the key
columns' distinct counts, clamped to the input rows), but as an upper
bound from declared facts only:

- keys from different scans (a join, or a self-join's two sides) multiply;
- a SQL key bounds the values only through its declared distinct values,
  not through the groups of rows the table declares, which do not say
  which columns form them;
- PromQL labels keep their series bound;
- every readout's values are at most the rows its input reads per
  evaluation: each source's series times the samples its window holds at
  its sample interval, or the rows it holds.
…values per scan, clamped to the rows read

GROUP BY dst with 300 dst values declared returns up to 300 values, not
the 6 groups of rows the table declares; an undeclared key is bounded by
the rows read; keys of joined scans multiply. A PromQL result keeps its
series bound.
…r scan, clamped to the rows read

A grouped SQL result's n_q was the groups its keys declare, at most the
groups of rows the table declares: GROUP BY dst with 300 dst values over
a table of 6 groups of rows gave 6, so a whole-result target split delta
over 6 instead of 300. n_q now comes from the memo's bound on the root
group (Memo::max_values_per_evaluation), the rule R uses: the product,
per scan, of the keys' declared distinct values, clamped to the rows the
grouping reads. A PromQL instant vector keeps its series bound. The
IR-side copy of the group rule in bounds is deleted.
…r join adds a NULL, a LIMIT caps the rows

m{side="l"} * on(x) group_left(instance) m{side="r"}, grouped by job and
instance, pairs a left series' job with a right series' instance: up to
10 x 10 values, not the 10 pairs one series holds (also group_right and
an offset side). flows LEFT JOIN hosts grouped by h.region returns the 6
regions and a NULL group, so the declared 6 do not bound it (also FULL
JOIN and two keys). A grouping over LIMIT 10 returns at most 10 values.
… side's scans; an outer join's nullable side is computed

A group_left/group_right match pairs each output series with one series
of the other side, as a join pairs rows: the labels it copies from that
side now move past the output side's scan ordinals (after_left_scans,
as for a join), so job and instance of two series of one metric multiply
instead of sharing one label-set count.

The columns of an outer join's nullable side (Left: right, Right: left,
Full: both) gain a NULL their source may not hold, so their lineage is
computed by the join; a key from them is bounded by the rows read.

LIMIT n over the whole input keeps at most n rows.

column_properties now lints wildcard enum arms: every lineage and sign
rule lists each variant, so a new operator states its rule.
…ta the declared facts allow

Each review round found another shape where Memo::max_values_per_evaluation
(the readout values R and a query's n_q) fell below the values a query can
return. This test checks the bounds against data directly: each query of
a corpus is lowered as planning lowers it, its Stage 0 memo built, and the
query run by the executor over datasets that keep to the declared source
facts (series, distinct values per label and label set, sample interval,
rows held). Worst-case seekers make the labels the facts bound only
together independent, crossed or aligned across series and rows, replace
series between the times a query reads, and split series into blocks by
literal labels so matches pair them one to one; random datasets add
more. Every bounded group's result rows, and the query's, must be at most
its bound.

The default corpus is the reviewed probe shapes (85 queries, 9 s); an
ignored variant runs larger datasets over the dag-viewer workloads and a
manifest of workloads. Cases the executor or the frontend cannot run yet
are listed by name, so a case joins the check once it can run.
SELECT dst, approx_percentile_cont(bytes, 0.9) AS q FROM flows ... GROUP
BY dst ORDER BY q DESC LIMIT 5 returns at most 5 values; the memo gives
96, the rows the window holds. Fails before the fix with Some(96).
`SELECT dst, approx_percentile_cont(bytes, 0.9) AS q FROM flows ... GROUP
BY dst ORDER BY q DESC LIMIT 5` returns at most 5 values, but
Memo::max_values_per_evaluation read the keys through the Limit and gave
the 96 rows the window holds. The bound now also takes the rows the
group itself returns (max_rows_per_evaluation), which caps a whole-input
LIMIT by n, as it already does for a LIMIT below the grouping.
…name the kept identity checks

The Join arm computed the right side's columns after the left's scans
even for a semi or anti join, whose output keeps only the left side's
columns. The lint header now names each kept `if let`/`matches!` and why
every other variant's answer is "no", and says that arithmetic_sign's
tuple match ends in an unchecked `_ => Unknown`, the safe answer there.
…ew-valued keys first

The value-bounds oracle counted a case as run when any one of its bounds
ran, so bounds that compile only alone (the n_q of a scalar() or vector()
operand, a DISTINCT, a quantile over count_values) went unchecked
silently. It now tracks each bound as `case:what` with the most values
any dataset returned: EXECUTOR_GAPS lists the bounds the executor cannot
run with the operator each lacks, and NEVER_REACHED the bounds that run
but return nothing on every dataset (none now). Any other unrun or
always-empty bound fails the test.

p13_unless read `qot(0.9, m) unless qot(0.5, m)`, empty on every dataset;
it now reads `a unless b` over two metrics.

The crossed dataset gave the fastest digit to the label sorted first, so
`dst` (300 values) cycled while `src` stayed fixed over the 96 rows a
window reads. Declared components are now crossed smallest first: against
the earlier bounds code the oracle finds j_full's 42 groups over a bound
of 36, and j_left's 7 over 6, on the crossed dataset.
…up expects KMV at its planned 600M-row shape

The δ split offers stored KLL windows at several k, so the KLL test
compares the smallest k offered with the next larger one instead of
fixed values.

The distinct rollup test asserted exact counts. Measured, KMV is 1.26x
faster at 60,000 rows per state (the planned shape) and exact is 1.39x
faster at 12,000. The test is renamed for what it asserts and records
the crossover.
@zzylol zzylol added bug Something isn't working cost-model Cost formulas, statistics, ranking, and selection labels Nov 16, 2026
…tinct rollup counts exactly at 12,000 rows per state and builds a KMV at 60,000

Measured on the executor over 132 builds, the KMV build is charged 0.62-0.84
of its time (at k = 4,323 and 60,000 rows per state, 553 ns per row charged
against 712 measured). The rollup tests vary the series' sample interval,
which sets the rows the planner prices.
…r compared hash

Refit on 132 executor builds (10 to 10,000 states, 1,200 to 60,000 rows per
state, 10,000 to 1,000,000 distinct values, k = 1,081 to 106,263): the
leave-one-setting-out error is 0.058, and measured / charged is 0.81-1.27,
against 1.17-1.63 at 65 and 0.50. The KMV merge keeps its own constants (65
per hash, 0.5 per compared hash), which its measured rollup matches.
Measured on the executor's SummaryMerge over stored KMV states (median of 3
runs on one pinned core), 5 to 200 states per group at k = 1,081 to 106,263:
the merge is charged 1.35x to 4.9x its time (200 states of 1,200 rows at
k = 1,081: 88,151 ns charged per state, 18,109 measured).
…compared hash, with states sharing hashes

Refit on 45 executor merges of stored KMV states (5 to 200 states per group,
1,200 to 60,000 rows each, k = 1,081 to 106,263): the leave-one-setting-out
error is 0.050, and measured / charged is 0.90-1.08, against 0.21-1.10 at 65
and 0.5. Each state retains the smallest hashes of the values it folded, so
merged states share hashes; StateInputRows now carries the folded column's
distinct values, which the merge uses for that. On 24 held-out rollups,
measured / charged is 0.84-1.14 (RMS log error 0.091, was 0.436).
@zzylol
zzylol force-pushed the fix/kmv-build-cost-refit branch from c9f874c to 8632711 Compare November 16, 2026 22:48
@zzylol zzylol changed the title fix(cost): charge a KMV build its measured time per row (110 + 0.62 per compared hash) fix(cost): charge KMV builds and merges their measured time (build 110 + 0.62, merge 2.6 + 0.43 per compared hash) Nov 16, 2026
… distinct-values plumbing and StateInputRows::merged

- The expected distinct values among n draws from V values,
  -V * exp_m1(-n / V), was written three times (exact distinct pairs, KMV
  compared hashes, KMV merge folded values); one private helper now.
- The KMV rollup merge test's producer declares the 1,000,000 users of
  the run it quotes, so it covers distinct_values reaching the merge.
- StateInputRows::merged keeps the larger distinct_values and None when
  either is unknown; a test checks both.
- Reflow two doc comments in rollup_controlled.rs.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working cost-model Cost formulas, statistics, ranking, and selection

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant