Repository navigation
fix(cost): charge sketch states and merges, raw batch visits, grouping keys, distinct counts and KMV at their measured costs, so the measured-best plan is selected - #1363
Conversation
e41cc1c to
3c0c826
Compare
b40ac65 to
c96d5b5
Compare
bb78057 to
91913fe
Compare
… k, output rows and carried labels as measured Rebased onto #1355's head with `column_magnitudes`: the PR's earlier commits (regret cases as failing tests, the fix, three tests kept under the new costs, the round-1 tests and fixes) applied as one, resolved against the base's declared value magnitudes, its no-wildcard rule in `operations_per_unit` and its restructured state-bytes tests. - A DDSketch state is charged the bytes it is expected to hold: the dense store over the span of its own rows under a named exponential spread (`exponential_values_magnitude_ratio`), not the library's worst case. - A KLL merge costs per byte of its input and merged state, a KLL estimate per byte, both fitted on the bytes the planner charges; the merged state is sized for the rows its group's states fold together (`state_input_rows`, carried by Stage 3's `logical_output` with the family). - A hash aggregate, a finalize and every estimate pay for each group's output row; a merge per output group; carried labels per row and label. - A summary state's row holds its labels and its own series identity (`state_row_bytes_beside_state`); `state_bytes_per_row` is the one rule for a state's bytes on an edge, compaction included. - Tests: the regret cases, a KLL merge within 2x of its measured time per series, the smallest offered KLL k selected, structural sharing, a disk-only objective, lognormal and narrow-gauge bands. The declared `column_magnitudes` range does not bound the DDSketch span yet; the next commits add that cap test-first.
… exact chain emits at their measured cost; a few-state KLL merge per group; a merge sized from all its inputs (failing)
…xact chain emits and few-state KLL merges at their measured cost; size a merge from all its inputs - DDSketch: the span is the smallest of the rows' exponential spread, the column's declared `column_magnitudes` and the assumed nine decades (`ValueMagnitudeRange::magnitude_ratio`). A gauge declared within [90, 110] is charged 1,248 bytes at any row count (holds 1,168). - An exact Min or Max state merges at 125 operations (measured 113–153 ns), a Sum or Count at 230. - A chain emits an exact state at 1,500 operations (measured 1.3–2.1 µs), a sketch state at 2,500 (`chain_emit_operations_per_state`). - A KLL merge pays 19,000 more per output group: the executor measures a merge of 5 states of 60 rows at 32–33 µs per series where the states' merges are charged 14 (`merge_output_operations` takes the family). - `StateInputRows::merged` sizes a merge's state from every input's rows and states per group, not the first input's. - Tests: #1313's known DDSketch gap in `priced_bytes_agree_with_the_executor` is removed (5,992 priced, 6,288 charged); a declared range never raises the bytes of rows that span less; values spread evenly in log over seven declared decades are under-charged 1.6x (pinned).
…L estimate after updates at its measured time; the rate and increase cells at 10 Hz build at query time; log-uniform DDSketch values over a declared range (failing) - An ingestion-time chain costs a time range's work per row besides its update: a Rate chain measures 87 ns per row at 10 Hz where the same Rate built at query time measures 66-75 and its time range 11. - A KLL quantile read from a state built by updates measures 13.4 us per group (300 rows, k = 547), after a merge 5.9. - `sum by (job) (rate|increase(requests_total[5m]))` at 10 Hz every 300 s with raw data builds the accumulator at query time (measured 0.86 against 0.95 per second for the stored windows). - The DDSketch under-charge is pinned on the case the planner sees: rows known, random order, log-uniform over seven declared decades, charged 0.33-0.70 of what the library holds.
…ead after updates at its measured cost; time range refit to 11 - A tumbling-window chain keeps each row it reads in its window's range at either timing: at ingestion time it assigns each arriving row its window, charged `TIME_RANGE_OPERATIONS_PER_ROW` as at query time. A Rate chain measures 87 ns per row at 10 Hz where the query-time Rate and its time range measure 66-76 and 11. - `TIME_RANGE_OPERATIONS_PER_ROW` is refit to 11 over 54 time ranges at 1 Hz and 10 Hz (100 to 10,000 series). The counter update (45) and the carried label (37) keep their values: refit on the same points they give 83 per row together (82 charged). Their per-row time follows the sample rate, not the rows per series, so no per-group term is added; the docs say why. - A KLL estimate reads a state built by updates (no merge since) at 1.8 per byte (`KLL_ESTIMATE_OPERATIONS_PER_BYTE_OF_UPDATED_STATE`), a merged one at 0.9: fitted on 11 executor nodes at 100 to 10,000 series, 300 to 36,000 rows per state. - The merge's fallback to the largest input state and the compaction's missing per-group term are documented where they apply.
…ain's window assignment, the KLL read after updates, Min/Max merges and exact chain emits in the design doc
…LL read after updates is nearly flat in its rows; the identical sum_over_time pair at 1 Hz keeps stored windows (failing) The per-row cost of a per-series hash aggregate, a query-time Rate build and an ingestion-time Rate chain follows the sample rate: each group's state is visited once per raw batch, 10 rows of a series at 1 Hz and 100 at 10 Hz. A KLL estimate over a state built by updates measures nearly the same at 300 and 3,000 rows per state. These replace the test of a time range charged per arriving row.
…sh aggregates, summary builds and chains; per-row constants refit at 1 Hz and 10 Hz; a KLL read after updates per unsorted item A new deployment input, ResourceSpeed::raw_batch_width_ms (10 s assumed by default), sets how many raw batches a grouping's input spans. Each group's state is visited once per batch (140 operations), and a chain also splits each group's rows of a batch by window (190) and routes each arriving row at ingestion time (8), replacing the time range charged per arriving row. The per-row constants are refit with the new term over 1 Hz and 10 Hz at 100 to 10,000 series; a hash aggregate's refit part is taken from its measure, so a grouping by several keys keeps its charge per key, and an exact distinct count, top-k or frequency statistic hashes each value as a key. A KLL estimate over a state built by updates costs 40 per unsorted item, at most k, instead of 1.8 per byte; a finalize charges its output row directly. A query-time chain sizes one window's states by the sources' rate, so Stage 3's baseline and plan_cost agree on the states a merge over it folds.
…put and the KLL read after updates in the design doc and library API; the rate and increase cells explained by the rate-dependent per-row cost
…e merge passes its state through; keys, measures, exact distinct counts and KMV builds and rollups at their measured costs; the years objective test at a pinned batch width (failing) - a_grouping_counts_raw_batches_from_the_time_its_input_spans and a_query_time_chain_sizes_its_windows_from_the_time_its_input_spans: a filter that keeps a tenth of each series' rows drops no raw batch. - a_merge_reading_one_state_per_group_passes_it_through: KLL and DDSketch merges of one state per group measure about 1 ms per 1,000 groups. - a_hash_aggregate_pays_each_further_key_and_measure_as_measured, an_exact_distinct_count_costs_more_per_row_with_more_distinct_values, a_kmv_build_costs_per_retained_hash_it_compares and a_kmv_rollup_merge_costs_per_hash_it_inserts: SQL aggregates, exact COUNT(DISTINCT) and KMV builds and rollups measured on the executor. - the_objective_is_the_plan_cost_on_one_off_queries_over_years pins raw_batch_width_ms at 30 min, plans with those cost inputs, and checks strictly that keeping raw data costs less than the views.
…s as pass-through; keys, measures, exact distinct counts and KMV at their measured costs; per-row, visit and chain constants refit on a grid - A grouping counts the raw batches its input arrives in from the time its raw rows span (ProducerOutput::raw_rows_span_ms: a scan's lookback, narrowed by a time range, kept by the operators that keep each row's window), and a query-time chain sizes its windows the same way, so a filter between the scan and the grouping drops no batch. - A SummaryMerge that reads one state per group passes each state through and pays only each group's row. - A hash aggregate pays 16 per row for its first key, 28 per further key and 4 per measure (SQL aggregates over 1 to 3 keys, 1 to 4 measures, 100 to 10,000 groups). - An exact distinct count or frequency statistic pays per square root of the distinct values its column takes and per value and pair it inserts (45 executor aggregates up to 3 M distinct values). - A KMV update and merge pay per retained hash they compare a hash with (asap_sketchlib's duplicate check); 22 builds and one rollup. - OPERATIONS_PER_GROUP_PER_RAW_BATCH 170, EXACT 29, COUNTER 33, DDSKETCH 210, KLL 84, CHAIN_OPERATIONS_PER_ARRIVING_ROW 14, refit by squared log error over 245 workloads (1 to 10 Hz, 100 to 10,000 series, 5 min to 1 h, every 60 and 300 s); the loss and the terms chosen by leave-one-setting-out error. The chain's own per group and batch term is deleted: it did not lower that error. - raw_batch_width_ms is the data store's batch width. - The two SQL distinct rollup tests now expect the exact counts, the plan measured cheaper; the Rate chain test takes this run's measurements.
…'s batch width, one-state merges, keys, exact distinct counts and KMV in the design doc and library API
…ts input's; a grouped exact distinct count pays its key and visits once; a sort, limit, dedup and concat pass on their raw rows' span (failing)
…s input's; exact distinct terms refit with the key and visits fixed; one merge row 1,100; raw rows' span through sort, limit, dedup and concat - A SummaryMerge passes one state per group through only when it keeps its input's groups (no group_by, or the input's own keys); a rollup whose estimate ties its input's is charged its merges. - The exact distinct count's own terms are refit with the grouping key, the measure, the group visits and the output row fixed at their constants (41 per row, 0.53 per square root of the distinct values, 72 per inserted entry), so a grouped count pays them once. - MERGE_OPERATIONS_PER_OUTPUT_GROUP 950 -> 1,100: lowers the error over every merge on the fitting grid and on the held-out settings; the 10,000-group range is stated in the docs. - A Sort, Limit and Dedup pass the raw rows' span on; a Concat of raw rows spans the longest of its inputs'. - Doc fixes: the validation wording, the KMV fit (28 builds, 63.8 + 0.51), and the distinct rollup test's justification at its own shape.
…ut's; the raw rows' span through sort, limit, dedup and concat; the merge paragraph rewrapped
499c276 to
e089d2f
Compare
|
Rebased with Validation on the merged tree: |
Summary
k. Finalize is charged 40 ns per group against 2.5 µs. The labels a per-series aggregate carries and a stored state holds are not charged. A Max state's merge and an exact chain's emit are over-charged. And a grouping's cost per row depends on the sample rate (1.15–1.39x its charge at 1 Hz, 0.75–0.92x at 10 Hz), soy11_identical_pairat 1 Hz picked an exact scan that measures 1.24x the stored plan.column_magnitudes, and nine decades.ResourceSpeed::raw_batch_width_ms; its 10 s default is an assumed value. A chain also pays per arriving row at ingestion time. The per-row, visit and chain constants are refit on 245 workloads (1 to 10 Hz, 100 to 10,000 series, 5 min to 1 h), with the loss and the terms chosen by leave-one-setting-out error. The time range is refit to 11 per row.asap_sketchlibdoes. All measured on the executor.k. A merged state's estimate stays at 0.9 per byte.quantile_over_time(0.99, latency[1h])every 60 s over 1,000 series at 10 Hz, with raw data kept. The planner now selects stored DDSketch windows, not the exact scan: predicted 41.3 cost/s against 73.4 for exact, measured 44.6 against 74.8, with 210 ms of query CPU per evaluation against 4,221. On the 192-decision sketch-benefit grid, the selected plan is within 1.05 of the measured best in all 192 decisions, worst 1.013 (fix(cost): charge each operation the executor's measured time, so a sketch plan is no longer predicted cheaper than a faster exact plan #1355: 173/192; round 3's head927aa85da: 192/192, worst 1.028), and in all 96 decisions of the review's 48 held-out workloads, worst 1.015 (927aa85da: 95/96, worst 1.065). Two SQL distinct-count rollups now count exactly: the KMV rollup they selected measures 393 s per evaluation against 14.7 s.Closes #1361. Closes #1362.
These keywords take effect only on a PR whose base is the default branch. This PR is stacked on #1355's branch, and #1355 does not contain these fixes, so after #1355 merges this PR must be retargeted to
mainfor the keywords to close the issues.Review guide
fix/cost-model-measured-unit-ratios(fix(cost): charge each operation the executor's measured time, so a sketch plan is no longer predicted cheaper than a faster exact plan #1355) atbb7805794, which includes feat(types, cost): cost DDSketch state from declared column_magnitudes #1324 (column_magnitudes).f8b0fe2e6: the PR as of round 1 (cc4b59342, on fix(cost): charge each operation the executor's measured time, so a sketch plan is no longer predicted cheaper than a faster exact plan #1355's earlier heade41cc1c5c), applied as one commit and resolved against the new base. It now also carries the three test compile fixes that round 2's head had in its test commit, so every commit compiles its tests (cargo check --tests -p asap-costatf8b0fe2e6: clean). The earlier commits are onbackup/fix/cost-model-sketch-benefit-errors-cc4b59342, and round 2's head onbackup/fix/cost-model-sketch-benefit-errors-b40ac657b.73dddd424: the round-1 functional findings and the declared-range cap, as failing tests.4c07b1c7d: their fix.a572ae507: the round-2 cost findings (C1, E1) as failing tests, and the re-pinned DDSketch test (C2).cda63582e: their fix.c96d5b507: the field doc and the design doc (D1). Round 2's head is onbackup/fix/cost-model-sketch-benefit-errors-c96d5b507.2d9556114: the round-3 findings (R1, R2, R3) as failing tests.db82a0393: their fix.927aa85da: the design doc, the library API and the reworded C1 explanation. Round 3's head is onbackup/fix/cost-model-sketch-benefit-errors-927aa85da.af84e3d66: the round-4 findings (L3, L4, M1, M2) as failing tests.ff3cb22bf: their fix and the refit.d09ce37dc: the design doc and the library API. Round 4's head is onbackup/fix/cost-model-sketch-benefit-errors-d09ce37dc.5dff04c1b: the round-5 polish findings (L1, the grouped exact distinct count, the raw rows' span) as failing tests.c036846d7: their fix, the merge row at 1,100 (L3) and the doc nits.499c276de: the design doc.crates/cost/src/node/operations_per_unit.rs,OPERATIONS_PER_GROUP_PER_RAW_BATCH(the fit),kmv_compared_hashes_per_row,exact_distinct_value_operations_per_row,hash_aggregate_operations_per_rowandmerge_operations_per_state. Incrates/cost/src/node/usage/mod.rs,ProducerOutput::raw_rows_span_ms,group_raw_batch_visitsand theSummaryMergearm. Thencrates/cost/src/node/state_bytes/mod.rs(ddsketch_state_bytes,StateInputRows::merged) andResourceSpeed::raw_batch_width_msincrates/cost/src/model.rs.state_input_rows: Noneandraw_rows_span_ms: Nonein theProducerOutputtest constructors, and the merge tests that add the per-group term.raw_rows_span_ms(a scan, a rawTimeRange, a materializedTimeShift; thenFilter,Project,TimeShift,Sort,Limit,Dedup, and the longest of aConcat's raw inputs), and the chain's rows per window at query time from it;max_distinct_per_evaluation, else every row's differs);StateLastWrittenByin theSummaryEstimatearm;keys_are_input_keys, which decides whether aSummaryMergepasses one state per group through;group_raw_batch_visits(at most one visit per row) and its floor of one batch at query time;state_input_rowson;state_bytes_per_row, which subtracts what a state's row holds beside its state;ddsketch_state_bytes.Round-4 review fixes
Review of
927aa85da:Validation. Each of the 35 settings (one series count, rate, window and interval together) is left out in turn, and its points are predicted by a fit on the rest. RMS log error of the left-out points: squared loss 0.147, absolute loss 0.150, so the squared loss is used. Leaving out each rate, each series count or each window instead gives 0.148, 0.172 and 0.148, with the same terms and the squared loss again at or below the absolute.
Chain terms (L2). With the chain's per group and batch term the error is 0.147 (the term fits to 15), without it 0.147, with only that term 0.155, with neither 0.165. So
CHAIN_WINDOW_OPERATIONS_PER_GROUP_PER_RAW_BATCHis deleted and the per arriving row term stays.Constants, fitted then rounded: per group and raw batch 140 → 170; one key and one measure of a hash aggregate 18 → 20; exact update 30 → 29; counter update 36 → 33; DDSketch update 190 → 210; KLL update 84 (unchanged); per arriving row 8 → 14. RMS log error over the 826 points: 0.155 → 0.143. No constant was set to keep a planner test passing.
production_sql_distinct_region_service_rolluppassing; exactCOUNT(DISTINCT)and KMV were not measuredExact distinct counts (45 executor aggregates, 300,000 and 3,000,000 rows, 100 to 100,000,000 distinct values, 1 to 1,000 groups): the executor interns each value in a table of every distinct value read, then keeps each group's (group, value) pairs. Of 26 forms, the one with the lowest leave-one-out error (0.215) is 61.3 + 0.4845 × √(distinct values) + 103.7 × (distinct values + pairs) ÷ rows per row. Round 5 refits its terms with the key, measure, visit and output-row terms fixed (see the polish table). The distinct values come from the column's declared
distinct_values_per_label, at mostmax_distinct_per_evaluationper group, else every row's differs.KMV (28 builds):
asap_sketchlibcompares each new hash with every hash a state retains until the state holdsk.kmv_compared_hashes_per_rowgives the expected count, within 1% of a simulation, and the build costs 63.8 + 0.51 per compared hash, charged 65 + 0.50 (leave-one-out error 0.107 squared, 0.200 absolute). A KMV merge inserts each input hash the same way: 348 s charged against 352 s measured for the rollup below.The test.
production_sql_distinct_region_service_rollupbecomesproduction_sql_distinct_region_service_counts_exactly. Both plans, as the planner sizes them, were executed over 12,000,000 rows of the test's shape (1,000 (region, service) groups, 1,000,000 distinct users): the two exact counts take 14.7 s per evaluation, and the KMV plan at k = 106,263 takes 393 s (its build 40.0 s, charged 36.5 s; the rollup the rest). The planner now selects the exact counts.rollup_controlled::production_distinct_rollup_finds_shared_kmv(the same two-level distinct count over the viewer's workload, with a time index) changed the same way and becomesproduction_distinct_rollup_counts_exactly.Executor issue. The grouped KMV build's 3.6–7.6 µs per row is
asap_sketchlib's linear duplicate scan, which #709 already tracks. The measurements are on #709 (comment), so no duplicate issue was filed. #1366 does not touch this path.<=was vacuous, and its result depended on the assumed batch widthraw_batch_width_msat 1,800,000, plans with those cost inputs, and checks with<again: the plan selected with raw data kept costs 4.7e5/s against 7.1e5/s for the views. The comment says that at the 10 s default every row is a visit, so the two plans are the same.raw_rows_span_msonProducerOutput(a scan's lookback, narrowed by aTimeRange, kept byFilter,ProjectandTimeShift) overraw_batch_width_ms. A query-time chain sizes its windows the same way. Tests:a_grouping_counts_raw_batches_from_the_time_its_input_spans(3,000 visits charged for 30,000 rows that keep a tenth of each series' 300 s; now 30,000) anda_query_time_chain_sizes_its_windows_from_the_time_its_input_spans(30 rows per state, now 6).a_merge_reading_one_state_per_group_passes_it_through(a KLL one-state merge of 1,000 groups was charged 21.2 ms against 1.0 ms measured). One-state merges: RMS log error 1.367 → 0.375 at the fitted settings and 1.356 → 0.152 at the held-out ones.SummaryMergeis now in the per-kind error table.d09ce37dc. The body says that lowering the exact update also lowers every unmeasured sketch's update (HLL, CMS, UnivMon, Hydra), which add their counters to it; it went from 40 to 30 in round 3 and to 29 here. KMV's update is now measured.The Rate chain test (
an_ingestion_time_chain_costs_more_per_row_at_a_lower_sample_rate) now uses this run's measurement, 106.3 and 87.6 ns per row (1.21x). The earlier run measured 125.6 and 85.7, as 1 Hz ingestion varies between runs.Round-5 polish
Review of
d09ce37dc(verdict ready, three low findings and nits):SummaryMergepasses one state per group through only when it keeps its input's groups: nogroup_by, orgroup_byequal to the input's own keys (every plain column but the time index, in any order;keys_are_input_keys). Otherwise it paysmerge_operations_per_stateper input state. Test:a_rollup_merges_its_states_when_its_group_estimate_ties_its_input(KMV states by (region, service) rolled up byservice, both estimated at 1,000 groups: charged 0.95 ms, as a pass-through, before). Therollup_controlledandproduction_rollup_workloadstests still pass; the KMV rollup inrollup_controlledis now charged 1,794 s per evaluation where it was 9.5 ms.production_distinct_rollup_counts_exactlyjustified with another workload's numbersMERGE_OPERATIONS_PER_OUTPUT_GROUPis raised to 1,100, which the fitting grid supports: over every merge on the grid the RMS log error of measured ÷ charged goes from 0.281 to 0.248 (one state per group 0.375 → 0.273, several 0.214 → 0.233), and on the held-out settings from 0.225 to 0.217. 1,200 fits the grid's one-state merges better (0.236) but raises the held-out error (0.230), so 1,100 was chosen. The doc states the range: at 10,000 groups one-state merges still measure a median 1.38–1.62x their charge (up to 2.29x for KLL), at 100 and 1,000 groups 0.96–1.20x.a_grouped_exact_distinct_count_pays_its_key_and_visits_once(byjobover 100 users, 60.2 ns per row measured, charged 83 before).an_exact_distinct_count_costs_more_per_row_with_more_distinct_valuesnow declares the 1,000,000 users that run measured, not the 259,000 of them its rows hold.499c276de.raw_rows_span_msgapsSort,LimitandDeduppass the span on; aConcatof raw rows spans the longest of its inputs', and one with an input that is not raw rows has none. Test:a_sort_limit_dedup_and_concat_pass_on_the_time_their_raw_rows_span.Verification of the polish (release builds on a 56-core host; harness runs pinned one per core on one socket, builds and tests on the other):
joint,qrank,quantandsotat 1 Hz and 10 Hz,max,cnt,rate,incr): regret ≤ 1.05 in 30 of 30, worst 1.000; 23 hits, 7 ties, 0 misses (error band 0.136 over the 47 plans). Every decision selects the same plan as round 4's sweep, and predicted costs move by at most 1%. A first run alongside the test builds on the other socket measured every plan about 1.6x higher; it also kept all 30 within 1.05 (worst 1.006), and was then repeated with nothing else running.DF_ARMS=exact,planned, 300 s of evaluations.git rebase d09ce37dc --exec 'cargo test --release --no-run -p asap-cost -p asap-types -p asap-stage3-plan-selection -p asap-integration-tests': all 3 new commits compile their tests.asap-cost --lib78 passed (75 atd09ce37dc), the 3 new tests among them;asap-stage3-plan-selection18 + 2 passed. Integration modulesrollup_controlled(1),production_rollup_workloads(3),stage3_objective(16) anddefault_cost_plan_ranking(14) pass at both heads;plan_sharingandpromql_to_post_asapfail the same 3 tests atd09ce37dcand here (nested_exact_sum_reuses_newest_tumbling_windows,slower_repeat_reuses_faster_sum,promql_quantile_of_rate_binds_kll_over_rate_accumulator).cargo fmt --all -- --checkis clean.cargo clippy -p asap-cost -p asap-stage3-plan-selection -p asap-integration-tests --all-targets -- -D warningsis clean with Rust 1.98. Two things stop clippy outside this round's changes:asap-typeshas an unusedStorageMediumimport onmain, and clippy 1.99 flags this PR'sEULER_GAMMAliteral (approx_constant), which 1.98 does not.Earlier review rounds
c0ba0abe5): KLL merge and estimate refitted on the bytes the planner charges; Stage 3'slogical_outputtakesstate_input_rowsfrom the candidate it takes the family from; onestate_bytes_per_rowand onecolumn_bytes; structuralplan_sharingandstage3_objectiveassertions.b40ac657b): the rate and increase cells at 10 Hz; the DDSketch under-charge pinned on the practical case; the KLL estimate after updates; commit hygiene; thecolumn_magnitudesfield doc.d09ce37dc): the grid refit, ties, keys and measures, exact distinct counts and KMV measured, the raw rows' span, one-state merges as pass-through.c96d5b507): the per group and raw batch visit shared by every grouping, withResourceSpeed::raw_batch_width_ms; the KLL estimate per unsorted item after updates;y11_identical_pairkeeps stored Sum windows; the finalize's output row charged directly.Each round's table is in the PR's edit history.
Behavior: before → after
quantile_over_time(0.99, latency[1h]), 1% relative error at δ = 0.01, every 60 s, raw data kept, defaultCostInputs:{"query_workload": {"language": "promql", "repeating_queries": [ {"query": "quantile_over_time(0.99, latency[1h])", "demand": {"fixed_interval_at": {"interval": 60000, "evaluation_phase": 0}}, "requirements": {"accuracy": {"max_error": 0.01, "failure_probability": 0.01, "accuracy_scope": "per_value"}}, "predictability": {"predictable": {"known_at": null}}}]}, "data_workload": {"arrival": {"ingesting": {"rows_per_second": 20000.0}}, "sources": {"latency": {"series_count": 1000, "metric_type": "gauge", "column_signs": {"value": "non_negative"}, "sample_interval": 100}}}}Scan→TimeRange(1h)→ per-series quantile)SummaryTumblingWindowChain→TimeRange(1h)→SummaryMerge→SummaryEstimate)This serves the MVP goal: the planner picks the plan the executor measures cheapest, and shows where sketch plans beat exact.
Interfaces
crates/cost/src/node/usage/mod.rs/// What a consumer reads of one producer node (stays). pub struct ProducerOutput<'a> { ... pub chain_tumbling_window_width_ms: Option<u64>, + /// What each summary state it outputs folds: a build's rows per state, a merge's rows over + /// every input's states per group, passed on by a `TimeRange` and the operators that keep + /// each row's window; `None` elsewhere or when unknown. + pub state_input_rows: Option<StateInputRows>, + /// The time, in ms, its raw output rows span: a scan's lookback (one second at ingestion, + /// the retention of raw data materialized there), narrowed by a `TimeRange`, passed on by + /// `Filter`, `Project`, `TimeShift`, `Sort`, `Limit` and `Dedup`, the longest of a `Concat`'s + /// raw inputs'; `None` for any other node. A grouping counts its raw batches from it. + pub raw_rows_span_ms: Option<u64>, } /// What one node uses and outputs (stays). pub struct NodeResourceUsageAndCost<'a> { ... pub chain_tumbling_window_width_ms: Option<u64>, + /// What each summary state it outputs folds (`ProducerOutput::state_input_rows`). + pub state_input_rows: Option<StateInputRows>, + /// The time its raw output rows span (`ProducerOutput::raw_rows_span_ms`). + pub raw_rows_span_ms: Option<u64>, pub work_ms_per_execution: f64, ... }crates/cost/src/node/state_bytes/mod.rscrates/cost/src/model.rs/// How fast one core does work, and the batches of raw rows its work is split into. pub struct ResourceSpeed { pub ms_per_cpu_op: f64, pub ms_per_scanned_byte: f64, + /// The time one batch of raw rows covers in the deployment's data store, which returns raw + /// rows in those batches: the executor looks up a series' group once per batch. Default + /// 10,000 (10 s), an assumed value; omitted from JSON, it takes that default. + /// `CostInputs::validate` refuses 0. + #[serde(default = "default_raw_batch_width_ms")] + pub raw_batch_width_ms: u64, }crates/types/src/workload/data.rscrates/cost/src/node/operations_per_unit.rs(crate-private)crates/cost/src/node/usage/mod.rs(private)Usage, end to end (
quantile_rank_without_raw_data_charges_the_larger_kll_moreanda_kll_merge_costs_more_at_a_larger_k):Algorithm changes
DDSketch state bytes
sketch_state_bytescharged the library's worst case over every order of inserts and merges for values over nine decades, or over the declared range: 18,752 B for any state of three or more values. The executor's states of latency data hold 2.9 KB (10 rows) to 5.9 KB (3,000 rows). feat(types, cost): cost DDSketch state from declared column_magnitudes #1324's declared range alone does not fix it, because a column's range is wider than one state's values.ddsketch_state_bytes):Stores: 1 when the values are proven non-negative or the state folds one row, else 2.
Span: the smallest of three magnitude ratios:
nrows,exponential_values_magnitude_ratio=n · H_n · e^γfor exponential samples;column_magnitudes;Without known rows, the span is the smaller of the last two.
Bytes: struct + stores × (the library's first block of buckets + 8 × (buckets spanned − 1)).
priced_bytes_agree_with_the_executoris gone (5,992 B priced, 6,288 B held).SummaryMerge, estimate, chain emit and output rows
state_input_rowsis recorded on builds and chains, and passed throughTimeRangeand the operators that keep each row's window. ASummaryMergesizes its state from every input (StateInputRows::merged).k. Any other sketch's estimate pays 1,000.TIME_RANGE_OPERATIONS_PER_ROW(11) per row. At ingestion time it routes each arriving row to its window's state, atCHAIN_OPERATIONS_PER_ARRIVING_ROW(8).Assumptions and limits: past
k, the KLL charge is the item array's capacity, so the smallerkis over-charged. At 10 × 360 states per series, the planner charges k = 547 and k = 1,404 1.9x apart, where they measure 3x apart. The executor also copies a group's KLL state before each merge.Group visits per raw batch
group_raw_batch_visits= min(input rows, groups × batches). The batches are the time the input's raw rows span (raw_rows_span_ms) overraw_batch_width_ms: at query time at least one; at ingestion time one second's share of a batch. An input that is not raw rows counts as one batch at query time. Every grouping pays 170 per visit; a chain pays nothing more per visit, and 14 per arriving row at ingestion time.max_over_time(latency[5m])over 1,000 series is charged 20 per row plus 30,000 visits × 170: 37 ns per row at 1 Hz and 21.7 at 10 Hz. Underlatency > 100, which keeps a tenth of each series' rows, the same 30,000 visits remain.Keys, distinct counts and KMV
distinct_values_per_label, at mostmax_distinct_per_evaluationtimes the groups, at most the rows read; the pairs are each group's expected distinct values among its rows.kmv_compared_hashes_per_row(k, rows per state, distinct values)per row. While the state retains fewer thankhashes, every update walks them; once full, only a hash below the largest is walked. A KMV merge inserts each retained input hash the same way, over all the hashes its group inserts. The distinct values are the folded column's declared ones, at mostmax_distinct_per_evaluation; without either, every row's differs.asap_sketchlib's linear duplicate scan (KMV insert and merge are O(k) and O(k²) in asap_sketchlib; unusable at the k the planner sizes #709); they must be measured again when it is fixed. Exact distinct counts were measured up to 3,000,000 distinct values; past that the lookup is charged as at 3,000,000.One-state merges
A
SummaryMergethat keeps its input's groups (nogroup_by, or the input's own keys) and whose input states are no more than its groups passes each state through (the executor keeps a group's first state unmerged) and paysMERGE_OPERATIONS_PER_OUTPUT_GROUP(1,100) per group, whatever the family. A rollup to other keys merges, even when its group estimate ties its input's. Measured: 0.94–1.0 ms per 1,000 groups for KLL, 0.97 for DDSketch, 0.87–1.31 for exact accumulators; at 10,000 groups 1.5–1.8 µs per group, a median 1.38–1.62x the charge.Carried labels and stored state rows
carried_label_operations_per_rowcharges 37 per input row for each string column of the output schema beyond the grouping keys. The counter update drops from 84 to 45, because the 84 was measured with one carried label.state_row_bytes_beside_state. Every other edge prices the series identity at 16 B (Planner charges less memory for a held OneEvaluation output than the executor holds #1301).Design scope
ProducerOutput::state_input_rowsandNodeResourceUsageAndCost::state_input_rows(cost, node usage): what each summary state a node outputs folds. This is a general property of any summary state flowing between nodes. It reusesStateInputRows.StateInputRows::merged(cost, state bytes): what a merged state folds, from any number of inputs. It is a method on the type whose data it combines.state_bytes_per_row,state_row_bytes_beside_stateandcolumn_bytes(cost, state bytes): one rule each for a state's bytes on an edge, the bytes beside it, and a column's price, for every family.exponential_values_magnitude_ratio(cost, state bytes): the named default spread when nothing is declared.ValueMagnitudeRange::magnitude_ratio(types, workload): a ratio of the declared range's own data, so it lives on that type.merge_output_operationsandchain_emit_operations_per_state(cost, operations per unit): per-family formulas kept beside their constants.KLL_ESTIMATE_OPERATIONS_PER_BYTE_OF_UPDATED_STATEand thestate_from_mergeargument ofestimate_operations_per_group(cost, operations per unit): whether a state's last writer was a merge is read from its producer's payload inusage, so no field is added.realize_from(integration tests): a test helper.ResourceSpeed::raw_batch_width_ms(cost, model): how the deployment batches raw rows, a calibration input beside the time per operation and per byte. It is general to any grouping over arriving or stored rows.StateLastWrittenBy(cost, operations per unit): whether an estimate reads a merged state or one built by updates, with the rows of the latter. It replaces thestate_from_mergeflag, which carried no rows.group_raw_batch_visits(cost, node usage): one rule shared by the three grouping operators.OPERATIONS_PER_GROUP_PER_RAW_BATCH,CHAIN_OPERATIONS_PER_ARRIVING_ROWandKLL_ESTIMATE_OPERATIONS_PER_UNSORTED_ITEM(cost, operations per unit): constants beside the others.ProducerOutput::raw_rows_span_msandNodeResourceUsageAndCost::raw_rows_span_ms(cost, node usage): the time a node's raw rows span, a property of any raw-row edge, set by the operators that read or narrow raw rows and passed on by the ones that keep each row's window (Operator::keeps_input_window), likechain_tumbling_window_width_ms.kmv_compared_hashes_per_row,exact_distinct_value_operations_per_rowandpass_through_merge_operations(cost, operations per unit): per-family formulas beside their constants.HASH_AGGREGATE_OPERATIONS_PER_KEYis replaced byHASH_AGGREGATE_OPERATIONS_PER_ROWandHASH_AGGREGATE_OPERATIONS_PER_FURTHER_KEY;CHAIN_WINDOW_OPERATIONS_PER_GROUP_PER_RAW_BATCHis deleted.keys_are_input_keys(cost, node usage): whether a merge'sgroup_bykeeps its input's groups, read from the input's schema; private beside theSummaryMergearm that uses it.Decisions and open questions
Decided
KLL coefficients: a refit, not a new size model (L4). The merge and estimate are refitted on the bytes
sketch_state_bytescharges, so a coefficient is applied to the same quantity it was fitted on. The alternative was to charge the merged KLL at what the library holds, and it was rejected for two reasons:kdepends on the merge history. At k = 547, it holds 4.8–13.5 KB for 3,000–36,000 rows across the 24 bench points, so no deterministic formula fits better than the capacity.k.The cost of the refit: k = 547's merge is over-charged about 1.8x per state in the microbenchmark (1.6x per series in the executor), and the k ratio is 1.9x where 3x is measured. The MILP still selects the smaller
k.Series identity price (L2): the minimal version. This PR shares one column price and the constant. Pricing every operator output edge at 64 B changes 71 of the 197 survey plans, so it needs its own measurement under Planner charges less memory for a held OneEvaluation output than the executor holds #1301, where the edges are named.
DDSketch default when nothing is declared (L1): the named exponential spread, with the planner's warning for undeclared magnitudes coming from feat(types, cost, planner): warn about each data fact a plan assumes because the workload does not declare it (DDSketch range, in-order arrival) #1365. This PR adds no warning of its own. The cap with the declared range is implemented here.
Raw batch width: a field on
ResourceSpeed, 10 s by default (R1). It is a deployment and calibration fact, so it sits with the speeds, not with the workload's data. Its default is stated as assumed in the field doc. It will be reported through feat(types, cost, planner): warn about each data fact a plan assumes because the workload does not declare it (DDSketch range, in-order arrival) #1365'sAssumedDataFactonce that merges; this PR adds no second warning mechanism.The chain's own terms (round 4). A chain pays the shared visit and 14 per arriving row at ingestion time. Its own per group and batch term is deleted: on the grid it fits to 15 and leaves the leave-one-setting-out error at 0.147.
The fit minimizes the squared log error (round 4). Chosen by leave-one-setting-out error, 0.147 against 0.150 for the absolute error. Round 3 chose the absolute error because the squared error flipped two planner cells; that is no reason to choose a loss.
Keys, measures, exact distinct counts and KMV are measured (round 4). Top-k still pays a further key's cost per row for hashing its value, not measured. The KMV terms charge
asap_sketchlib's current duplicate scan (KMV insert and merge are O(k) and O(k²) in asap_sketchlib; unusable at the k the planner sizes #709), so the planner picks what the executor runs fastest today; they are to be measured again when KMV insert and merge are O(k) and O(k²) in asap_sketchlib; unusable at the k the planner sizes #709 is fixed.Ties (round 4). The error band is the RMS log error of measured ÷ predicted cost over the sweep's plans (0.135). A decision whose nearest other plan is predicted within it is reported as a tie, not as a hit or a miss.
Open
AssumedDataFactwarning once that merges.raw_batch_width_ms's default as anAssumedDataFactafter feat(types, cost, planner): warn about each data fact a plan assumes because the workload does not declare it (DDSketch range, in-order arrival) #1365 merges.Verification
All runs used a release build on a 56-core host. Each harness run was pinned to its own core, 12 at a time on one socket; benches, test binaries and the plan survey ran on the other socket. Forced arms went through an uncommitted Stage 3 hook, as in the earlier rounds. "Before" is round 3's head
927aa85da.Sketch-benefit sweep (192 decisions) and the review's 48 held-out workloads
A decision is labelled a tie when the selected plan and the nearest other plan are predicted within the model's error band (RMS log error of measured ÷ predicted cost over the sweep's 293 plans: 0.135). Hits and misses are the other decisions, with regret ≤ 1.05 or above it.
{job="j3"}selectors)927aa85da. Its one decision above 1.05,max__5m__1hz__60sat 300 series with raw data kept (1.065), is a tie: its two plans were predicted 4.7% apart.Per node: RMS log error of measured ÷ charged, before → after
Fitted settings are the fitting grid above. Held-out settings are the review's 48 workloads. "Before" costs the same DAGs with
927aa85da's model through the harness's re-costing, so both columns divide the same measurements. Each cell gives the RMS log error, with the range of measured ÷ charged in parentheses.{job="j3"}: n{job="j3"}selectorsAggregateSummaryAggSummaryAggSummaryAggSummaryMerge, several states per groupSummaryMerge, one state per groupTimeRangeBy rate, the refit kinds go from 0.213 to 0.198 at 1 Hz, 0.163 to 0.149 at 2 Hz, 0.101 to 0.090 at 5 Hz and 0.082 to 0.082 at 10 Hz on the fitting grid; held out, 0.220 to 0.198, 0.190 to 0.158, 0.114 to 0.088 and 0.096 to 0.106. The
{job="j3"}selectors measure about a tenth of their charge before and after: a scan ignores label matchers (not this PR; follow-up).Showcase gate
21/21 pass: the selected plan uses at most 1.05x the exact plan's query CPU per evaluation and total CPU, and every answer is within 1% (
DF_ARMS=exact,planned, 300 s of evaluations). The closest areq4b_max__1m_1hz_10s(exact scan, 1.05 and 1.04) andq1_quantile__1m_1hz_10s(1.02).Accuracy
498 arms over the sweep and the held-out workloads, none failed. Worst DDSketch relative error 0.99998% (target 1%). Worst KLL rank error 0.90% on the sweep; on the held-out workloads 1.23%, on
qrank__5m__2hz__60s(one of 6 evaluations over 1,000 series, at δ = 0.01 per value). The review's run of927aa85dameasured the same 1.23% on the same arms: the KLL'skis sized by its target, which this PR does not change.Plan-change survey (197 workloads with the viewer workloads, plus the 12 paper workloads)
927aa85da→ this head, each workload planned once by each build. Which workloads are installable is unchanged in both sets (197: 152 installable, 26 not installable, 16 with no plan, 3 that crash on both).197 set, 12 plans change:
y03_offset30m_threen17_max_min_overy15_three_reads_rawWhy: dropping the chain's own per group and batch term (190) makes an ingestion-time chain cheaper per arriving row, for example 29 + 14 + 17 = 60 at 1 Hz where round 3 charged 30 + 8 + 33 = 71, so stored windows win where they were within a few percent.
n17is within 5% of round 3's plan.y15measures 1.13x the exact scan in both builds: the exact scan is predicted 1.43 against 0.94 measured (follow-up).p20_noqt(no query-time candidates allowed) moves from an exact scan to stored 60 s Sum windows, for the same reason.example1(SQL) answers a per-series sum with a query-time exact accumulator instead of a hash aggregate: the hash aggregate's row and its visits rose (18 → 20, 140 → 170), the accumulator's update fell (30 → 29).n04_sql_distinct_windowkeeps its exact plan; its cost doubles (0.254 → 0.524) as the exact distinct count is now charged by its distinct values, and one projection stops being kept for one evaluation.c09_topk_rate_noacc,c22_topk_count_over_gauge,random-mixed-seed-862-long-workload-02,-05,-06andrandom-mixed-seed-862-workload-03-defcapskeep every operator; only whether an estimate, projection or binary operator is kept for one evaluation changes, within 1% of the predicted cost.Paper set, 4 plans change within the same plan kind (the four range-heavy workloads): whether estimates and finalizes are kept for one evaluation changes, and the predicted cost rises 1.2%.
No plan in either set uses KMV, so the KMV merge charge changes none of them (a second survey after it: no change).
Tests (release, compared by name)
927aa85daagainst this head, every test binary of the changed crates:927aa85da→ this headcargo fmt --all -- --check: cleancargo clippy -p asap-cost -p asap-stage3-plan-selection -p asap-types -p asap-integration-tests --all-targets -- -D warnings: cleangit rebase 927aa85da --exec 'cargo test --no-run -p asap-cost -p asap-types -p asap-stage3-plan-selection -p asap-integration-tests': the 3 new commits compile their tests; the 9 below were checked in round 4asap-cost --lib)cost_inputs_json) passed → the samesummary_sharing::identical_ungrouped_queries_share_their_producersfails on the base too)joint_search_equivalence×2,materialized_views_equal_the_stored_outputs_main_listed,plan_sharing×2,promql_quantile_of_rate_binds_kll_over_rate_accumulator);crate_graph, which readsCargo.tomland the harness edits, failed only in the base runcorpus_snapshots, unchanged) + 17 passed, 56 failed (atodo!in the CLI) → the sameChanged-part tests by name, all passing on this head:
node::usage::tests::a_grouping_counts_raw_batches_from_the_time_its_input_spansnode::usage::tests::a_query_time_chain_sizes_its_windows_from_the_time_its_input_spansnode::usage::tests::a_merge_reading_one_state_per_group_passes_it_throughnode::usage::tests::a_rollup_merges_its_states_when_its_group_estimate_ties_its_inputnode::usage::tests::a_grouped_exact_distinct_count_pays_its_key_and_visits_oncenode::usage::tests::a_sort_limit_dedup_and_concat_pass_on_the_time_their_raw_rows_spannode::usage::tests::a_hash_aggregate_pays_each_further_key_and_measure_as_measurednode::usage::tests::an_exact_distinct_count_costs_more_per_row_with_more_distinct_valuesnode::usage::tests::a_kmv_build_costs_per_retained_hash_it_comparesnode::usage::tests::a_kmv_rollup_merge_costs_per_hash_it_insertsnode::usage::tests::an_ingestion_time_chain_costs_more_per_row_at_a_lower_sample_rate,a_per_series_aggregate_costs_more_per_row_at_a_lower_sample_rate,a_query_time_rate_build_costs_more_per_row_at_a_lower_sample_rate,a_grouping_visits_each_group_once_per_raw_batch_of_the_declared_width,a_query_time_chain_pays_the_time_range_over_the_rows_it_readsproduction_rollup_workloads::production_sql_distinct_region_service_counts_exactlyrollup_controlled::production_distinct_rollup_counts_exactlystage3_objective::the_objective_is_the_plan_cost_on_one_off_queries_over_yearsdefault_cost_plan_rankingtestTest-first evidence. At
af84e3d66(the tests on927aa85da's code) these fail, with the rest ofasap-costpassing:At
5dff04c1b(round 5's tests ond09ce37dc's code) the three new tests fail and the other 75 ofasap-cost --libpass: the tied rollup charged 0.95 ms as a pass-through, the grouped distinct count 83.3 ns per row against 60.2 measured, and aSortover raw rows passing no span.Follow-ups
raw_batch_width_ms's default reported as anAssumedDataFactonce feat(types, cost, planner): warn about each data fact a plan assumes because the workload does not declare it (DDSketch range, in-order arrival) #1365 merges.{job="j3"}selector is charged like the unfiltered one (measured about a tenth of its charge).y15_three_reads_raw's exact scan is predicted 1.43 against 0.94 measured, so its stored plan measures 1.13x the exact scan, before and after this round.KMV_UPDATE_OPERATIONS_PER_ROWandKMV_OPERATIONS_PER_COMPARED_HASHagain.y11_identical_pairis still a near tie in prediction (0.2738 against 0.2777) where it measures 1.19–1.25x apart. The exact scan is under-charged 0.89x: its scanned bytes are predicted 1.5x low, andTimeRangeat 1 Hz measures 1.07–1.28x its charge.state_input_rowsthrough aTimeRange, two edges from the chain that sets them. When two chains with the same θ fold different rows per state, the MILP's objective can differ fromplan_cost. This PR removes the case it found (a query-time chain's window sized by its retained windows), but the two-edge dependency stays.