Repository navigation
feat(types, cost): cost DDSketch state from declared column_magnitudes - #1324
Merged
Merged
Conversation
This was referenced Nov 15, 2026
zzylol
force-pushed
the
feat/source-facts-magnitudes-group-counts
branch
from
November 16, 2026 01:29
577c7d0 to
02e6e61
Compare
This was referenced Nov 16, 2026
SourceFacts gains column_magnitudes (ValueMagnitude {min_abs, max_abs}
per column). A DDSketch state built from a column its producer passes
on unchanged, from sources that all declare it, is priced at the bins
that range spans instead of the assumed 1e9 range. Invalid magnitudes
are a WorkloadError::InvalidValueMagnitude naming source and column.
StateInputRows gains value_magnitude next to non_negative, filled for SummaryAgg and SummaryTumblingWindowChain from the declared column magnitudes. sketch_shape and summary_shape keep main's signatures.
…for sign and range - Read `column_magnitudes` from the series' own entry, not its family's: a histogram family's range no longer bounds `foo_sum`, `foo_count` or `foo_bucket`. - One rule picks the folded source column (`folded_source_column`); both `non_negative` and `value_magnitude_range` come from it. `declared_non_negative` takes the column, keeping the counter case for `value`, so a SQL column declared non-negative and a filter over a non-negative scan now cost one DDSketch store. - Rename `ValueMagnitude` to `ValueMagnitudeRange` (and the error variant, the bounds function and the `StateInputRows` field); one `DDSKETCH_ASSUMED_MAGNITUDE` constant; simpler union fold. - Docs: exact bin count, per-series lookup, and the too-narrow-range warning; "cost" instead of "price" in the new comments.
…etch cost
- Add workload::bounds::declared_sign (and declared_column_sign for the
memo, which holds only the sources): the column_signs entry, else
NonNegative for a counter's value, else Unknown. The memo's Scan arm
and declared_non_negative both use it, so a counter declaring
{"value": "unknown"} is Unknown in the memo and its DDSketch is costed
with two stores. declared_counter is deleted.
- Move the folded-column rule to ProducerOutput::unchanged_column.
- Fold declared_value_magnitude_range without collecting a Vec.
- Rename DDSKETCH_ASSUMED_MAGNITUDE to DDSKETCH_ASSUMED_MAGNITUDE_RANGE
and the union test to declared_value_magnitude_ranges_...
- Docs: column_signs is read by DDSketch costing too.
…gnitude tests carries no chain width The rebased tests construct ProducerOutput, which main now gives a chain_tumbling_window_width_ms field; a source scan's producer reads from no chain, so None.
…est's StateInputRows declares no value magnitudes
zzylol
force-pushed
the
feat/source-facts-magnitudes-group-counts
branch
from
November 16, 2026 04:08
02e6e61 to
68074d0
Compare
Contributor
Author
|
Rebased onto main 9c752dd before merge (head 68074d0, backup branch backup/feat/source-facts-magnitudes-group-counts-02e6e6199). One extra commit: #1330's new Hydra/exact state-bytes test builds a |
This was referenced Nov 16, 2026
Merged
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
[1, 1e9]), whatever the data, because the data workload cannot declare a value range.SourceFactsgainscolumn_magnitudes, bounds on each column's non-zero value magnitudes, read per series name. A DDSketch built from a source column passed on unchanged is costed at the bins that range spans. The same folded column now also decides the sign (one negative store or two), by the one rule the memo already uses to sign aScan's columns (declared_sign). So a SQL column declared non-negative and a filter over a non-negative scan get one store, and a counter that declares itsvalueunknowngets two, as the memo says.quantile_over_time(0.99, latency[1h])every 60 s over 1,000 series at 10 Hz, withvaluedeclared in [0.001, 10000]: each stored DDSketch state drops from 18,752 B to 15,072 B. The stored-window plan drops from 143.05 to 114.99 cost/s. The plan does not flip: the query-time plan still wins at 3.72/s. That also needs [Design] Price CPU and latency from a measured cost table per state family (scan, update, merge), not built-in unit costs #1314.Part of #1313 (the declared value range, proposal 1).
Review guide
main(3e434c760, after refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237). Head:02e6e6199. Rebased onto main after refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237: see the section below.declared_signincrates/types/src/workload/bounds.rs(with the memo'sScanarm instages/logical/group/column_properties.rs),ProducerOutput::unchanged_columnandfolded_valuesincrates/cost/src/node/usage/mod.rs, then theDDSketcharm ofsketch_shapeincrates/cost/src/node/state_bytes/mod.rs.value_magnitude_range: Noneat the otherStateInputRowsconstructors, thedeclared_non_negative(.., "value")call sites inhistogram_metadata.rs, and the docs.weightis a column its producer passes on unchanged (unchanged_columns: a scan, a filter, a sort, a time range, a projection passing it on under its own name). The sign then needs every source to declare that column non-negative (declared_sign: itscolumn_signsentry, else non-negative for a counter'svalue). The range needs every source to declare it under its own series name. Over several sources, the widest range is used.Rebased onto main after #1237
Base
main3e434c760(was59365ce80); head02e6e6199(was577c7d060, kept asbackup/feat/source-facts-magnitudes-group-counts-577c7d060).crates/types/src/workload/mod.rs(first and second commits)pub use data::{…}list: main addedONE_SECOND_MS, this PRValueMagnitude(renamedValueMagnitudeRangein the second commit)StreamNorms, ValueMagnitudeRange, ONE_SECOND_MSOne more commit,
02e6e6199: main gaveProducerOutputachain_tumbling_window_width_msfield, so the column-magnitude tests'ProducerOutputincrates/cost/src/node/usage/tests.rssets it toNone(its producer, a scan or a filter, reads from no chain).Semantic check of #1237's names: this PR calls none of
least_retention_ms_per_execution,ExecutionCosts::retention_ms_for_need,RefreshWindow::computed_fromor theErrorContracthelpers.folded_valuesandunchanged_columnreadProducerOutputfields main did not change.Re-run after the rebase (release build, compared by test name with main
3e434c760):cargo check --workspace --all-targetscargo fmt --all -- --checkcargo clippy -p asap-cost -p asap-types -p asap-executor -p asap-frontend-promql -p asap-stage2-physical-optimization -p asap-integration-tests -p asap-planner --all-targets -- -D warningscargo test --no-fail-fast -p asap-cost -p asap-types -p asap-executor -p asap-frontend-promql -p asap-stage2-physical-optimization -p asap-stage3-plan-selection -p asap-planner -p asap-integration-tests -p asap-devtools --lib --testscargo test -p asap-types -p asap-cost --docThe plan comparison under Verification ran before the rebase (main
59365ce80against577c7d060) and was not repeated.Behavior: before → after
Workload (data facts):
Query:
quantile_over_time(0.99, latency[1h]), relative error 0.01, δ 0.01, every 60 s, ingesting 10,000 rows/s, defaultCostInputs.column_magnitudes)raw_data_retained: true: selected planraw_data_retained: false: stored chainSign, before → after (α = 0.01, 600 rows per state):
approx_percentile_cont(latency, 0.99)over SQL tablerequestsdeclaring{"latency": "non_negative"}valueonly)requests'latencywhen the table declares{"value": "non_negative"}valuewas read for any raw-sample producer)Filteron a scan oflatencydeclaring{"value": "non_negative"}ScanorTimeRangeproducer counted)requests_totaldeclaring{"value": "unknown"}Unknown)foo_countof a histogramfoothat declaresvaluein [0.001, 10]foo_count's own entry, else the assumed [1, 1e9]; the family's range is not usedWhen nothing is declared, nothing changes.
Design scope
ValueMagnitudeRange(struct) andSourceFacts::column_magnitudestypes::workload::data, next tocolumn_signsWorkloadError::InvalidValueMagnitudeRangetypes::workload::planningdeclared_value_magnitude_range(free function)types::workload::bounds, next todeclared_non_negativeanddeclared_seriesDataWorkload; a union over sources, like its neighboursdeclared_sign(pub) anddeclared_column_sign(pub(crate)) (free functions)types::workload::bounds, declared data factsScanarm callsdeclared_column_sign(it holds only the sources), cost callsdeclared_signthroughdeclared_non_negative. Replaces the memo's inline rule and the deleteddeclared_counterStateInputRows::value_magnitude_range(field)cost::node::state_bytes, next tonon_negativeProducerOutput::unchanged_column(method)unchanged_columnsit readsNo new files.
irgains nostagesimport.Interfaces
crates/types/src/workload/data.rspub struct SourceFacts { pub series_count: Option<u64>, pub metric_type: Option<PrometheusMetricType>, - /// The declared sign of each column's values, keyed by column name - /// (default: none declared), e.g. `{"value": "non_negative"}` for a - /// latency gauge. A `Scan`'s `column_signs` reads it; an undeclared - /// column is `Unknown`, except a PromQL counter's value, which is - /// non-negative without a declaration. The deployment must ensure it. - /// Omitted from JSON when empty. + /// The declared sign of each column's values, keyed by column name + /// (default: none declared), e.g. `{"value": "non_negative"}` for a + /// latency gauge. A `Scan`'s `column_signs` reads it, and so does + /// DDSketch costing, which gives a state over a non-negative column one + /// store instead of two (both by + /// [`declared_sign`](crate::workload::bounds::declared_sign)); an + /// undeclared column is `Unknown`, except a PromQL counter's value, + /// which is non-negative without a declaration. The deployment must + /// ensure it. + /// Omitted from JSON when empty. #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] pub column_signs: BTreeMap<String, ValueSign>, + /// Bounds on the magnitude of each column's non-zero values, keyed by + /// column name (default: none declared), e.g. `{"value": {"min_abs": + /// 0.001, "max_abs": 10000.0}}` for a latency in seconds. Read per + /// series name: a histogram or summary family's range does not apply to + /// its `_sum`, `_count` or `_bucket` series. A DDSketch state built from + /// the column unchanged is costed at the bins those magnitudes span (the + /// bucket indices from ⌊log_γ min_abs⌋ to ⌊log_γ max_abs⌋, at least one) + /// instead of an assumed range of nine decades. The deployment must + /// ensure it: a range that is too narrow can make Stage 3 pick a finer + /// and much larger sketch than the default range would. Omitted from + /// JSON when empty. + #[serde(default, skip_serializing_if = "BTreeMap::is_empty")] + pub column_magnitudes: BTreeMap<String, ValueMagnitudeRange>, pub sample_interval: Option<DurationMs>, /* other fields unchanged */ } +/// Bounds on the magnitude of one column's non-zero values: +/// `min_abs <= |v| <= max_abs` for every non-zero `v`, with +/// `0 < min_abs <= max_abs`, both finite. E.g. `{"min_abs": 0.001, +/// "max_abs": 10000.0}` for latencies from 1 ms to about 3 h in seconds. +#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +pub struct ValueMagnitudeRange { + /// The smallest magnitude of a non-zero value. + pub min_abs: f64, + /// The largest magnitude of a value. + pub max_abs: f64, +}crates/types/src/workload/mod.rspub use data::{ DataArrival, DataWorkload, LabelSetDistinctValues, PrometheusMetricType, Rate, SourceFacts, - StreamNorms, + StreamNorms, ValueMagnitudeRange, };crates/types/src/workload/planning.rspub enum WorkloadError { + /// Bad input: a `column_magnitudes` entry that is not + /// `0 < min_abs <= max_abs` with both finite. + #[error( + "source {source_name}: column_magnitudes for column {column} must have \ + 0 < min_abs <= max_abs, both finite, got min_abs {min_abs}, max_abs {max_abs}" + )] + InvalidValueMagnitudeRange { + source_name: String, + column: String, + min_abs: f64, + max_abs: f64, + }, /* other variants unchanged */ }crates/types/src/workload/bounds.rscrates/cost/src/node/state_bytes/mod.rs#[derive(Debug, Clone, Copy, PartialEq)] pub struct StateInputRows { pub rows_per_state: Option<u64>, - /// Whether every value folded is proven non-negative (raw data samples of - /// sources declared non-negative), so a DDSketch fills no negative store. + /// Whether every value folded is proven non-negative (a source column + /// passed on unchanged, which every source declares non-negative), so a + /// DDSketch fills no negative store. pub non_negative: bool, + /// Bounds on `|value|` of the non-zero values folded into the state; + /// `None` when not declared, which costs a DDSketch over the assumed + /// `DDSKETCH_ASSUMED_MAGNITUDE_RANGE`. E.g. `Some(ValueMagnitudeRange { + /// min_abs: 0.001, max_abs: 10000.0 })` for a latency in seconds. + pub value_magnitude_range: Option<ValueMagnitudeRange>, } -/// Ratio of the largest to the smallest value magnitude a DDSketch is -/// assumed to see: nine decades, e.g. latencies from 1 ns to 1 s, priced as -/// magnitudes in `[1, 1e9]` (its bins depend on the ratio alone). An -/// assumption, not workload evidence: the data workload declares no value -/// range. -const DDSKETCH_ASSUMED_VALUE_RANGE: f64 = 1e9; +/// The range of value magnitudes a DDSketch is assumed to see when its +/// column declares no `column_magnitudes`: nine decades, e.g. latencies from +/// 1 ns to 1 s, costed as magnitudes in `[1, 1e9]` (its bins depend on the +/// ratio alone). An assumption, not workload evidence. +const DDSKETCH_ASSUMED_MAGNITUDE_RANGE: ValueMagnitudeRange = ValueMagnitudeRange { + min_abs: 1.0, + max_abs: 1e9, +};sketch_shapeandsummary_shapekeep their signatures; only the DDSketch arm readsvalue_magnitude_range.node_resource_usage_and_costfillsnon_negativeandvalue_magnitude_rangeforSummaryAggandSummaryTumblingWindowChainfrom one folded source column.crates/cost/src/node/usage/mod.rsUsage (as the PR's tests and the run above do):
Algorithm changes
DDSketch state bytes
DdSketch::estimated_memory_bytes(α, rows, 1.0, 1e9, negatives)becomesestimated_memory_bytes(α, rows, min_abs, max_abs, negatives)when a range is declared. The bins are the bucket indices from ⌊log_γ min_abs⌋ to ⌊log_γ max_abs⌋, at least one, for γ = (1 + α) / (1 − α): 807 for α = 0.01 over [0.001, 10000] (indices −346 to 460). This sketchlib DDSketch has no bin limit, so there is no cap.column_magnitudes).rateoutput, gets no declared range and no declared sign.DDSketch sign (one store or two)
ScanorTimeRange, readingcolumn_signs["value"]whatever column was folded.declared_signgivesNonNegativefor the folded source column (theweightcolumn, passed on unchanged) in every source: itscolumn_signsentry, else non-negative for a counter'svalue. The memo'sScanarm signs columns by the same rule, so cost and the memo agree (a counter declaring{"value": "unknown"}isUnknownin both). A weight that is not a plain column is not treated as non-negative.Decisions and open questions
Decided
value_magnitude_range: Option<ValueMagnitudeRange>onStateInputRows, next tonon_negative, not an extrasketch_shape/summary_shapeargument.SourceFacts::distinct_values_per_label_setalready declares them, e.g.{"labels": ["instance", "job"], "distinct_values": 5000}costsby (instance, job)at 5,000 instead of 10 × 1,000.column_magnitudesis read from each series' own entry; a family'scolumn_signsstill applies to its series (a sign carries over to_sumand_count, a range does not).Verification
Tests (each with what it catches; "new" is against
main)typesa_series_magnitudes_come_from_its_own_entry_not_its_family(new)foo_count/foo_sum/foo_bucket; failed before the round-1 fix (foo_countgot [0.001, 10] instead of its own [1, 1e6])typesdeclared_value_magnitude_ranges_are_the_union_over_the_sources(new)typescolumn_magnitudes_must_be_positive_ordered_and_finite(new)typesa_scan_signs_its_columns_by_the_declared_sign(new)Scansigns drifting fromdeclared_sign; a counter declaring{"value": "unknown"}signed non-negativetypesplanninground trip (extended)column_magnitudeslost in JSON, or written when emptycostddsketch_state_bytes_follow_the_declared_value_magnitude_range(new)costa_ddsketch_over_a_scan_is_costed_at_its_declared_value_magnitude_range(new)node_resource_usage_and_cost; a range used for a column the producer does not pass on unchangedcosta_ddsketch_over_a_sql_column_declared_non_negative_keeps_one_store(new)valueinstead of the folded SQL column; fails onmaincosta_ddsketch_over_a_filter_on_a_non_negative_scan_keeps_one_store(new)Filter, or kept for a computed column; fails onmaincosta_ddsketch_over_a_counter_declaring_its_value_unknown_keeps_both_stores(new)valueunknown(the memo saysUnknown); written first and seen failing at b198c6a (18,752 B instead of 37,360 B)frontend-promqlhistogram_metadata(updated call sites)_sum/_countof a declared family losing the counter signcargo test -p asap-types -p asap-cost -p asap-frontend-promql --all-features --no-fail-fastcargo clippy -p asap-types -p asap-cost --all-targets --all-features -- -D warnings,cargo fmt --all -- --checkPlan changes against
main. Each workload was planned to Stage 3 withmain(59365ce) and with this PR's head (577c7d0), recording the selected plan's cost and a hash of the selected DAG (through a temporary print inselect_plan, not committed; the CLI's own output still stops attodo!("#1207 PR 9")). This is a direct comparison withmain. Comparing with the round-1 head gives the same answer, because none of these workloads declarescolumn_magnitudesand none has a counter declaring itsvaluesign, so only the sign rule could move a plan.random-mixed-seed-862workload-03 and-x5workload-03. Each has the same cost on both binaries, only the DAG hash differs. Rerunning the head binary three times gives both ofmain's hashes for workload-03, so these are equal-cost ties, not caused by this PR.http.latencynon-negative, but theirapprox_percentile_cont(latency, …)queries carry rank-error targets, which DDSketch does not serve, so the DDSketch candidates whose costs fell are not selected.Review status
All four round-2 reviews are done (abstraction, layering and functional approve with nits; efficiency approves, with planning time and MILP size unchanged). Round-2 fixes, in 577c7d0:
declared_non_negativerepeated the memo'sScansign rule and disagreed with it on a counter declaring{"value": "unknown"}declared_sign(declared_column_signfor the memo, which holds only the sources);declared_non_negativeisdeclared_sign(..) == NonNegative;declared_counterdeleted; test written first and seen failing; no plan changesDDSKETCH_ASSUMED_MAGNITUDEholds a rangeDDSKETCH_ASSUMED_MAGNITUDE_RANGE, docs updateddeclared_value_magnitudes_are_…declared_value_magnitude_ranges_are_the_union_over_the_sourcesmain; all new tests marked; the plan check is againstmainfolded_source_columnclosure reads only producer dataProducerOutput::unchanged_column;data.rsdoc re-wrappedcolumn_signsdocs did not mention costSourceFacts::column_signsdoc and itsinput-output-workflow.mdrow say DDSketch costing reads it, bydeclared_signdeclared_value_magnitude_rangecollected aVecto reducetry_folddeclared_non_negativelooked upseries_factstwicedeclared_signlooks it up oncelatency AS l) falls back to two stores and the assumed rangeRound 1 (abstraction, layering, functional), fixed in b198c6a:
_sum,_count,_bucketseries (functional repro:foo_countstate predicted 9,552 B, measured up to 12,432 B)data.sources.get(source); test written first and seen failing; docs say "read per series name"declared_non_negativetakes the columnceil(log_γ(max_abs / min_abs))SourceFactsdoc andinput-output-workflow.mdcolumn_magnitudesand ininput-output-workflow.mdValueMagnitudeholds a rangeValueMagnitudeRange(alsoInvalidValueMagnitudeRange,declared_value_magnitude_range,StateInputRows::value_magnitude_range); fieldcolumn_magnitudeskeptValueMagnitudeRangetry_foldoverOption<Option<_>>try_fold, round 2)Follow-ups
SELECT latency AS l …), so its declared sign and range still apply. Todayunchanged_source_columnspasses a column on only under its own name, so such a DDSketch is costed with two stores over the assumed range (conservative: 112,080 B predicted vs 56,256 B for one store).Scansign arm again;declared_signis the one place to change.