Skip to content

feat(types, cost): cost DDSketch state from declared column_magnitudes - #1324

Merged
zzylol merged 6 commits into
mainfrom
feat/source-facts-magnitudes-group-counts
Nov 16, 2026
Merged

zzylol merged 6 commits into
mainfrom
feat/source-facts-magnitudes-group-counts

Conversation

@zzylol

@zzylol zzylol commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: a DDSketch state is costed for values spanning nine decades ([1, 1e9]), whatever the data, because the data workload cannot declare a value range.
  • Change: SourceFacts gains column_magnitudes, bounds on each column's non-zero value magnitudes, read per series name. A DDSketch built from a source column passed on unchanged is costed at the bins that range spans. The same folded column now also decides the sign (one negative store or two), by the one rule the memo already uses to sign a Scan's columns (declared_sign). So a SQL column declared non-negative and a filter over a non-negative scan get one store, and a counter that declares its value unknown gets two, as the memo says.
  • Effect: quantile_over_time(0.99, latency[1h]) every 60 s over 1,000 series at 10 Hz, with value declared in [0.001, 10000]: each stored DDSketch state drops from 18,752 B to 15,072 B. The stored-window plan drops from 143.05 to 114.99 cost/s. The plan does not flip: the query-time plan still wins at 3.72/s. That also needs [Design] Price CPU and latency from a measured cost table per state family (scan, update, merge), not built-in unit costs #1314.

Part of #1313 (the declared value range, proposal 1).

Review guide

  • Base: main (3e434c760, after refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237). Head: 02e6e6199. Rebased onto main after refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R) (#1207 PR 7) #1237: see the section below.
  • Read first: declared_sign in crates/types/src/workload/bounds.rs (with the memo's Scan arm in stages/logical/group/column_properties.rs), ProducerOutput::unchanged_column and folded_values in crates/cost/src/node/usage/mod.rs, then the DDSketch arm of sketch_shape in crates/cost/src/node/state_bytes/mod.rs.
  • Mechanical: value_magnitude_range: None at the other StateInputRows constructors, the declared_non_negative(.., "value") call sites in histogram_metadata.rs, and the docs.
  • Careful review: which column a summary folds. Both facts apply only when the summary's weight is a column its producer passes on unchanged (unchanged_columns: a scan, a filter, a sort, a time range, a projection passing it on under its own name). The sign then needs every source to declare that column non-negative (declared_sign: its column_signs entry, else non-negative for a counter's value). The range needs every source to declare it under its own series name. Over several sources, the widest range is used.

Rebased onto main after #1237

Base main 3e434c760 (was 59365ce80); head 02e6e6199 (was 577c7d060, kept as backup/feat/source-facts-magnitudes-group-counts-577c7d060).

File Conflict Resolution
crates/types/src/workload/mod.rs (first and second commits) the pub use data::{…} list: main added ONE_SECOND_MS, this PR ValueMagnitude (renamed ValueMagnitudeRange in the second commit) both kept: StreamNorms, ValueMagnitudeRange, ONE_SECOND_MS

One more commit, 02e6e6199: main gave ProducerOutput a chain_tumbling_window_width_ms field, so the column-magnitude tests' ProducerOutput in crates/cost/src/node/usage/tests.rs sets it to None (its producer, a scan or a filter, reads from no chain).

Semantic check of #1237's names: this PR calls none of least_retention_ms_per_execution, ExecutionCosts::retention_ms_for_need, RefreshWindow::computed_from or the ErrorContract helpers. folded_values and unchanged_column read ProducerOutput fields main did not change.

Re-run after the rebase (release build, compared by test name with main 3e434c760):

Scope Command Result Newly failing vs main
check cargo check --workspace --all-targets clean –
fmt cargo fmt --all -- --check clean –
clippy cargo clippy -p asap-cost -p asap-types -p asap-executor -p asap-frontend-promql -p asap-stage2-physical-optimization -p asap-integration-tests -p asap-planner --all-targets -- -D warnings clean –
tests cargo test --no-fail-fast -p asap-cost -p asap-types -p asap-executor -p asap-frontend-promql -p asap-stage2-physical-optimization -p asap-stage3-plan-selection -p asap-planner -p asap-integration-tests -p asap-devtools --lib --tests 1074 passed, 65 failed, 9 ignored (main: 1065, 65, 9) none; the same 65 fail on main; the 9 new tests pass
doc tests cargo test -p asap-types -p asap-cost --doc 8 passed none

The plan comparison under Verification ran before the rebase (main 59365ce80 against 577c7d060) and was not repeated.

Behavior: before → after

Workload (data facts):

"latency": {"series_count": 1000, "metric_type": "gauge", "sample_interval": 100,
            "column_signs": {"value": "non_negative"},
            "column_magnitudes": {"value": {"min_abs": 0.001, "max_abs": 10000.0}}}

Query: quantile_over_time(0.99, latency[1h]), relative error 0.01, δ 0.01, every 60 s, ingesting 10,000 rows/s, default CostInputs.

Before (no column_magnitudes) After (declared [0.001, 10000])
raw_data_retained: true: selected plan query-time DDSketch, 3.7214/s same plan, 3.7212/s (peak memory 18.75 MB → 15.07 MB)
raw_data_retained: false: stored chain 143.05/s, average memory 1.144 GB 114.99/s, average memory 919 MB
Bytes per DDSketch state (α = 0.01) 18,752 (1,037 bins) 15,072 (807 bins)

Sign, before → after (α = 0.01, 600 rows per state):

Summary input Before After
approx_percentile_cont(latency, 0.99) over SQL table requests declaring {"latency": "non_negative"} two stores (the sign was read for value only) one store
a DDSketch over SQL table requests' latency when the table declares {"value": "non_negative"} one store (wrong: value was read for any raw-sample producer) two stores
DDSketch over a Filter on a scan of latency declaring {"value": "non_negative"} two stores (only a Scan or TimeRange producer counted) one store
a counter requests_total declaring {"value": "unknown"} one store (the counter type won over the declaration; the memo said Unknown) two stores, as the memo says
foo_count of a histogram foo that declares value in [0.001, 10] costed at the assumed [1, 1e9] (no ranges existed) costed at foo_count's own entry, else the assumed [1, 1e9]; the family's range is not used

When nothing is declared, nothing changes.

Design scope

New item Scope Why it is general
ValueMagnitudeRange (struct) and SourceFacts::column_magnitudes declared data facts, types::workload::data, next to column_signs a range of magnitudes holds for any numeric column of any source (PromQL or SQL); not tied to DDSketch
WorkloadError::InvalidValueMagnitudeRange workload input validation, types::workload::planning rejects any malformed range, naming the source and column
declared_value_magnitude_range (free function) types::workload::bounds, next to declared_non_negative and declared_series reads only DataWorkload; a union over sources, like its neighbours
declared_sign (pub) and declared_column_sign (pub(crate)) (free functions) types::workload::bounds, declared data facts the one rule for a source column's declared sign, for any source and column; the memo's Scan arm calls declared_column_sign (it holds only the sources), cost calls declared_sign through declared_non_negative. Replaces the memo's inline rule and the deleted declared_counter
StateInputRows::value_magnitude_range (field) Stage 3 cost inputs, cost::node::state_bytes, next to non_negative what sizes a summary state besides its parameters; only the DDSketch arm reads it today
ProducerOutput::unchanged_column (method) Stage 3 cost of one node, on the producer whose schema and unchanged_columns it reads one rule for "which source column does a summary over this producer fold", used by both the sign and the range

No new files. ir gains no stages import.

Interfaces

crates/types/src/workload/data.rs

 pub struct SourceFacts {
     pub series_count: Option<u64>,
     pub metric_type: Option<PrometheusMetricType>,
-    /// The declared sign of each column's values, keyed by column name
-    /// (default: none declared), e.g. `{"value": "non_negative"}` for a
-    /// latency gauge. A `Scan`'s `column_signs` reads it; an undeclared
-    /// column is `Unknown`, except a PromQL counter's value, which is
-    /// non-negative without a declaration. The deployment must ensure it.
-    /// Omitted from JSON when empty.
+    /// The declared sign of each column's values, keyed by column name
+    /// (default: none declared), e.g. `{"value": "non_negative"}` for a
+    /// latency gauge. A `Scan`'s `column_signs` reads it, and so does
+    /// DDSketch costing, which gives a state over a non-negative column one
+    /// store instead of two (both by
+    /// [`declared_sign`](crate::workload::bounds::declared_sign)); an
+    /// undeclared column is `Unknown`, except a PromQL counter's value,
+    /// which is non-negative without a declaration. The deployment must
+    /// ensure it.
+    /// Omitted from JSON when empty.
     #[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
     pub column_signs: BTreeMap<String, ValueSign>,
+    /// Bounds on the magnitude of each column's non-zero values, keyed by
+    /// column name (default: none declared), e.g. `{"value": {"min_abs":
+    /// 0.001, "max_abs": 10000.0}}` for a latency in seconds. Read per
+    /// series name: a histogram or summary family's range does not apply to
+    /// its `_sum`, `_count` or `_bucket` series. A DDSketch state built from
+    /// the column unchanged is costed at the bins those magnitudes span (the
+    /// bucket indices from ⌊log_γ min_abs⌋ to ⌊log_γ max_abs⌋, at least one)
+    /// instead of an assumed range of nine decades. The deployment must
+    /// ensure it: a range that is too narrow can make Stage 3 pick a finer
+    /// and much larger sketch than the default range would. Omitted from
+    /// JSON when empty.
+    #[serde(default, skip_serializing_if = "BTreeMap::is_empty")]
+    pub column_magnitudes: BTreeMap<String, ValueMagnitudeRange>,
     pub sample_interval: Option<DurationMs>,
     /* other fields unchanged */
 }
 
+/// Bounds on the magnitude of one column's non-zero values:
+/// `min_abs <= |v| <= max_abs` for every non-zero `v`, with
+/// `0 < min_abs <= max_abs`, both finite. E.g. `{"min_abs": 0.001,
+/// "max_abs": 10000.0}` for latencies from 1 ms to about 3 h in seconds.
+#[derive(Debug, Clone, Copy, PartialEq, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct ValueMagnitudeRange {
+    /// The smallest magnitude of a non-zero value.
+    pub min_abs: f64,
+    /// The largest magnitude of a value.
+    pub max_abs: f64,
+}

crates/types/src/workload/mod.rs

 pub use data::{
     DataArrival, DataWorkload, LabelSetDistinctValues, PrometheusMetricType, Rate, SourceFacts,
-    StreamNorms,
+    StreamNorms, ValueMagnitudeRange,
 };

crates/types/src/workload/planning.rs

 pub enum WorkloadError {
+    /// Bad input: a `column_magnitudes` entry that is not
+    /// `0 < min_abs <= max_abs` with both finite.
+    #[error(
+        "source {source_name}: column_magnitudes for column {column} must have \
+         0 < min_abs <= max_abs, both finite, got min_abs {min_abs}, max_abs {max_abs}"
+    )]
+    InvalidValueMagnitudeRange {
+        source_name: String,
+        column: String,
+        min_abs: f64,
+        max_abs: f64,
+    },
     /* other variants unchanged */
 }

crates/types/src/workload/bounds.rs

-/// Whether `data` declares every sample of `metric` non-negative: the
-/// metric is declared a counter, or its `value` column is declared
-/// non-negative ([`SourceFacts::column_signs`](crate::workload::SourceFacts::column_signs)).
-pub fn declared_non_negative(data: &DataWorkload, metric: &str) -> bool;
+/// The sign `data` declares for `source`'s `column`: its
+/// [`SourceFacts::column_signs`] entry (a histogram or summary family's
+/// applies to its series, [`DataWorkload::series_facts`]), else
+/// `NonNegative` for a counter's `value`, else `Unknown`. The memo signs a
+/// `Scan`'s columns by this rule, and DDSketch costing reads it. E.g. a
+/// counter `requests_total` gives `NonNegative` for `value`, or `Unknown`
+/// when it declares `{"value": "unknown"}`; a SQL table declaring
+/// `{"latency": "non_negative"}` gives `NonNegative` for `latency`.
+pub fn declared_sign(data: &DataWorkload, source: &str, column: &str) -> ValueSign;
+
+/// [`declared_sign`] from the facts and series type
+/// [`DataWorkload::series_facts`] finds for the source, for a caller that
+/// holds only the sources: `Unknown` when there are none.
+pub(crate) fn declared_column_sign(
+    series_facts: Option<(&SourceFacts, Option<PrometheusMetricType>)>,
+    column: &str,
+) -> ValueSign;
+
+/// Whether `data` declares every value of `source`'s `column` non-negative
+/// ([`declared_sign`]).
+pub fn declared_non_negative(data: &DataWorkload, source: &str, column: &str) -> bool;
+
-/// Whether `data` declares `metric` a counter, itself or as the `_sum` or
-/// `_count` series of a declared histogram or summary. ...
-fn declared_counter(data: &DataWorkload, metric: &str) -> bool;
+
+/// The range of magnitudes of `column`'s non-zero values over every source
+/// in `sources`, from each one's own declared `column_magnitudes`
+/// ([`SourceFacts::column_magnitudes`](crate::workload::SourceFacts::column_magnitudes)):
+/// the smallest `min_abs` and the largest `max_abs` among them. Each series
+/// is read under its own name, so a histogram family `foo`'s range does not
+/// bound `foo_sum`, `foo_count` or `foo_bucket`. `None` when `sources` is
+/// empty or some source declares none for `column`. E.g. sources declaring
+/// `[0.001, 10]` and `[0.01, 100]` give `[0.001, 100]`.
+pub fn declared_value_magnitude_range<'a>(
+    data: &DataWorkload,
+    sources: impl IntoIterator<Item = &'a str>,
+    column: &str,
+) -> Option<ValueMagnitudeRange>;

crates/cost/src/node/state_bytes/mod.rs

 #[derive(Debug, Clone, Copy, PartialEq)]
 pub struct StateInputRows {
     pub rows_per_state: Option<u64>,
-    /// Whether every value folded is proven non-negative (raw data samples of
-    /// sources declared non-negative), so a DDSketch fills no negative store.
+    /// Whether every value folded is proven non-negative (a source column
+    /// passed on unchanged, which every source declares non-negative), so a
+    /// DDSketch fills no negative store.
     pub non_negative: bool,
+    /// Bounds on `|value|` of the non-zero values folded into the state;
+    /// `None` when not declared, which costs a DDSketch over the assumed
+    /// `DDSKETCH_ASSUMED_MAGNITUDE_RANGE`. E.g. `Some(ValueMagnitudeRange {
+    /// min_abs: 0.001, max_abs: 10000.0 })` for a latency in seconds.
+    pub value_magnitude_range: Option<ValueMagnitudeRange>,
 }
 
-/// Ratio of the largest to the smallest value magnitude a DDSketch is
-/// assumed to see: nine decades, e.g. latencies from 1 ns to 1 s, priced as
-/// magnitudes in `[1, 1e9]` (its bins depend on the ratio alone). An
-/// assumption, not workload evidence: the data workload declares no value
-/// range.
-const DDSKETCH_ASSUMED_VALUE_RANGE: f64 = 1e9;
+/// The range of value magnitudes a DDSketch is assumed to see when its
+/// column declares no `column_magnitudes`: nine decades, e.g. latencies from
+/// 1 ns to 1 s, costed as magnitudes in `[1, 1e9]` (its bins depend on the
+/// ratio alone). An assumption, not workload evidence.
+const DDSKETCH_ASSUMED_MAGNITUDE_RANGE: ValueMagnitudeRange = ValueMagnitudeRange {
+    min_abs: 1.0,
+    max_abs: 1e9,
+};

sketch_shape and summary_shape keep their signatures; only the DDSketch arm reads value_magnitude_range. node_resource_usage_and_cost fills non_negative and value_magnitude_range for SummaryAgg and SummaryTumblingWindowChain from one folded source column.

crates/cost/src/node/usage/mod.rs

+impl ProducerOutput<'_> {
+    /// The source column `weight` reads, when this producer passes that
+    /// column on unchanged from the sources: the column whose values a
+    /// summary built over this producer folds. E.g. `value` for a DDSketch
+    /// over a scan of `latency` or over a filter on that scan; `None` for a
+    /// computed weight such as a rate.
+    pub(crate) fn unchanged_column(&self, weight: &SummaryInputExpr) -> Option<&str>;
+}

Usage (as the PR's tests and the run above do):

// The data workload declares latency values between 1 ms and 10,000 s.
let workload: PlanningWorkload = serde_json::from_str(QUANTILE_WORKLOAD_JSON)?;
// Validation rejects e.g. {"min_abs": 0.0, ...} with InvalidValueMagnitudeRange.
workload.validate()?;
// Stage 3 costs each candidate. For the SummaryTumblingWindowChain over Scan(latency),
// node_resource_usage_and_cost resolves the chain's weight column (`value`), sees it among
// the scan's unchanged columns (ProducerOutput::unchanged_column), and reads from that one column
// declared_non_negative(.., "latency", "value") = true and
// declared_value_magnitude_range(.., ["latency"], "value") = [0.001, 10000].
// sketch_shape then costs each state at 807 bins and one store: 15,072 B.
let (plan, report) = plan_installable(UserInput::new(&workload, FrontendInput::Promql,
    PlanningModels::builtin().with_deployment_capabilities(&no_raw_data))).await?;
// report.cost_per_second == Some(114.99); average_memory_bytes ≈ 9.19e8.

Algorithm changes

DDSketch state bytes

  • Problem: at α = 0.01, every DDSketch state with ≥ 3 rows was costed at 1,037 bins (18,752 B), even for data spanning fewer decades.
  • Before / after: DdSketch::estimated_memory_bytes(α, rows, 1.0, 1e9, negatives) becomes estimated_memory_bytes(α, rows, min_abs, max_abs, negatives) when a range is declared. The bins are the bucket indices from ⌊log_γ min_abs⌋ to ⌊log_γ max_abs⌋, at least one, for γ = (1 + α) / (1 − α): 807 for α = 0.01 over [0.001, 10000] (indices −346 to 460). This sketchlib DDSketch has no bin limit, so there is no cap.
  • Correctness: the value is still the library's upper bound over any insert/merge order, for values inside the declared range. Answers do not depend on the range: a declaration only changes the cost. A range that is too narrow can make Stage 3 pick a finer α, whose state on real data is much larger than predicted (documented on column_magnitudes).
  • Complexity: unchanged (one map lookup per source per summary node).
  • Assumptions and limits: the bytes are still the worst-case bound, not the expected bytes. [Bug] DDSketch memory is charged at the 1e9 value-range bound, about 6.4× what the executor holds #1313's proposals 2 (measured bytes) and 3 (expected vs. bound) are not in this PR. A computed column, such as a rate output, gets no declared range and no declared sign.

DDSketch sign (one store or two)

  • Before: non-negative only when the producer was a Scan or TimeRange, reading column_signs["value"] whatever column was folded.
  • After: non-negative when declared_sign gives NonNegative for the folded source column (the weight column, passed on unchanged) in every source: its column_signs entry, else non-negative for a counter's value. The memo's Scan arm signs columns by the same rule, so cost and the memo agree (a counter declaring {"value": "unknown"} is Unknown in both). A weight that is not a plain column is not treated as non-negative.

Decisions and open questions

Decided

  1. The value range is a field value_magnitude_range: Option<ValueMagnitudeRange> on StateInputRows, next to non_negative, not an extra sketch_shape / summary_shape argument.
  2. No new field for the distinct value combinations of a label set: SourceFacts::distinct_values_per_label_set already declares them, e.g. {"labels": ["instance", "job"], "distinct_values": 5000} costs by (instance, job) at 5,000 instead of 10 × 1,000.
  3. column_magnitudes is read from each series' own entry; a family's column_signs still applies to its series (a sign carries over to _sum and _count, a range does not).

Verification

Tests (each with what it catches; "new" is against main)

Test Catches
types a_series_magnitudes_come_from_its_own_entry_not_its_family (new) a histogram family's range applied to foo_count/foo_sum/foo_bucket; failed before the round-1 fix (foo_count got [0.001, 10] instead of its own [1, 1e6])
types declared_value_magnitude_ranges_are_the_union_over_the_sources (new) a wrong union over sources, or a range when one source declares none or the column differs
types column_magnitudes_must_be_positive_ordered_and_finite (new) accepting zero, negative, reversed, infinite or NaN bounds; an error not naming the source and column
types a_scan_signs_its_columns_by_the_declared_sign (new) the memo's Scan signs drifting from declared_sign; a counter declaring {"value": "unknown"} signed non-negative
types planning round trip (extended) column_magnitudes lost in JSON, or written when empty
cost ddsketch_state_bytes_follow_the_declared_value_magnitude_range (new) a wrong bin count (hand-computed 807 vs 1,037 at α = 0.01); library bytes for in-range values above the cost
cost a_ddsketch_over_a_scan_is_costed_at_its_declared_value_magnitude_range (new) the range not wired through node_resource_usage_and_cost; a range used for a column the producer does not pass on unchanged
cost a_ddsketch_over_a_sql_column_declared_non_negative_keeps_one_store (new) the sign read for value instead of the folded SQL column; fails on main
cost a_ddsketch_over_a_filter_on_a_non_negative_scan_keeps_one_store (new) the sign dropped behind a pass-through Filter, or kept for a computed column; fails on main
cost a_ddsketch_over_a_counter_declaring_its_value_unknown_keeps_both_stores (new) cost giving a counter one store when it declares value unknown (the memo says Unknown); written first and seen failing at b198c6a (18,752 B instead of 37,360 B)
frontend-promql histogram_metadata (updated call sites) _sum/_count of a declared family losing the counter sign
Command Result
cargo test -p asap-types -p asap-cost -p asap-frontend-promql --all-features --no-fail-fast 508 passed, 0 failed (types lib 159, types integration 79, cost lib 43, frontend-promql 197 + others)
cargo clippy -p asap-types -p asap-cost --all-targets --all-features -- -D warnings, cargo fmt --all -- --check clean

Plan changes against main. Each workload was planned to Stage 3 with main (59365ce) and with this PR's head (577c7d0), recording the selected plan's cost and a hash of the selected DAG (through a temporary print in select_plan, not committed; the CLI's own output still stops at todo!("#1207 PR 9")). This is a direct comparison with main. Comparing with the round-1 head gives the same answer, because none of these workloads declares column_magnitudes and none has a counter declaring its value sign, so only the sign rule could move a plan.

Set Compared Same cost and plan Different
197-workload harness set (functional, offset, stored-shift, windowed SQL, random-mixed) 197 (178 select a plan) 195 2, both ties (below)
Paper workload components (RangeHeavy, RollupHeavy, both 8-user versions; L1–L5) 294 294 0
Whole paper workloads (the same four) 4 4 0
  • The 2 differences are random-mixed-seed-862 workload-03 and -x5 workload-03. Each has the same cost on both binaries, only the DAG hash differs. Rerunning the head binary three times gives both of main's hashes for workload-03, so these are equal-cost ties, not caused by this PR.
  • No selected plan or cost changes. The paper RollupHeavy workloads declare SQL http.latency non-negative, but their approx_percentile_cont(latency, …) queries carry rank-error targets, which DDSketch does not serve, so the DDSketch candidates whose costs fell are not selected.

Review status

All four round-2 reviews are done (abstraction, layering and functional approve with nits; efficiency approves, with planning time and MILP size unchanged). Round-2 fixes, in 577c7d0:

Finding Status
A4 / layering N1 (medium/low): declared_non_negative repeated the memo's Scan sign rule and disagreed with it on a counter declaring {"value": "unknown"} fixed: one declared_sign (declared_column_sign for the memo, which holds only the sources); declared_non_negative is declared_sign(..) == NonNegative; declared_counter deleted; test written first and seen failing; no plan changes
A N1 / layering N3: DDSKETCH_ASSUMED_MAGNITUDE holds a range renamed DDSKETCH_ASSUMED_MAGNITUDE_RANGE, docs updated
A N2: test name declared_value_magnitudes_are_… renamed declared_value_magnitude_ranges_are_the_union_over_the_sources
layering N2: body compared with round 1 every table compares with main; all new tests marked; the plan check is against main
layering N4: interface diff missed the const doc and the re-export both added
layering N5: folded_source_column closure reads only producer data moved to ProducerOutput::unchanged_column; data.rs doc re-wrapped
layering N6: column_signs docs did not mention cost SourceFacts::column_signs doc and its input-output-workflow.md row say DDSketch costing reads it, by declared_sign
efficiency N1: declared_value_magnitude_range collected a Vec to reduce folds directly with try_fold
efficiency N2: declared_non_negative looked up series_facts twice gone: declared_sign looks it up once
functional N1 (low): a renamed column (latency AS l) falls back to two stores and the assumed range not changed here: conservative, and the by-name rule predates this PR; possible follow-up below

Round 1 (abstraction, layering, functional), fixed in b198c6a:

Finding Status
F1 / L1 (medium): a histogram family's range applied to its _sum, _count, _bucket series (functional repro: foo_count state predicted 9,552 B, measured up to 12,432 B) fixed: read from data.sources.get(source); test written first and seen failing; docs say "read per series name"
A1 / L3 (medium/low): two rules for the folded column (sign vs range) fixed: one folded-column rule; declared_non_negative takes the column
L2 (medium): no Design scope section added
F2 / L4: bin count stated as ceil(log_γ(max_abs / min_abs)) fixed: exact formula in code docs, SourceFacts doc and input-output-workflow.md
F3: a too-narrow range can pick a finer, much larger sketch documented on column_magnitudes and in input-output-workflow.md
L5: ValueMagnitude holds a range renamed ValueMagnitudeRange (also InvalidValueMagnitudeRange, declared_value_magnitude_range, StateInputRows::value_magnitude_range); field column_magnitudes kept
L6: "price" in title and comments "cost" in the title and the new comments
A2: assumed range built inline one assumed-range constant of type ValueMagnitudeRange
A3: try_fold over Option<Option<_>> replaced (now a direct try_fold, round 2)
L7: interface diff drifted from the code the diff above is copied from the code

Follow-ups

SourceFacts gains column_magnitudes (ValueMagnitude {min_abs, max_abs}
per column). A DDSketch state built from a column its producer passes
on unchanged, from sources that all declare it, is priced at the bins
that range spans instead of the assumed 1e9 range. Invalid magnitudes
are a WorkloadError::InvalidValueMagnitude naming source and column.
StateInputRows gains value_magnitude next to non_negative, filled for
SummaryAgg and SummaryTumblingWindowChain from the declared column
magnitudes. sketch_shape and summary_shape keep main's signatures.
…for sign and range

- Read `column_magnitudes` from the series' own entry, not its family's:
  a histogram family's range no longer bounds `foo_sum`, `foo_count` or
  `foo_bucket`.
- One rule picks the folded source column (`folded_source_column`); both
  `non_negative` and `value_magnitude_range` come from it.
  `declared_non_negative` takes the column, keeping the counter case for
  `value`, so a SQL column declared non-negative and a filter over a
  non-negative scan now cost one DDSketch store.
- Rename `ValueMagnitude` to `ValueMagnitudeRange` (and the error variant,
  the bounds function and the `StateInputRows` field); one
  `DDSKETCH_ASSUMED_MAGNITUDE` constant; simpler union fold.
- Docs: exact bin count, per-series lookup, and the too-narrow-range
  warning; "cost" instead of "price" in the new comments.
…etch cost

- Add workload::bounds::declared_sign (and declared_column_sign for the
  memo, which holds only the sources): the column_signs entry, else
  NonNegative for a counter's value, else Unknown. The memo's Scan arm
  and declared_non_negative both use it, so a counter declaring
  {"value": "unknown"} is Unknown in the memo and its DDSketch is costed
  with two stores. declared_counter is deleted.
- Move the folded-column rule to ProducerOutput::unchanged_column.
- Fold declared_value_magnitude_range without collecting a Vec.
- Rename DDSKETCH_ASSUMED_MAGNITUDE to DDSKETCH_ASSUMED_MAGNITUDE_RANGE
  and the union test to declared_value_magnitude_ranges_...
- Docs: column_signs is read by DDSketch costing too.
…gnitude tests carries no chain width

The rebased tests construct ProducerOutput, which main now gives a chain_tumbling_window_width_ms field; a source scan's producer reads from no chain, so None.
…est's StateInputRows declares no value magnitudes
@zzylol
zzylol force-pushed the feat/source-facts-magnitudes-group-counts branch from 02e6e61 to 68074d0 Compare November 16, 2026 04:08
@zzylol
zzylol merged commit a162936 into main Nov 16, 2026
2 of 3 checks passed
@zzylol

zzylol commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main 9c752dd before merge (head 68074d0, backup branch backup/feat/source-facts-magnitudes-group-counts-02e6e6199). One extra commit: #1330's new Hydra/exact state-bytes test builds a StateInputRows without value_magnitude_range; it now sets None. fmt, workspace clippy -D warnings and asap-cost/asap-types lib tests (51 + 176) pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cost-model Cost formulas, statistics, ranking, and selection enhancement New feature or request workload Batch, repeating, or multi-query workload modeling

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant