Skip to content

feat(types, cost, planner): warn about each data fact a plan assumes because the workload does not declare it (DDSketch range, in-order arrival) - #1365

Merged
zzylol merged 3 commits into
mainfrom
feat/assumed-fact-warnings
Nov 16, 2026
Merged

zzylol merged 3 commits into
mainfrom
feat/assumed-fact-warnings

Conversation

@zzylol

@zzylol zzylol commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: when the data workload does not declare a fact, the planner plans with a written default and does not say so. For example, quantile_over_time(0.99, latency_ms[10m]) at relative error 0.01, with no column_magnitudes, gets a DDSketch costed for magnitudes in [1, 1e9]. A repeating rate with no max_out_of_order_ms gets an ingestion-time counter state that assumes its samples arrive in timestamp order. If they do not, answers may be wrong by an amount no accuracy target bounds, or ingestion may fail.
  • Change: a plan now carries one AssumedDataFact per undeclared fact it rests on. Each one gives the default it used, the sources it applies to, and whether the accuracy target depends on it. The fact is emitted where the default is applied: the cost model for the DDSketch range, Stage 2 for in-order arrival. The planner facade returns the facts in SelectedPlanReport::assumed_data_facts, and asap_planner prints each one as a warning.
  • Effect: plans do not change (survey of 209 workloads, 0 differences in selected plan or cost). Of the 178 planned workloads in the 197 set, 42 now warn about the DDSketch range (cost only) and 54 about in-order arrival (answers may be wrong beyond any accuracy target). Each of the 4 planned paper workloads carries both warnings.

Review guide

Behavior: before → after

The rule (decided): when a data fact is not declared, the planner uses a written default. The plan output names that assumption. When the answers depend on it, the output also says what happens if it does not hold.

Input: sum by (job) (rate(http_requests_total[1m])) every minute, with "arrival": {"ingesting": {"rows_per_second": 1000.0}} and no max_out_of_order_ms. The deployment keeps no raw data (raw_data_retained: false).

before after
selected plan ingestion-time rate state the same plan
SelectedPlanReport::assumed_data_facts (field did not exist) [{sources: ["http_requests_total"], assumption: in_order_arrival}]
asap_planner output no warning warning: assumed data fact: the data arrival declares no max_out_of_order_ms, so the samples of source http_requests_total that an ingestion-time rate, increase or irate state reads are taken to arrive in timestamp order; if this does not hold, answers may be wrong by an amount no accuracy target bounds (a late sample can be read as a counter reset), or ingestion may fail
asap_planner JSON no field stage3_plan_selection.assumed_data_facts: [{"sources": ["http_requests_total"], "assumption": "in_order_arrival"}], beside the same text in warnings

For a DDSketch over latency_ms's value, whose magnitudes are not declared, the warning reads: assumed data fact: source latency_ms declares no column_magnitudes for value, so a DDSketch state over it is costed for magnitudes in [1, 1000000000]; this affects cost only, the accuracy guarantee does not depend on it. Declaring column_magnitudes or max_out_of_order_ms removes the matching warning and leaves the plan as it was. With max_out_of_order_ms declared, the planner takes the bound as handled; the executor does not read it yet, so a late sample inside the bound still fails the build (#1367).

Why the DDSketch range affects cost only. The asap_sketchlib DdSketch at the pinned rev 21b6357 has no maximum bucket count and never collapses buckets. Its store grows in 128-bucket chunks (Buckets::ensure and add_one), and merge grows it the same way. try_add rejects only non-finite values and magnitudes beyond ddsketch_indexable_bounds(α). The executor's DDSketch updater refuses those as input errors (summary_kernels/factory/ddsketch_domain.rs), so they never land in a wrong bucket. Magnitudes below the indexable minimum (about 2e-308 at α = 0.01) would go into the zero bucket, and the executor rejects those too. So α holds for every accepted value, whatever its range. The assumed range only sets the state bytes and memory that Stage 3 costs.

Interfaces

crates/types/src/workload/assumed_data_fact.rs, re-exported from asap_types::workload

+/// One data fact the data workload does not declare, which the planner replaced by a written default.
+pub struct AssumedDataFact {
+    /// The sources the assumption is about (those that do not declare the fact), in name order.
+    pub sources: Vec<String>,
+    /// What was assumed in place of the undeclared fact.
+    pub assumption: DataFactAssumption,
+}
+
+/// The default the planner used for one undeclared fact. JSON: snake_case variant names.
+pub enum DataFactAssumption {
+    /// No `column_magnitudes` for `column` (None: computed values), so a DDSketch state is costed for `assumed_range`. Cost only.
+    DDSketchValueMagnitudeRange { column: Option<String>, assumed_range: ValueMagnitudeRange },
+    /// No `max_out_of_order_ms`, so the PromQL samples an ingestion-time rate/increase/irate state reads are taken to arrive in order.
+    /// If one arrives late, answers may be wrong by an amount no accuracy target bounds, or ingestion may fail.
+    InOrderArrival,
+}
+
+impl AssumedDataFact {
+    /// Whether the plan's answers may be wrong beyond its accuracy target when the assumption does not hold.
+    pub fn accuracy_target_depends_on_it(&self) -> bool;
+}
+/// The warning line ("assumed data fact: ..."); no sources reads "the data workload".
+impl fmt::Display for AssumedDataFact;

crates/types/src/workload/data.rs

 impl DataArrival {
     /// Rows arriving per second, when the data is ingesting or mixed.
     pub fn rows_per_second(&self) -> Option<f64>;
+    /// How late a row may arrive, when the data is ingesting or mixed and declares it.
+    pub fn max_out_of_order_ms(&self) -> Option<u64>;
 }

crates/cost/src/node/usage/mod.rs, crates/cost/src/node/state_bytes/mod.rs, crates/cost/src/plan/cost.rs

 /// One node's cost per second of wall time, with how it was measured. The type stays; this PR adds a field.
 pub struct NodeCost {
     pub operator: OperatorKind,
     pub timing: ExecutionTiming,
     pub cost_per_second: Option<f64>,
     pub cpu_core_seconds_per_second: f64,
     pub multiplier: u64,
     pub detail: String,
     pub cost_source: NodeCostSource,
+    /// The undeclared data facts its cost assumed, each once (empty when none).
+    pub assumed_data_facts: Vec<AssumedDataFact>,
 }
+/// The magnitude range `sketch_shape` assumes for a state, the one place that default is applied:
+/// Some([1, 1e9]) for a DDSketch with α in (0, 1) whose values declare none.
+pub fn assumed_magnitude_range(params: &SketchParams, input_rows: StateInputRows) -> Option<ValueMagnitudeRange>;
+impl PlanCost {
+    /// The data facts its nodes' costs assumed, each once, in node order.
+    pub fn assumed_data_facts(&self) -> Vec<AssumedDataFact>;
+}

crates/stage2-physical-optimization/src/candidate.rs, re-exported from the crate root

+/// The in-order arrival `plan` assumed: its ingestion-time counter states over PromQL samples when `max_out_of_order_ms` is undeclared.
+pub fn assumed_in_order_arrival(plan: &PhysicalASAPDAG, data: &DataWorkload) -> Option<AssumedDataFact>;

crates/planner/src/installable/mod.rs, re-exported from asap_planner

 /// The selected plan's cost, for reporting. The type stays; this PR adds a field.
 pub struct SelectedPlanReport {
     pub cost_per_second: Option<f64>,
     pub predicted_usage: PredictedUsage,
+    /// The undeclared data facts the plan assumed, each once (`assumed_data_facts`); omitted when empty.
+    #[serde(default, skip_serializing_if = "Vec::is_empty")]
+    pub assumed_data_facts: Vec<AssumedDataFact>,
 }
+/// The selected plan's assumed data facts: those of its cost, then Stage 2's in-order arrival.
+pub fn assumed_data_facts(plan: &PhysicalASAPDAG, cost: &PlanCost, data: &DataWorkload) -> Vec<AssumedDataFact>;

asap_planner: each fact adds a warning: line to the text output and a string to stage3_plan_selection.warnings in the JSON output; the JSON output also carries the facts as records in stage3_plan_selection.assumed_data_facts (omitted when there are none).

Usage (from the_report_names_a_ddsketch_s_assumed_magnitude_range, crates/planner/tests/runtime_install.rs):

// What the user writes: a repeating relative-error p99 over latency_ms. The source declares
// the sign of its values but no column_magnitudes. The deployment keeps no raw samples.
let workload: PlanningWorkload = serde_json::from_str(r#"{"query_workload": {"language": "promql",
  "repeating_queries": [{"query": "quantile_over_time(0.99, latency_ms[10m])",
    "demand": {"fixed_interval_at": {"interval": 600000, "evaluation_phase": 0}},
    "requirements": {"accuracy": {"accuracy_metric": "relative", "max_error": 0.01}},
    "predictability": {"predictable": {"known_at": null}}}]},
  "data_workload": {"arrival": {"ingesting": {"rows_per_second": 1000.0}},
    "sources": {"latency_ms": {"series_count": 1000, "sample_interval": 1000,
      "column_signs": {"value": "non_negative"}}}}}"#)?;
// The deployment input: no raw data retained, so an ingestion-time state serves the query.
let capabilities = DeploymentCapabilities { raw_data_retained: false, ..DeploymentCapabilities::default() };
// The planner call. Stage 3 selects a DDSketch plan. plan_cost costs its build through
// node_resource_usage_and_cost, which records the assumed [1, 1e9] range on that node's NodeCost.
// assumed_data_facts gathers it. Stage 2's in-order arrival adds nothing: the plan has no counter state.
let (plan, report) = plan_installable(UserInput::new(&workload, FrontendInput::Promql, models)).await?;
// What the user gets back beside the plan: one cost-only assumed fact.
assert_eq!(report.assumed_data_facts, vec![AssumedDataFact {
    sources: vec!["latency_ms".into()],
    assumption: DataFactAssumption::DDSketchValueMagnitudeRange {
        column: Some("value".into()),
        assumed_range: ValueMagnitudeRange { min_abs: 1.0, max_abs: 1e9 },
    },
}]);
// Declaring "column_magnitudes": {"value": {"min_abs": 0.1, "max_abs": 100000.0}} gives no fact.

Design scope

New item Scope Why general
workload/assumed_data_fact.rs: AssumedDataFact, DataFactAssumption data workload (asap_types::workload), next to the facts it stands in for One record for an undeclared fact that any stage replaces with a default. The cost model, Stage 2 and the facade all read it. Each new default adds a variant.
NodeCost::assumed_data_facts, PlanCost::assumed_data_facts, assumed_magnitude_range cost model (Stage 3 cost) A cost assumption is recorded where the node is costed, the only place that knows the folded column and its sources.
assumed_in_order_arrival; private IngestionRowOrder and ingestion_row_order Stage 2 physical (candidate.rs, replacing counter_state_rows_ordered) The ingestion-time offer and the report both call ingestion_row_order, so they cannot drift apart. IngestionRowOrder says how any ingestion-time build gets its rows in order: not needed or bounded, assumed, or not at all.
assumed_data_facts, SelectedPlanReport::assumed_data_facts planner facade The facade runs the stages, so it combines their facts into the plan output.

No Stage 1 change. The inventory found no default in Stage 1 sizing that affects accuracy. Every guarantee that needs an undeclared fact (stream norms, max_distinct_per_evaluation, values per evaluation) rejects or prunes the candidate instead.

Algorithm changes

None. No search, sizing, pruning or cost result changes; the survey shows identical plans and costs. The only new logic is reporting. assumed_in_order_arrival walks the selected DAG once, from all ingestion-time counter states down to their scans with one visited set: O(nodes + edges).

Inventory: where an undeclared fact or a missing measurement is replaced by a default (main 863153e)

Site Default Affects Warning
cost/src/node/state_bytes/mod.rs DDSKETCH_ASSUMED_MAGNITUDE_RANGE [1, 1e9] when column_magnitudes is undeclared cost only (the store has no maximum, see above) this PR
stage2-physical-optimization/src/candidate.rs counter_state_rows_ordered PromQL samples arrive in order when max_out_of_order_ms is undeclared answers (if a sample arrives late, answers may be wrong by an amount no accuracy target bounds, or ingestion may fail) this PR
cost/src/node/state_bytes/mod.rs StateInputRows::rows_per_state: None KLL at full capacity; DDSketch and KMV for unbounded rows cost only; a worst-case bound, not an assumed value none needed
cost/src/node/usage/mod.rs folded_values (non_negative) an undeclared sign is Unknown: a DDSketch gets two stores, and Count-Min with weights not proven non-negative is rejected cost only; conservative none needed
types/src/workload/bounds.rs group_count_bound groups = min(declared counts, series_count, rows read) cost only #1364
cost/src/node/usage/mod.rs per-series groups, instant-selector rows rows read when series_count is undeclared cost only #1364
types/src/workload/bounds.rs source_rows_per_ms the source's series share of rows_per_second, or all of it cost only #1364
cost/src/node/state_bytes/mod.rs ASSUMED_HEAP_KEY_BYTES = 63, ASSUMED_VALUE_KEY_BYTES = 17, row_bytes 16-byte labels and strings cost only #1364
cost/src/model.rs illustrative unit costs and speeds; frozen_calibration_factor = 1 illustrative values cost only #1364
measured cost table missing or partial (formulas, merge costed as one CPU op) analytical formulas cost only #843 (asap_run is not on main yet)
stream norms (Count-Min / Count Sketch) none: an absolute target finds no guarantee; a target with no metric accepts absolute_stream_l1/l2 and reports it in accepted_accuracy_per_query no default value n/a
max_distinct_per_evaluation, values per evaluation (G) none: no guarantee, or the readout is pruned (ValuesPerEvaluationUnknown) rejects n/a
Hydra and Count-Min sizing Hydra is never offered (its parameter list is empty); Count-Min ε is relative to the stream norm, with no assumed norm no default value n/a
accuracy evidence when none is declared, the theoretical guarantee is used (a proof, not a measurement); KLL's empirical constants calibrate the library, not the data no data default n/a

Decisions and open questions

  • Decided: warnings cover only the selected plan, not every candidate that was costed. An unselected candidate's assumption cannot affect what is installed. A cost assumption may have influenced the choice, but if the selected plan rests on it, the selected plan's warning names it.
  • Decided: a DDSketch whose bytes come from the measured table gets no range warning. The measured bytes reflect the range of the benchmark data, not the workload's, but that still affects cost only.
  • Decided: a conservative treatment of an unknown fact (unknown sign, unknown rows per state) is not an assumed value and gets no warning.
  • Needs reviewer decision: none.

Verification

Changed-part tests, run locally at the PR head:

Scope Result vs main Notes
cargo test -p asap-types --lib 177 passed +1 new the_warning_names_the_sources_the_default_and_the_accuracy_effect: the warning text for one source, several sources and none, for computed values, and the accuracy clause
cargo test -p asap-cost --lib 54 passed +3 new; all failed before their change a_ddsketch_over_an_undeclared_magnitude_range_names_the_assumed_range (the fact is named; none when declared); a_ddsketch_assumption_names_only_the_sources_that_declare_no_range; the_assumed_ddsketch_range_is_named_only_when_the_bytes_use_it (no range for α = 1.5, costed as an empty sketch)
cargo test -p asap-stage2-physical-optimization 15 passed unchanged covers the offer rule, now ingestion_row_order
cargo test -p asap-planner --test planner 52 passed, 1 failed, 1 ignored +2 new. The failure, summary_sharing::identical_ungrouped_queries_share_their_producers (1 vs 2 deployments), also fails on main. the_report_names_the_data_facts_the_plan_assumed (in-order arrival; with the emitter removed it fails, [] vs one fact; a declared max_out_of_order_ms gives no fact and the same plan; an empty list is omitted from the JSON and a report without it still reads); the_report_names_a_ddsketch_s_assumed_magnitude_range (a DDSketch plan; declared magnitudes give no fact and still a DDSketch)
cargo test -p asap-devtools --test devtools -- warned_about 3 failed same as main asap_planner panics on main at todo!("#1207 PR 9 ...") (planning_stages_text.rs, stage2_pruned_candidates_json), so the existing a_counter_function_over_a_declared_gauge_is_warned_about_in_text fails too. The two new tests, a_selected_ddsketch_over_undeclared_magnitudes_is_warned_about and an_ingestion_time_rate_over_undeclared_arrival_order_is_warned_about, check the text and JSON output. They also check stage3_plan_selection.assumed_data_facts. They can pass only after #1207 PR 9 lands.
cargo fmt --all; cargo clippy -p asap-types -p asap-cost -p asap-stage2-physical-optimization -p asap-planner -p asap-devtools -p asap-stage3-plan-selection --all-targets -- -D warnings clean

Survey (release builds of main 863153e and this head; run_default_search_over, then plan_cost and assumed_data_facts on the selected plan):

Set Workloads Planned Selected plan or cost differs DDSketch range warnings In-order arrival warnings (affect accuracy)
197 set 197 178 (16 no plan, 3 harness crashes; the same on both sides) 0 42 54
paper sets (RangeHeavy/RollupHeavy, ± counter, ± 8 users, ± no raw data) 12 4 (4 no plan, 4 SQL catalog errors; the same on both sides) 0 4 4

No workload carries more than one warning of each kind.

After the round-1 fixes, a 13-workload sample (DDSketch only, in-order only, both, none, o11y_all14, 3 paper workloads without raw data) gives the same selected nodes, cost_per_second, MILP variables and rows, status and kinds of fact as before the fixes and as main. The MILP objective of o11y_all14 differs only in the last bit, as it does between repeated runs of main.

Follow-ups

Round-1 review fixes (88c0e94)

Finding Fix
3a (medium) in-order wording The warning ends "if this does not hold, answers may be wrong by an amount no accuracy target bounds (a late sample can be read as a counter reset), or ingestion may fail". The InOrderArrival and accuracy_target_depends_on_it docs are executor-neutral. The bool stays.
1a one predicate ingestion_row_order → IngestionRowOrder::{Ordered, AssumedOrdered, Unordered}; the Stage 2 offer and assumed_in_order_arrival both call it. counter_state_rows_ordered and is_counter_state are deleted.
1b default in one place sketch_shape reads value_magnitude_range.or_else(|| assumed_magnitude_range(..)); assumed_magnitude_range names no range for α outside (0, 1). New test the_assumed_ddsketch_range_is_named_only_when_the_bytes_use_it.
1c folded_values returns the folded column; assumed_magnitude_range_fact takes it.
1d add_assumed_data_fact deleted; the dedup is inlined in PlanCost::assumed_data_facts.
2a The report_of doc is back on report_of.
2b "the samples of X that an ingestion-time rate, increase or irate state reads".
2c, 4b Design notes under #1364 in Follow-ups.
3b No sources reads "the data workload"; covered in the types test.
3c Filed #1367 (executor reorder buffer); linked above.
4a One visited set for the whole walk: O(nodes + edges).
5a asap_planner JSON emits stage3_plan_selection.assumed_data_facts as records; the devtools tests assert it.
5b Planner tests: a declared max_out_of_order_ms gives no fact and the same plan; declared magnitudes give no fact and still a DDSketch. Written first; they passed before the fixes (no behaviour change).
5c #[serde(default, skip_serializing_if = "Vec::is_empty")]; the planner test checks the empty list is omitted and the report reads back. It failed before the fix.
6a Body: one shared rule (now true), the in-order effect, the measured-bytes decision, the DDSketch lower bound.

Tests at 88c0e94: types lib 177 passed; cost lib 54 passed; stage2 15 passed; asap-planner --test planner 52 passed, 1 failed (summary_sharing::identical_ungrouped_queries_share_their_producers, fails on main too); devtools warned_about 3 failed at the same todo!("#1207 PR 9") as main. cargo fmt --all and cargo clippy on the six crates with -D warnings: clean.

…because the workload does not declare it

A plan now names every undeclared data fact it rests on, with the default
used and whether the accuracy target depends on it:
- a DDSketch state over a column without column_magnitudes is costed for
  the assumed [1, 1e9] (cost only: the DDSketch store grows without
  collapsing, so its alpha holds for any range);
- an ingestion-time rate, increase or irate state over PromQL samples
  without a declared max_out_of_order_ms takes them to arrive in order
  (the plan may miss its accuracy target if they do not).

The cost model records the first on NodeCost, Stage 2 reports the second
next to the rule that relies on it, and the planner facade gathers both
into SelectedPlanReport::assumed_data_facts; asap_planner prints each as
a warning.
… warnings

- The in-order arrival warning and its type doc are executor-neutral and
  say what a late sample does: answers may be wrong by an amount no
  accuracy target bounds, or ingestion may fail.
- Stage 2's offer and the in-order report read one rule,
  `ingestion_row_order`; the report walks the plan once with one visited
  set.
- The DDSketch default range is applied only through
  `assumed_magnitude_range`, which names no range for an α costed as an
  empty sketch.
- `folded_values` returns the folded column; the dedup helper is inlined
  into `PlanCost::assumed_data_facts`.
- An empty source list reads "the data workload".
- `SelectedPlanReport::assumed_data_facts` defaults when absent and is
  omitted when empty; `asap_planner` JSON also emits
  `stage3_plan_selection.assumed_data_facts` as records.
- Planner tests: a declared `max_out_of_order_ms` gives no fact and the
  same plan; declared magnitudes give no fact and still a DDSketch.
The DDSketch's column range now comes from its declared column_magnitudes,
else from assumed_magnitude_range, the one place the default range is
applied; the assumed-range fact is still emitted for each DDSketch build
that uses it. A test producer fills the new ProducerOutput fields.
@zzylol
zzylol force-pushed the feat/assumed-fact-warnings branch from 88c0e94 to 4830776 Compare November 16, 2026 14:18
@zzylol
zzylol merged commit 8e65fca into main Nov 16, 2026
1 of 3 checks passed
@zzylol

zzylol commented Nov 16, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main 4cb4110c1 (after #1343, #1344, #1350, #1355 and #1363). Head 483077670 (was 88c0e948d); backup branch backup/feat/assumed-fact-warnings-88c0e948d.

Conflicts and how they were resolved:

Validation on the merged tree: cargo fmt --all -- --check, cargo check --workspace --all-targets and cargo clippy --workspace --all-targets -- -D warnings pass. cargo test -p asap-types -p asap-cost -p asap-stage2-physical-optimization -p asap-planner -p asap-devtools compared by name with main: the 6 new tests pass, including runtime_install::the_report_names_a_ddsketch_s_assumed_magnitude_range and the_report_names_the_data_facts_the_plan_assumed. No existing test newly fails. The two new devtools tests fail at todo!("#1207 PR 9") in asap_planner, like the other devtools tests on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accuracy Accuracy requirements, guarantees, and propagation cost-model Cost formulas, statistics, ranking, and selection enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant