Repository navigation
feat(types, cost, planner): warn about each data fact a plan assumes because the workload does not declare it (DDSketch range, in-order arrival) - #1365
Conversation
…because the workload does not declare it A plan now names every undeclared data fact it rests on, with the default used and whether the accuracy target depends on it: - a DDSketch state over a column without column_magnitudes is costed for the assumed [1, 1e9] (cost only: the DDSketch store grows without collapsing, so its alpha holds for any range); - an ingestion-time rate, increase or irate state over PromQL samples without a declared max_out_of_order_ms takes them to arrive in order (the plan may miss its accuracy target if they do not). The cost model records the first on NodeCost, Stage 2 reports the second next to the rule that relies on it, and the planner facade gathers both into SelectedPlanReport::assumed_data_facts; asap_planner prints each as a warning.
… warnings - The in-order arrival warning and its type doc are executor-neutral and say what a late sample does: answers may be wrong by an amount no accuracy target bounds, or ingestion may fail. - Stage 2's offer and the in-order report read one rule, `ingestion_row_order`; the report walks the plan once with one visited set. - The DDSketch default range is applied only through `assumed_magnitude_range`, which names no range for an α costed as an empty sketch. - `folded_values` returns the folded column; the dedup helper is inlined into `PlanCost::assumed_data_facts`. - An empty source list reads "the data workload". - `SelectedPlanReport::assumed_data_facts` defaults when absent and is omitted when empty; `asap_planner` JSON also emits `stage3_plan_selection.assumed_data_facts` as records. - Planner tests: a declared `max_out_of_order_ms` gives no fact and the same plan; declared magnitudes give no fact and still a DDSketch.
The DDSketch's column range now comes from its declared column_magnitudes, else from assumed_magnitude_range, the one place the default range is applied; the assumed-range fact is still emitted for each DDSketch build that uses it. A test producer fills the new ProducerOutput fields.
88c0e94 to
4830776
Compare
|
Rebased onto Conflicts and how they were resolved:
Validation on the merged tree: |
Summary
quantile_over_time(0.99, latency_ms[10m])at relative error 0.01, with nocolumn_magnitudes, gets a DDSketch costed for magnitudes in [1, 1e9]. A repeatingratewith nomax_out_of_order_msgets an ingestion-time counter state that assumes its samples arrive in timestamp order. If they do not, answers may be wrong by an amount no accuracy target bounds, or ingestion may fail.AssumedDataFactper undeclared fact it rests on. Each one gives the default it used, the sources it applies to, and whether the accuracy target depends on it. The fact is emitted where the default is applied: the cost model for the DDSketch range, Stage 2 for in-order arrival. The planner facade returns the facts inSelectedPlanReport::assumed_data_facts, andasap_plannerprints each one as a warning.Review guide
main(863153e, includes feat(types, cost): cost DDSketch state from declared column_magnitudes #1324). Not stacked.crates/types/src/workload/assumed_data_fact.rs(the type and the warning text). Then read the two emitters:assumed_magnitude_range_factincrates/cost/src/node/usage/mod.rs;assumed_in_order_arrivalincrates/stage2-physical-optimization/src/candidate.rs.counter_state_rows_orderedis replaced byingestion_row_order, which returnsOrdered,AssumedOrderedorUnordered. The Stage 2 offer keeps everything butUnordered, and the report namesAssumedOrdered. The rule is the same.summary_buildis now generic over the child id.asap_plannercomputes the selected plan'splan_costonce and uses it for both kinds of warning.sketch_shapenow appliesDDSKETCH_ASSUMED_MAGNITUDE_RANGEonly throughassumed_magnitude_range; the new model must keep reading the range from there, or emit the sameDDSketchValueMagnitudeRangefact.Behavior: before → after
The rule (decided): when a data fact is not declared, the planner uses a written default. The plan output names that assumption. When the answers depend on it, the output also says what happens if it does not hold.
Input:
sum by (job) (rate(http_requests_total[1m]))every minute, with"arrival": {"ingesting": {"rows_per_second": 1000.0}}and nomax_out_of_order_ms. The deployment keeps no raw data (raw_data_retained: false).ratestateSelectedPlanReport::assumed_data_facts[{sources: ["http_requests_total"], assumption: in_order_arrival}]asap_planneroutputwarning: assumed data fact: the data arrival declares no max_out_of_order_ms, so the samples of source http_requests_total that an ingestion-time rate, increase or irate state reads are taken to arrive in timestamp order; if this does not hold, answers may be wrong by an amount no accuracy target bounds (a late sample can be read as a counter reset), or ingestion may failasap_plannerJSONstage3_plan_selection.assumed_data_facts:[{"sources": ["http_requests_total"], "assumption": "in_order_arrival"}], beside the same text inwarningsFor a DDSketch over
latency_ms'svalue, whose magnitudes are not declared, the warning reads:assumed data fact: source latency_ms declares no column_magnitudes for value, so a DDSketch state over it is costed for magnitudes in [1, 1000000000]; this affects cost only, the accuracy guarantee does not depend on it. Declaringcolumn_magnitudesormax_out_of_order_msremoves the matching warning and leaves the plan as it was. Withmax_out_of_order_msdeclared, the planner takes the bound as handled; the executor does not read it yet, so a late sample inside the bound still fails the build (#1367).Why the DDSketch range affects cost only. The
asap_sketchlibDdSketchat the pinned rev 21b6357 has no maximum bucket count and never collapses buckets. Its store grows in 128-bucket chunks (Buckets::ensureandadd_one), andmergegrows it the same way.try_addrejects only non-finite values and magnitudes beyondddsketch_indexable_bounds(α). The executor's DDSketch updater refuses those as input errors (summary_kernels/factory/ddsketch_domain.rs), so they never land in a wrong bucket. Magnitudes below the indexable minimum (about 2e-308 at α = 0.01) would go into the zero bucket, and the executor rejects those too. So α holds for every accepted value, whatever its range. The assumed range only sets the state bytes and memory that Stage 3 costs.Interfaces
crates/types/src/workload/assumed_data_fact.rs, re-exported fromasap_types::workloadcrates/types/src/workload/data.rsimpl DataArrival { /// Rows arriving per second, when the data is ingesting or mixed. pub fn rows_per_second(&self) -> Option<f64>; + /// How late a row may arrive, when the data is ingesting or mixed and declares it. + pub fn max_out_of_order_ms(&self) -> Option<u64>; }crates/cost/src/node/usage/mod.rs,crates/cost/src/node/state_bytes/mod.rs,crates/cost/src/plan/cost.rs/// One node's cost per second of wall time, with how it was measured. The type stays; this PR adds a field. pub struct NodeCost { pub operator: OperatorKind, pub timing: ExecutionTiming, pub cost_per_second: Option<f64>, pub cpu_core_seconds_per_second: f64, pub multiplier: u64, pub detail: String, pub cost_source: NodeCostSource, + /// The undeclared data facts its cost assumed, each once (empty when none). + pub assumed_data_facts: Vec<AssumedDataFact>, } +/// The magnitude range `sketch_shape` assumes for a state, the one place that default is applied: +/// Some([1, 1e9]) for a DDSketch with α in (0, 1) whose values declare none. +pub fn assumed_magnitude_range(params: &SketchParams, input_rows: StateInputRows) -> Option<ValueMagnitudeRange>; +impl PlanCost { + /// The data facts its nodes' costs assumed, each once, in node order. + pub fn assumed_data_facts(&self) -> Vec<AssumedDataFact>; +}crates/stage2-physical-optimization/src/candidate.rs, re-exported from the crate rootcrates/planner/src/installable/mod.rs, re-exported fromasap_planner/// The selected plan's cost, for reporting. The type stays; this PR adds a field. pub struct SelectedPlanReport { pub cost_per_second: Option<f64>, pub predicted_usage: PredictedUsage, + /// The undeclared data facts the plan assumed, each once (`assumed_data_facts`); omitted when empty. + #[serde(default, skip_serializing_if = "Vec::is_empty")] + pub assumed_data_facts: Vec<AssumedDataFact>, } +/// The selected plan's assumed data facts: those of its cost, then Stage 2's in-order arrival. +pub fn assumed_data_facts(plan: &PhysicalASAPDAG, cost: &PlanCost, data: &DataWorkload) -> Vec<AssumedDataFact>;asap_planner: each fact adds awarning:line to the text output and a string tostage3_plan_selection.warningsin the JSON output; the JSON output also carries the facts as records instage3_plan_selection.assumed_data_facts(omitted when there are none).Usage (from
the_report_names_a_ddsketch_s_assumed_magnitude_range,crates/planner/tests/runtime_install.rs):Design scope
workload/assumed_data_fact.rs:AssumedDataFact,DataFactAssumptionasap_types::workload), next to the facts it stands in forNodeCost::assumed_data_facts,PlanCost::assumed_data_facts,assumed_magnitude_rangeassumed_in_order_arrival; privateIngestionRowOrderandingestion_row_ordercandidate.rs, replacingcounter_state_rows_ordered)ingestion_row_order, so they cannot drift apart.IngestionRowOrdersays how any ingestion-time build gets its rows in order: not needed or bounded, assumed, or not at all.assumed_data_facts,SelectedPlanReport::assumed_data_factsNo Stage 1 change. The inventory found no default in Stage 1 sizing that affects accuracy. Every guarantee that needs an undeclared fact (stream norms,
max_distinct_per_evaluation, values per evaluation) rejects or prunes the candidate instead.Algorithm changes
None. No search, sizing, pruning or cost result changes; the survey shows identical plans and costs. The only new logic is reporting.
assumed_in_order_arrivalwalks the selected DAG once, from all ingestion-time counter states down to their scans with one visited set: O(nodes + edges).Inventory: where an undeclared fact or a missing measurement is replaced by a default (main 863153e)
cost/src/node/state_bytes/mod.rsDDSKETCH_ASSUMED_MAGNITUDE_RANGEcolumn_magnitudesis undeclaredstage2-physical-optimization/src/candidate.rscounter_state_rows_orderedmax_out_of_order_msis undeclaredcost/src/node/state_bytes/mod.rsStateInputRows::rows_per_state: Nonecost/src/node/usage/mod.rsfolded_values(non_negative)types/src/workload/bounds.rsgroup_count_boundseries_count, rows read)cost/src/node/usage/mod.rsper-series groups, instant-selector rowsseries_countis undeclaredtypes/src/workload/bounds.rssource_rows_per_msrows_per_second, or all of itcost/src/node/state_bytes/mod.rsASSUMED_HEAP_KEY_BYTES= 63,ASSUMED_VALUE_KEY_BYTES= 17,row_bytescost/src/model.rsillustrative unit costs and speeds;frozen_calibration_factor= 1asap_runis not on main yet)absolutetarget finds no guarantee; a target with no metric acceptsabsolute_stream_l1/l2and reports it inaccepted_accuracy_per_querymax_distinct_per_evaluation, values per evaluation (G)ValuesPerEvaluationUnknown)Decisions and open questions
Verification
Changed-part tests, run locally at the PR head:
cargo test -p asap-types --libthe_warning_names_the_sources_the_default_and_the_accuracy_effect: the warning text for one source, several sources and none, for computed values, and the accuracy clausecargo test -p asap-cost --liba_ddsketch_over_an_undeclared_magnitude_range_names_the_assumed_range(the fact is named; none when declared);a_ddsketch_assumption_names_only_the_sources_that_declare_no_range;the_assumed_ddsketch_range_is_named_only_when_the_bytes_use_it(no range for α = 1.5, costed as an empty sketch)cargo test -p asap-stage2-physical-optimizationingestion_row_ordercargo test -p asap-planner --test plannersummary_sharing::identical_ungrouped_queries_share_their_producers(1 vs 2 deployments), also fails on main.the_report_names_the_data_facts_the_plan_assumed(in-order arrival; with the emitter removed it fails,[]vs one fact; a declaredmax_out_of_order_msgives no fact and the same plan; an empty list is omitted from the JSON and a report without it still reads);the_report_names_a_ddsketch_s_assumed_magnitude_range(a DDSketch plan; declared magnitudes give no fact and still a DDSketch)cargo test -p asap-devtools --test devtools -- warned_aboutasap_plannerpanics on main attodo!("#1207 PR 9 ...")(planning_stages_text.rs,stage2_pruned_candidates_json), so the existinga_counter_function_over_a_declared_gauge_is_warned_about_in_textfails too. The two new tests,a_selected_ddsketch_over_undeclared_magnitudes_is_warned_aboutandan_ingestion_time_rate_over_undeclared_arrival_order_is_warned_about, check the text and JSON output. They also checkstage3_plan_selection.assumed_data_facts. They can pass only after #1207 PR 9 lands.cargo fmt --all;cargo clippy -p asap-types -p asap-cost -p asap-stage2-physical-optimization -p asap-planner -p asap-devtools -p asap-stage3-plan-selection --all-targets -- -D warningsSurvey (release builds of main 863153e and this head;
run_default_search_over, thenplan_costandassumed_data_factson the selected plan):No workload carries more than one warning of each kind.
After the round-1 fixes, a 13-workload sample (DDSketch only, in-order only, both, none,
o11y_all14, 3 paper workloads without raw data) gives the same selected nodes,cost_per_second, MILP variables and rows, status and kinds of fact as before the fixes and as main. The MILP objective ofo11y_all14differs only in the last bit, as it does between repeated runs of main.Follow-ups
max_out_of_order_ms(no reorder buffer), so with the bound declared a late counter sample inside it still fails the ingestion-time build. Fix in the executor; the planner's offer stays.AssumedDataFactchannel. Design notes for it:series_count,sample_interval, distinct counts, rows per source) fit as newDataFactAssumptionvariants; the exhaustive match inaccuracy_target_depends_on_itforces a decision for each.AssumedFactand a sibling type such asAssumedCostInput.AssumedDataFactlives inasap_types::workloadbeside the facts it stands in for, though it is a planning output. If non-data assumptions join it, move it to a report or plan scope then.Round-1 review fixes (88c0e94)
InOrderArrivalandaccuracy_target_depends_on_itdocs are executor-neutral. The bool stays.ingestion_row_order→IngestionRowOrder::{Ordered, AssumedOrdered, Unordered}; the Stage 2 offer andassumed_in_order_arrivalboth call it.counter_state_rows_orderedandis_counter_stateare deleted.sketch_shapereadsvalue_magnitude_range.or_else(|| assumed_magnitude_range(..));assumed_magnitude_rangenames no range for α outside (0, 1). New testthe_assumed_ddsketch_range_is_named_only_when_the_bytes_use_it.folded_valuesreturns the folded column;assumed_magnitude_range_facttakes it.add_assumed_data_factdeleted; the dedup is inlined inPlanCost::assumed_data_facts.report_ofdoc is back onreport_of.asap_plannerJSON emitsstage3_plan_selection.assumed_data_factsas records; the devtools tests assert it.max_out_of_order_msgives no fact and the same plan; declared magnitudes give no fact and still a DDSketch. Written first; they passed before the fixes (no behaviour change).#[serde(default, skip_serializing_if = "Vec::is_empty")]; the planner test checks the empty list is omitted and the report reads back. It failed before the fix.Tests at 88c0e94: types lib 177 passed; cost lib 54 passed; stage2 15 passed;
asap-planner --test planner52 passed, 1 failed (summary_sharing::identical_ungrouped_queries_share_their_producers, fails on main too); devtoolswarned_about3 failed at the sametodo!("#1207 PR 9")as main.cargo fmt --allandcargo clippyon the six crates with-D warnings: clean.