Skip to content

fix(accuracy): relative error through sum and avg of values of one sign - #1302

Merged
zzylol merged 2 commits into
mainfrom
fix/sign-conditioned-relative-sum-avg
Nov 14, 2026
Merged

zzylol merged 2 commits into
mainfrom
fix/sign-conditioned-relative-sum-avg

Conversation

@zzylol

@zzylol zzylol commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Problem: avg had only an absolute error contract and sum had none, so a relative-error sketch readout could not feed either. avg by (job) (quantile_over_time(0.99, data[300s])) with max_error 0.05 over non-negative data and no raw data retained failed with "selection MILP infeasible".
  • Change: sum and avg (and the exact sum accumulator) now have a relative contract with factor 1 when the values they aggregate are of one sign. The sign comes from the existing GroupProperties::column_signs, which is derived through the operators below.
  • Effect: that query plans again with a DDSketch chain (cost 207.17/s). With raw data retained, it moves from an exact plan at 26.42/s to a DDSketch plan at 23.41/s.

Review guide

Behavior: before → after

The user writes one PromQL workload entry, with data facts declaring data's value non-negative:

{"query": "avg by (job) (quantile_over_time(0.99, data[300s]))",
 "demand": {"fixed_interval_at": {"interval": 1000, "evaluation_phase": 0}},
 "requirements": {"accuracy": {"max_error": 0.05, "failure_probability": 0.3, "accuracy_scope": "whole_result"}}}
"sources": {"data": {"series_count": 1000, "metric_type": "gauge", "sample_interval": 100,
                     "column_signs": {"value": "non_negative"}}}
Deployment Before After
raw data not retained NoValidCandidate: the selection MILP is infeasible DDSketch chain, 7 nodes, 207.17/s
default (raw retained) exact query-time quantile, 4 nodes, 26.42/s DDSketch, 6 nodes, 23.41/s

Why: the 0.99-quantile of non-negative values is non-negative, so avg averages values of one sign. A DDSketch readout's relative error ε then gives a relative error of at most ε on the mean. Before this PR, the metric rule cut every DDSketch (relative) and KLL (rank) readout edge into avg, and only exact candidates were left. This is a correctness/MVP need: before the #1207 stack (3608c22), the old exact_average allowed this case through relative_fold(..., needs_sign = true). The regression came with #1207 PR 4 (70b0abc).

Interfaces

None. The only addition is a private helper in crates/accuracy/src/error_contract/mod.rs:

+/// Whether the values of column `column` (resolved by the caller) of
+/// `group`'s output all have one sign (`column_signs` `NonNegative` or
+/// `NonPositive`, derived through the operators below `group`).
+fn values_of_one_sign(memo: &Memo, group: MemoGroupId, column: usize) -> bool;

Usage (end to end, as in the example above): the PromQL frontend lowers the query to Aggregate{Avg} ← Aggregate{Quantile 0.99, per series} ← Range 300s ← Scan data. Memo::new derives column_signs: the Scan's value is NonNegative from the source facts, and measure_sign(Quantile{q: 0.99}, NonNegative) = NonNegative. The metric rule (ErrorMetricsOfMemo) asks error_contract(memo, avg, Relative). That call reaches the new arm, and the arm resolves the sample value column of the quantile group, and values_of_one_sign(memo, quantile_group, column) returns true, so the contract is [1.0]. The DDSketch readout's edge into avg passes, Stage 1 Pass 2 builds the chain, and Stage 3 selects the DDSketch plan in the table above. The user gets a plan whose accepted accuracy is "relative (DDSketch)".

Algorithm changes

Error contract of sum / avg / exact sum accumulator

  • Problem: see the example above. The plan is infeasible without raw data, and it is exact when a sketch would be cheaper.
  • Before:
    1. Aggregate: Count → β 0; Min/Max → β 1 in absolute or relative error; Avg → β 1 in absolute error; anything else, including Sum → no contract.
    2. Exact accumulator: Count → β 0; Min/Max → β 1; Sum → none.
  • After: the same, plus:
    1. Aggregate Sum or Avg in relative error → β 1 when values_of_one_sign(memo, child, col).
    2. SummaryAgg / SummaryTumblingWindowChain with ExactAggregate(Sum) in relative error → β 1 when the update's weight is a column and that column is of one sign. This arm is what the rollup through sum and count accumulators uses.
  • Correctness: if every x_i has the same sign and |x̂_i − x_i| ≤ ε|x_i|, then |Σ(x̂_i − x_i)| ≤ ε Σ|x_i| = ε|Σx_i|. Dividing by an exact count keeps this, so the same bound holds for the mean. The sign is that of the column the aggregate reads in its child group (column_signs). It is derived by the existing propagation (measure_sign, quantile_sign, expression_sign, state_columns, readout_columns): a quantile with 0 ≤ q ≤ 1, a min, max, avg or sum of values of one sign keeps that sign; abs and counts are non-negative; a negation flips the sign; a difference, or an input of unknown sign, gives Unknown, which means no relative contract. Absolute and rank metrics are unchanged: a sum's absolute errors add up, so a sum still has no absolute contract.
  • Complexity: one column_signs lookup per Sum/Avg measure per metric check, O(1).
  • Assumptions and limits: a declared sign is trusted, because the deployment must ensure it (SourceFacts::column_signs). An exact sum accumulator whose weight is a constant, a tuple or an entity identity gets no relative contract.

Decisions and open questions

Needs reviewer decision

  1. Two sketch-plan workloads (random-mixed-seed-862-w02-q2-q7-mvp, random-mixed-seed-862-w01-q3-q8-mvp) go from "selection MILP infeasible" to an installation error: QueryRunNotExecutable: aggregate output names differ from measures. Planned alone, their avg(quantile_over_time) query now plans (DDSketch, 6.41e-6/s). The error comes from their topk by (region) (4, increase|rate(data_total[…])) query, which fails the same way on base when planned alone. It is a pre-existing bug, hidden until now by the infeasible avg query. Should I file a follow-up issue?
  2. The planner snapshot suites (corpus_snapshots, sketch_plan_snapshots) cannot run on main: asap_planner reaches todo!("#1207 PR 9 (#1205) …"). No snapshot is regenerated here; the plan changes were measured with a library harness instead (Verification).

Decided:

  • Review follow-ups applied in a second commit after the rebase onto 09a1a2d: the helper takes a resolved column in both callers (no None = "sample value"); readout_caps::query_levels's doc names sum; the test imports ColumnRef. After it: asap-accuracy, stage1 pass1/pass2 and asap-planner --lib pass (31/37/28/12), fmt and clippy -D warnings clean.

Verification

Scope Result Newly failing vs base Notes
asap-accuracy --lib 31 passed none 2 new tests, both failing on base
asap-stage1-pass1-per-computation-candidates 37 passed none
asap-stage1-pass2-reuse-candidates 28 passed none
asap-stage2-physical-optimization 15 passed none
asap-stage3-plan-selection 5 + 2 passed none
asap-planner --lib 12 passed none view_refreshes_compile_only_the_nodes_below_them was failing on base, now passes
asap-planner tests/main.rs 19 passed, 2 failed, 1 ignored none the 2 failures (summary_sharing::identical_ungrouped_queries_share_their_producers, runtime_install::check_compiles_reports_the_executors_error) fail identically on base
asap-integration-tests 163 passed, 7 failed, 4 ignored none the 7 failures fail with identical messages on base
asap-devtools --test corpus_snapshots --include-ignored (release) 2 failed none todo! in asap_planner on base too
cargo fmt --all --check, cargo clippy -p asap-accuracy --all-targets -D warnings clean

New tests:

  • error_contract::tests::sums_and_averages_carry_relative_error_over_values_of_one_sign: sum, avg and an exact sum accumulator by job over a DDSketch p99 readout have relative factor 1 when latency is non-negative or non-positive, and none when its sign is unknown (values that may be of mixed sign). sum and the exact sum accumulator have no absolute contract, avg keeps its absolute contract, and none of them has a rank contract.
  • metric_rule::tests::avg_of_readouts_of_values_of_one_sign_keeps_the_ddsketch_edge: avg(quantile_over_time(0.99, m[5m])) at a relative target keeps the DDSketch readout edge for non-negative and non-positive m, and still cuts the KLL edge (rank). The existing avg_of_readouts_under_a_relative_target_reads_exact_only remains the unknown-sign case.

Plan changes: an unstaged harness called plan_installable on 41 workloads (o11y_bench corpus, sketch-plan manifest with its deployments and cost inputs, dag-viewer PromQL memo examples, plan-sharing fixtures, fuzz seeds, and 4 variants of the example above), on base and on this branch. 34 are unchanged. The changed ones are below; the "Committed snapshot" column is the cost in the committed snapshot from before the regression:

Workload Base This PR Committed snapshot
sketch quantile-grouped-mvp (avg by (region) (quantile_over_time(0.9, data[5m]))) infeasible DDSketch, 1.96628e-6/s DDSketch, 1.96600e-6/s
sketch random-mixed-seed-862-long-w01-q3-mvp (avg by (region) (quantile_over_time(0.95, data[300s]))) infeasible DDSketch, 1.90766e-5/s DDSketch, 1.90435e-5/s
sketch random-mixed-seed-862-w02-q2-q7-mvp infeasible installation error from the topk query (decision 1) DDSketch + exact, 6.44592e-6/s
sketch random-mixed-seed-862-w01-q3-q8-mvp infeasible same as above
example above, raw data not retained infeasible DDSketch, 207.168/s
example above, raw retained exact, 26.420/s DDSketch, 23.406/s
example above with sum by (job), raw retained exact, 26.420/s DDSketch + exact sum accumulators, 23.406/s
example above with sign unknown exact, 26.420/s exact, 26.420/s (unchanged)

Each change is a quantile of a source declared non-negative, aggregated by sum or avg. The sketch plans match the pre-regression snapshots up to small cost drift from later cost-model changes. Every changed plan is cheaper or newly feasible.

Follow-ups

@zzylol zzylol added bug Something isn't working error-estimation accuracy Accuracy requirements, guarantees, and propagation optimizer Workload optimization and plan selection labels Nov 14, 2026
When the values a sum or avg aggregates are all of one sign (column_signs
NonNegative or NonPositive, derived through the operators below), a
relative error eps on each input bounds the result's relative error by
eps. Add that contract (factor 1) to the Aggregate arm for Sum and Avg
and to the exact sum accumulator (SummaryAgg / chain). Unknown sign
keeps no relative contract.

Restores the DDSketch plans of avg-of-quantile workloads that became
infeasible without raw data (view_refreshes_compile_only_the_nodes_below_them).

Part of #1207.
…h callers

Each caller resolves its column (the Aggregate arm its measure's column or
the sample value; the exact-sum arm its update weight), so the helper no
longer gives None the meaning "sample value". Name sum among the
reductions in readout_caps::query_levels's doc; import ColumnRef in the
test.
@zzylol
zzylol force-pushed the fix/sign-conditioned-relative-sum-avg branch from deb1d63 to 7fc7650 Compare November 14, 2026 20:09
@zzylol
zzylol merged commit 4a37273 into main Nov 14, 2026
3 of 5 checks passed
zzylol added a commit that referenced this pull request Nov 15, 2026
… output; targets in context copies (#1207 PR 7)

avg by (job)(quantile_over_time(0.99, data[300s])) at relative 0.05
over non-negative data had no plan without raw data: Pass 2 listed every
DDSketch group as exact-only for it, and Stage 3 now enforces the list
(X). The avg's exact-accumulator path reads count(quantile): a Project
above the count passes no metric, so the count's group got none, and the
walk and its closure marked the quantile and every sketch below it
exact-only. A count's value does not depend on its input's error (error
factor 0 in every metric), so a consumer now passes every metric in
which an input's error does not reach its output, and the exact-only
closure does not cross such an input. The DDSketch plan is valid: #1302
made a relative bound on avg hold for values of one sign, and data is
declared non-negative.

A store always keeps a stored result's newest output, and plan_cost
charges at least one retained output, but Stage 3 let a stored result
read only at its newest output retain 0 (an ingestion-time chain read by
a stored ingestion-time TimeShift: 0.007/s apart). Each candidate with a
retention cost now retains at least its output interval
(ExecutionCosts::newest_output_retention_ms): the linear retention cost
is taken at no less than one output, (R)'s least need, the bound and the
reported retention are floored at it.

Context copies also compare each query's accuracy target (metric, ε, δ,
scope), so the accuracy constraint (K) holds alike in both copies: the
same queries at rank 0.01 every 10 s and rank 0.05 every 20 s are not
copies.
zzylol added a commit that referenced this pull request Nov 15, 2026
… output; targets in context copies (#1207 PR 7)

avg by (job)(quantile_over_time(0.99, data[300s])) at relative 0.05
over non-negative data had no plan without raw data: Pass 2 listed every
DDSketch group as exact-only for it, and Stage 3 now enforces the list
(X). The avg's exact-accumulator path reads count(quantile): a Project
above the count passes no metric, so the count's group got none, and the
walk and its closure marked the quantile and every sketch below it
exact-only. A count's value does not depend on its input's error (error
factor 0 in every metric), so a consumer now passes every metric in
which an input's error does not reach its output, and the exact-only
closure does not cross such an input. The DDSketch plan is valid: #1302
made a relative bound on avg hold for values of one sign, and data is
declared non-negative.

A store always keeps a stored result's newest output, and plan_cost
charges at least one retained output, but Stage 3 let a stored result
read only at its newest output retain 0 (an ingestion-time chain read by
a stored ingestion-time TimeShift: 0.007/s apart). Each candidate with a
retention cost now retains at least its output interval
(ExecutionCosts::newest_output_retention_ms): the linear retention cost
is taken at no less than one output, (R)'s least need, the bound and the
reported retention are floored at it.

Context copies also compare each query's accuracy target (metric, ε, δ,
scope), so the accuracy constraint (K) holds alike in both copies: the
same queries at rank 0.01 every 10 s and rank 0.05 every 20 s are not
copies.
zzylol added a commit that referenced this pull request Nov 16, 2026
…}, h_v and constraint (R) (#1207 PR 7) (#1237)

* refactor(stage3): select over execution contexts: x_{κ,v}, y_{κ,u,c,i}, h_v and constraint (R); asap-cost computes once per context (#1207 PR 7)

* feat(stage3): a query reads the groups Pass 2 lists as exact-only for it through exact candidates only: per-query reads a_{q,e} and constraint (X) (#1207 PR 7)

* fix(stage3): one run rate shared with plan_cost, request contexts per anchor, retention and scale over kept executions, per-query bound (#1207 PR 7)

- asap-cost: context_executions_per_second is the one run-rate function
  Stage 3 and plan_cost both call; plan_cost walks through OneEvaluation
  nodes, so an `@ end()` / `@ start()` subtree runs once per request in
  both. `@ start()` requests count once per distinct start time.
- Inputs of `@ end()` and `@ start()` shifts run in contexts of their own,
  apart from each other and from plain evaluations.
- h_v and its (R) rows exist only for a candidate with a kept execution;
  the objective scale is over kept coefficients only.
- The greedy bound leaves out of each query's trees only what that query
  must not read.
- A query-time Scan that is not materialized has its read cost on its
  edges; a materialized one keeps a per-execution read.
- SelectableExecutions (renamed) holds executions, edges by execution
  position, edges per input and the answering executions; milp/retention.rs
  holds h_v and (R); QueryId moves to selectable.rs.
- pi_v is stored per second retained; each edge's need is computed once
  and stored on ExecutionCosts; the request overhead is added in
  select_plan; forbidden answering executions are filtered before the
  MILP instead of (X) rows z <= 0.
- PhysicalProperties::is_across_evaluations, PhysicalCandidate::own_schedule,
  AtModifier derives Hash and Ord, Consumer::QueryRequest carries the anchor.

* docs(stage3): list exact_only.rs and milp/retention.rs, and (X), in the Stage 3 rows (#1207 PR 7)

* fix(stage3): the greedy bound counts the least retention an execution pays, pi_v times its least need over its uses (#1207 PR 7)

* perf(stage3): leave out a query-time execution with lifetime None that its OneEvaluation twin dominates (#1207 PR 7)

* perf(stage3): leave out (B) and (D) rows the one-reader equality (F) implies, and scale raw data on disk in its rows, per execution (#1207 PR 7)

* fix(stage3): an anchored input runs once per request of the queries that reach its shift (#1207 PR 7)

* fix(stage3): an ingestion-time view's pi_v is one more output's cost over the window it maintains (#1207 PR 7)

* fix(stage3,cost): an instant query's @ end() input runs in its evaluation context (#1207 PR 7)

An instant query's request ends at its evaluation, so the input of an
`@ end()` shift read by an instant query is computed in the evaluation
context, shared with the plain reads on its grid, in Stage 3's contexts
and in `plan_cost` alike (`anchor_is_evaluation_time`). Range queries
and `@ start()` keep their request contexts.

* fix(cost): an @ start() input runs once per request of each distinct schedule and span (#1207 PR 7)

* refactor(stage3,cost): one walk of retained reads, one rule for a context's queries, dominance in its own module (#1207 PR 7)

- `crate::retention::reads_back`: every read of an execution's rows (edge
  needs and query-output spans), folded by costing, (R),
  `least_retention_ms` and the greedy bound; `ms_to_seconds` and
  `ExecutionCosts::retention_cost_per_second` for the ms-to-seconds step.
- `asap_cost::plan::execution_demand::queries_of_context`: Stage 3 and
  `computations` pick a context's queries with it.
- `milp/dominance.rs` holds `dominated_by_one_evaluation_twin`, over two
  lists per (context, logical candidate) and the selectable edges.
- `SelectableExecutions`: `SelectableEdge`, `edges_out_of` next to
  `edges_into` (used by the components, (X) and dominance), private
  fields with getters, a map of context ids.
- `raw_data_disk_rows` gives the raw-data rows to the scale and the
  constraints; `has_one_reader`; the scaled `raw_data_on_disk` doc.
- `delivers_on` compares with `own_schedule` (no `% 0`).
- asap-cost: the `Scan` arm and the retained raw rows share `raw_rows`.

* perf(stage3): h_v is at least the least need over an execution's uses times x (#1207 PR 7)

* fix(stage3,cost): rate a context by the queries whose walk reaches it; @ start() per span (#1207 PR 7)

- `asap_cost::plan::execution_demand::query_reads`: how each query reads
  the nodes below it, per (query, anchor), over a plan or a memo.
  `consumers` (plan_cost) walks plan nodes with it, Stage 3 memo groups
  (`SelectableExecutions::readers_of`). A run in an evaluation context is
  rated by the queries reading its group in their evaluation context, not
  by every query reaching the group: an instant `m[5m] @ end()` beside a
  range `m[5m] @ end() offset 10m` cost 0.3642/s in the MILP for a plan
  `plan_cost` charged 0.1542/s. Stage 3 walks every candidate of a group,
  `plan_cost` the selected plan; the difference is documented.
- `@ start()`: `evaluation_rate` of the request schedules per span,
  summed over spans; `anchored_request_recurrence` deleted.
- `ExecutionCosts::least_need_ms_per_execution`, computed once, read by
  the greedy bound and the h_v >= need * x rows.
- The bound uses `has_one_reader`; the query-time `Scan` reads
  `raw_rows_back`; `Candidates` holds its edge; `SelectedEdge`;
  `is_across_evaluations` wrapper inlined; module docs and
  planner-layering.md list dominance.rs.

* test(types): an ingestion-time view without a window delivers on every context (#1207 PR 7)

* refactor(types,cost,stage3): place shared rules where their facts live; one raw-row-store reader rule (#1207 PR 7)

- IR: `Operator::chain_tumbling_window_width_ms` and
  `Operator::request_anchor`; Stage 3, asap-cost and Pass 2 call them.
- Logical: `EvaluationGrid`, `LogicalASAPMemoCandidates::own_schedule`
  (an `EvaluationGrid`) and `query_must_not_read`; `Memo::output_groups`.
  `PhysicalCandidate` keeps `delivers_on` only.
- Physical: `PhysicalProperties::has_one_reader_per_context`,
  `PhysicalASAPMemoCandidates::operator`,
  `ExecutionContext::EVERY_ARRIVING_ROW`, and the context rules
  `queries_of_context`, `anchor_is_evaluation_time` and `query_reads`
  (moved from asap-cost, which keeps depending on types only).
- Workload: `QueryWorkloadEntry::evaluation_grid_ms` and
  `request_time_grid_ms`.
- Plan: `PhysicalASAPDAGNode::keeps_raw_rows_at_ingestion` (a `Scan`, or a
  `TimeShift` without `@` over one), with `asap_cost::reads_one_batch_of`:
  an ingestion-time reader of a raw-row store reads one batch, in Stage 3
  and now in `plan_cost` too (it read the whole store there); the cost
  model's raw-row rule uses the same predicate.
- asap-cost: `output_interval_ms` is public; Stage 3's Δ of π_v calls it.
- Stage 3: `SelectableExecutions::reads_back` and `Reader`; the
  `is_across_evaluations` wrapper and `is_query_time_scan`'s memo walk
  replaced by `PhysicalASAPMemoCandidates::operator`.

* test(cost,integration): an ingestion-time reader of raw rows kept at ingestion reads one batch; two queries reading one anchored group (#1207 PR 7)

* perf(stage3): a copy of an equivalent evaluation context shares the original's variables (#1207 PR 7)

The same queries on several grids (one user every 60 s, another every
300 s) give one evaluation context per grid with the same executions and
edges, each cost a fixed multiple of the other's. The MILP chose each
copy freely, so HiGHS branched over both: c16 130 s, RangeHeavy-8users
222 s on seed 0.

copies_in_equivalent_contexts finds such copies (same candidates, kept
alike, same memo edges from corresponding producers, paired queries with
the same outputs, exact-only groups and span, costs proportional), and
selection_variables gives a copy's executions, edges and queries the
original's x, y and z. Exact: some optimum selects alike in both copies
(give both the cheaper one's selection; retention and raw data do not
grow). Plan costs are unchanged; c16 drops to about 10 s.

* refactor(types,cost,stage3): one refresh rule for an ingestion-time stored result (#1207 PR 7)

Three rules said when an ingestion-time stored result refreshes and
disagreed: Stage 3 (own_schedule: a chain at phase 0, anything else its
group's evaluation grid), the installed plan (refresh_window: a chain
(w, s mod w), anything else the union of the windows below it) and the
cost model (maintained_window_ms: the first range or chain below). An
ingestion-time sum(TimeRange(5m)) read every 60 s ran its inputs on a
60 s pipeline and served the 60 s reader, while the plan refreshed it
every 5 min; a chain over TimeShift(30s) had phase 0 in Stage 3 and
30 s in the plan.

The plan's rule is the one rule: RefreshWindow::computed_from over any
TimeIndexedPlan. PhysicalASAPDAG::refresh_window, Stage 3's ingestion
pipeline context, the stored times delivers_on checks
(PhysicalASAPMemoCandidates::stored_times; delivers_on moves to the
candidates, whose data it reads) and both cost paths (plan_cost and
Stage 3's ingestion_window_ms, its width) use it. The candidates
implement TimeIndexedPlan; Stage 3's per-edge Candidates delegates to
it. shift_back_ms_below follows every alternative producer and needs
them to agree. maintained_window_ms and refresh_windows are deleted.

* fix(stage3,types,cost): a query's own context for context copies; one stored-times rule; review fixes (#1207 PR 7)

A query is evaluated in its own evaluation context, but the context-copy
search put it in the context of its first answering execution. With an
AcrossEvaluations candidate listed first that is Materialized(v), so two
evaluation contexts whose queries read different exact-only groups had
empty query lists, matched, and shared their variables: Q0 every 10 s
and Q1 every 20 s that must read the exact quantile then selected alike,
at 9 instead of 8. SelectableExecutions now records each query's
evaluation context in new (evaluation_context), the copy search reads
it, and selection_variables asserts that every shared z is a real
(query, answering execution) pair.

The copy search moves out of dominance into milp/equivalent_contexts.rs
(symmetry reduction) and returns SharedVariables. Contexts are bucketed
by a canonical key (candidates, kept, least need, in-edges and query
signatures), so only the cost proportionality check compares two
contexts; the out-edge and input-count checks it implied are a
debug_assert. Dominance and the copy search share one relative
tolerance (COST_RELATIVE_TOLERANCE).

stored_times uses RefreshWindow::computed_from for any chain, at either
timing (a query-time chain over TimeShift(30s) is stored at phase 30 s,
as at ingestion time), and its group's evaluation grid for any other
query-time result; own_schedule is deleted. EvaluationGrid gets
From<RefreshWindow>, which delivers_on uses.

A Filter or Project between a chain and a TimeRange keeps the chain's
tumbling-window width in the cost model, as a TimeShift does: the rule is
Operator::keeps_input_window (exhaustive), which also passes the windows
a TimeRange selected through them. Operator::chain_tumbling_window_width_ms
and Operator::request_anchor list every operator.

Also: QueryWorkloadEntry::range_span_ms at the three span sites;
Memo::output_groups reads QueryRoot::operator_refs and output_references
is deleted; PhysicalASAPMemoCandidates::is_query_time_scan; one
ONE_SECOND_MS in asap_types::workload; docs name Stage 2's output where
they said Stage 3 candidates; comments state rules, not process.

* fix(accuracy,stage3): a count's input is not exact-only; one retained output; targets in context copies (#1207 PR 7)

avg by (job)(quantile_over_time(0.99, data[300s])) at relative 0.05
over non-negative data had no plan without raw data: Pass 2 listed every
DDSketch group as exact-only for it, and Stage 3 now enforces the list
(X). The avg's exact-accumulator path reads count(quantile): a Project
above the count passes no metric, so the count's group got none, and the
walk and its closure marked the quantile and every sketch below it
exact-only. A count's value does not depend on its input's error (error
factor 0 in every metric), so a consumer now passes every metric in
which an input's error does not reach its output, and the exact-only
closure does not cross such an input. The DDSketch plan is valid: #1302
made a relative bound on avg hold for values of one sign, and data is
declared non-negative.

A store always keeps a stored result's newest output, and plan_cost
charges at least one retained output, but Stage 3 let a stored result
read only at its newest output retain 0 (an ingestion-time chain read by
a stored ingestion-time TimeShift: 0.007/s apart). Each candidate with a
retention cost now retains at least its output interval
(ExecutionCosts::newest_output_retention_ms): the linear retention cost
is taken at no less than one output, (R)'s least need, the bound and the
reported retention are floored at it.

Context copies also compare each query's accuracy target (metric, ε, δ,
scope), so the accuracy constraint (K) holds alike in both copies: the
same queries at rank 0.01 every 10 s and rank 0.05 every 20 s are not
copies.

* fix(stage3,accuracy,cost,types): round-5 review fixes: context copies to the context's largest cost, one metric rule per input, plan children order (#1207 PR 7)

- Context copies compare costs to COST_RELATIVE_TOLERANCE of the copy's
  largest cost, alpha from that pair: a small edge cost, a difference of
  large costs, keeps their rounding (RollupHeavy-8users flows: -2.36e-15
  and -4.71e-16, ratio 0.200000000026), which stopped copies sharing.
- One metric rule per input: ErrorContract::input_error_reaches_output and
  ErrorMetricsOfMemo::error_metrics_allowed_on_input, used by the exact-only
  walk and closure, the edges and the readout caps; a count's input edge
  from a readout holds in any metric.
- Newest-output floor applied once: most_read_ms floored where built,
  ExecutionCosts::retention_ms_for_need for edge and output needs.
- Operator::map_children applies f in children() order, so a plan's
  Project over q with scalar(count(q)) keeps q as its child; plan_cost
  takes a node's producers in its payload's input order.
- Layering: From<RefreshWindow> for EvaluationGrid in plan;
  ExecutionContext::is_outside_evaluations; QuerySignature with named
  fields; QueryWorkloadEntry::evaluation_grid; exact-only field docs.
- Abstraction: key_of returns queries in signature order; tolerance
  comments; range_span_ms for @ start() spans.

* fix(stage3,types,cost,accuracy): round-6 review fixes: context copies compare kept costs only, IN subquery order, producers from the payload (#1207 PR 7)

- E1-b: proportional_costs pairs only kept executions and edges between
  kept ones, so a left-out execution no longer sets the copy tolerance;
  alpha must be positive. Test: a left-out range costing 1e12 merged two
  contexts (optimum 2.5 instead of 2.421).
- N1 gap: ScalarExpr::map_parts maps an IN's subquery before its left
  side, as operator_refs lists them; a test checks map_children follows
  children() for every operator with scalar references.
- L-1: least_need_ms_per_execution -> least_retention_ms_per_execution.
- A2: plan_cost reads producers from PhysicalASAPDAGNode::producers
  instead of sorting edges. A3: unretained nodes built once. A4:
  outputs_of[&query]. L-2: input_error_reaches_output is pub(crate) and
  indexes directly.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

accuracy Accuracy requirements, guarantees, and propagation bug Something isn't working error-estimation optimizer Workload optimization and plan selection

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant