You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Stage 3 under-predicts CPU ~740×: measured costs price only sketch kernels, not scans, per-row work or merges #761
ASAPQuery-backend ran the planner-layering PromQL examples on 2,000 series (90 minutes of data, 15 s samples, raw data retained). It ran Stage 3's selected plan with AWS Fargate prices and sketch-bench's measured operation costs, using a release build and perf sampling.
Stage 3 predicted
Measured
Data plane CPU for the selected plan
0.072 core-seconds
53 core-seconds (about 740× more)
Peak memory, 10,000-series plan with no raw data
1.06 GiB
1.69 GiB
The sketch kernels that measured_operation_costs prices are only 3–7% of the measured CPU. perf breaks down the rest:
Row handling in the executor: allocation and free 32%, row clones 11%, row drops 5%, grouping keys 9%, per-row memory accounting 9%, batch validation 2%.
Raw-sample transfer: reading raw ranges back from Prometheus as JSON is about 15.5%, about 0.89 µs per sample.
Scans and merges: Scan rows 5.6%, building kept panes 7.4%, merging and copying stored panes about 4%.
Because those costs are missing, Stage 3 underprices query-time plans that rebuild state from raw samples. With raw data retained it selected no ingestion-time outputs at all. The two quantile queries then rebuild 1,000 per-series KLLs at every evaluation, taking 57 and 313 ms in-process, against 31 and 62 ms when they read stored panes.
Cause
Stage 3's CPU estimate prices each operator by its kernel operations: sketch updates, merges and estimates, through measured or analytical per-operation costs. It does not price:
Scanning raw samples: per-sample decode and transfer from the data source, with the cost depending on where the raw data lives (local vs remote, JSON vs binary).
Per-row operator overhead: materializing, grouping, hashing and moving rows between operators, which is independent of the kernel.
Merging and copying stored state: reading N panes and merging them per evaluation, beyond the kernel's own merge cost.
Memory overheads: row layout, label strings and decoded buffers on top of the summary state.
Proposal
Add per-sample and per-row cost terms to MeasuredOperationCost: scan per sample, per-row operator overhead, and merge or copy per stored pane. A runtime measures these and supplies them, so the Planner stays deployment-agnostic and only consumes numbers.
Keep the analytical defaults for these terms, but make them realistic: derive them from a reference run instead of zero.
Report predicted vs measured per plan in the stage_pipeline output, so the gap stays visible.
Acceptance
With the runtime-measured terms supplied, Stage 3's predicted CPU for the example above is within 2× of the measured 53 core-seconds.
A test pins that the scan and per-row terms change the choice between rebuilding from raw samples and reading stored panes, in the direction the measurements show.
Related
#754 (kept results almost never selected), #726 (sketch memory sized by full capacity), #734 (measured operation costs). ASAPQuery-backend design-doc PR #826 has the full measurement ("Overhead of the data plane" section).
Done. A measured Scan entry per phase, priced per sample. The runtime supplies the number for where its raw data lives (local or remote, JSON or binary).
OperatorKind::Scan, OperatorCost.phase; test external_raw_reads_make_the_query_time_raw_plan_costlier
Cause 2: per-row operator overhead
Done. Every physical operator kind can have a measured entry (ns_per_unit per input row, ns_per_invocation per computation), including its row handling. Per-request overhead is ns_per_evaluation.
OperatorKind, MeasuredOperationCosts
Cause 3: merging and copying stored state
Done.SummaryMerge is priced per state read, PaneChain per row, at each phase.
test compaction_prices_merges_from_the_measured_merge
Cause 4: memory overheads
Partly. Summary state is sized per family; query-time working memory is not yet charged under cost model 1.
Proposal 2: realistic defaults from a reference run
Planned. An executor-side microbench writes a measured table with provenance. Runtime-specific costs (remote JSON reads, allocator) stay in the table the runtime supplies, not in the Planner's formulas.
Predicted side done (SelectedPlanReport.predicted_usage: totals, per node, per query). stage_pipeline doesn't print per-node CPU yet. The measured side is the runtime's.
Activity
zzylol commentedon Oct 21, 2026
Closing as covered by #789. The Planner side of this issue is implemented there; the rest is tracked in the issues below.
Status on
14394adaScanentry per phase, priced per sample. The runtime supplies the number for where its raw data lives (local or remote, JSON or binary).OperatorKind::Scan,OperatorCost.phase; testexternal_raw_reads_make_the_query_time_raw_plan_costlierns_per_unitper input row,ns_per_invocationper computation), including its row handling. Per-request overhead isns_per_evaluation.OperatorKind,MeasuredOperationCostsSummaryMergeis priced per state read,PaneChainper row, at each phase.compaction_prices_merges_from_the_measured_mergeSelectedPlanReport.predicted_usage: totals, per node, per query). stage_pipeline doesn't print per-node CPU yet. The measured side is the runtime's.external_raw_reads_make_the_query_time_raw_plan_costlier,calibration_keeps_unmeasured_nodes_on_the_measured_scaleFollow-ups
Scan,TimeRangeorAggregateentry. Without a table, a raw-sample scan is priced at about 4 ns, against about 890 ns measured here.cpu_core_seconds_per_second), operator and timing in stage_pipeline's Stage 3 cost output.