Skip to content

test(e2e): answer #509's PromQL examples like the Planner executor - #822

Draft
zzylol wants to merge 5 commits into
feat/query-enginefrom
test/e2e-examples
Draft

zzylol wants to merge 5 commits into
feat/query-enginefrom
test/e2e-examples

Conversation

@zzylol

@zzylol zzylol commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Stack 5/5. Base: #821.

Why

First milestone of executing the Planner's plan: the data plane's answers must equal what the Planner executor computes for the same selected plan.

What

data_plane/tests/e2e_examples.rs runs the real data_plane binary for #509's PromQL Examples 1, 3a, 3b, 4a and 4b:

  • data goes in through Remote Write;
  • a stand-in Prometheus serves the raw ranges (and answers forwarded queries with an empty vector, so a query sent before its panes are built is retried);
  • each answer is compared with the Planner executor running the selected plan with nothing materialized.

control_plane::plan_selected_at_query_time gives that reference: the Stage 2 candidate P<n> implementing the selected candidate's logical candidate L<n> with nothing materialized, sized for the failure probabilities and with confirmed weights as Stage 3 does for the selected one. Same summaries and parameters, everything at query time.

examples/planner-layering-example3a.json (Pattern A, ad hoc at T) and -example4a.json (Pattern A, monthly) are added in the current workload format.

Example Deployment Selected Stored definitions Result
1 default P1 (exact, query time) 0 exact
3a default P485 (KLL k=1246, query time) 0 within 10 (one query off by 1)
3b default P2 (KLL, query time) 0 exact
3b no raw data 1-min KLL panes at ingestion 1 exact
4a default P4125 (KLL k=1246, query time) 0 within 10 (exact this run)
4b no raw data P3-m1 (10-min KLL panes at ingestion) 1 exact

3a/4a match within a value tolerance of 10, about twice KLL's 0.005 rank error: their five-year windows hold more samples than k, and asap_sketchlib's KLL seeds compaction from the thread RNG, so two builds above k can differ.

How

Per example: plan with plan_workload (the plan installed) and plan_selected_at_query_time (the reference), start data_plane --allowed-lateness-ms 0, Remote Write every sample, poll each planned query until x-asap-execution: plan, and compare series by series with the reference run through QueryEngine over the same samples.

Key code interfaces added / modified

// control_plane/src/lib.rs
/// The selected plan with nothing materialized: the Stage 2 candidate that
/// implements the same logical candidate with every summary built at query
/// time, sized as Stage 3 sizes the selected one.
pub fn plan_selected_at_query_time(workload: &PlanningWorkload, inputs: &DeploymentInputs)
    -> Result<InstalledPlan, PlanError>;

Before this PR

No test checked that the data plane's answers match the Planner's.

After this PR

Every example's planned queries are answered by the plan and match the Planner executor (numbers below).

Usage

# Run the five examples against the real data_plane binary; --nocapture prints
# each series as "<query>: <labels> plan <data plane> executor <reference> difference <d>".
cargo test -p data_plane --test e2e_examples -- --nocapture
sum by (job) (rate(http_requests_total[1m])): {"job": "api"} plan 2.9999999999999996 executor 2.9999999999999996 difference 0
topk by (job) (10, sum_over_time(http_requests_total[1m])): {"job": "web"} plan 3330 executor 3330 difference 0
quantile_over_time(0.99, latency_ms[5y]): {"service": "a"} plan 987 executor 988 difference -1
quantile_over_time(0.99, latency_ms[3y] offset 2y): {"service": "b"} plan 10986 executor 10986 difference 0
quantile_over_time(0.99, latency_ms[5m]): {"service": "a"} plan 36 executor 36 difference 0
quantile_over_time(0.99, latency_ms[60m]): {"service": "b"} plan 1237 executor 1237 difference 0
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 378.70s

Verification

  • cargo test -p data_plane --test e2e_examples (5 tests, about 6 minutes in a debug build: Examples 3a/4a enumerate thousands of candidates).
  • cargo test -p control_plane -p data_plane (control plane adds reference_plan_materializes_nothing), cargo fmt --all -- --check, cargo clippy --workspace --all-targets -- -D warnings.

This was referenced Oct 7, 2026
…ecution stack

The backend now executes ASAPPlanner's selected PhysicalASAPDAG as given
(#509): the Planner runs Stages 0-3, the backend supplies deployment inputs.

- Pin every Planner crate the backend uses to ASAPPlanner main (58999df2):
  asap-types, asap-frontend-promql, asap-logical-asap-optimization,
  asap-plan-selection, asap-executor. The sync script updates all five.
- control_plane: lower a PromQL PlanningWorkload, run Stages 1-3
  (run_stage1_to_stage3) under DeploymentInputs (DeploymentCapabilities and
  PlanSelectionSettings), and write the selected candidate's DAG, with each
  PaneChain expanded into its panes, plus the query each root answers
  (plan.json).
- data_plane: load and validate plan.json, accept Remote Write, forward
  queries to Prometheus until the query engine lands.
- Remove the backend's own plan selection, placement pricing, ERP, SDS,
  precompute and query engines, ClickHouse, OTel and OpAMP paths, which were
  built on the removed Planner APIs.
…puts

Each MatAtIngestion output the query side reads (the executor's
materialization frontier) reads one tumbling pane of raw samples. Outputs
that differ only in which pane they read share one SummaryDefinition,
identified by a SHA-256 of their sub-DAG with node ids blanked and offset
TimeShifts skipped. Each output binds to its definition with an offset: at
evaluation time T it reads the pane ending at T - offset. SummaryStore keeps
built panes by definition and end. A plan with a node kept across
evaluations is rejected: the backend declares it cannot keep one.

#509 Example 4b on a deployment without raw data: its six 10-minute KLL
panes become one definition with offsets 0..50 min.
An engine thread owns the installed plan (Planner IR holds Rcs) and receives
Remote Write samples from the HTTP server. For each stored summary
definition it buffers its Scan metric's samples by tumbling pane
(end - width, end], like PromQL ranges. When the watermark (newest sample
minus --allowed-lateness-ms) passes a pane's end, it runs the plan's Scan
over them (applying label matchers) and the definition's build from its
TimeRange input, then stores the output. Panes without samples are stored
empty; the first pane seen is skipped because it may be partial; samples for
built panes are dropped and counted. A full engine queue answers Remote
Write with 503 so senders retry.
Each plan root is compiled once with two kinds of inputs: stored outputs,
bound at evaluation time T to their definition's pane ending at T - offset,
and query-time raw ranges (TimeRange <- TimeShift* <- Scan), bound to the
metric's samples in (T - offset - range, T - offset] fetched from
Prometheus and filtered by the plan's Scan. An instant query whose text is a
planned query is answered with a Prometheus vector and x-asap-execution:
plan. Unplanned queries are forwarded; planned ones that cannot be answered
(pane missing or off the pane grid, fetch or execution failure) are
forwarded with x-asap-forward-reason.
For Examples 1, 3a, 3b, 4a and 4b: plan the workload, start the data_plane
binary against a Prometheus holding generated samples, Remote Write the
same samples, and require every planned query to be answered by the plan
with the Planner executor's result for the selected logical candidate with
nothing materialized (control_plane::plan_selected_at_query_time). 3b also
runs, and 4b only runs, on a deployment without raw data, so their panes are
built from Remote Write. Exact results must match exactly; Example 3a/4a's
five-year KLL windows exceed k and KLL compaction is randomized, so they
must agree within twice the 0.005 rank target.
@zzylol
zzylol force-pushed the feat/query-engine branch from 6f7b53c to b1b1e5a Compare October 18, 2026 15:40
@zzylol
zzylol force-pushed the test/e2e-examples branch from 4ab777a to f7faf1b Compare October 18, 2026 15:40
@zzylol zzylol added area: data plane Runtime ingestion, storage, and query-serving data paths integration: ASAPPlanner ASAPPlanner API, IR, candidate, and selected-plan integration quality: accuracy Exact and approximate accuracy contracts, error bounds, and routing test: end-to-end Process-boundary, differential, conformance, and full-stack tests labels Oct 18, 2026
@zzylol
zzylol force-pushed the feat/query-engine branch 3 times, most recently from 1bdf7e9 to afa5bc6 Compare October 20, 2026 23:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: data plane Runtime ingestion, storage, and query-serving data paths integration: ASAPPlanner ASAPPlanner API, IR, candidate, and selected-plan integration quality: accuracy Exact and approximate accuracy contracts, error bounds, and routing test: end-to-end Process-boundary, differential, conformance, and full-stack tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant