Skip to content

Pin mixed revisions between stored window boundaries - #808

Closed
zzylol wants to merge 1 commit into
fix/type-logical-candidates-upfrontfrom
fix/placement-pricing-lag
Closed

zzylol wants to merge 1 commit into
fix/type-logical-candidates-upfrontfrom
fix/placement-pricing-lag

Conversation

@zzylol

@zzylol zzylol commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #811.

Why

Remote-write revision pinning required a stored window ending exactly at the query evaluation timestamp. Mixed queries between window boundaries fell back before bounded-lag execution could select the latest complete window.

What

Pin one compatible revision with complete stored windows within each mixed binding's configured lag bound (default one slide). Retain only admitted windows; stored-only queries still require exact interval coverage.

Before this PR

A mixed query evaluated three seconds after a window boundary fell back despite a complete stored window within its lag bound.

After this PR

The query fetches raw input at its requested timestamp, combines it with the admitted stored window, and reports the three-second lag. Incomplete newer windows do not shadow complete older windows.

Validation

  • The new off-grid process regression failed before the fix and passes after it.
  • Formatting, workspace clippy, workspace library tests, control-plane tests, and all 30 compatibility process tests pass.
  • Independent agent review checked revision consistency, group completeness, lag bounds, and preservation of exact stored-only reads; no blockers remained.

🤖 Generated with Claude Code

Final stack validation at #809 / Planner a049a3b: formatting, clippy, workspace library tests, all control-plane tests, and all 30 process tests pass (122.91s).

@zzylol
zzylol marked this pull request as draft September 30, 2026 15:03
@zzylol
zzylol force-pushed the fix/placement-pricing-lag branch from 8c8aeda to e899d10 Compare September 30, 2026 15:04
@zzylol
zzylol changed the base branch from feat/per-state-mixed-placement to fix/type-logical-candidates-upfront September 30, 2026 15:04
@zzylol
zzylol force-pushed the fix/type-logical-candidates-upfront branch from 2ce2942 to aaa7876 Compare September 30, 2026 15:17
Select the newest complete stored window from one compatible revision at off-grid evaluation times, while retaining exact coverage for stored-only queries. The new +3s process regression failed before the fix and passes afterward.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@zzylol
zzylol force-pushed the fix/placement-pricing-lag branch from e899d10 to ec8786e Compare September 30, 2026 15:18
@zzylol
zzylol marked this pull request as ready for review September 30, 2026 15:25
@zzylol

zzylol commented Oct 7, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by the #818 → #822 stack (draft). The backend no longer selects plans: it executes the PhysicalASAPDAG that ASAPPlanner selects (#509), with the Planner pinned at one stack revision. Leaving this open for now; close it when the new stack is accepted.

🤖 Generated with Claude Code

@zzylol

zzylol commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #818, which replaced the old backend with the stack that executes ASAPPlanner's selected plan. Closing.

@zzylol zzylol closed this Oct 20, 2026
zzylol added a commit that referenced this pull request Oct 21, 2026
zzylol added a commit that referenced this pull request Oct 21, 2026
…coverage, ignore_accuracy_guarantees, pane-width SDS keys, --stream-start-ms (#844)

* feat(sds): adopt ASAPPlanner #787: the plan's definition id and full selection

Pin ASAPPlanner 4111e620 (#787: StoredOutput gives its SummaryCoverage,
definition id and selection, and its inputs instead of a pane).

- A definition's identity is the plan's coverage.definition_id; the SDS's
  own sub-DAG hash is gone (and with it the sha2 dependency).
- The definition id leaves the selection out, so outputs of one id with
  other label constraints are other SummaryDefinitions (DefinitionKey: id
  and label constraints), and a summary instance is keyed by the id and its
  concrete Selection: the label constraints and the pane end the plan's
  relative time binds to.
- A definition's pane width and each output's offset come from its
  selection's relative time; its RawSampleWindow is the stored output's
  raw-sample input; its stored inputs are the plan's
  inputs.stored_outputs. A selection that is not one left-open time box, or
  several raw-sample windows, is refused.

* feat(deps): pin ASAPPlanner main ef52e175; plan with ignore_accuracy_guarantees

- Both Planner crates move to ef52e175 (sketchlib planner-pin/0.4.0-plus:
  interpolated and signed DDSketch quantiles, top-k with series labels,
  Stage 3's executor compile check, joint search, shared stored summaries).
- control_plane: --plan-selection-settings (the Planner's
  PlanSelectionSettings JSON) and --ignore-accuracy-guarantees; prints the
  queries accepted without a guarantee. --measured-operation-costs reads
  MeasuredOperationCosts.
- serde_json float_roundtrip: a plan file reloads to the plan it was written
  from (its predicted usage is floats).
- data_plane/tests/planner_executor_answers.rs: the data plane answers as
  the Planner's executor for DDSketch panes over zero and signed samples,
  interpolated quantiles, top-k labels, counter top-k without guarantees,
  and jointly planned shared states.

* feat(deps): pin ASAPPlanner main 05fb36e4 (#808: faster installable-plan compile check)

* fix(sds): panes of one definition id and two widths are separate definitions

A jointly planned workload can store one computation (one definition id,
the same label constraints) in panes of two widths, e.g. per-series sums in
1-second panes for `sum_over_time(data[1m])` every second and in 30-second
panes for `sum_over_time(data[20m])` every 30 s. The SDS keyed definitions
by (id, label constraints), so both shared one definition with the first
width: the 20-minute query merged forty 1-second panes and answered 1/30 of
its sum.

DefinitionKey and Selection now carry the pane width: a definition is (id,
label constraints, pane width), and a summary instance's concrete selection
is (label constraints, width, end), the interval (end - width, end].

* feat(precompute): --stream-start-ms builds the first pane that starts at the stream's start

The first pane each stored output sees is skipped, since it may have started
before the data plane did. When the producers start their stream at the
data plane's start (the benchmark feeds from an aligned start S), that pane
is complete, and a query whose window reaches back to S was refused: e.g.
`rate(data_total[20m])` and `sum_over_time(data[20m])` at S + 20 min.

A producer can now declare where its stream starts (PrecomputeOptions::
stream_start_ms, `--stream-start-ms`): a first pane starting at or after it
is built; one starting before it is still skipped.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant