Retained raw data as concrete state groups per lookback: offsets, @ and a retention limit #955

Problem, by example

The cases below come from reading Stage 3 on main (plan-selection/src/lib.rs), not from a planner run.

1. An offset is not retained. For rate(m[5m] offset 1h) evaluated at query time, the selector reads raw samples from (t − 65m, t − 60m], so 65 minutes of m must be kept. Stage 3 keeps and prices 5 minutes. windows_read follows a TimeShift but "moves a window without widening it", so query_time_lookback_ms returns 5 minutes. The plan is priced as if it needed one thirteenth of the raw data it reads. @ T has the same gap: the window it reads lies now − T back.

2. Retention is all or nothing. The deployment declares DeploymentCapabilities::raw_data_retained: bool.

• With true, a query-time scan of any window is valid, even when the deployment actually keeps, say, 15 minutes of raw data, as a short-retention Prometheus does.
• With false, every query-time scan is rejected (raw_data_violation).

So no deployment can say "15 minutes kept", and no plan is rejected or repriced because its horizon exceeds what is kept.

3. The horizon is a plan-level afterthought. Stage 3 already charges raw data once per source, at the longest lookback of any query-time scan of that source (the raw_data map in price_at_scale, priced at cost_per_disk_byte_second). That is the right quantity, but it is computed only after a whole plan is fixed. The memo has no raw-horizon site, although the design doc of #509 (memo section, Pass 2 row) lists "Raw horizon: a source's raw-data horizon site on every candidate that reads a raw window of it". In fact the code has no coupling-site type at all. So a reader choosing between "scan 1 hour of raw m at query time" and "maintain a pane chain at ingestion" is not told that another reader already pays for that hour, or that it alone would extend the horizon.

Proposal: retained raw data as a concrete state group per lookback

Requirement of a candidate

Every candidate that reads raw rows of source S at query time declares a lookback r(S), the oldest raw data it reads, measured back from its evaluation time:

Ingestion-time readers declare nothing: they read rows as they arrive.

Retained raw data as concrete state groups (decided 2026-10-27)

There is no new site type. The memo already represents a shared choice by concrete state groups, one per threshold (memo/state_groups.rs: readers needing KLL k ≥ 345 and k ≥ 200 get concrete KLL(345) and KLL(200) groups). Retained raw data uses the same interface:

• For a source S, one concrete group raw window(S, H) per distinct lookback H among the candidates that read S at query time. It stands for the raw rows of S kept for H back from the evaluation time.
• A reader with lookback r gets one candidate per raw window(S, H) with H ≥ r. The candidate reads that group and keeps only its own window, (t − o − w, t − o] or (T − w, T].
• The group's own cost is its retention, charged once whatever the number of readers:

retention(S, H) = bytes_per_second(S) × H × cost_per_disk_byte_second

Choosing the horizon becomes choosing which raw window group the readers read, exactly as choosing a shared sketch size is choosing a KLL(k) group. The search then sees "this reader extends the horizon from 5m to 1h" as the price difference between reading raw window(S, 5m) and raw window(S, 1h), instead of a maximum Stage 3 recomputes after the plan is fixed. Today's per-source maximum (raw_data in price_at_scale) is what the search obtains when every reader of S reads one group.

This is also the shared-producer shape of #954: the covering need is the longest lookback, and the derivation is the reader's own window filter.

Deployment capability

raw_data_retained: bool becomes a horizon limit per source, defaulting to the current behavior:

/// How much raw data the deployment keeps, per source (default: all; `None` = unbounded).
raw_data_retention: RawRetention,   // was raw_data_retained: bool

enum RawRetention {
    /// Kept as long as any plan needs: today's `true`.
    Unbounded,
    /// Nothing kept after arrival: today's `false`.
    None,
    /// At most this much per source, e.g. a short-retention Prometheus.
    Limit(HashMap<SourceName, Duration>),
}

A candidate whose lookback exceeds the limit is invalid, with the lookback and the limit as the rejection reason. raw_data_violation generalizes from "any query-time scan" to "a lookback over the limit". When raw data is already kept by an external store at no marginal cost, the deployment sets a zero disk price for raw data, and the limit still bounds which raw window groups exist.

Relation to #951

The shared scan of #951 reads the union of its readers' windows. That union's oldest point is exactly the lookback the shared scan declares. Both rules use one function for a reader's window, (t − o − w, t − o] or (T − w, T], so they cannot disagree.

Scope and acceptance

In scope:

• the lookback function (with offsets, @, instant selectors, SQL ts bounds);
• raw window(S, H) groups through the state-group interface, their reader candidates, and their pricing in Stage 3 (replacing the after-the-fact raw_data maximum);
• the RawRetention capability;
• raw_data_violation generalized to a limit;
• the docs: propose workload-wide planning, summary sharing, and materialization #509 design doc's description of raw horizon as raw window groups.

Out of scope:

• tiered storage (memory vs disk for raw data).

Acceptance:

• Example 1 fails first: rate(m[5m] offset 1h) at query time keeps and prices 65 minutes of m, not 5.
• With Limit(15m), a 1-hour query-time scan is rejected with a reason naming both durations, and a plan that maintains the same query at ingestion is selected instead.
• Two readers of one source with lookbacks 5m and 1h both read raw window(S, 1h) when that is cheapest, and the 1h retention is charged once, not 65 minutes.
• Default Unbounded and None reproduce today's selected plans and costs on the existing tests (stored_data_without_raw_data_retained_is_bad_input, the raw_data_retained: false tests in plan-selection, materialization_e2e).

Related

The design doc of #509 (memo section, Pass 2 row: "Raw horizon"); #951 (shared scan: the union of readers' windows); query_time_lookback_ms, windows_read, raw_data_violation and price_at_scale in plan-selection/src/lib.rs.