Skip to content

Enforce absolute deadlines for auction response collection - #1433

Open
prk-Jr wants to merge 13 commits into
mainfrom
fix/1015-auction-deadlines
Open

prk-Jr wants to merge 13 commits into
mainfrom
fix/1015-auction-deadlines

Conversation

@prk-Jr

@prk-Jr prk-Jr commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Start auction operations at dispatch and bound additional network waiting by the original deadline, including headers and complete bodies.
  • Preserve complete available provider bids after a slow origin through one nonblocking snapshot; time out unfinished providers and enforce the earlier of auction and mediator cutoffs.
  • Exercise actual delayed/trickling network paths automatically under Viceroy, workerd, and Spin.

Changes

Files Change
Core platform HTTP/error/test support Fail-closed bounded collection, deadline errors, completed-result snapshots, and aligned test behavior. Remove the unused total-request-deadline capability.
core/src/auction/orchestrator.rs Preserve provider attribution/count/timing, skip exhausted mediation, check mediator cutoff after launch and before parsing, retain local winners, and log expected expiry separately from transport failures.
Provider and Prebid/APS/mock integration launch sites Explicit intermediary-cache bypass for personalized auction requests.
Fastly adapter platform and deadline collector Timed SDK readiness waits, bounded incremental buffering, attributed eviction, direct payload transfer, and isolation of unsupported individual requests.
Axum adapter platform Retain task ownership through timeout, snapshot finished tasks, abort unfinished work.
Cloudflare/Spin adapter platform Start cancelable background operations at dispatch, retain complete results, and probe the result channel at timer expiry. Spin limits response buffering incrementally to 8 MiB.
Adapter runtime fixtures, Node runners, workflow files Real SDK network tests, fixture feature gates and WASM lint, exact Fastly test-count enforcement, and checksums for downloaded runtime tools.
Publisher, guides, CHANGELOG, technical plan Explain deadline semantics, operator impact, resource/cancellation limits, and test seams.

Crate paths above are relative to crates/trusted-server-. fastly-sys is inherited from workspace dependencies at the locked SDK version; web-time supplies the shared clock.

Closes

Closes #1015

Spec

Not a feature. Implements #1015. Technical plan: docs/superpowers/plans/2026-10-22-absolute-auction-deadlines.md.

Deadline decisions and limits

Provider bids complete at the final nonblocking snapshot remain eligible even after tmax; mediation starts only with remaining budget and rejects responses collected after its cutoff. This bounds additional network waiting rather than eligibility by an unavailable upstream arrival timestamp.

Fastly does not use an empty handle list or millisecond header polling. One missing cache-bypass marker or malformed pending request fails only that provider, retaining healthy siblings. Retained payloads share a 10 MiB cap; pressure rejects the largest partial response under its own name. Payload vectors move directly into platform bodies. This cap excludes host/allocator overhead; the final nonblocking drain uses local CPU bounded by payload size, not a separate CPU-time ceiling.

Fastly closes acquired bodies and abandons pending requests because its installed SDK cannot cancel them. Other adapters abort owned tasks; Cloudflare also aborts fetch. Upstream side effects cannot be undone. Runtime scheduling and timer granularity can add overshoot; local parsing, ranking, HTML work, and delivery remain outside the network cutoff.

Review findings and proof

  • Real worker fixtures prove dispatch overlaps a 200 ms origin with an 80 ms auction budget. Completed responses survive delayed collection; incomplete headers/bodies time out.
  • A mixed-provider Fastly regression failed when one unsupported request aborted collection, then passed with per-provider attribution. An actual SDK request without cache bypass is included in the five real network cases.
  • Worker timer/result-channel race regressions failed before the final probe and passed afterward. Spin's real oversized-response case proves incremental body-limit enforcement and attribution.
  • A 12-case production-orchestrator regression covers planned/legacy, synchronous/split, slow Immediate/Pending launch, and a completed response returned after mediator expiry. Old post-wait checks accepted the late response; fixed paths reject it before parsing and retain provider/local winners and elapsed timing.
  • The Fastly harness rejects a zero-test run and requires exactly five passing network cases.
  • CI exposed a missing Clippy component in the new worker job. The job now explicitly installs Clippy through the shared toolchain action before fixture linting; an independent reviewer verified the prerequisite chain.
  • Fresh independent deadline, cookie, and cross-PR reviews found no remaining concrete correctness findings after fixes. The manually polled publisher fixture tests injection order; real runtime fixtures establish autonomous timer behavior.

Test plan

  • Full native core/integration/doctests and all five adapter aliases, including reusable Fastly sandboxes.
  • All five real Fastly network cases, four workerd cases, and five Spin cases.
  • Both runtime-fixture WASM builds and explicit clippy checks.
  • Production Axum deadline/cancellation/snapshot tests; worker channel and Spin body-limit tests.
  • Cross-adapter parity, native build-digest tests/lint, and all eight clippy aliases.
  • 1,211 JS tests, JS format, and 13-module build.
  • Docs/Markdown format, WASM clock lint, changed-workflow lint, documentation-proposal checks, Rust format, and diff checks.
  • Optimized Fastly WASM build.
  • Independent review and verified cross-PR composition.
  • Actual combined tree: 2,982 native core tests and 2,976 Viceroy core tests passed, plus integration suites and doctests.

Exploratory tsc --noEmit and whole-repository actionlint report pre-existing errors in unchanged files; they are not counted as passing checks. Local verification uses Rust 1.98.1, Viceroy 0.21.0, and Node 24.12.0; the repository pins Node 24.20.0. Runtime checks use local SDK runtimes, not production Fastly/Cloudflare deployments.

Checklist

  • Changes follow AGENTS.md conventions.
  • No Python code, secrets, customer data, or new operator settings.
  • Deadline, attribution, memory-pressure, consent composition, and mediation behavior have regression coverage.
  • Direct dependencies and resource limits are justified above.

@prk-Jr prk-Jr self-assigned this Oct 22, 2026
Comment thread crates/trusted-server-adapter-fastly/tests/deadline_origin.py Fixed
@prk-Jr
prk-Jr marked this pull request as ready for review October 26, 2026 09:29

@ChristianPavilonis ChristianPavilonis left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved, with one non-blocking maintenance suggestion: share the runtime-neutral deadline helper and its regression tests between Spin and Cloudflare to keep future race fixes aligned, while retaining runtime-specific timers and cancellation in each adapter.

Comment thread crates/trusted-server-adapter-spin/src/platform.rs Outdated

@dhruv8sh dhruv8sh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a careful, well-tested rework of auction deadline handling. Replacing the dormant has_enforceable_total_request_deadline capability with a single select_until / wait_until contract that every adapter implements is the right shape. The default implementation refuses with Unsupported instead of falling back to an unbounded wait, which is the correct safety default. The nonblocking snapshot on expiry keeps bids that already arrived when the publisher origin is slow, and the mediator cutoff (the earlier of the auction deadline and the mediator's own timeout) is now checked at all three points in both the sync and split paths.

The adapter work holds up on a close read:

  • Fastly: the timed fastly_async_io::select gets a positive timeout (.max(1)), so the host can never treat it as unbounded. Each partial body is capped, and so is the total across bodies. Invalid handles fail only their own provider.
  • Axum: handles are borrowed across tokio::time::timeout, so expiry can't drop siblings that already finished.
  • Cloudflare and Spin: the final try_recv closes the race between the timer and the response channel.

The real-runtime tests under Viceroy, workerd and Spin are a big step up from stub-only coverage.

CI status: all checks pass.

None of the comments below is blocking. The one I'd most like a reply on is the Fastly-only cache-bypass rule (comment 1).

Comment thread crates/trusted-server-adapter-fastly/src/platform/deadline.rs
Comment thread crates/trusted-server-core/src/auction/orchestrator.rs Outdated
Comment thread crates/trusted-server-adapter-spin/src/platform.rs Outdated
Comment thread CHANGELOG.md Outdated
Comment thread docs/guide/auction-orchestration.md Outdated
Comment thread crates/trusted-server-adapter-fastly/Cargo.toml
@aram356 aram356 added this to the 202611 milestone Oct 29, 2026
prk-Jr and others added 3 commits October 29, 2026 22:21
Share pending-operation ownership and deadline channel collection between
Cloudflare and Spin while retaining runtime timers and cancellation.

Require cache bypass in bounded HTTP test collection like Fastly, preserve
healthy siblings, and update auction test providers. Replace four elapsed
wall-clock assertions with checks that unbounded work was skipped. Clarify
bid eligibility and coordinated Fastly SDK updates.
Preserve current creative asset/click settings, renderer diagnostics, and
CI Node/domain checks alongside the reviewed deadline changes.

Resolve the adapter lockfile entries by retaining web-time and main's
removal of redundant direct JS dependencies. Keep the new click-rewrite
configuration row and the original dispatch-deadline description.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Enforce absolute auction deadlines during dispatched provider collection

4 participants