Repository navigation
Conversation
Add EID value types, an additive KV schema for provider-written EIDs with owner and expiry, EID-level fields on the OpenRTB Eid, and a resolver that serves only live UIDs to auctions, x-ts-eids, identify and the admin auction preview. Auction EIDs merge per (source, inserter, matcher, mm, ext) with first-occurrence UID dedupe. KV writes over 16 KiB are refused. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Top-level document navigations on the publisher route receive a single-use write permit; finalize plans one row change (browser ingest, provider results, prune, size cap) and commits it once without retry. Non-navigation requests and redirects never write, integration proxies no longer mint ECs, and new ECs are created by the page-load write instead of at generation. Withdrawal tombstones only the active EC and keeps the EC cookie when the tombstone does not land; the withdrawal completion marker is the one deliberate extra write. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Add the IdentityProvider/ProviderKind interface and [[ec.identity_providers]] config, response validation, a deadline-bounded LookupTransport over select_until, the IdentityCoordinator with consent, allowlist and budget checks, the trusted-server-identity-providers crate (no kinds yet), and Fastly wiring that dispatches before the origin fetch and collects before the single page-load write. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Orphan recovery minted a new EC ID and then discarded the navigation's provider results because they were looked up under the orphaned ID. Lookups depend on the browser, not the EC ID, and the orphan has no row, so retarget them to the replacement as the spec requires. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Collection that starts after its deadline takes one nonblocking snapshot. A request still in the headers state moved to the body state without reading bytes, which counted as no progress, so a response that had fully arrived was abandoned unread and reported as a timeout. Count entering the body state as progress. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
A response body the host already holds can report not ready until the host services async I/O once, so an expired snapshot abandoned identity responses that arrived well before the deadline and reported them as timeouts. Give each pending body one host select capped at 1 ms before abandoning it. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
…not fill Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Page-load writes only run on real navigations, which browsers mark with both sec-fetch-dest: document and sec-fetch-mode: navigate. Without the mode header the consent withdrawal scenario never got a write permit, so the EC cookie was kept. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
A lookup sends the EC ID to the provider before finalize can decline the EC, so GPC, sale opt-out and unknown-jurisdiction requests must not dispatch. Matches the gate pull sync already applies. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
… read LiveRamp envelopes are about 700 bytes, so the 512-byte limit dropped every one from KV and the auction. Rows written under a higher limit now load with the oversized IDs removed instead of failing as a whole. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
304f395 to
2de6d0d
Compare
|
@dhruv8sh, directionally the layout here is not compatible with V2 (the 55 pull requests tracked in #777, merged in order). This pull request creates one shared The V2 spec is identifier-modules and pluggable-providers, with the folder layout in integrations-crate, all in #1342. The associated pull requests are #1382 (a module hands core its identifiers, starting with the Prebid EID cookie) and #1343 (the Edge Cookie as a module, with the HMAC module built in). Could you evaluate the V2 spec and those pull requests before progressing this one? |
…readable rows Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
…or gate Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
|
CREATOR: From the server-side identity plan (#1559, |
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
…ints it Providers now declare sends_ec_id (default true). On a navigation that mints an EC, page-phase lookups run within identity_budget_ms for providers whose requests do not carry the EC ID, against the row the navigation creates, and their results go into that create. A failed or colliding create drops them, having exposed no EC. Providers that send the EC ID, user.id and after-response lookups still wait for a later navigation. Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
Signed-off-by: dhruv8sh <dhruv8sh@proton.me>
ChristianPavilonis
left a comment
There was a problem hiding this comment.
Summary
Adds server-side identity-provider collection and consolidates navigation persistence into a single page-load write, with an optional after-response update. One reproduced timing defect can renew a cached provider opt-out for another 30 days without a fresh provider response.
Blocking
🔧 wrench
- Cached opt-outs can accidentally renew for another 30 days — see inline at
crates/trusted-server-core/src/ec/write_plan.rs:149–150.
Verification and scope
- Native core library: 3,559 passed, 2 ignored; includes the 556 passing focused EC tests (not an additional independent total).
- Identity-provider crate: 5 passed.
- Fastly reusable-sandbox suite: 260 passed, 5 network fixtures ignored in that run; those 5 deadline network fixtures passed separately.
- Fastly verification used the pinned Viceroy 0.21.0. An initial attempt with Viceroy 0.17.0 failed before tests because it lacked
fastly_http_req::pending_req_header_insert; that environment mismatch is not a PR defect. git diff --checkpassed.- A standalone probe through the actual coordinator and planner reproduced the inline finding. The proposed multi-file fix has not been implemented or tested.
- Local Axum/Cloudflare/Spin suites were not repeated; their hosted results are reported below.
Coverage limitation: complete, untruncated full-file coverage of every unchanged portion of the very large publisher/settings files has not been established. This is not a claim of an exhaustive full-file audit. Previously fixed feedback and tracked rollout deferrals are not repeated as new findings.
CI Status
- integration tests: PASS
- browser integration tests: PASS
- trace runtime and browser acceptance: PASS
- integration tests (Fastly EC lifecycle): PASS
- CodeQL: PASS
- cargo test (ts CLI, native) (macos-latest): PASS
- docs-proposal scripts: PASS
- cargo fmt: PASS (required)
- Analyze (javascript-typescript): PASS
- release scripts: PASS
- Analyze (actions): PASS
- CI Node pin guard: PASS
- HTML EOF runtime (Fastly): PASS
- ts dev lint domains (changed lines): PASS
- cargo test (ts CLI, native) (ubuntu-latest): PASS
- cargo test (cross-adapter parity): PASS
- CLAUDE.md symlink guard: PASS
- format-typescript: PASS (required)
- cargo check/build/test (spin native + wasm32-wasip1): PASS
- cargo test: PASS (required)
- format-docs: PASS (required)
- cargo check (cloudflare native + wasm32-unknown-unknown): PASS
- worker-deadlines (cloudflare): PASS
- cargo test (axum native): PASS
- Analyze (rust): PASS
- prepare integration artifacts: PASS
- vitest: PASS
- worker-deadlines (spin): PASS
- Analyze (python): PASS
- Analyze (javascript-typescript): PASS
| if lookup.status == LookupStatus::Skipped(SkipReason::OptedOut) { | ||
| entry.remember_opt_out(&lookup.provider, now); |
There was a problem hiding this comment.
🔧 Cached opt-outs can accidentally renew for another 30 days
The coordinator represents both a fresh provider opt-out and a skip caused by remembered opt-out state as Skipped(OptedOut) (coordinator/mod.rs). If that remembered state expires between dispatch and finalization, plan_write first prunes it, then these lines recreate it for another 30 days—even though the provider was never contacted. Finalization obtains a fresh timestamp, so a navigation crossing the expiry boundary can take this path.
I reproduced this through the actual coordinator and planner:
dispatch time: 1000
original expiration: 1001
finalization time: 1002
provider contacted: false
new expiration: 2593002
The provider consequently remains suppressed instead of becoming eligible again. This affects page-load collection once providers are enabled.
Proposed fix: distinguish fresh opt-out results from cached skips. Only fresh results should establish opt-out memory; expired cached skips should be discarded. For example, after introducing a distinct LookupStatus::OptedOut and mapping fresh provider opt-out results to it:
if lookup.status == LookupStatus::Skipped(SkipReason::OptedOut)
&& !entry.opted_out(&lookup.provider, now)
{
continue;
}
if lookup.status == LookupStatus::OptedOut {
entry.remember_opt_out(&lookup.provider, now);
}Update the existing concurrent-opt-out guard to recognize fresh opt-out results as well, preserve its protection against restoring IDs over a concurrent opt-out, and add an expiration-crossing regression test alongside opt_out_is_remembered_without_extending_it.
Apply manually — this requires coordinated changes in the result type, coordinator, planner, and tests, so it cannot be safely expressed as one contiguous suggestion. The reproduction is verified; the proposed multi-file fix has not been implemented or tested.
Brings PR #1487 onto PR #1526's base (PR #1506's EdgeZero cutover, PR #1614, PR #1525 and PR #1526's awaitable EC persistence) and keeps PR #1487's behavior in the new base's structure. How each area was resolved: - Provider lookups: the lookup transport now sends through EdgeZero's outbound client in the auction fan-out's shape. `LookupTransport::start` takes every built request and starts them as one batch with `HttpClient::start_batch_until` and a cutoff of the identity budget, taken on the client's own clock; `gather` reads the batch until it finishes, maps slots back to providers and reports unresolved slots as timeouts. `PlatformLookupTransport` and the coordinator's web-time `Deadline` are replaced by `OutboundLookupTransport`; requests keep the 64 KiB response cap, cache bypass and strict TLS. - Deleted modules: `platform/http.rs` and the Fastly `platform/deadline.rs` stay deleted. PR #1487's final-snapshot change to the Fastly deadline select (a 1 ms surfacing wait per pending body and counting arrived headers as progress) has no counterpart in EdgeZero's batch and is not ported, nor is its doc-only change to `select_until`. - Async KV: `ec/kv.rs`, `ec/mod.rs` and `ec/finalize.rs` take PR #1487's versions converted to PR #1526's async store; the page-load and after-response writes, the write planner, after-response commit, batch sync's provider-owned check and their tests await the store. Unchanged tests take PR #1526's async versions, and PR #1526's non-Send yielding store test is kept. `generate_if_needed` stays async as on the base. - Fastly entry point: `main.rs` takes the base's lifecycle. `finalize` collects the navigation's page-phase lookups inside the async EC finalize under its one `block_on`, using the partner registry built once in `AppState`. After-response lookups run in post-delivery work, after pull sync, on the post-delivery client and clock. Lookups still travel through the request-local slot, because an in-flight batch is not `Send`; the slot is now cleared before each request and after delivery. - Fastly app: keeps the base's local 501/405 identify and batch-sync answers, publisher origin options and response-body scope owner, with PR #1487's partner registry, identity coordinators, navigation permit and dispatch. - Other adapters, core call sites and tests: kept both sides; PR #1487's tests that used the removed platform HTTP client or a custom auction provider now use the scripted outbound client, and the Fastly identity tests route with the host geo lookup the entry point's `prepare` runs. - Cargo: the lockfile adds only the identity-providers crate to the base's. - Docs and CHANGELOG: auction collection text follows the base; PR #1487's identity entries and spec now describe the EdgeZero batch transport.
|
For the identity integration chain (#1608), this PR now stacks on #1526, which carries #1525, #1614 and #1506's EdgeZero cutover. Two merges port it onto that base:
Behavior that changes with the move onto EdgeZero's client:
Tests changed beyond the async conversion: the transport tests run on The base is now Checked locally at |
#1487's head 64674f9 stacks on #1526's b53c6e2 and merged #1557's 76e1dd5; each merge resolved its own conflicts, so this merge needs no resolution. It brings server-side identity provider collection with one page-load write (#1567), with lookups on EdgeZero's outbound client and the KV code on #1526's async store.
Brings PR #1488's vendor identity providers onto PR #1487's new head, which carries PR #1487 onto PR #1526 (EdgeZero cutover, local partner API answers, HTTP auction opt-in and awaitable EC persistence). How each area was resolved: - Coordinator: PR #1488's cookie reads, GDPR source exclusion, opt-out memory and merged results run inside PR #1487's batch-shaped dispatch. The lookups a navigation still needs are built first and start together in one EdgeZero batch; `PendingLookups::is_empty` counts started requests along with cookie, generated and early results. - Fastly entry point: `main.rs` takes PR #1487's EdgeZero version. The provider cookies PR #1488 hands to after-response lookups go into the after-response `LookupContext` in post-delivery work; the request-side cookie capture merged unchanged. - Async KV: PR #1488's new write-plan and identify tests await the store. - Providers crate, CLI and settings: merged without conflict; providers stay pure and make no HTTP calls of their own. - Cargo: the textual merge pointed `crypto-common` at a `getrandom` the lockfile no longer held; cargo resolved it to the locked 0.4.2. The rest of the lockfile delta is PR #1488's own.
Brings PR #1489's pull-sync provider and EID mirror onto PR #1488's new head, which carries PR #1487 onto PR #1526 (EdgeZero cutover, local partner API answers, HTTP auction opt-in and awaitable EC persistence). How each area was resolved: - Pull sync: core `ec/pull_sync.rs` stays deleted, so the base's EdgeZero group sender and its tests go with it. Pull sync runs as PR #1489's `pull-sync` after-response provider, sent through the post-delivery client in the lookup batch. Post-delivery work is now telemetry, then after-response lookups; the separate pull-sync step, its graph factory and the unused clock are removed, and PR #1489's "nothing due" check runs before the live row read. `require_identity_graph`, now used only by tests, is test-only. - Coordinator: PR #1489's dispatch order and per-navigation pull-sync cap run inside PR #1487's batch-shaped dispatch; the cap counts each queued pull-sync request. - Fastly app: `EcFinalizeState` drops both the old `services` and `is_real_browser`; the bot gate moves into after-response capture as in PR #1489, with the base's request services and async batch sync. - Async KV and finalize: PR #1489's removals in `kv.rs` (snapshot partner upsert) and the pull-sync marker stand; the mirror and the inner finalize are awaited, and PR #1489's new tests await the store. - Publisher: keeps the base's EdgeZero origin request without the marker preload inputs PR #1489 removed. - Providers crate: PR #1489's pull-sync providers and PR #1488's SharedID cookie defaulting both apply when the phase coordinators are built. - Docs and CHANGELOG: the base's auction wording with PR #1489's identity bullets; the onboarding step names after-response lookups.
…ntity #1489's head a7262c7 stacks on #1488 (8183c3c), which stacks on #1487 (64674f9); each merged the one below it and resolved its own conflicts, so this merge needs no resolution. It completes the provider stack (#1567): the vendor identity providers and cookie hooks (#1488), and the bounded identity mirror with pull sync as an after-response provider on EdgeZero's outbound client (#1489).
…ity plan Both integration branches now carry every in-review issue's non-draft pull request. DEC-20 records the new rule that a conflict is fixed in the pull request that lands second, and DEC-21 the merge order the user chose: the Trusted Server pull requests stack linearly on #1506 (stack 1621), so T1, #1525 and #1526 land after the cutover. The spec's Integration branches section and the plan's State section list the new heads, and the wave rows for T1, #1525, #1526, #1506 and #1487 follow the new order.
Summary
atype,ext) and serves them unchanged into OpenRTB and Prebid, so bidders see the same shape the provider returned.This PR registers no provider kinds yet, so none of the provider paths run in production until the vendor PR lands. Follow-up PRs stack on this branch: mirror/pull sync (#1489), then the per-vendor provider implementations (#1488).
Changes
ec/eid_types.rs,ec/kv_types.rs,ec/kv.rs,openrtb.rs,ec/prebid_eids.rsec/write_plan.rs,ec/page_load.rs,ec/finalize.rs,publisher.rsec/provider.rs,ec/coordinator/*,ec/registry.rs,settings.rs,config.rs, newtrusted-server-identity-providerscrate, adapterapp.rs/main.rsec/coordinator/after_response.rs, adaptermain.rsdocs/guide/configuration.md,docs/guide/edge-cookies.md,docs/guide/integrations/prebid.md,trusted-server.example.tomlec.identity_providersconfiguration docs and exampledocs/superpowers/specs/2026-10-30-s2s-eid-collection-design.mdCollection
collect = "page"(default) runs before the response withinidentity_budget_ms(default 200).collect = "after_response"runs after the response is sent withinidentity_after_response_budget_ms(default 2000).tcf_vendor_id, when set, is checked only where GDPR applies or a TCF string is present. A KV read failure skips every provider. When a provider's vendor consent is denied, its stored IDs are also withheld from bidders,x-ts-eidsand/identify.opt_outsfield on the row, and the provider is not called again until it lapses. Older builds ignore the field.rate_limit_per_houris per EC hash, so clearing cookies does not reset it. On Fastly it is enforced per minute with a minimum of one, so values below 60 behave like 60.Storage and writes
ts-eidscookie, so an opted-out ID cannot come back from the browser.main) and expires the cookie once every tombstone is in place.row_fulland continues.Behaviour changes from
mainSec-Fetch-Dest: documentandSec-Fetch-Mode: navigate, the same check that permits its row write. Clients that send no Fetch Metadata no longer get a new EC./auctionor page-bids request body go to bidders but are never persisted.user.idand no provider lookup runs for it until a later navigation finds its row.Cleanup
ProviderResponsedebug output hides the body and any request context.Closes
Part of #1246 and #1245; the provider implementations that close them come in follow-up PRs.
Spec
Spec:
docs/superpowers/specs/2026-10-30-s2s-eid-collection-design.md, included in this PR.Test plan
cargo test-fastly && cargo test-fastly-reuse && cargo test-axum && cargo test-cloudflare && cargo test-spinclippy-fastly,-axum,-cloudflare,-cloudflare-wasm,-spin-native,-spin-wasm,-cli,-codegen)cargo fmt --all -- --checkscripts/lint-wasm-clock.shcd crates/trusted-server-js/lib && npx vitest run(plusnpm run formatandnpm run lint)./scripts/test-cli.sh, domain lint againstorigin/main, CI Node pin guard, docs-proposal scriptsfastly compute serveChecklist
unwrap()in production code — useexpect("should ...")logmacros (notprintln!)Closes #1567