Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 13 additions & 12 deletions docs/reporting-provisional-observations.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,13 @@

This caller-ownership release requires all reporting readers, admission and workers to stop during upgrade. Earlier additive or rolling procedures below apply only to pre-ownership releases; use the [maintenance and recovery contract](reporting-caller-ownership-migration.md) for retained state.

The producer records every successful scheduled provisional read as a new
immutable revision, including a read whose rows are unchanged. A durable
The producer records every successful scheduled provisional read as a durable
observation. Changed content creates an immutable revision; unchanged content
retains the current snapshot and advances the checkpoint without a notification. A durable
reservation freezes the acquisition before source work starts. Its successful
observation, revision rows, ledger change, notifications and next checkpoint
commit together. A retry of the same reserved acquisition returns the existing
revision instead of appending another observation.
observation and next checkpoint commit together with any revision rows, ledger
change and notification. A retry of the same reserved acquisition returns the
existing observation instead of appending another one.

This is a producer and persistence change. Complete service rollout qualification
is still pending, including activation of previously retired work, source-object
Expand Down Expand Up @@ -54,20 +55,20 @@ same atomic operations. Custom producer stores must implement
`ProvisionalObservationStore` from `adcp.reporting.ledger.provisional` together
with `RestatementCheckpointStore`.

The packaged `reporting/ledger/required_provisional_schema.json` describes all
27 objects added by this extension. Bootstrap and each reservation, observation
The packaged `reporting/ledger/required_provisional_schema.json` describes the
required objects. Bootstrap and each reservation, observation
read, and atomic observation commit validate those objects, including immutable
triggers and their function definitions. Validation also runs with notifications
disabled. Missing, disabled or changed objects fail closed with
`PROVISIONAL_SCHEMA_UNREADY`; reinstall or repair the schema before resuming
production. Notification readiness alone does not establish observation readiness:
historical notification-only schemas remain valid for their original operations.

Existing revision tables, exact-read row layout and pinned migration manifests
are unchanged. Each observation has its own revision ID and revision-specific
wire content hash. Unchanged source content retains its source content identity;
it does not mean that the revision ID or wire hash is reused. Historical readers
can still read rows directly from `reporting_revision_rows` for every revision.
Existing revision tables and exact-read row layout are unchanged. An unchanged
observation points to the retained snapshot revision. The observation table's
revision ID uniqueness constraint is removed by `create_schema()` so multiple
observations can reference the same immutable revision. Historical readers can
still read rows directly from `reporting_revision_rows` for every revision.

This change retains per-revision row payloads. Content-addressed reuse of staged
source objects is a separate pending change; complete payload deduplication has
Expand Down
36 changes: 36 additions & 0 deletions docs/reporting-source-stable-finality.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
# Source stable package finality

ESA's paid media package definition requires a 90 day source requery window,
content change publication, and official close only after one day of unchanged
source data at or beyond the 90 day age. Configure the SDK's provisional
snapshot offering with `restatement_window="P90D"`, a source safe
`restatement_cadence`, `official_close_lag="P90D"`, and
`official_stability_duration="P1D"`. Keep the provisional offering and add a
distinct `AuthoritativeOfferingV1` for the official read. Set its
`correction_policy="immutable_correction"` and choose a correction window that
matches the definition. Wire both offering IDs into `ProducerOfferings`.

The snapshot adapter should return the complete package rows and control totals
on every scheduled requery, including an explicit zero-row result when the
package has no activity. An unchanged read advances the durable observation,
checkpoint and next due time; it does not create a revision, ledger change or
notification. A changed read publishes a snapshot that supersedes the current
snapshot. The final snapshot recheck at the 90 day boundary is inclusive.

For an official read, the source adapter must return `InlineFetchResult` with
`finality_evidence_ref` identifying authoritative source evidence and
`finality_stable_since` naming when that source content last changed. The
producer requires both, checks that the source has been stable for the declared
duration, and waits for the period age, provisional boundary and last changed
snapshot's stability duration. A provisional observation alone never establishes
official finality. An official attempt with missing or premature evidence records
a nonpublishing checkpoint, so its retry uses a fresh source execution key and
read cutoff. The official offering's readiness fields should reflect when
the authoritative source can answer. An official correction after close must
be published through the existing adjustment flow, not as another snapshot or
replacement official revision.

ESA should adopt the first SDK release containing this change, expected to be
`adcp==10.0.0b21` after its release. The PostgreSQL producer also needs the
matching `adcp-reporting-storage` release containing the observation constraint
migration. Do not deploy the SDK change with an earlier storage package.
Original file line number Diff line number Diff line change
Expand Up @@ -1805,12 +1805,12 @@ async def get_provisional_observation(
async def commit_provisional_observation(
self,
observation: ProvisionalObservation,
revision: ReportingRevisionRecord,
revision: ReportingRevisionRecord | None,
rows: Sequence[dict[str, Any]],
) -> ReportingRevisionRecord:
acquisition = observation.acquisition
key = (acquisition.account_id, acquisition.obligation_id, acquisition.ordinal)
if (
if revision is not None and (
revision.account_id != acquisition.account_id
or revision.reporting_obligation_id != acquisition.obligation_id
or revision.reporting_revision_id != observation.revision_id
Expand Down Expand Up @@ -1840,6 +1840,7 @@ async def commit_provisional_observation(
if (
retained_observation.acquisition != acquisition
or existing[0] != observation.revision_id
or retained_observation.manifest_json != observation.manifest_json
):
raise LedgerConflictError("OBSERVATION_CONFLICT", "observation replay differs")
retained_revision = await self.get_revision(
Expand All @@ -1851,7 +1852,11 @@ async def commit_provisional_observation(
)
# Reuse the immutable publication replay checks while holding
# the observation's account lock and transaction.
return await self.commit_revision(revision, rows)
return (
await self.commit_revision(revision, rows)
if revision is not None
else retained_revision
)
reserved = await (
await connection.execute(
"SELECT payload FROM reporting_provisional_acquisitions"
Expand All @@ -1867,7 +1872,38 @@ async def commit_provisional_observation(
)
if checkpoint is not None and checkpoint.next_observation != acquisition.ordinal:
raise LedgerConflictError("OBSERVATION_CONFLICT", "observation ordinal changed")
committed = await self.commit_revision(revision, rows)
if revision is None:
if acquisition.predecessor_revision_id != observation.revision_id:
raise LedgerConflictError(
"OBSERVATION_CONFLICT", "snapshot predecessor differs"
)
committed = await self.get_revision(
account_id=acquisition.account_id,
reporting_revision_id=observation.revision_id,
)
if (
committed is None
or committed.finality != "snapshot"
or committed.reporting_obligation_id != acquisition.obligation_id
):
raise LedgerConflictError(
"HISTORY_UNAVAILABLE", "snapshot predecessor is missing"
)
current = await (
await connection.execute(
"SELECT 1 FROM reporting_revisions"
" WHERE account_id=%s AND reporting_obligation_id=%s"
" AND finality='snapshot' AND supersedes_reporting_revision_id=%s"
" LIMIT 1",
(*key[:2], observation.revision_id),
)
).fetchone()
if current is not None:
raise LedgerConflictError(
"OBSERVATION_CONFLICT", "snapshot predecessor is no longer current"
)
else:
committed = await self.commit_revision(revision, rows)
await self.record_restatement_checkpoint(
RestatementCheckpoint(
acquisition.account_id,
Expand All @@ -1881,7 +1917,7 @@ async def commit_provisional_observation(
"INSERT INTO reporting_provisional_observations"
" (account_id,reporting_obligation_id,ordinal,reporting_revision_id,payload)"
" VALUES (%s,%s,%s,%s,%s::jsonb)",
(*key, revision.reporting_revision_id, _json(observation.to_wire())),
(*key, observation.revision_id, _json(observation.to_wire())),
)
return committed

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ CREATE TABLE IF NOT EXISTS reporting_provisional_observations (
account_id TEXT COLLATE "C" NOT NULL,
reporting_obligation_id TEXT COLLATE "C" NOT NULL,
ordinal BIGINT NOT NULL,
reporting_revision_id TEXT COLLATE "C" NOT NULL UNIQUE,
reporting_revision_id TEXT COLLATE "C" NOT NULL,
payload JSONB NOT NULL,
PRIMARY KEY (account_id, reporting_obligation_id, ordinal),
CONSTRAINT provisional_observation_revision_fk
Expand All @@ -32,6 +32,9 @@ CREATE TABLE IF NOT EXISTS reporting_provisional_observations (
(account_id, reporting_obligation_id, ordinal)
);

ALTER TABLE reporting_provisional_observations
DROP CONSTRAINT IF EXISTS reporting_provisional_observations_reporting_revision_id_key;

CREATE OR REPLACE FUNCTION reporting_provisional_immutable()
RETURNS trigger LANGUAGE plpgsql AS $$
BEGIN
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -67,10 +67,6 @@
"enabled": true,
"fingerprint": "6e0009dfd0c8c76adc077b6f9e8bc866b476342946a107c8cba342ee3c6ec602"
},
"constraint:reporting_provisional_observations.reporting_provisional_observations_reporting_revision_id_key": {
"enabled": true,
"fingerprint": "dface19184e3b1196ac3f05865ce9fd2c1fde32593a604fd8e59e0e7b37525e7"
},
"function:reporting_provisional_immutable()": {
"enabled": true,
"fingerprint": "5f46da14407d6b8acab20bf770940bcae96698b27a941eab2d1f4aa5c271484b"
Expand All @@ -87,10 +83,6 @@
"enabled": true,
"fingerprint": "cc1a138288cf600afcffe140e7001d4e44d1a762ba2b6feeea6c5615c2e1c204"
},
"index:reporting_provisional_observations.reporting_provisional_observations_reporting_revision_id_key": {
"enabled": true,
"fingerprint": "423b8e84d59c9f7c8a580f1b8f97058e964b191a9c4a2dfd0e1e95e00b1872c2"
},
"table:reporting_provisional_acquisitions": {
"enabled": true,
"fingerprint": "1f824779ff80f110344420b019786663d8c9beaad230da90e0439795e734ccda"
Expand Down
33 changes: 32 additions & 1 deletion src/adcp/reporting/inline_source.py
Original file line number Diff line number Diff line change
Expand Up @@ -284,6 +284,12 @@ class InlineFetchResult:
scheduling correctly after a restart.
"""

finality_evidence_ref: str | None = field(default=None, kw_only=True)
"""Source evidence reference for an authoritative close."""

finality_stable_since: datetime | None = field(default=None, kw_only=True)
"""Source attestation of when authoritative content last changed."""

cell_availability: Mapping[str, Mapping[str, MetricEvidence]] | None = field(
default=None, kw_only=True
)
Expand Down Expand Up @@ -819,6 +825,17 @@ async def execute(
unavailable_status="delayed",
)

if request.require_finality_evidence and (
not answer.finality_evidence_ref or answer.finality_stable_since is None
):
return ReportingSourceExecutorResult.failed(
ReportingSourceErrorV1(
code="PROVIDER_TRANSIENT",
retry="retryable",
safe_message="source has not supplied official finality evidence",
)
)

return await self._publish(request, answer, cancel=cancel)

# -- fetch ----------------------------------------------------------
Expand Down Expand Up @@ -1021,6 +1038,8 @@ async def _publish(
control_totals=control_totals,
warnings=warnings,
provisional_until=result.provisional_until,
finality_evidence_ref=result.finality_evidence_ref,
finality_stable_since=result.finality_stable_since,
)

manifest_bytes = encode_source_batch_manifest_v1(manifest)
Expand Down Expand Up @@ -1196,6 +1215,8 @@ def _seal_manifest(
control_totals: list[SourceControlTotalV1],
warnings: list[str],
provisional_until: datetime | None,
finality_evidence_ref: str | None,
finality_stable_since: datetime | None,
) -> SourceBatchManifestV1:
coverage = SourceBatchCoverageV1(
denominator_fingerprint=request.coverage.denominator_fingerprint,
Expand Down Expand Up @@ -1230,9 +1251,19 @@ def _seal_manifest(
basis=(
"provisional_observation"
if request.publication_class == "PROVISIONAL_SNAPSHOT"
else "elapsed_settlement_window"
else (
"provider_declared"
if finality_evidence_ref
else "elapsed_settlement_window"
)
),
observed_at=finality_at,
evidence_ref=finality_evidence_ref
if request.publication_class == "AUTHORITATIVE"
else None,
stable_since=finality_stable_since
if request.publication_class == "AUTHORITATIVE"
else None,
provisional_until=(
provisional_until
if request.publication_class == "PROVISIONAL_SNAPSHOT"
Expand Down
Loading
Loading