Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,7 @@ To add or change a rule, read `docs/engineering/skills/rules-registry.md`.

### Observability

- Checked: `sim.observability.log-fields-through-log-extra`
- `sim.observability.failures-never-break-requests`: Invalid observability configuration fails at build or deploy time, and telemetry failures at runtime never fail a request or computation. (doc-only, not checked: follow it yourself)
- `sim.observability.header-transport-only`: observability_id travels only in the X-PolicyEngine-Observability-Id header, never in a simulation JSON body, and polling never invents one. (doc-only, not checked: follow it yourself)
- `sim.observability.id-is-diagnostic-only`: observability_id has diagnostic meaning only: never use it for idempotency, authorization, database identity, routing or business logic. (doc-only, not checked: follow it yourself)
Expand Down
1 change: 1 addition & 0 deletions changelog.d/c1-34-v2-telemetry-record-ids.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Each v2 Modal function's telemetry operation now names the API v1 row its call works on and its own Modal call, so an operation that Modal interrupted leads straight to its row: `database_table` (`household_jobs`, `report_runs` for a report run or a household report run, or `simulation_runs`), `database_row_id` and `modal_call_id`. A US segment worker names the simulation run of the worker that started it. The IDs go to the operation's log records and span, not to metrics.
1 change: 1 addition & 0 deletions changelog.d/c1-34-v2-telemetry-record-ids.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The Simulation Entrypoint's log records now carry their fields. 31 of its 34 log calls that pass fields passed them only as standard-library `extra=` keys, which the observability runtime doesn't emit, so their records in Cloud Logging named no job, report, run or error type. Every log call now passes its fields through `log_extra` in the shared observability library, and a call about an API v1 row names it with `database_table` (`household_jobs`, `reports`, `report_runs` or `simulations`) and `database_row_id`: a key such as `household_job_id` would be dropped by the runtime's key filter. A budget window's `started_years` are now joined into one string, since the runtime drops lists. The Entrypoint also sets its package logger to INFO, so its INFO records (a deferred budget-window year, a refused bearer token) reach the runtime: the root logger's WARNING level discarded them.
30 changes: 30 additions & 0 deletions docs/engineering/skills/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,12 +16,42 @@ Use each identifier for its defined scope:
| `evaluation_id` | One Stage 12 report | Stage 12 report construction | Yes, as functional report state |
| `simulation_execution_id` | One Stage 12 baseline or reform simulation | Stage 12 coordinator | Yes, as functional simulation state |
| `submission_claim_id` | One API v1 attempt to acquire ownership of a simulation submission | API v1 | Only where the simulation request is stored |
| `database_table` | The API v1 table holding the row a v2 Modal function call, or a Simulation Entrypoint log record, is about: `household_jobs`, `report_runs` (a report run or a household report run), `simulation_runs`, `reports` or `simulations` | The code writing the record, as a fixed name | No: telemetry only |
| `database_row_id` | That row's UUID | The simulation service, which creates the row | Yes, as the row's ID |
| `modal_call_id` | One Modal function call of a v2 Modal function | Modal | Only where the row records the call that claimed it; a segment worker's call is recorded nowhere, and its `database_row_id` names the simulation run its simulation worker's call claimed |

Each v2 Modal function's operation (`src/modal/v2_app.py`) names its
`modal_call_id` and, when its call works on a detailed-save row, that row's
`database_table` and `database_row_id`; a US segment worker names the
simulation run its simulation worker records. They go to the operation's log
records and span only, never to metrics, where each distinct ID would make a
separate series. The observability package drops every attribute whose key
contains `household`, `reform`, `payload` or `person`, among others, so no key
for these IDs names one. Do not use `job_id` for a v2 call: it names one
annual Modal invocation of the v1 executor.

`observability_id` is a canonical UUID string and has diagnostic meaning only.
Do not use it for idempotency, authorization, database identity, routing, or
business logic. Do not create one for health checks, version queries, invalid
requests, missing jobs, or ordinary status requests.

## Standard-library log fields

The observability runtime's standard-library log handler emits only a log
record's `policyengine_attributes`: any other key passed in `extra=` stays on
the Python `LogRecord` and never reaches Cloud Logging. In the Simulation
Entrypoint, pass every log call's fields through `log_extra(...)` from
`policyengine_simulation_observability.log_records`, which puts them in both
places (a field named like a `LogRecord` attribute, such as `name`, goes only
to `policyengine_attributes`, since `logging` refuses it), and name a row with `database_row_attributes(table, row_id)` (rule
`sim.observability.log-fields-through-log-extra`; its check reads only the
field names written as keywords in the `log_extra(...)` call, not keys unpacked
from a dictionary with `**`). Each field must be a
string, a number or a boolean; the runtime drops a list. The Entrypoint sets
its package logger to INFO: the root logger stays at WARNING, which would
discard INFO records before the handler sees them. Some Stage 12 runtime and
contract-library logs still pass plain `extra=` fields, which are not emitted.

## HTTP transport

Transport `observability_id` only in
Expand Down
10 changes: 8 additions & 2 deletions docs/generated/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ Every rule that governs `policyengine-sim-api` is listed here with how it is enf

| Tier | enforced | ratcheted | advisory | doc-only | Total |
| --- | ---: | ---: | ---: | ---: | ---: |
| mechanical | 65 | 0 | 0 | 14 | 79 |
| mechanical | 66 | 0 | 0 | 14 | 80 |
| metric | 1 | 0 | 0 | 0 | 1 |
| judgment | 0 | 0 | 0 | 0 | 0 |

**Blocking:** 66 of 80 rules.
**Blocking:** 67 of 81 rules.

## Gaps

Expand Down Expand Up @@ -311,6 +311,7 @@ flowchart LR
| `sim.observability.failures-never-break-requests` | Invalid observability configuration fails at build or deploy time, and telemetry failures at runtime never fail a request or computation. | mechanical | doc-only | no | nothing | — | — |
| `sim.observability.header-transport-only` | observability_id travels only in the X-PolicyEngine-Observability-Id header, never in a simulation JSON body, and polling never invents one. | mechanical | doc-only | no | nothing | — | — |
| `sim.observability.id-is-diagnostic-only` | observability_id has diagnostic meaning only: never use it for idempotency, authorization, database identity, routing or business logic. | mechanical | doc-only | no | nothing | — | — |
| `sim.observability.log-fields-through-log-extra` | Every Simulation Entrypoint log call that passes fields passes them through log_extra, so they reach the emitted record, and no field name written as a keyword in that call contains a word the observability package drops (household, reform, payload, person and others). | mechanical | enforced | yes | pytest `projects/policyengine-simulation-entry/tests/test_log_fields.py` | `pr.yml:test` | yes |
| `sim.observability.registered-stages-only` | Span names come from the stage plans in policyengine_simulation_observability.stages; runtime code never adds span-name string literals. | mechanical | doc-only | no | nothing | — | — |

```mermaid
Expand All @@ -319,10 +320,15 @@ flowchart LR
gap_unenforced>"not enforced"]
r_sim_observability_header_transport_only("sim.observability.header-transport-only (doc-only)")
r_sim_observability_id_is_diagnostic_only("sim.observability.id-is-diagnostic-only (doc-only)")
r_sim_observability_log_fields_through_log_extra("sim.observability.log-fields-through-log-extra (enforced)")
e_pytest_projects_policyengine_simulation_entry_tests_test_log_fields_py["pytest projects/policyengine-simulation-entry/tests/test_log_fields.py"]
j_CI_pr_yml_test["CI pr.yml:test"]
r_sim_observability_registered_stages_only("sim.observability.registered-stages-only (doc-only)")
r_sim_observability_failures_never_break_requests -.-> gap_unenforced
r_sim_observability_header_transport_only -.-> gap_unenforced
r_sim_observability_id_is_diagnostic_only -.-> gap_unenforced
r_sim_observability_log_fields_through_log_extra --> e_pytest_projects_policyengine_simulation_entry_tests_test_log_fields_py
e_pytest_projects_policyengine_simulation_entry_tests_test_log_fields_py --> j_CI_pr_yml_test
r_sim_observability_registered_stages_only -.-> gap_unenforced
```

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
"""Telemetry fields that name an API v1 row, and log fields that are emitted.

``policyengine-observability``'s standard-library log handler emits only a
log record's ``policyengine_attributes``; every other key passed in
``extra=`` stays on the standard-library record alone. It then drops each
attribute whose key contains ``household``, ``reform``, ``payload`` or
``person``, among other words. So a row is named by the table holding it
(``database_table``) and its UUID (``database_row_id``), never by a key such
as ``household_job_id``.
"""

from __future__ import annotations

import logging
from typing import TYPE_CHECKING, Final, Literal

if TYPE_CHECKING:
from uuid import UUID

DATABASE_TABLE_ATTRIBUTE = "database_table"
DATABASE_ROW_ID_ATTRIBUTE = "database_row_id"
MODAL_CALL_ID_ATTRIBUTE = "modal_call_id"

# The API v1 tables whose rows telemetry names.
DatabaseTable = Literal[
"household_jobs", "report_runs", "reports", "simulation_runs", "simulations"
]
LogFieldValue = str | int | float | bool

# ``logging`` raises on an ``extra`` key naming one of a record's own
# attributes, or the two it sets while formatting.
_LOG_RECORD_ATTRIBUTES: Final = frozenset(
vars(logging.LogRecord("", logging.INFO, "", 0, "", None, None))
) | {"message", "asctime"}


def database_row_attributes(table: DatabaseTable, row_id: UUID) -> dict[str, str]:
"""Return the attributes naming row ``row_id`` of API v1's ``table``."""
return {DATABASE_TABLE_ATTRIBUTE: table, DATABASE_ROW_ID_ATTRIBUTE: str(row_id)}


def log_extra(**fields: LogFieldValue) -> dict[str, object]:
"""Return a logging call's ``extra`` whose ``fields`` reach the emitted record.

Each field also stays on the standard-library record, where other
handlers and tests read it, unless its name is one of the record's own
attributes: ``logging`` would raise for that one, and these calls are
often in error handlers. A field must be a string, a number or a
boolean: the runtime drops anything else.
"""
record_fields = {
name: value
for name, value in fields.items()
if name not in _LOG_RECORD_ATTRIBUTES
}
return {**record_fields, "policyengine_attributes": dict(fields)}
137 changes: 137 additions & 0 deletions libs/policyengine-simulation-observability/tests/test_log_records.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
"""Standard-library log fields that reach the observability runtime's records."""

from __future__ import annotations

import logging
from uuid import UUID

import pytest
from policyengine_observability import (
CustomLogDestination,
DeploymentIdentity,
LoggingConfig,
ObservabilityConfig,
OTelConfig,
ServiceIdentity,
configure,
)

ROW_ID = UUID("00000000-0000-4000-8000-00000000a001")


@pytest.fixture
def written_records():
"""Configure a runtime that keeps every record, standard-library ones too."""
records: list[dict] = []

class ListRecordWriter:
def write(self, record: dict) -> None:
records.append(record)

runtime = configure(
ObservabilityConfig(
service=ServiceIdentity(
name="policyengine-simulation-observability-test",
namespace="policyengine.test",
version="test",
role="test",
),
deployment=DeploymentIdentity(environment="test", platform="local"),
logging=LoggingConfig(
destinations=(
CustomLogDestination(
name="test-records",
writer_factory=ListRecordWriter,
delivery="inline",
),
),
capture_standard_library=True,
),
otel=OTelConfig(enabled=False),
)
)
yield records
runtime.shutdown()


def _record_with_message(records: list[dict], message: str) -> dict:
(record,) = [item for item in records if item.get("message") == message]
return record


def test_fields_in_log_extra_reach_the_emitted_record(written_records) -> None:
from policyengine_simulation_observability.log_records import (
database_row_attributes,
log_extra,
)

logging.getLogger("policyengine_simulation_entry.test").warning(
"household_call_unconfirmed",
extra=log_extra(
error_type="ConnectionError",
**database_row_attributes("household_jobs", ROW_ID),
),
)

record = _record_with_message(written_records, "household_call_unconfirmed")
assert record["attributes"] == {
"error_type": "ConnectionError",
"database_table": "household_jobs",
"database_row_id": str(ROW_ID),
}


def test_log_extra_keeps_its_fields_on_the_standard_library_record(caplog) -> None:
# Other handlers, and tests, read them there.
from policyengine_simulation_observability.log_records import log_extra

with caplog.at_level(logging.INFO):
logging.getLogger("policyengine_simulation_entry.test").info(
"budget_window_year_deferred", extra=log_extra(year=2027)
)

(record,) = caplog.records
assert record.year == 2027
assert record.policyengine_attributes == {"year": 2027}


def test_plain_extra_fields_are_not_emitted(written_records) -> None:
# Why ``log_extra`` exists: the runtime's log handler keeps only
# ``policyengine_attributes``.
logging.getLogger("policyengine_simulation_entry.test").warning(
"economy_state_unreadable", extra={"error_type": "OperationalError"}
)

record = _record_with_message(written_records, "economy_state_unreadable")
assert "attributes" not in record


def test_a_household_key_is_dropped_even_in_policyengine_attributes(
written_records,
) -> None:
# Why rows are named by ``database_table`` and ``database_row_id``.
from policyengine_simulation_observability.log_records import log_extra

logging.getLogger("policyengine_simulation_entry.test").warning(
"household_job_not_failed",
extra=log_extra(household_job_id=str(ROW_ID), error_type="OperationalError"),
)

record = _record_with_message(written_records, "household_job_not_failed")
assert record["attributes"] == {"error_type": "OperationalError"}


def test_a_field_named_like_a_log_record_attribute_is_emitted_without_raising(
written_records,
) -> None:
# ``logging`` refuses an ``extra`` key that names a LogRecord attribute,
# and these calls sit in error handlers, where raising would turn a
# handled failure into an unhandled one.
from policyengine_simulation_observability.log_records import log_extra

logging.getLogger("policyengine_simulation_entry.test").warning(
"budget_window_not_started", extra=log_extra(name="window", module="entry")
)

record = _record_with_message(written_records, "budget_window_not_started")
assert record["attributes"] == {"name": "window", "module": "entry"}
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
OBSERVABILITY_ID_HEADER,
normalize_observability_id,
)
from policyengine_simulation_observability.log_records import log_extra
from policyengine_simulation_observability.observability import (
init_simulation_observability,
)
Expand Down Expand Up @@ -126,6 +127,8 @@
]

logger = logging.getLogger(__name__)
# Every module of the package logs under this logger.
PACKAGE_LOGGER_NAME = __name__.rpartition(".")[0]
SIMULATION_DATABASE_READINESS_TIMEOUT_SECONDS = 3.0
# The database ends each statement of a household route that runs longer.
HOUSEHOLD_STATEMENT_TIMEOUT_MILLISECONDS = 5000
Expand Down Expand Up @@ -174,12 +177,12 @@ async def request_context(
except Exception:
logger.exception(
"simulation_entry_unhandled_request",
extra={
"request_id": request_id,
"method": request.method,
"path": route_template(request),
extra=log_extra(
request_id=request_id,
method=request.method,
path=route_template(request),
**request_identifiers(request),
},
),
)
response = Response(
content="Internal Server Error",
Expand Down Expand Up @@ -257,10 +260,10 @@ async def submit_temporary_stage12_report(
except Exception as error: # noqa: BLE001 # every integration failure becomes a 503
logger.error( # noqa: TRY400 # Stage 12 failure logs carry only the error type; a traceback could expose request or calculation data
"stage12_direct_submission_failed",
extra={
"request_id": request.state.request_id,
"error_type": type(error).__name__,
},
extra=log_extra(
request_id=request.state.request_id,
error_type=type(error).__name__,
),
)
return _stage12_unavailable_response(
"Stage 12 direct execution is unavailable."
Expand Down Expand Up @@ -314,11 +317,11 @@ async def get_temporary_stage12_report(
except Exception as error: # noqa: BLE001 # any status-read failure becomes a 503
logger.error( # noqa: TRY400 # Stage 12 failure logs carry only the error type; a traceback could expose request or calculation data
"stage12_direct_status_failed",
extra={
"request_id": request.state.request_id,
"evaluation_id": str(evaluation_id),
"error_type": type(error).__name__,
},
extra=log_extra(
request_id=request.state.request_id,
evaluation_id=str(evaluation_id),
error_type=type(error).__name__,
),
)
return _stage12_unavailable_response(
"Stage 12 report status is unavailable."
Expand Down Expand Up @@ -717,6 +720,9 @@ async def lifespan(_: FastAPI) -> AsyncIterator[None]:
platform="google_cloud_run",
environment=runtime_settings.environment,
)
# The root logger stays at WARNING, which would discard this package's
# INFO records before the observability runtime's handler sees them.
logging.getLogger(PACKAGE_LOGGER_NAME).setLevel(logging.INFO)

runtime_backend = backend or OldGatewayBackend(runtime_settings, runtime=runtime)
authenticate = auth_dependency or CallerAuthenticator(runtime_settings, runtime)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
import jwt
from fastapi import Depends, HTTPException, status
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
from policyengine_simulation_observability.log_records import log_extra

from policyengine_simulation_entry.schemas import CallerIdentity

Expand Down Expand Up @@ -79,7 +80,7 @@ def __call__(
reason = type(error).__name__
logger.info(
"invalid_simulation_entry_bearer_token",
extra={"error_type": reason},
extra=log_extra(error_type=reason),
)
runtime.event(
"simulation_entry_auth_rejected",
Expand Down
Loading
Loading