Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
18beb50
Report run completion leaves its four cliff columns null (failing)
anth-volk Nov 17, 2026
36a5c94
Stop writing report_runs' four cliff columns on completion
anth-volk Nov 17, 2026
1c159cc
A worker calculates the cliff result its call names, once (failing)
anth-volk Nov 17, 2026
3a53c7a
Store and grant each simulation run's cliff calculations (failing)
anth-volk Nov 17, 2026
2f6cabf
Calculate a simulation run's cliff gap and share as its cliff result
anth-volk Nov 17, 2026
103a6d9
The stuck-run sweep fails a dead cliff calculation (failing)
anth-volk Nov 17, 2026
c087563
Sweep simulation runs' cliff results left by a call that ended
anth-volk Nov 17, 2026
3e5cdcd
Document each simulation run's cliff calculation
anth-volk Nov 17, 2026
e37c443
A context naming no cliff result keeps its JSON (failing)
anth-volk Nov 17, 2026
dbda6e2
Leave an unset cliff_result_id out of the call context's JSON
anth-volk Nov 17, 2026
64fbba2
A failed read during a split cliff calculation is logged (failing)
anth-volk Nov 17, 2026
c225773
Log a failed cliff result read during a split calculation
anth-volk Nov 17, 2026
ea9032e
Name cliff results wherever the docs list the sweep's and roles' tables
anth-volk Nov 17, 2026
beeb2d2
Expect the sweep's fourth kind and cliff_impacts' status column
anth-volk Nov 17, 2026
e9c9f50
Cover year and service contexts' cliff rows; tidy sweep wording
anth-volk Nov 17, 2026
c57a1f7
Test that merged region groups give the national cliffs; fix two texts
anth-volk Nov 17, 2026
6fdd18c
Grant cliff_impacts in the missing-privileges rule's proof fixtures
anth-volk Nov 17, 2026
19889bd
Test CliffImpact with the service's mapped classes (failing)
anth-volk Nov 18, 2026
a72ac62
Map CliffImpact in tables, beside the service's other classes
anth-volk Nov 18, 2026
d7bb735
Test a split cliff calculation's stop and its one-worker case
anth-volk Nov 18, 2026
b249742
A cliff calculation's operation names its cliff result (failing)
anth-volk Nov 18, 2026
4b7ece5
Name the cliff result in a cliff calculation's operation
anth-volk Nov 18, 2026
8f0a0f7
Pin that a cliff calculation's segments name no row
anth-volk Nov 18, 2026
495bf7e
Call API v1 by its name in the cliff result's text
anth-volk Nov 19, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/simulation-deploy.reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1187,8 +1187,8 @@ jobs:
MODAL_ENVIRONMENT: ${{ inputs.modal_environment }}
run: ../../.github/scripts/stage12-v2-default-verify.sh

# The scheduled sweep that fails report runs, simulation runs and
# household jobs nothing will finish. Its app is unversioned and its
# The scheduled sweep that fails report runs, simulation runs, household
# jobs and cliff results nothing will finish. Its app is unversioned and its
# schedule starts when it deploys, so it deploys here, after every
# check, with the release's Stage 12 default; if it fails, the next
# step puts the default back.
Expand Down
1 change: 1 addition & 0 deletions changelog.d/c1-38-run-cliff-results.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
A v2 simulation worker can now calculate a simulation run's own cliff result: a call whose context names a row of API v1's `cliff_impacts` calculates that run's simulation again with .py's cliff columns and records its weighted cliff gap and cliff share, once, on that row, and the stuck-run sweep fails one whose call ended. The worker's operation record names `cliff_impacts` and the row in telemetry. A report run's completion no longer writes `report_runs`' four cliff columns. Nothing starts a cliff calculation yet, and a report asking for cliffs is still refused.
8 changes: 4 additions & 4 deletions docs/engineering/skills/database-access.md
Original file line number Diff line number Diff line change
Expand Up @@ -170,8 +170,8 @@ and this repository has no migration chain of its own
- the Stage 12 comparison tables, whose live schema deployment validation
compares with the Stage 12 library's SQLAlchemy mappings.
- the tables the simulation service writes (`simulations`, `reports`,
`simulation_runs`, `report_runs`, `budget_window_runs` and
`household_jobs`), modelled in `libs/policyengine-simulation-persistence`.
`simulation_runs`, `report_runs`, `budget_window_runs`, `household_jobs`
and `cliff_impacts`), modelled in `libs/policyengine-simulation-persistence`.
The service connects as its own role, which policyengine-api's privilege
manifest grants.
- the policyengine-api tables a calculation is read from (`households`,
Expand All @@ -182,8 +182,8 @@ and this repository has no migration chain of its own
models of the tables it writes.
- the run tables the v2 Modal coordinator, workers, household functions and
stuck-run sweep write as the Modal worker role: `report_runs`,
`simulation_runs`, `budget_window_runs`, the result tables and
`household_jobs`. They use the same models, in
`simulation_runs`, `budget_window_runs`, the result tables,
`household_jobs` and `cliff_impacts`. They use the same models, in
`libs/policyengine-simulation-persistence` (`tables.py` and
`result_tables.py`, which `sim.persistence.models-match-public-api`
compares with the Public API's tables), through the stores in
Expand Down
11 changes: 7 additions & 4 deletions docs/engineering/skills/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,16 +16,19 @@ Use each identifier for its defined scope:
| `evaluation_id` | One Stage 12 report | Stage 12 report construction | Yes, as functional report state |
| `simulation_execution_id` | One Stage 12 baseline or reform simulation | Stage 12 coordinator | Yes, as functional simulation state |
| `submission_claim_id` | One API v1 attempt to acquire ownership of a simulation submission | API v1 | Only where the simulation request is stored |
| `database_table` | The API v1 table holding the row a v2 Modal function call, or a Simulation Entrypoint log record, is about: `household_jobs`, `report_runs` (a report run or a household report run), `simulation_runs`, `reports` or `simulations` | The code writing the record, as a fixed name | No: telemetry only |
| `database_table` | The API v1 table holding the row a v2 Modal function call, or a Simulation Entrypoint log record, is about: `household_jobs`, `report_runs` (a report run or a household report run), `simulation_runs`, `cliff_impacts`, `reports` or `simulations` | The code writing the record, as a fixed name | No: telemetry only |
| `database_row_id` | That row's UUID | The simulation service, which creates the row | Yes, as the row's ID |
| `modal_call_id` | One Modal function call of a v2 Modal function | Modal | Only where the row records the call that claimed it; a segment worker's call is recorded nowhere, and its `database_row_id` names the simulation run its simulation worker's call claimed |

Each v2 Modal function's operation (`src/modal/v2_app.py`) names its
`modal_call_id` and, when its call works on a detailed-save row, that row's
`database_table` and `database_row_id`; a US segment worker names the
simulation run its simulation worker records. They go to the operation's log
records and span only, never to metrics, where each distinct ID would make a
separate series. The observability package drops every attribute whose key
simulation run its simulation worker records. A segment of a cliff calculation
names no row: it joins the simulation worker's trace, whose operation names the
cliff result, or, if it starts too late to join, links to that trace with a span
link, so its log records then carry another trace ID. The attributes go to the
operation's log records and span only, never to metrics, where each distinct ID
would make a separate series. The observability package drops every attribute whose key
contains `household`, `reform`, `payload` or `person`, among others, so no key
for these IDs names one. Do not use `job_id` for a v2 call: it names one
annual Modal invocation of the v1 executor.
Expand Down
58 changes: 50 additions & 8 deletions docs/migration/stage-12-modal-worker-foundation.md
Original file line number Diff line number Diff line change
Expand Up @@ -367,6 +367,42 @@ stored output's descriptor without calculating, and for a run another call
holds, or one that failed, it returns `{"deduplicated": true,
"simulation_run_id", "status"}`.

**A simulation run's cliff result** (C1 · 24, answer 4b). Each simulation run
can have one cliff result, its cliff gap and cliff share, which every report
naming the run reads: a row of API v1's `cliff_impacts`, one per
calculation. Nothing starts one yet. The simulation service is to insert it
pending, start the country's simulation worker (`run_single_simulation_us` or
`run_single_simulation_uk`) with the run's own simulation input, built as a
standalone run's, and a context naming the row
(`Stage12InvocationContext.cliff_result_id`, left out of a context that names
none, which older workers refuse), and record the call (C1 · 39). The worker
claims the row as a run is claimed, and refuses, failing it with
`cliff_result_input_mismatch`, an input that is not the simulation of the
row's run on that run's year and versions; the calculation then checks those
versions are the installed bundle's, so a result never comes from other
versions than its run's. It plans the simulation with the country model's
default columns and .py's three cliff columns (`resolve_cliff_output_plan`),
calculates it again from its dataset and policy, since `cliff_gap` reads
marginal tax rates, which recalculate each household's net income with extra
earnings, and writes .py's cliff gap (the weighted sum of `cliff_gap`) and
cliff share (the weighted count of people `is_on_cliff` over the weighted
count of `is_adult`) only while it holds the row; a calculation with no adults
fails with `cliff_share_undefined`. A US national calculation is split into
the 20 region groups whenever its run's own calculation would be
(`should_segment_cliff_calculation`): each person's cliff columns depend only
on their own household's calculation, each group keeps its households'
weights, and the share is taken once, from the merged rows. A second start,
or a row another call holds or that finished, calculates nothing and returns
`{"deduplicated": true, "cliff_result_id", "status"}`; the call that completed
it returns its values again when Modal restarts it. A failure fails the row
with its redacted cause; the stuck-run sweep fails one whose call ended
without an outcome, and a failed row never counts, so the next request starts
another calculation. The worker writes nothing to the simulation run, its
output file or any report run. API v1's revision that adds
`cliff_impacts` must be applied in an environment before this release deploys
there: the stuck-run sweep lists the table on every pass, and the service's
`/ready` privilege check names it.

**What a simulation run stores** (C1 · 24, answer 4). A run's output file
depends on its simulation alone, never on a report naming it
(`resolve_simulation_output_plan`): every column that doesn't depend on the
Expand All @@ -375,10 +411,11 @@ columns; the labour-supply response columns only when the simulation's own
policy turns a response on; in the US, the two inputs .py's labour-supply
output reads from either side (`self_employment_income`,
`weekly_hours_worked`); and never the cliff columns, so a US national run keeps
its region-group split. Until cliffs come back as a calculation of each
simulation run (a later stage), a report run asking for them
(`include_cliffs: true`) fails before any worker starts, with
`cliff_impacts_unavailable`, and touches none of its simulation runs. The
its region-group split. Cliffs are a calculation of each simulation run of
their own (above). A report run asking for them (`include_cliffs: true`) still
fails before any worker starts, with `cliff_impacts_unavailable`, and touches
none of its simulation runs, and its completion never writes `report_runs`'
four cliff columns. The
coordinator gives each worker its simulation's
stored plan, and keeps the report's own plan, from its pair, for what the
aggregation reads. A reused or waited-for file must hold what the report reads
Expand Down Expand Up @@ -615,9 +652,12 @@ Segmentation applies by default when both simulation inputs describe a US
national society-wide calculation and the shared output plan does not require
cliff analysis or labor-supply responses. `segmented=false` selects the
single-container calculation explicitly. UK calculations, subnational US
calculations, cliff analysis, and labor-supply-response calculations also use
the single-container path. `segmented=true` is accepted, but it does not
override those numerical eligibility conditions. A US national simulation run
calculations, a report's cliff analysis, and labor-supply-response
calculations also use the single-container path. A simulation run's own cliff
calculation is the exception: it is split whenever its run's own calculation
is (`should_segment_cliff_calculation`), and its cliff gap and share are taken
from the merged rows. `segmented=true` is accepted, but it does not override
those numerical eligibility conditions. A US national simulation run
started on its own (role `standalone`) is segmented by the same rule. It never
uses the precomputed baseline files, which serve only a report run's
current-law baseline, planned as its simulation run stores its output, so it
Expand Down Expand Up @@ -947,7 +987,9 @@ report run and its simulation runs are), counting a job's age from its
creation: a pending job with no call
recorded ten minutes after its creation, a pending or running job once
Modal reports its recorded call ended, and any job still unfinished 24 hours
after its creation, whatever Modal reports.
after its creation, whatever Modal reports. It judges simulation runs' cliff
results last, by the same rule, counting from a result's first claim or,
while it is pending, its creation.

- Image: the CA store, the frozen `modal-simulation-image` dependency group
and the source packages, without the bundle's datasets or the Stage 12
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,9 +73,11 @@ class Stage12InvocationContext(StrictContractModel):

A detailed-save call works on the Public API's run tables instead of the
comparison path's records: a report coordinator's call names its report
run and both simulation runs in ``report_run``, and a simulation worker's
call names the one simulation run it claims in ``simulation_run_id``.
Its simulation output files go to their own bucket, which keeps them.
run and both simulation runs in ``report_run``, a simulation worker's
call names the one simulation run it claims in ``simulation_run_id``, and
a simulation worker's call that calculates a run's cliff gap and share
names the cliff result it claims in ``cliff_result_id``. Its simulation
output files go to their own bucket, which keeps them.
"""

contract_version: Literal[1] = 1
Expand All @@ -97,9 +99,14 @@ class Stage12InvocationContext(StrictContractModel):
production_function_call_id: ContractText | None = None
created_at: datetime
retention_expires_at: datetime
# Comparison runs leave both unset and record only the Stage 12 tables.
# Comparison runs leave all three unset and record only the Stage 12 tables.
report_run: ReportRunIds | None = None
simulation_run_id: UUID | None = None
# Left out of the JSON when unset: a worker deployed before the field
# refuses it, even as null, so every other call keeps its JSON.
cliff_result_id: UUID | None = Field(
default=None, exclude_if=lambda value: value is None
)

@model_validator(mode="after")
def validate_retention(self) -> Stage12InvocationContext:
Expand All @@ -111,21 +118,37 @@ def validate_retention(self) -> Stage12InvocationContext:
def validate_report_run_is_direct(self) -> Stage12InvocationContext:
"""Reject runs on a shadow run compared with a production call.

A call names a report run, for its coordinator, or a simulation run,
for a worker, not both.
A call names at most one of a report run, for its coordinator, a
simulation run, for a worker, and a cliff result, for a worker
calculating a run's cliffs.
"""
if self.records_runs and self.production_function_call_id:
msg = "a run compared with a production function call has no report run"
raise ValueError(msg)
if self.report_run is not None and self.simulation_run_id is not None:
msg = "a call names a report run or a simulation run, not both"
if len(self._named_records()) > 1:
msg = (
"a call names one of a report run, a simulation run and a "
"cliff result, not two"
)
raise ValueError(msg)
return self

@property
def records_runs(self) -> bool:
"""Whether this is a detailed-save call, which works on the run tables."""
return self.report_run is not None or self.simulation_run_id is not None
return bool(self._named_records())

def _named_records(self) -> tuple[object, ...]:
"""Return the report run, simulation run and cliff result the call names."""
return tuple(
record
for record in (
self.report_run,
self.simulation_run_id,
self.cliff_result_id,
)
if record is not None
)


ErrorCode = Annotated[str, Field(min_length=1, max_length=64)]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,21 @@ def test_a_later_year_is_calculated_on_its_own_year_and_runs() -> None:
assert later_context.retention_expires_at == LATER_CREATED_AT + timedelta(days=30)


def test_a_later_years_context_names_no_cliff_result_in_its_json() -> None:
"""A v2 application deployed before cliff results refuses the field, even null."""
report, context = _first_year()

_, later_context = year_calculation(
report,
context,
year=2029,
run_ids=LATER_YEAR_RUNS,
created_at=LATER_CREATED_AT,
)

assert "cliff_result_id" not in later_context.model_dump(mode="json")


def test_nothing_else_differs_between_the_years() -> None:
report, context = _first_year()

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
REPORT_RUN_ID = UUID("00000000-0000-0000-0000-0000000000a1")
BASELINE_RUN_ID = UUID("00000000-0000-0000-0000-0000000000b1")
REFORM_RUN_ID = UUID("00000000-0000-0000-0000-0000000000c1")
CLIFF_RESULT_ID = UUID("00000000-0000-0000-0000-0000000000f1")


def _context_fields(**overrides: Any) -> dict[str, Any]:
Expand Down Expand Up @@ -141,7 +142,7 @@ def test_only_a_detailed_save_call_records_runs() -> None:


def test_a_context_names_a_report_run_or_a_simulation_run_not_both() -> None:
with pytest.raises(ValidationError, match="report run or a simulation run"):
with pytest.raises(ValidationError, match="one of a report run, a simulation run"):
Stage12InvocationContext(
**_context_fields(
report_run=_run_ids().model_dump(mode="json"),
Expand All @@ -150,6 +151,62 @@ def test_a_context_names_a_report_run_or_a_simulation_run_not_both() -> None:
)


def test_a_cliff_workers_context_names_the_cliff_result_its_call_claims() -> None:
context = Stage12InvocationContext(
**_context_fields(cliff_result_id=str(CLIFF_RESULT_ID))
)

assert context.cliff_result_id == CLIFF_RESULT_ID
assert (context.report_run, context.simulation_run_id) == (None, None)
assert context.records_runs
restored = Stage12InvocationContext.model_validate(context.model_dump(mode="json"))
assert restored.cliff_result_id == CLIFF_RESULT_ID


@pytest.mark.parametrize(
"named_run",
[
{},
{"report_run": _run_ids().model_dump(mode="json")},
{"simulation_run_id": str(BASELINE_RUN_ID)},
],
ids=["comparison", "report-run", "simulation-run"],
)
def test_a_context_naming_no_cliff_result_keeps_its_json(named_run) -> None:
"""Workers deployed before the field refuse it, even as null (extra fields)."""
context = Stage12InvocationContext(**_context_fields(**named_run))

serialized = context.model_dump(mode="json")

assert "cliff_result_id" not in serialized
assert Stage12InvocationContext.model_validate(serialized) == context


@pytest.mark.parametrize(
"named_run",
[
{"report_run": _run_ids().model_dump(mode="json")},
{"simulation_run_id": str(BASELINE_RUN_ID)},
],
ids=["report-run", "simulation-run"],
)
def test_a_context_naming_a_cliff_result_names_no_run(named_run) -> None:
with pytest.raises(ValidationError, match="one of a report run, a simulation run"):
Stage12InvocationContext(
**_context_fields(cliff_result_id=str(CLIFF_RESULT_ID), **named_run)
)


def test_a_shadow_comparison_run_cannot_record_a_cliff_result() -> None:
with pytest.raises(ValidationError, match="production function call"):
Stage12InvocationContext(
**_context_fields(
production_function_call_id="fc-production",
cliff_result_id=str(CLIFF_RESULT_ID),
)
)


def test_a_shadow_comparison_run_cannot_record_a_simulation_run() -> None:
with pytest.raises(ValidationError, match="production function call"):
Stage12InvocationContext(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,12 @@

# The API v1 tables whose rows telemetry names.
DatabaseTable = Literal[
"household_jobs", "report_runs", "reports", "simulation_runs", "simulations"
"cliff_impacts",
"household_jobs",
"report_runs",
"reports",
"simulation_runs",
"simulations",
]
LogFieldValue = str | int | float | bool

Expand Down
Loading
Loading