Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
30c488f
Merge C1 · 23's local-area work (98a6144f) for the shared local-area …
anth-volk Nov 13, 2026
2d16835
Test the area columns of a UK simulation planned alone on the local-a…
anth-volk Nov 13, 2026
54e9990
Plan the area columns for a UK-wide or nation simulation on the local…
anth-volk Nov 13, 2026
7f60a6b
Merge C1 · 23's local-area work (71d1a553): full results plus areas o…
anth-volk Nov 13, 2026
7fe0c8d
Test a report run naming one simulation run on both sides (failing)
anth-volk Nov 13, 2026
dabc962
Accept a report run naming one simulation run on both sides
anth-volk Nov 13, 2026
c477783
Test the rows of a report run naming one simulation run on both sides…
anth-volk Nov 13, 2026
c4f5ca0
Store a simulation run's rows once when it is both sides of a report run
anth-volk Nov 13, 2026
4e40e3f
Test a coordinator whose report run names one simulation run on both …
anth-volk Nov 13, 2026
7699c2e
Calculate a simulation run named on both sides of a report run once
anth-volk Nov 13, 2026
87214a2
Describe a report run naming one simulation run on both sides
anth-volk Nov 13, 2026
31ee50f
Merge C1 · 23's local-area work (a4b8cadf): areas and withheld codes …
anth-volk Nov 13, 2026
2dd84d8
Test a coordinator that waits for a simulation run another call holds…
anth-volk Nov 13, 2026
9b981c2
Test failing a simulation run only while it records an ended call (fa…
anth-volk Nov 13, 2026
79ec967
Test a refused worker start, and drop the release write's tests (fail…
anth-volk Nov 13, 2026
b594dcf
Wait for a simulation run another call holds, and fail no shared run …
anth-volk Nov 13, 2026
a324835
Describe a pre-empted call's baseline worker as left running in a reu…
anth-volk Nov 13, 2026
be8845e
Test each simulation run's stored output plan (failing)
anth-volk Nov 13, 2026
7b3759c
Give each worker its simulation's stored output plan
anth-volk Nov 13, 2026
7af2c7d
Test refusing a report run that asks for cliff impacts (failing)
anth-volk Nov 13, 2026
b84d203
Refuse a report run that asks for cliff impacts, for now
anth-volk Nov 13, 2026
3fa13f3
Test a failed probe of a call Modal reports running (failing)
anth-volk Nov 13, 2026
02e22a5
Count a failed probe as a call's end only once Modal says it ended
anth-volk Nov 13, 2026
35f5552
Test a waited-for run naming another dataset than the other side's (f…
anth-volk Nov 13, 2026
90acd0a
Check a waited-for run's dataset and region against the other side's
anth-volk Nov 13, 2026
9a4ac7d
Test the age limit failing a run unfinished report runs name (failing)
anth-volk Nov 13, 2026
1f9221a
Let the sweep's age limit fail a run whatever report runs name it
anth-volk Nov 13, 2026
5aaa2bd
Fix review round 1's quality findings for shared simulation runs
anth-volk Nov 13, 2026
1eec8f2
Test a comparison report with one execution ID refused unwritten (fai…
anth-volk Nov 13, 2026
3f79675
Refuse a comparison report with one execution ID before claiming it
anth-volk Nov 13, 2026
836e024
Cover a run moved to a new holding call, one left unstarted, and one …
anth-volk Nov 13, 2026
227d700
Merge C1 · 23's pushed branch (PolicyEngine/policyengine-sim-api#1068…
anth-volk Nov 13, 2026
8898924
Format the one-run-on-both-sides condition as the pinned Ruff does
anth-volk Nov 13, 2026
c115814
Test a household function sharing simulation runs (failing)
anth-volk Nov 13, 2026
dba7035
Let the household function share simulation runs as the coordinator does
anth-volk Nov 13, 2026
5330fbe
Test the precomputed US baselines are planned as simulation runs stor…
anth-volk Nov 13, 2026
aa5b3f6
Plan the precomputed US baselines as simulation runs store them
anth-volk Nov 13, 2026
e62deca
Test the age limit leaves an unclaimed run a report run names (failing)
anth-volk Nov 13, 2026
909b0bf
Let the age limit ignore naming report runs only for runs with a call
anth-volk Nov 13, 2026
d6b2593
Test a failed probe reads the waited-for run before asking Modal (fai…
anth-volk Nov 13, 2026
bb8390b
Read a waited-for run before asking Modal about a failed probe
anth-volk Nov 13, 2026
063fa48
Merge C1 · 23's pushed branch (PolicyEngine/policyengine-sim-api#1068…
anth-volk Nov 13, 2026
f91ac41
Cover a forced report run starting workers on its fresh runs
anth-volk Nov 13, 2026
84847da
Test a household side recorded by a call that keeps running (failing)
anth-volk Nov 13, 2026
4005da1
Read a household side's run before each probe of its holding call
anth-volk Nov 13, 2026
dd6b440
Cover more of the household and coordinator waits
anth-volk Nov 13, 2026
9be3b57
Test what a household side reports when its wait window closes (failing)
anth-volk Nov 13, 2026
b24594f
Report a household side as it stands when its wait window closes
anth-volk Nov 13, 2026
077c65f
Name the forced-run test after the state it covers
anth-volk Nov 13, 2026
fdebe66
Fix review round 4's findings for the household tests and texts
anth-volk Nov 14, 2026
db8a72f
Add the changelog fragment for shared simulation runs
anth-volk Nov 14, 2026
95a93e1
Merge main into the shared simulation runs branch
anth-volk Nov 16, 2026
ffa7d76
Merge main into the shared simulation runs branch
anth-volk Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/c1-26-shared-simulation-runs.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Stage 12 report runs, economy and household, can share simulation runs. A report run's coordinator now starts a worker for any simulation run that is pending with no Modal call, whichever report run created it, and waits for a run another call holds instead of cancelling or releasing that call; once that call ends, it reads the run's stored file, fails with the run's own recorded failure, or fails the run if the call ended without recording an outcome. The household function first calculates the sides it can claim, then waits the same way, reading the run before each probe, for up to 780 seconds from its start. A failing, cancelled or pre-empted coordinator or household function no longer fails simulation runs it doesn't hold, which other report runs may share. A report run may name one simulation run on both sides, and that run is calculated once. Each simulation run now stores an output that depends on its simulation alone: the labour-supply response columns only when its own policy turns a response on (a report reads them as zeros otherwise), in the US the two inputs the labour-supply output reads, and never the cliff columns. So a report run asking for cliffs fails with `cliff_impacts_unavailable` until cliffs return as a calculation of each simulation run. The precomputed US baselines are planned the same way, so the first deployment computes them again under new cache identities, and a comparison-path call's US national baseline (a shadow comparison with a production job, or the operator-only `POST /internal/stage12/reports`) no longer reads them. The stuck-run sweep's 24-hour age limit now fails a simulation run that records a Modal call whatever report runs name it; a run no call has claimed still waits for the report runs naming it.
111 changes: 87 additions & 24 deletions docs/migration/stage-12-modal-worker-foundation.md
Original file line number Diff line number Diff line change
Expand Up @@ -310,15 +310,20 @@ run that a pending or running report run names: those are the coordinator's.
The coordinator claims the report run under its own Modal call ID, then checks
the input against it: a report input or prefix naming other runs, or a report
run row naming other simulation runs, fails the report run with
`report_run_input_mismatch`. If a worker it starts finds its run held by a
worker an earlier attempt started, which Modal never cancelled, the coordinator
waits for that worker instead. It uses the stored output of each simulation run
`report_run_input_mismatch`. If a worker it starts finds its run held by
another call, the coordinator waits for that run instead (see "Shared
simulation runs" below). It uses the stored output of each simulation run
that already succeeded (see the reuse of succeeded runs above), fails the
report run with the run's own recorded failure if a run it must calculate
already failed (`simulation_run_failed` if it recorded none), and starts a
worker for each other one with a context naming only that
simulation run (`simulation_run_id`). It aggregates both outputs and completes
the report run with its result rows. It returns
the report run with its result rows. A report run may name one simulation run
as both its baseline and its reform, as a report comparing a simulation with
itself does; both inputs then carry that run's ID and the same policy. The
coordinator starts one worker for it, or reads its stored output once, uses
that output as both sides, and stores the run's poverty and inequality rows
once. It returns
`{"report_run_id", "status"}`, or the same marked `deduplicated` without running
when another call holds the report run or it has finished.

Expand All @@ -329,10 +334,8 @@ its `simulation_runs` row (pending, no call), starts the country's
both the run's ID) and a context naming the run in `simulation_run_id`, with
an `artifact_prefix` that contains the run's ID (for example
`stage-12-runs/<environment>/<YYYY>/<MM>/<simulation_run_id>`), and records the
call as for a report run. A prefix without it fails the run. The worker plans the simulation's outputs
as for a report comparing it with itself, so a report whose plan needs more
columns, for example a reform that turns on labour supply responses, can't
reuse it and fails with `reused_simulation_output_incompatible`.
call as for a report run. A prefix without it fails the run. The worker plans the simulation's
outputs with its stored plan, as a report run's coordinator does (below).

**A re-run.** The service re-runs a stored report or simulation with the same
two calls, by run ID. It creates the record's new run (a report run with
Expand All @@ -354,18 +357,61 @@ stored output's descriptor without calculating, and for a run another call
holds, or one that failed, it returns `{"deduplicated": true,
"simulation_run_id", "status"}`.

**What a simulation run stores** (C1 · 24, answer 4). A run's output file
depends on its simulation alone, never on a report naming it
(`resolve_simulation_output_plan`): every column that doesn't depend on the
other side, the area columns on the local-area dataset and the US budget
columns; the labour-supply response columns only when the simulation's own
policy turns a response on; in the US, the two inputs .py's labour-supply
output reads from either side (`self_employment_income`,
`weekly_hours_worked`); and never the cliff columns, so a US national run keeps
its region-group split. Until cliffs come back as a calculation of each
simulation run (a later stage), a report run asking for them
(`include_cliffs: true`) fails before any worker starts, with
`cliff_impacts_unavailable`, and touches none of its simulation runs. The
coordinator gives each worker its simulation's
stored plan, and keeps the report's own plan, from its pair, for what the
aggregation reads. A reused or waited-for file must hold what the report reads
from its side (`side_read_plan`); a side that stores no responses the report
reads is read with them as zeros. A run that succeeded before this rule, under
a report's plan, is reused when it holds what the new report reads from it,
and refused (`reused_simulation_output_incompatible`) otherwise. The
precomputed US baselines are planned the same way, so a report run's
current-law baseline finds its precomputed files; a comparison-path call (a
shadow comparison with a production job, or the operator-only
`POST /internal/stage12/reports`) still plans its baseline with its report's
plan, so its US national baseline no longer finds them and calculates all 20
region groups.

**Shared simulation runs.** Identical requests share simulation runs, so a
report run may name a run another report run's worker is calculating (C1 · 24,
answer 5). The coordinator starts a worker on each run that is pending with no
call, whichever report run created it: the worker's claim lets one call hold
the run, and a second worker returns `deduplicated`. It waits for a run that is
pending or running under another call, without cancelling or releasing that
call: it probes the call, and once the call has ended reads the run again. A
run that succeeded is read from its stored file and checked as a reused run is;
one that failed fails the report run with its recorded failure; one still
unfinished under the call that ended is failed while it still records that call
(`fail_simulation_run_held_by`, `modal_call_ended`), so every coordinator
waiting for it stops. Each waiting report run holds a coordinator container
(25 at most) while it waits, within its own wait window.

**Pre-emption and cancellation.** Modal runs a pre-empted call again under the
same call ID, and each claim lets that call resume. A pre-empted coordinator
cancels its unfinished workers and records nothing; the restarted coordinator
asks each earlier worker that holds a simulation run to stop, releases that run
(pending, no call, keeping its first start) and starts a new worker on it. A
run that worker finished meanwhile is used as it finished, or fails the report
with its own cause. A failed or cancelled coordinator fails its report run and the
simulation runs it left unfinished, never one that succeeded. The stuck-run
sweep judges the run tables only, leaves a report run's simulation runs to its
coordinator while the report run is unfinished, and never fails a run whose
call Modal still reports pending, which includes a pre-empted one, before the
run's maximum age of 24 hours.
records nothing and cancels no worker; the restarted coordinator waits for each
run an earlier worker holds, as for any run another call holds. A failed or
cancelled coordinator fails its report run only: its workers finish, and their
outputs stay reusable, since other report runs may name their runs. A worker's
run is failed by its coordinator only for its own failure, while that worker
holds it. The stuck-run sweep judges the run tables only, leaves a simulation
run to the coordinators of the unfinished report runs naming it, and never
fails a run whose call Modal still reports pending, which includes a
pre-empted one, before the run's maximum age of 24 hours. That age limit
fails a simulation run that records a call whatever report runs name it,
since new ones may keep joining a run whose call Modal never ends; a run no
call has claimed waits for the report runs naming it, as a budget window's
later year does for the earlier years.

**Applications deployed before this change** reject both calls before writing
anything: their `coordinate_report` requires an evaluation record, and their
Expand Down Expand Up @@ -561,9 +607,9 @@ calculations, cliff analysis, and labor-supply-response calculations also use
the single-container path. `segmented=true` is accepted, but it does not
override those numerical eligibility conditions. A US national simulation run
started on its own (role `standalone`) is segmented by the same rule. It never
uses the precomputed baseline files, which serve only a report's current-law
baseline under that report's output plan, so it calculates all 20 region
groups.
uses the precomputed baseline files, which serve only a report run's
current-law baseline, planned as its simulation run stores its output, so it
calculates all 20 region groups.

The Stage 12 coordinator still persists only two logical simulation records:
one baseline and one reform. Each eligible logical simulation call performs
Expand Down Expand Up @@ -851,9 +897,26 @@ report run, never both. The function then:
Both sides are always calculated, so each run has its own outcome. A report
run another call holds, or one already finished, returns `deduplicated`, the
report run's ID and its status, without calculating. A restarted call keeps a
side it already recorded. An error, a refused payload included, or Modal's
cancellation fails the report run and each of its simulation runs left
unfinished, with a summary naming only the error's type, then propagates; a
side it already recorded.

Identical household requests share simulation runs (C1 · 24, answer 8), so,
as a report run's coordinator does, the function waits for a side whose run
another call holds, once the sides it can claim are done: it uses the run once
it succeeds, takes its recorded failure as the side's, claims and calculates
it here if it is left pending with no call, and fails it, while it still
records that call, only once Modal reports the call ended without an outcome
(`modal_call_ended`). The holding call is another household function,
which goes on to its other side and its own waits once it has recorded the
run, so the wait reads the run before each probe and takes a finished run at
once. A side still held
`HOUSEHOLD_SIDE_WAIT_SECONDS` (780) after the call's start, inside the
function's 900-second Modal timeout, fails the report run and is left to its
call; a waiting call holds one of the household functions' containers. A run
named on both sides, as a report comparing a policy with itself names one, is
calculated once and is both sides' outcome. An error, a refused payload
included, or Modal's cancellation fails the report run and each simulation run
this call holds, with a summary naming only the error's type, then propagates;
a run it never claimed, which other report runs may share, is left as it is. A
pre-emption records nothing. An error in the report run's claim, or in the
read of its simulation runs, fails the report run if the claim applied
before the error, and leaves its simulation runs to the stuck-run sweep.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,11 +44,22 @@ class HouseholdReportRunCalculation(StrictHouseholdModel):

@model_validator(mode="after")
def validate_one_household(self) -> HouseholdReportRunCalculation:
"""Require two sides to record different runs of one household calculation."""
"""Require two sides to calculate one household the same way.

They record different simulation runs, except a report comparing a
policy with itself, which names that simulation's one run on both
sides: its two sides are then one calculation, policy included.
"""
if self.reform is None:
return self
if self.reform.calculation_id == self.baseline.calculation_id:
msg = "the baseline and the reform must record different simulation runs"
if (
self.reform.calculation_id == self.baseline.calculation_id
and self.reform.policy != self.baseline.policy
):
msg = (
"a household report naming one simulation run on both sides "
"must give both the same policy"
)
raise ValueError(msg)
baseline = self.baseline.model_dump(include=_SHARED_FIELDS)
reform = self.reform.model_dump(include=_SHARED_FIELDS)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,11 @@
class ReportRunIds(StrictContractModel):
"""The Public API's report run and simulation runs one report call records on.

The simulation service creates the three rows before it starts the
coordinator. The coordinator records the report run's status and results,
and each worker it starts claims and records its own simulation run.
The simulation service creates the rows before it starts the coordinator.
The coordinator records the report run's status and results, and each
worker it starts claims and records its own simulation run. A report
comparing a simulation with itself names that simulation's one run on
both sides (``names_one_simulation_run``).
"""

report_run_id: UUID
Expand All @@ -37,17 +39,20 @@ class ReportRunIds(StrictContractModel):

@model_validator(mode="after")
def validate_distinct_runs(self) -> ReportRunIds:
"""Require the report run and both simulation runs to be different rows."""
run_ids = {
self.report_run_id,
"""Require the report run to be another row than its simulation runs."""
if self.report_run_id in {
self.baseline_simulation_run_id,
self.reform_simulation_run_id,
}
if len(run_ids) != 3: # noqa: PLR2004 # one report run and two simulation runs
msg = "report run IDs must name three different runs"
}:
msg = "the report run must differ from its simulation runs"
raise ValueError(msg)
return self

@property
def names_one_simulation_run(self) -> bool:
"""Whether the baseline and the reform are the same simulation run."""
return self.baseline_simulation_run_id == self.reform_simulation_run_id

def simulation_run_id(self, role: SimulationRole) -> UUID:
"""Return the simulation run of ``role``; a standalone role has none."""
if role is SimulationRole.BASELINE:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,9 @@ class ReportExecutionInput(StrictContractModel):
"""A baseline and reform simulation pair to run and aggregate into one report.

Both simulations must belong to this report, carry the baseline and reform
roles and distinct execution IDs, and differ only in their policy.
roles, and differ only in their policy. Their execution IDs differ, except
in a report that names one simulation run on both sides, whose two inputs
are then the same simulation, policy included.
"""

contract_version: Literal[1] = 1
Expand All @@ -174,8 +176,9 @@ class ReportExecutionInput(StrictContractModel):
def require_aligned_simulations(self) -> ReportExecutionInput:
"""Check the two simulations form one comparable report.

Roles must be baseline and reform, both must belong to this report with
distinct execution ids, and everything but the policy must match.
Roles must be baseline and reform, both must belong to this report,
and everything but the policy must match. Inputs sharing one
execution ID name one simulation run, so their policies match too.
"""
if self.baseline.role is not SimulationRole.BASELINE:
msg = "baseline input must use the baseline role"
Expand All @@ -189,8 +192,14 @@ def require_aligned_simulations(self) -> ReportExecutionInput:
if self.reform.evaluation_id != self.evaluation_id:
msg = "reform evaluation_id must match the report"
raise ValueError(msg)
if self.baseline.simulation_execution_id == self.reform.simulation_execution_id:
msg = "baseline and reform simulation identifiers must differ"
if (
self.baseline.simulation_execution_id == self.reform.simulation_execution_id
and self.baseline.policy != self.reform.policy
):
msg = (
"a report naming one simulation run on both sides must give "
"both the same policy"
)
raise ValueError(msg)
for field_name in (
"population",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -91,9 +91,22 @@ def test_a_calculation_round_trips_through_json() -> None:
assert HouseholdReportRunCalculation.model_validate(payload) == calculation


def test_each_side_is_its_own_calculation() -> None:
with pytest.raises(ValidationError, match="different simulation runs"):
_calculation(reform=_request(BASELINE_RUN_ID))
def test_one_simulation_run_may_be_both_sides_of_one_calculation() -> None:
# A household report comparing a policy with itself names that
# simulation's one run on both sides.
calculation = _calculation(reform=_request(BASELINE_RUN_ID))

assert [request.calculation_id for _, request in calculation.sides()] == [
BASELINE_RUN_ID,
BASELINE_RUN_ID,
]


def test_one_simulation_run_on_both_sides_must_have_one_policy() -> None:
with pytest.raises(ValidationError, match="one simulation run on both sides"):
_calculation(
reform=_request(BASELINE_RUN_ID, policy={"parameters": CTC_REFORM})
)


@pytest.mark.parametrize(
Expand Down
Loading
Loading