Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
144de5b
Test that household job and report run reads name their Modal call (f…
anth-volk Nov 12, 2026
af5ec0c
Name the Modal call recorded on a household job or report run in its …
anth-volk Nov 12, 2026
dd9b55c
Test the helper that lists a Modal call's inputs and their containers…
anth-volk Nov 12, 2026
7a01b07
List a Modal call's inputs with the container that runs each
anth-volk Nov 12, 2026
3b776bd
Test the staging preemption check, its deploy job and its ready check…
anth-volk Nov 12, 2026
693de57
Stop staging calculations' Modal containers on every staging deploy
anth-volk Nov 12, 2026
a429b71
Document the staging preemption check and the reads' Modal call
anth-volk Nov 12, 2026
614b3e7
Test review round 1's fixes to the staging preemption check (failing)
anth-volk Nov 12, 2026
1a3387a
Fix review round 1: follow a job after a refused stop, wait out a war…
anth-volk Nov 12, 2026
b85b914
Test that the preemption job's time limit covers the waits between at…
anth-volk Nov 12, 2026
68ae26b
Give the preemption job 120 minutes, past its waits between attempts
anth-volk Nov 12, 2026
d6e8e17
Test review round 2's fixes to the staging preemption check (failing)
anth-volk Nov 12, 2026
50724bc
Test that the preemption job's time limit covers its Modal commands' …
anth-volk Nov 12, 2026
165db2c
Fix review round 2: require completion after the stop, wait out every…
anth-volk Nov 12, 2026
af18a1a
Test review round 3's quality fixes to the staging preemption check (…
anth-volk Nov 12, 2026
e53b3be
Fix review round 3's findings: a completion margin, unanswered listin…
anth-volk Nov 12, 2026
d0d041d
Test review round 4's fixes to the staging preemption check (failing)
anth-volk Nov 12, 2026
921743a
Fix review round 4: wait for the rerun's warm container too
anth-volk Nov 12, 2026
6aa9ed0
Test review round 5's fixes to the staging preemption check (failing)
anth-volk Nov 12, 2026
7d71756
Fix review round 5: wait a fixed time before a new attempt, refuse em…
anth-volk Nov 12, 2026
b6a68df
Re-wrap the preemption check's paragraph on missed attempts
anth-volk Nov 12, 2026
d17ab4e
Test review round 6's quality fixes to the staging preemption check (…
anth-volk Nov 12, 2026
3a2dc5f
Fix review round 6's quality findings in the staging preemption check
anth-volk Nov 13, 2026
d574a1c
Fix review round 7's quality findings in the preemption check's tests
anth-volk Nov 13, 2026
b746920
Merge main into v2plan/c1-20-staging-preemption-check
anth-volk Nov 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
461 changes: 461 additions & 0 deletions .github/scripts/simulation-staging-preemption-check.sh

Large diffs are not rendered by default.

13 changes: 13 additions & 0 deletions .github/scripts/simulation-staging-preemption-ready.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
#!/bin/bash
# Require the staging preemption check to have succeeded in a staging
# deployment, and to have been skipped in any other.
# Required env vars: REQUESTED ("true" in a staging deployment) and
# CHECK_RESULT (the staging_preemption_check job result).

set -euo pipefail

if [[ "${REQUESTED}" == "true" ]]; then
test "${CHECK_RESULT}" = "success"
else
test "${CHECK_RESULT}" = "skipped"
fi
80 changes: 79 additions & 1 deletion .github/workflows/simulation-deploy.reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1290,6 +1290,78 @@ jobs:
SIMULATION_TEST_AUTH_CLIENT_SECRET: ${{ secrets.SIMULATION_ENTRYPOINT_TEST_AUTH_CLIENT_SECRET }}
run: .github/scripts/simulation-staging-economy-check.sh "${{ needs.deploy_entrypoint.outputs.candidate_url }}"

staging_preemption_check:
name: Stop staging calculations' Modal containers and require them to succeed
# Every staging deployment calculates the staging household, then
# creates the staging economy check's report, through the candidate, and
# stops the Modal container running each one's call with `modal
# container stop` once the household function or coordinator has claimed
# it. That sends SIGINT and gives the input to another container under
# the same call ID, as a pre-emption does; each must still succeed. It
# runs after the economy check, so the report reuses that check's
# simulation runs and stopping the coordinator costs no new simulation.
# The script refuses any Modal environment but staging, and stops only a
# container Modal lists there. Each calculation writes rows in the
# staging database, as the other staging checks' do.
if: >-
${{
!cancelled() &&
inputs.deployment_environment == 'staging' &&
needs.deploy_entrypoint.result == 'success' &&
needs.authenticated_test.result == 'success' &&
needs.staging_economy_check.result == 'success' &&
((inputs.deploy_stage12_v2 && needs.publish_stage12_v2_default.result == 'success') ||
(!inputs.deploy_stage12_v2 && needs.publish_stage12_v2_default.result == 'skipped'))
}}
needs: [deploy_entrypoint, authenticated_test, publish_stage12_v2_default, staging_economy_check]
runs-on: ubuntu-latest
# Longer than both steps' worst case together, so the script fails
# first: about 126 minutes, with every request and Modal command taking
# its full time limit (three attempts of up to 17.5 minutes for the
# household and 22.5 for the report, and a 90-second wait before each
# new attempt), and the install.
timeout-minutes: 150
environment: ${{ inputs.release_environment }}
steps:
- uses: actions/checkout@v6

- uses: actions/setup-python@v6
with:
python-version: "3.13"

- uses: astral-sh/setup-uv@v8.1.0
with:
version: "0.13.6"

- name: Install the Modal client
working-directory: projects/policyengine-simulation-executor
run: uv sync --frozen

- name: Stop a staging household calculation's container and require it to succeed
working-directory: projects/policyengine-simulation-executor
env:
STAGING_HOUSEHOLD_ID: ${{ vars.SIMULATION_STAGING_HOUSEHOLD_ID }}
SIMULATION_TEST_AUTH_ISSUER: ${{ secrets.SIMULATION_ENTRYPOINT_AUTH_ISSUER }}
SIMULATION_TEST_AUTH_AUDIENCE: ${{ secrets.SIMULATION_ENTRYPOINT_AUTH_AUDIENCE }}
SIMULATION_TEST_AUTH_CLIENT_ID: ${{ secrets.SIMULATION_ENTRYPOINT_TEST_AUTH_CLIENT_ID }}
SIMULATION_TEST_AUTH_CLIENT_SECRET: ${{ secrets.SIMULATION_ENTRYPOINT_TEST_AUTH_CLIENT_SECRET }}
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
MODAL_ENVIRONMENT: ${{ inputs.modal_environment }}
run: ../../.github/scripts/simulation-staging-preemption-check.sh household "${{ needs.deploy_entrypoint.outputs.candidate_url }}"

- name: Stop a staging report's coordinator container and require it to succeed
working-directory: projects/policyengine-simulation-executor
env:
SIMULATION_TEST_AUTH_ISSUER: ${{ secrets.SIMULATION_ENTRYPOINT_AUTH_ISSUER }}
SIMULATION_TEST_AUTH_AUDIENCE: ${{ secrets.SIMULATION_ENTRYPOINT_AUTH_AUDIENCE }}
SIMULATION_TEST_AUTH_CLIENT_ID: ${{ secrets.SIMULATION_ENTRYPOINT_TEST_AUTH_CLIENT_ID }}
SIMULATION_TEST_AUTH_CLIENT_SECRET: ${{ secrets.SIMULATION_ENTRYPOINT_TEST_AUTH_CLIENT_SECRET }}
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
MODAL_ENVIRONMENT: ${{ inputs.modal_environment }}
run: ../../.github/scripts/simulation-staging-preemption-check.sh report "${{ needs.deploy_entrypoint.outputs.candidate_url }}"

finalize_routing:
name: Finalize the Modal routing publication
# Finish the durable journal after every gate, including failed uploads
Expand Down Expand Up @@ -1348,7 +1420,7 @@ jobs:
# The reporter only reads job results. Journal completion happens before
# it, so a failure or cancellation here cannot strand a publication.
if: ${{ !cancelled() }}
needs: [deploy_entrypoint, update_routing, integration, authenticated_test, promote_entrypoint, publish_latest, finalize_routing, publish_stage12_v2_default, staging_household_check, staging_economy_check]
needs: [deploy_entrypoint, update_routing, integration, authenticated_test, promote_entrypoint, publish_latest, finalize_routing, publish_stage12_v2_default, staging_household_check, staging_economy_check, staging_preemption_check]
runs-on: ubuntu-latest
defaults:
run:
Expand Down Expand Up @@ -1389,6 +1461,12 @@ jobs:
CHECK_RESULT: ${{ needs.staging_economy_check.result }}
run: .github/scripts/simulation-staging-economy-ready.sh

- name: Check the staging preemption check result
env:
REQUESTED: ${{ inputs.deployment_environment == 'staging' }}
CHECK_RESULT: ${{ needs.staging_preemption_check.result }}
run: .github/scripts/simulation-staging-preemption-ready.sh

cleanup_entrypoint_revisions:
name: Remove superseded Cloud Run revisions
if: ${{ !cancelled() && inputs.promote_entrypoint && needs.deployment_ready.result == 'success' }}
Expand Down
1 change: 1 addition & 0 deletions changelog.d/1062.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Every staging deploy stops the Modal container running a household calculation and a small economy report, as a pre-emption does, and fails unless both still succeed; `GET /household-calculations/{household_job_id}` and the run in `GET /reports/{report_id}` now name the Modal call recorded on them (`modal_call_id`).
3 changes: 2 additions & 1 deletion libs/policyengine-simulation-persistence/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -111,7 +111,8 @@ call's ID, and `fail_unstarted_report_run` and
`fail_unstarted_simulation_run` fail a run at once, each only while it is
`pending` with no call ID; failing a report run also fails the simulation
runs created for it. `economy_run_reads.py` reads a report, or a simulation,
with its newest run's state.
with its newest run's state; a report run's state includes the Modal call
recorded on it.

`EconomyRunStore` runs them as the Simulation Entrypoint does, each step in
a transaction of its own: `read_report_inputs` reads the catalog version,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,11 @@ class SimulationRunState:

@dataclass(frozen=True)
class ReportRunState:
"""A report run's state, and the simulation runs it compares."""
"""A report run's state, the simulation runs it compares, and its call.

``modal_call_id`` is the coordinator's Modal call recorded on the run,
or None before the service has recorded one.
"""

id: UUID
status: ReportRunStatus
Expand All @@ -56,6 +60,7 @@ class ReportRunState:
completed_at: datetime | None
baseline_simulation_run_id: UUID
reform_simulation_run_id: UUID | None
modal_call_id: str | None


@dataclass(frozen=True)
Expand Down Expand Up @@ -104,6 +109,7 @@ def read_economy_report(
report_runs.c.completed_at,
report_runs.c.baseline_simulation_run_id,
report_runs.c.reform_simulation_run_id,
report_runs.c.modal_call_id,
)
.where(report_runs.c.report_id == report_id)
.order_by(report_runs.c.created_at.desc(), report_runs.c.id.desc())
Expand Down Expand Up @@ -178,4 +184,5 @@ def _report_run_state(run: RowMapping) -> ReportRunState:
completed_at=run["completed_at"],
baseline_simulation_run_id=run["baseline_simulation_run_id"],
reform_simulation_run_id=run["reform_simulation_run_id"],
modal_call_id=run["modal_call_id"],
)
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,52 @@ def test_a_failed_newest_run_is_read_with_its_error(public_api_database_url) ->
assert report.run.completed_at is not None


def test_a_report_run_is_read_with_the_modal_call_it_records(
public_api_database_url,
) -> None:
"""A run names the Modal call recorded on it, and no call before one is.

The staging preemption check finds the coordinator's container from it.
"""
with rolled_back_transaction(public_api_database_url) as connection:
economy = insert_economy_catalog(connection)
report_id, run_ids = _report_with_runs(connection, economy)
for run_id in run_ids:
connection.execute(
update(report_runs)
.where(report_runs.c.id == run_id)
.values(modal_call_id=f"fc-{run_id.hex}")
)
population = economy.population()
uncalled_run_id = insert_report_run_naming(
connection,
economy,
(
insert_simulation_run(connection, population),
insert_simulation_run(connection, population),
),
ReportRunOutcome(status=ReportRunStatus.PENDING),
)
uncalled_report_id = (
connection.execute(
report_runs.select().where(report_runs.c.id == uncalled_run_id)
)
.one()
.report_id
)
act_as_the_service(connection)

report = read_economy_report(connection, report_id)
uncalled_report = read_economy_report(connection, uncalled_report_id)

assert report is not None
assert report.run is not None
assert report.run.modal_call_id == f"fc-{report.run.id.hex}"
assert uncalled_report is not None
assert uncalled_report.run is not None
assert uncalled_report.run.modal_call_id is None


def test_a_simulation_is_read_with_its_newest_run(public_api_database_url) -> None:
with rolled_back_transaction(public_api_database_url) as connection:
economy = insert_economy_catalog(connection)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -177,6 +177,7 @@ def test_a_report_and_a_simulation_are_read_with_their_newest_runs():
"completed_at": "2026-11-12T15:04:00Z",
"baseline_simulation_run_id": SIDE_IDS[1],
"reform_simulation_run_id": SIDE_IDS[3],
"modal_call_id": "fc-coordinator",
},
}
simulation = {"simulation_id": REPORT_ID, "run": None}
Expand All @@ -192,5 +193,6 @@ def test_a_report_and_a_simulation_are_read_with_their_newest_runs():
run = report_response.parsed.run
assert run is not None
assert run.status is ReportRunStatus.SUCCEEDED
assert run.modal_call_id == "fc-coordinator"
assert isinstance(simulation_response.parsed, EconomySimulationState)
assert simulation_response.parsed.to_dict() == simulation
Original file line number Diff line number Diff line change
Expand Up @@ -74,6 +74,7 @@ def test_a_job_keeps_its_result_document_whole():
"year": 2026,
"policyengine_version": "6.17.1",
"country_package_version": "2.2.1",
"modal_call_id": "fc-1",
"result": document,
"error_message": None,
"created_at": "2026-11-10T12:00:00Z",
Expand All @@ -84,6 +85,7 @@ def test_a_job_keeps_its_result_document_whole():
parsed = HouseholdCalculationState.from_dict(state)

assert parsed.status is HouseholdJobStatus.SUCCEEDED
assert parsed.modal_call_id == "fc-1"
assert parsed.to_dict()["result"] == document


Expand Down
61 changes: 61 additions & 0 deletions projects/policyengine-simulation-entry/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,67 @@ check has succeeded, later checks on that version reuse its simulation runs,
so they check the coordinator's use of stored outputs rather than the
workers', which the v2 deployment's own validation calls.

### The staging preemption check

Every staging deployment runs `staging_preemption_check` once the economy
check has succeeded. `.github/scripts/simulation-staging-preemption-check.sh`
runs twice, from the executor project, whose environment has the Modal
client:

- `household` calculates the staging household that
`SIMULATION_STAGING_HOUSEHOLD_ID` names, as the household check does;
- `report` creates the economy check's report again (labelled "Staging
preemption check"), whose simulation runs reuse that check's, so it costs
no new simulation.

Once the household job or report run is `running`, that is once the
household function or coordinator has claimed it, the check takes the
`modal_call_id` its read answers, asks Modal's call graph which container
runs that call (`src/modal/utils/describe_call_inputs.py`), and stops that
container with `modal container stop`. Modal sends the container SIGINT and
gives the input to another container under the same call ID, as it does
after a pre-emption. The check requires the record to succeed under that
call, completed at least 5 seconds after the stop, and the call graph to
show the call's input finished in another container. A record completed
sooner proves nothing, and that attempt counts as missed: the stopped
container may have written the success before the signal reached it, and
Modal's rerun then finds the record finished and only returns its summary.
A rerun in a new container can't finish within 5 seconds of the stop (both
times are counted in whole seconds). Run again, the household function or
coordinator resumes its claim. The report's simulation runs are reused, so
its coordinator starts no worker, and the check doesn't cover a coordinator
starting its workers again.

A calculation that finishes before its container is stopped, for example on
a warm container, proves nothing, so the check submits a new one, up to
three attempts. Each new attempt starts 90 seconds after the last one ended.
Modal stops a container left idle for 60 seconds by default, and these
functions don't change that. A container that finished the last calculation
unstopped went idle when the record finished, so the wait covers it, and the
next calculation starts in a new container. A container that Modal ran a
stopped call in again goes idle only when that rerun returns, which may be
after the attempt ended, so the wait covers it only when the rerun returns
within about 30 seconds of the attempt's end. A record that fails, or
doesn't finish within its deadline (10 minutes for the household, 15 for the
report, per attempt), fails the check at once. Modal refuses to stop a
container that has ended; when a stop fails, the check doesn't stop it again
or submit another calculation, but follows the record to its end, so a
record left running by a container that ended for another reason, such as
running out of memory, fails the check at its deadline. Every Modal command
may take 60 seconds.

It uses the repository's Modal token (the repository secrets
`MODAL_TOKEN_ID` and `MODAL_TOKEN_SECRET`), which must be allowed to read
call graphs and to list and stop containers. Production's deploy jobs use
the same token, and `modal container stop` takes a container ID in any
environment, so the script refuses any `MODAL_ENVIRONMENT` but `staging` and
stops only a container that `modal container list --env staging` lists. It
also uses the authenticated tests' client credentials. `deployment_ready`
requires the check to pass in staging, so a failure stops the production
deployment.
Each attempt adds a household job, or a report and its run, to the staging
database.

## Candidate qualification

For each eligible revision, the workflow first deploys `beta`
Expand Down
11 changes: 7 additions & 4 deletions projects/policyengine-simulation-entry/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,8 +85,10 @@ default worker, creates a `household_jobs` row and starts the household
function's call with `spawn`, answering 202 with the job's `job_id` and
`status`. Each submission creates a new job.
`GET /household-calculations/{household_job_id}` returns the job's status,
the household function's `result` document once it has one, and its
`error_message`.
the household function's `result` document once it has one, its
`error_message`, and `modal_call_id`, the household function's Modal call
recorded on the job (null until the service records one). Modal runs a
pre-empted call again under the same ID.

A submission is refused before anything is written: with 400 and a `code`
when the household, policy or catalog version for the default worker's .py
Expand Down Expand Up @@ -185,8 +187,9 @@ reads.

`GET /reports/{report_id}` and `GET /simulations/{simulation_id}` return the
record and its newest run (created last, ties to the larger ID): its status,
versions, error message and times, and a report run's simulation runs. The
Public API reads results from the tables itself.
versions, error message and times, and a report run's simulation runs and
`modal_call_id`, the coordinator's Modal call recorded on it (null until the
service records one). The Public API reads results from the tables itself.

## Household reports

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -226,7 +226,12 @@ class SimulationRunRead(_Answer):


class ReportRunRead(_Answer):
"""A report run's state, and the simulation runs it compares."""
"""A report run's state, the simulation runs it compares, and its call.

``modal_call_id`` is the coordinator's Modal call recorded on the run,
or null before the service has recorded one. Modal runs a pre-empted
call again under the same ID.
"""

run_id: UUID
status: ReportRunStatus
Expand All @@ -238,6 +243,7 @@ class ReportRunRead(_Answer):
completed_at: datetime | None
baseline_simulation_run_id: UUID
reform_simulation_run_id: UUID | None
modal_call_id: str | None


class EconomyReportState(_Answer):
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,9 @@
function, answering 201 with the simulation's ID, its run's ID and status.
- ``GET /reports/{report_id}`` and ``GET /simulations/{simulation_id}``
(``simulation:read``) return the record with its newest run's status,
versions, error message and times. The Public API reads results from the
tables itself; these reads let a deploy check follow a run.
versions, error message and times, and a report run's coordinator call.
The Public API reads results from the tables itself; these reads let a
deploy check follow a run.

A refused request answers 400 (a row it names, or a function it needs, is
missing; ``code`` says which) or 422 (its fields, or the stored values it
Expand Down Expand Up @@ -376,6 +377,7 @@ def _report_state(stored: StoredReport) -> EconomyReportState:
completed_at=run.completed_at,
baseline_simulation_run_id=run.baseline_simulation_run_id,
reform_simulation_run_id=run.reform_simulation_run_id,
modal_call_id=run.modal_call_id,
),
)

Expand Down Expand Up @@ -652,7 +654,8 @@ def _read_routes(
summary="Get a report's newest run",
description=(
"Return the report, its two simulations and its newest run's "
"status, versions, error message and times."
"status, versions, error message, times and coordinator's Modal "
"call (null until one is recorded)."
),
operation_id="get_report",
response_model=EconomyReportState,
Expand Down
Loading
Loading