Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
257e9be
Test the restart check's reads and writes for unstarted runs (failing)
anth-volk Nov 12, 2026
8ad78f6
List, mark and read back runs whose Modal call never started
anth-volk Nov 12, 2026
a95c14b
Test restarting runs and household jobs whose call never started (fai…
anth-volk Nov 12, 2026
7ede4eb
Start a run or household job again when its Modal call never started
anth-volk Nov 12, 2026
fa2236a
Build a restarted call in a module of its own
anth-volk Nov 12, 2026
116780d
Test the scheduled restart check, its telemetry and the sweep's age (…
anth-volk Nov 12, 2026
ace6c64
Run the restart check every two minutes beside the stuck-run sweep
anth-volk Nov 12, 2026
83d0b80
Describe the restart check and the sweep's 15 minutes
anth-volk Nov 12, 2026
6aacd1a
Test that a restart check waiting on every call ends within its timeo…
anth-volk Nov 12, 2026
5a8f7e9
Restart at most 20 runs a check, so a check ends within its timeout
anth-volk Nov 12, 2026
d5ee476
Test review round 1's restart findings (failing)
anth-volk Nov 12, 2026
5e158d3
Fix review round 1's restart findings
anth-volk Nov 12, 2026
1768f26
Test review round 2's restart findings (failing)
anth-volk Nov 12, 2026
b0db539
Fix review round 2's restart findings
anth-volk Nov 12, 2026
a9fb34e
Test review round 3's restart findings (failing)
anth-volk Nov 12, 2026
80a2e56
Fix review round 3's restart findings
anth-volk Nov 12, 2026
a78cf82
Test that the unexpected-error log carries its run to the runtime (fa…
anth-volk Nov 12, 2026
3798044
Fix review round 4's restart findings
anth-volk Nov 12, 2026
b006791
Describe every way a restart is left or deferred
anth-volk Nov 12, 2026
7a3095d
Merge main into v2plan/c1-19-restart-unstarted-runs
anth-volk Nov 12, 2026
7f8882f
Test restarting a budget window whose years never started (failing)
anth-volk Nov 12, 2026
e923c88
Restart a budget window whose waiting years never started
anth-volk Nov 12, 2026
3d2c6f3
Test review round 6's budget-window restart findings (failing)
anth-volk Nov 13, 2026
c63d733
Fix review round 6's budget-window restart findings
anth-volk Nov 13, 2026
5232960
Test review round 7's budget-window restart findings (failing)
anth-volk Nov 13, 2026
7a9de4a
Fix review round 7's budget-window restart findings
anth-volk Nov 13, 2026
da413de
Merge main into v2plan/c1-19-restart-unstarted-runs, with #1057's hou…
anth-volk Nov 13, 2026
6dd8308
Test restarting a household report run whose call never started (fail…
anth-volk Nov 13, 2026
da89c8e
Restart a household report run whose call never started
anth-volk Nov 13, 2026
a6753ef
Test round 8's household report restart findings (failing)
anth-volk Nov 13, 2026
c069a2f
Fix round 8's household report restart findings
anth-volk Nov 13, 2026
337b881
Merge main into v2plan/c1-19-restart-unstarted-runs, with #1059's re-…
anth-volk Nov 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 20 additions & 8 deletions .github/workflows/simulation-deploy.reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -1189,22 +1189,34 @@ jobs:
run: ../../.github/scripts/stage12-v2-default-verify.sh

# The scheduled sweep that fails report runs, simulation runs and
# household jobs nothing will finish. Its app is unversioned and its
# schedule starts when it deploys, so it deploys here, after every
# check, with the release's Stage 12 default; if it fails, the next
# step puts the default back.
# household jobs nothing will finish, and the restart check that starts
# again those whose Modal call never started. Their app is unversioned
# and its schedules start when it deploys, so it deploys here, after
# every check, with the release's Stage 12 default; if it fails, the
# next step puts the default back.
# Only from main: a dispatch from another branch, which may be behind
# main, leaves the environment's sweep as it is. With no status
# main, leaves the environment's app as it is. With no status
# function, the condition also requires every earlier step to pass.
# Its database URL comes from the Modal secret
# Their database URL comes from the Modal secret
# policyengine-modal-worker-database, which the v2 workers' deploy in
# deploy_stage12_v2 already requires.
- name: Deploy the stuck-run sweep
# deploy_stage12_v2 already requires. The image bakes in the Modal
# environment the restart check starts calls in, and the v2 workers'
# telemetry settings, from the repository variables they use.
- name: Deploy the stuck-run sweep and the restart check
if: ${{ github.ref == 'refs/heads/main' }}
working-directory: projects/policyengine-simulation-executor
env:
MODAL_TOKEN_ID: ${{ secrets.MODAL_TOKEN_ID }}
MODAL_TOKEN_SECRET: ${{ secrets.MODAL_TOKEN_SECRET }}
SIMULATION_V2_MODAL_ENVIRONMENT: ${{ inputs.modal_environment }}
OBSERVABILITY_SERVICE_NAMESPACE: ${{ vars.OBSERVABILITY_SERVICE_NAMESPACE }}
OBSERVABILITY_TRACE_PROJECT_ID: ${{ vars.OBSERVABILITY_TRACE_PROJECT_ID }}
OBSERVABILITY_LOGGING_PROJECT_ID: ${{ vars.OBSERVABILITY_LOGGING_PROJECT_ID }}
OBSERVABILITY_LOG_NAME: ${{ vars.OBSERVABILITY_LOG_NAME }}
OBSERVABILITY_GOOGLE_WORKLOAD_IDENTITY_PROVIDER: ${{ vars.OBSERVABILITY_GOOGLE_WORKLOAD_IDENTITY_PROVIDER }}
OBSERVABILITY_GOOGLE_SERVICE_ACCOUNT_EMAIL: ${{ vars.OBSERVABILITY_GOOGLE_SERVICE_ACCOUNT_EMAIL }}
OTEL_EXPORTER_OTLP_ENDPOINT: ${{ vars.OBSERVABILITY_OTLP_ENDPOINT }}
POLICYENGINE_OTEL_GOOGLE_AUDIENCE: ${{ vars.OBSERVABILITY_OTLP_GOOGLE_AUDIENCE }}
run: uv run --frozen modal deploy --env="${{ inputs.modal_environment }}" src/modal/run_sweep_app.py

# Writes back the manifest the move replaced, only while the stored
Expand Down
1 change: 1 addition & 0 deletions changelog.d/c1-19-restart-unstarted-runs.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
A restart check in the stuck-run sweep's Modal application starts the Modal call of a household job, economy or household report run, or standalone economy simulation run again when the run stayed pending with no call recorded 3 to 10 minutes after its creation, as when the Simulation Entrypoint stopped before starting its call. It also starts a budget window's waiting years again when the window stopped with none under way, as the Simulation Entrypoint starts a new window's first years, and fails at once, with the reason, a window whose first waiting year's call can't be built again. Its telemetry counts every restart and how it ended. The sweep now fails a run still pending with no call at 15 minutes instead of 10.
66 changes: 62 additions & 4 deletions docs/migration/stage-12-modal-worker-foundation.md
Original file line number Diff line number Diff line change
Expand Up @@ -371,7 +371,7 @@ run's maximum age of 24 hours.
anything: their `coordinate_report` requires an evaluation record, and their
context model refuses `simulation_run_id`. Modal accepts the call, which fails
at once, and the run stays pending with its call recorded until the sweep fails
it (`modal_call_ended`), at least ten minutes after its creation. A call
it (`modal_call_ended`), at least 15 minutes after its creation. A call
reaches such an application only while the manifest's default, or
`STAGE12_V2_WORKER_VERSION`, names one: during the deployment of this change,
if it adds a newer .py version, until it moves the default to it, or while a
Expand Down Expand Up @@ -422,7 +422,7 @@ single-year report's runs, and the reverse.
- **The stuck-run sweep.** It leaves a waiting year of an unfinished window to
the window, counts a year started later from when its call was recorded,
fails the window when it fails one of its years, and settles a window that
nothing has changed for ten minutes with no year under way: from its years
nothing has changed for 15 minutes with no year under way: from its years
when they have all ended, or as stalled (`budget_window_stalled`) when years
are still waiting.

Expand Down Expand Up @@ -811,8 +811,8 @@ The stuck-run sweep (`src/modal/run_sweep_app.py`) fails household jobs
nothing will finish by the rule it applies to report runs and simulation
runs (a household report run and its simulation runs are judged as any
report run and its simulation runs are), counting a job's age from its
creation: a pending job with no call
recorded ten minutes after its creation, a pending or running job once
creation: a pending job with no call recorded 15 minutes after its
creation, a pending or running job once
Modal reports its recorded call ended, and any job still unfinished 24 hours
after its creation, whatever Modal reports.

Expand Down Expand Up @@ -842,6 +842,64 @@ validation like any other validation failure: the manifest is not published,
and, as described under "Release and deployment constraints", the deployment
does not promote the new Simulation Entrypoint revision.

**Restarting a run whose call never started.** The stuck-run sweep's Modal
application also runs a restart check every two minutes, as the Modal worker
role. It lists, at most 20 a check, the household jobs, report runs and
standalone simulation runs created 3 to 10 minutes ago that are still pending
with no call recorded and that nothing has changed since their insert, as
when the Simulation Entrypoint stopped between creating a run and starting
its call. For each, on a thread, it reads back what the entrypoint built the
call from and builds the same call with the entrypoint's own builders, on the
worker of the .py version the row records. It then sets the row's
`updated_at`, so the row is never restarted twice, starts the call and
records its ID while the row is still pending with none. A row it can't start
again (its worker left the manifest, its stored rows changed, or Modal
refuses the call) is failed with `run_not_started`, a report run with its
pending simulation runs. A row is left unmarked for the next check when its
rows or manifest can't be read now, its rebuild raises an unexpected error
(logged with its traceback), its mark can't be written, its call can't be
rebuilt and its failure can't be written, or the check's time is up. After
three minutes a check marks no more rows, so it ends within its five-minute
timeout without leaving a marked row with no call. The sweep's 15 minutes come after the restart window, so a second miss
is still failed with `run_not_started`. The restart code is the simulation
service's (`policyengine_simulation_entry.run_restarts`, `run_restart_calls`
and `run_restart_check`, on `policyengine_simulation_persistence`), which the
image adds as source.

A budget window's years are never restarted one by one. A window stops when
no year is under way and years still wait: the entrypoint stopped before
starting its first years, or a year's coordinator before starting the next.
The check restarts a window that stopped 3 to 10 minutes ago, with its
waiting years untouched, as the entrypoint starts a new window: it rebuilds
the earliest waiting year's call from its stored rows, then advances the
window (`BudgetWindowStore.advance_window`) with the entrypoint's year
starter (`YearStarts`), which derives each later year's call from the
first's. Just before the advance's first start it marks that year, so the
window is restarted once each time it stops; an advance whose lock or reads
fail marks nothing, and the next check takes the window up again. Its
statements are bounded by the check's statement timeout. A window whose
first call can't be rebuilt is failed at once through the same advance: that
year records `run_not_started` and why, and the window which year failed,
unless the year changed before the advance took the window's lock.

A household report run is restarted as its own kind: the check reads the
report run's household, year, versions and each side's policy, reads the
household and the policies' values again with the entrypoint's reader
(`HouseholdReportStore.read_inputs`), and rebuilds its one household
function call with `household_report_calculation`, started with
`start_report_run`. A run whose .py version's catalog version is no longer
the one its simulation runs were created on, or whose household or policy
is gone or can no longer be calculated, is failed with `run_not_started`
and why, with its simulation runs.

Each check is a `v2_run_restart_check` operation and each row it takes up a
`v2_run_restart` operation, with the row's kind, ID, age and outcome, and
each outcome is also logged as a `v2_run_restart_outcome` event. A deferred
row is taken up again by the next check, so the distinct run IDs are the
runs that needed a restart that a check took up; a row no check took up, in
a burst larger than the checks in its window had time for, is failed by the
sweep with `run_not_started` instead. The events say how each ended.

## Temporary direct runner endpoint

> **Temporary Stage 12 interface:** The authenticated routes in this section
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ class RunConfiguration(StrEnum):
STAGE12_SIMULATION = "stage12_simulation"
STAGE12_SEGMENT = "stage12_segment"
V2_HOUSEHOLD = "v2_household"
V2_RUN_RESTART = "v2_run_restart"


class Stage(StrEnum):
Expand Down Expand Up @@ -116,6 +117,8 @@ class Stage(StrEnum):
V2_HOUSEHOLD_COUNTRY_MODEL_LOAD = "v2_household_country_model_load"
V2_HOUSEHOLD_INPUT_NORMALIZATION = "v2_household_input_normalization"
V2_HOUSEHOLD_CALCULATION = "v2_household_calculation"
V2_RUN_RESTART_CHECK = "v2_run_restart_check"
V2_RUN_RESTART = "v2_run_restart"


@dataclass(frozen=True)
Expand Down Expand Up @@ -287,6 +290,12 @@ def name(self, stage: Stage) -> str:
Stage.V2_HOUSEHOLD_CALCULATION,
),
),
# One scheduled restart check, then each run or household job it
# lists because its Modal call never started.
RunConfiguration.V2_RUN_RESTART: StagePlan(
RunConfiguration.V2_RUN_RESTART,
(Stage.V2_RUN_RESTART_CHECK, Stage.V2_RUN_RESTART),
),
}
)

Expand All @@ -302,3 +311,4 @@ def name(self, stage: Stage) -> str:
STAGE12_SIMULATION_STAGES = RUN_STAGE_REGISTRY[RunConfiguration.STAGE12_SIMULATION]
STAGE12_SEGMENT_STAGES = RUN_STAGE_REGISTRY[RunConfiguration.STAGE12_SEGMENT]
V2_HOUSEHOLD_STAGES = RUN_STAGE_REGISTRY[RunConfiguration.V2_HOUSEHOLD]
V2_RUN_RESTART_STAGES = RUN_STAGE_REGISTRY[RunConfiguration.V2_RUN_RESTART]
11 changes: 11 additions & 0 deletions libs/policyengine-simulation-observability/tests/test_stages.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import pytest
from policyengine_simulation_observability.stages import (
RUN_STAGE_REGISTRY,
V2_RUN_RESTART_STAGES,
RunConfiguration,
Stage,
)
Expand Down Expand Up @@ -66,3 +67,13 @@ def test_v2_household_plan_measures_model_load_inputs_and_each_policy() -> None:
Stage.V2_HOUSEHOLD_CALCULATION,
)
assert household.name(Stage.V2_HOUSEHOLD_CALCULATION) == "v2_household_calculation"


def test_v2_run_restart_plan_measures_each_check_and_each_restart() -> None:
"""The scheduled check is one operation; each row it lists is one more."""
restart = RUN_STAGE_REGISTRY[RunConfiguration.V2_RUN_RESTART]

assert restart is V2_RUN_RESTART_STAGES
assert restart.stages == (Stage.V2_RUN_RESTART_CHECK, Stage.V2_RUN_RESTART)
assert restart.name(Stage.V2_RUN_RESTART_CHECK) == "v2_run_restart_check"
assert restart.name(Stage.V2_RUN_RESTART) == "v2_run_restart"
43 changes: 43 additions & 0 deletions libs/policyengine-simulation-persistence/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,49 @@ reusable run of its own simulation, never giving both sides the same run.
`create_report_rerun`, `create_window_rerun` and `create_simulation_rerun`
run the inserts in a transaction of their own.

## Restarting runs whose call never started

`run_restart_records.py` holds the restart check's reads and writes. The
check runs as the Modal worker role, whose grants cover them: `SELECT` on
the run tables and their inputs, and `UPDATE` of `updated_at`,
`modal_call_id`, `status`, `error_message` and `completed_at`.

- `list_unstarted_runs` returns the oldest rows, up to a limit, created in a
window that are still `pending` with no Modal call and whose `updated_at`
still equals `created_at`: household jobs, report runs outside a budget
window whose baseline is an economy simulation, household report runs
(`household_report_run`, a kind of their own), and runs of economy
simulations that no report run names (standalone runs).
- `list_idle_budget_windows` returns the budget windows that stopped
(`IdleBudgetWindow`, a type apart from the rows): pending or running, with
no year under way and years still waiting untouched, last changed in a
window. `read_earliest_waiting_year` names the year a restart starts the
window from, and `mark_waiting_year` marks it just before its first
start.
- `mark_restart` sets a listed row's `updated_at` to the statement's clock
while it is still unstarted, so it is listed and marked only once.
- `read_unstarted_report_run` and `read_unstarted_simulation_run` read back
what the Simulation Entrypoint built a call from: each side's simulation
run, policy and dynamic, with the values read against the run's own
catalog version, the region, dataset and year the run records, the
report's options and the run's versions. `StoredRunUnreadableError` says
the rows no longer describe one calculation: a side's region or dataset
is gone, or the two sides differ. The values are read with
`read_simulation_reform`, the reader a submission uses.
- `read_unstarted_household_report_run` reads back a household report run:
its report's household and year, the run's versions, each side's
simulation run and policy, and the catalog version the sides were created
on. The entrypoint reads the household and the policies' values again
with `HouseholdReportStore.read_inputs`.
- `read_report_run_simulation_run_ids` returns the simulation runs a report
run names, so one that can't be rebuilt is failed with them, and
`read_run_call` returns a row's status and recorded call, which says whose
call holds it after a restart.

`RunRestartStore` runs each in a transaction of its own. The check records
a call, or fails a row, with `EconomyRunStore`'s and `HouseholdJobStore`'s
own conditional writes.

## Reading a policy

`read_policy_reform` (`policy_reform.py`) reads a stored policy as the reform
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,34 @@
combine_reforms,
read_dynamic_reform,
read_policy_reform,
read_simulation_reform,
)
from policyengine_simulation_persistence.run_restart_records import (
RESTART_TABLES,
EconomyScope,
IdleBudgetWindow,
RestartKind,
RunCall,
StoredEconomySide,
StoredHouseholdReportRun,
StoredHouseholdReportSide,
StoredReportRun,
StoredRunUnreadableError,
StoredSimulationRun,
UnstartedRun,
WaitingYear,
list_idle_budget_windows,
list_unstarted_runs,
mark_restart,
mark_waiting_year,
read_earliest_waiting_year,
read_report_run_simulation_run_ids,
read_run_call,
read_unstarted_household_report_run,
read_unstarted_report_run,
read_unstarted_simulation_run,
)
from policyengine_simulation_persistence.run_restart_store import RunRestartStore
from policyengine_simulation_persistence.tables import (
LATEST_YEAR,
MAX_BUDGET_WINDOW_YEARS,
Expand All @@ -150,6 +177,7 @@
"HOUSEHOLD_REPORT_TYPE",
"LATEST_YEAR",
"MAX_BUDGET_WINDOW_YEARS",
"RESTART_TABLES",
"SERVICE_TABLE_PRIVILEGES",
"SERVICE_UPDATABLE_COLUMNS",
"BudgetWindowInputs",
Expand All @@ -164,6 +192,7 @@
"EconomyReportState",
"EconomyRequest",
"EconomyRunStore",
"EconomyScope",
"EconomySimulationInputs",
"EconomySimulationState",
"HouseholdCalculationInputs",
Expand All @@ -175,6 +204,7 @@
"HouseholdReportSimulation",
"HouseholdReportStore",
"HouseholdRerunSide",
"IdleBudgetWindow",
"MissingPolicyParametersError",
"NewBudgetWindowReport",
"NewEconomyReport",
Expand All @@ -197,8 +227,11 @@
"ReportSimulation",
"RerunRecordDeletedError",
"RerunSide",
"RestartKind",
"ReusableRun",
"RunCall",
"RunInProgressError",
"RunRestartStore",
"RunReuse",
"RunVersions",
"SideDefinition",
Expand All @@ -209,13 +242,21 @@
"SimulationRunState",
"SimulationStatus",
"SimulationType",
"StoredEconomySide",
"StoredHousehold",
"StoredHouseholdReportRun",
"StoredHouseholdReportSide",
"StoredReportRun",
"StoredRunUnreadableError",
"StoredSimulationRun",
"UnknownCatalogVersionError",
"UnknownDatasetError",
"UnknownDynamicError",
"UnknownHouseholdError",
"UnknownPolicyError",
"UnknownRegionError",
"UnstartedRun",
"WaitingYear",
"WindowRerunYear",
"WindowSide",
"WindowYearDefinition",
Expand All @@ -240,13 +281,18 @@
"insert_report_rerun",
"insert_simulation_rerun",
"insert_window_rerun",
"list_idle_budget_windows",
"list_unstarted_runs",
"lock_reruns_of",
"mark_restart",
"mark_waiting_year",
"missing_table_privileges",
"normalized_database_url",
"read_catalog_version_id",
"read_dataset",
"read_dataset_by_id",
"read_dynamic_reform",
"read_earliest_waiting_year",
"read_economy_report",
"read_economy_simulation",
"read_household_job",
Expand All @@ -255,8 +301,14 @@
"read_policy_reform",
"read_region",
"read_report_definition",
"read_report_run_simulation_run_ids",
"read_run_call",
"read_simulation_definition",
"read_simulation_reform",
"read_stored_household",
"read_unstarted_household_report_run",
"read_unstarted_report_run",
"read_unstarted_simulation_run",
"record_household_call",
"record_report_run_call",
"record_simulation_run_call",
Expand Down
Loading
Loading