Skip to content

Start a run or household job again when its Modal call never started - #1063

Draft
anth-volk wants to merge 32 commits into
mainfrom
v2plan/c1-19-restart-unstarted-runs
Draft

anth-volk wants to merge 32 commits into
mainfrom
v2plan/c1-19-restart-unstarted-runs

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1058

What this does

When the Simulation Entrypoint creates a household job, economy or household report run, or standalone simulation run but stops before starting its Modal call, nothing records a call for it. Until now the stuck-run sweep failed such a run with run_not_started at 10 minutes. With this change, a restart check starts the run's call once more, and the sweep fails it only if that restart misses too.

Where it runs. It is a second scheduled function, restart_unstarted_runs, in the stuck-run sweep's Modal app (policyengine-simulation-run-sweep). It runs every two minutes as the Modal worker role, from the sweep's Modal secret policyengine-modal-worker-database. The restart code itself is the simulation service's (policyengine_simulation_entry.run_restarts, run_restart_calls, run_restart_check), on policyengine_simulation_persistence (run_restart_records, run_restart_store). The image adds both as local sources.

What one check does:

  1. Lists, oldest first and at most 20, the rows created 3 to 10 minutes ago that are still pending with no modal_call_id and whose updated_at still equals created_at:

    • household jobs;
    • report runs outside a budget window whose baseline is an economy simulation;
    • household report runs (household_report_run, a kind of their own: one household function call calculates both sides);
    • runs of economy simulations that no report run names (standalone runs).

    A report's own simulation runs are its coordinator's to start, so they are never listed.

  2. Reads back, on a thread, what the submission built the call from, each side's values read against the run's own catalog version with the submission's own reader. Then it rebuilds the call with the submission's own builders (report_run_calculation, simulation_run_calculation, prepare_household_job, household_report_calculation). It uses the manifest's worker for the .py version the row records, even when that worker is no longer the default.

    • The rebuilt payload equals the original. The context differs only in its request ID (run-restart-<uuid>).
    • A household job whose rebuilt job would record anything else is not restarted: for example, a changed default year or catalog version.
  3. Sets the row's updated_at (with clock_timestamp()), so the row is restarted at most once, even by two checks at once. Then it starts the call and records its ID with the submission's own conditional write.

  4. Counts the restart as done when the row then records the check's call. A row recording another call was claimed by that call first.

After three minutes a check marks no more rows and leaves the rest for the next check. A marked row then needs only its call's start and a few bounded statements, so the check ends within the function's five-minute timeout, and every row it marks gets its call recorded before the sweep's 15 minutes.

When a restart can't go ahead:

  • A row whose call can't be rebuilt is failed at once with run_not_started: …. That covers a worker that left the manifest, stored rows that changed, and a policy that is gone. A report run is failed together with its simulation runs that are still pending with no call, as its submission would fail it.
  • So is a row whose restarted call Modal refuses, or doesn't confirm within ten seconds.
  • A row whose rows or manifest can't be read right now is left unmarked, and the next check tries it again. So is a row whose rebuild hits an unexpected error, which is logged with its traceback (v2_run_restart_unexpected_error); the check goes on to the next row.
  • A late original call claims a row only while the row records no call or its own. Once the restart records its call, the late call's claim is refused. If the late call claims first, the restarted call's claim is refused instead.

Budget windows (from #1054). A window's years are never restarted one by one. A window stops when no year is under way and years still wait: the entrypoint stopped before starting its first years, or a year's coordinator stopped before starting the next. Each check also lists the windows that stopped 3 to 10 minutes ago, with their waiting years untouched. For each, it:

  1. rebuilds the earliest waiting year's coordinator call from its stored rows;
  2. advances the window (BudgetWindowStore.advance_window) with the entrypoint's own year starter (YearStarts), as the entrypoint starts a new window's first years. The starter derives each later year's call from the first's and shares one wait for Modal, and the advance starts at most five years under the window's row lock;
  3. marks that year just before the advance's first start, so the window is restarted once each time it stops. An advance whose lock or reads fail starts and marks nothing, and the next check takes the window up again.

A year Modal refuses fails the window, as it would at the entrypoint's start. A window whose first call can't be rebuilt is failed at once through the same advance: the starter refuses that year with YearStartRefused, so the year and its own simulation runs record run_not_started: … and why, and the window records which year failed (budget_window_year_failed). An unexpected error in a window's restart is logged with its traceback and ends only that window's restart. The window advance's statements are bounded by the check's 5-second statement timeout.

Idle windows are a type of their own (IdleBudgetWindow), kept apart from the rows the check marks and records calls on.

Household report runs (from #1057). The check reads back the report's household and year, the run's versions, each side's simulation run and policy, and the catalog version the sides were created on (read_unstarted_household_report_run). It reads the household and the policies' values again with the entrypoint's reader (HouseholdReportStore.read_inputs), rebuilds the one household function call with household_report_calculation on the worker of the run's .py version, and starts it with start_report_run. The call is recorded, or the run failed with its simulation runs, with the same report run writes. A run whose .py version's catalog version is no longer the one its runs were created on, whose worker records other versions, or whose household or policy is gone or can no longer be calculated, is failed with run_not_started and why.

The sweep now waits 15 minutes (DEFAULT_MINIMUM_AGE, was 10) before failing a pending run with no call. That is after the restart window, so a run whose restart started is judged by its call, and a second miss is still failed with run_not_started.

Telemetry (how often runs need a restart):

  • Each check is a v2_run_restart_check operation. The counts are set on it as restart_listed, restart_restarted, restart_failed, and so on.
  • Each row a check takes up is a v2_run_restart operation with run_kind, run_id and run_age_seconds. A deferred row is taken up again by the next check, so the runs whose call never started are the distinct run_id values.
  • Each outcome is set on that operation (restart_outcome, restart_error_type). It is also logged as a v2_run_restart_outcome event, at WARNING when the row was failed, couldn't be read or couldn't be recorded.
  • The function's return value is the same counts, which Modal's dashboard shows per run.

Smaller changes:

  • ManifestEconomyWorkers and ManifestHouseholdWorkers gain worker_for_version.
  • BudgetWindowStore takes statement_timeout_milliseconds, set local to each transaction, and its year starters may raise YearStartRefused(error_message) to fail a year with a message of their own. The service's and the coordinators' starters are unchanged.
  • YearStarts (from budget_window_submissions) is public, so the restart starts a window's years with the entrypoint's own starter.
  • The entry package's __init__ loads create_app on first use. Before, importing any of its modules imported the FastAPI app, whose PyJWT and cryptography the sweep's image doesn't install.
  • The executor's dev group gains the entry and persistence packages. The deploy job's uv sync installs them for modal deploy, which adds them to the sweep image as local sources. The executor's own Docker image is built without that group, so it doesn't need them. The relock adds only those two.

What waits for other pull requests

Once each merges, main is merged into this branch and the restart covers what it adds.

Deployment

The deploy step Deploy the stuck-run sweep and the restart check in publish_stage12_v2_default deploys the app, as before. It runs only from main, after every check. Each variable the changed code or workflow reads or declares:

Variable Provider Wiring Confirmed
MODAL_TOKEN_ID, MODAL_TOKEN_SECRET GitHub Actions secrets The step's env: (unchanged), for modal deploy. Yes, unchanged
SIMULATION_V2_MODAL_ENVIRONMENT The reusable workflow's modal_environment input (staging or main, from simulation-deploy.yml) The step's env:. run_sweep_app.py bakes it into the image's environment at deploy time. The restart function reads it, then maps it to the deployment label (staging or production) that a call's context and artifact prefix name. Yes. The input is used by the same step's command.
OBSERVABILITY_*, OTEL_EXPORTER_OTLP_ENDPOINT, POLICYENGINE_OTEL_GOOGLE_AUDIENCE GitHub Actions repository variables (vars.OBSERVABILITY_*) The step's env:, the same eight entries as the v2 workers' deploy step. modal_image_environment() bakes them into the image. Yes. The variables exist at repository level.
MODAL_WORKER_DATABASE_URL Modal secret policyengine-modal-worker-database Attached to both scheduled functions, with required_keys. Yes. The sweep already requires it in staging and main.
MODAL_IDENTITY_TOKEN, MODAL_TASK_ID, MODAL_REGION Modal, at run time Read by the observability runtime. Provided by Modal

Telemetry export needs an operator step (not done). The modal-api-v1 workload identity provider (project policyengine-observability) admits only these app names:

  • policyengine-simulation-gateway;
  • names matching policyengine-simulation-py…;
  • names matching policyengine-simulation-v2-py….

So the sweep's app can't get Google credentials. Until its condition also admits policyengine-simulation-run-sweep, the restart check's records reach Modal's own logs but not Cloud Logging or the collector.

Tests

All run by node ID; CI runs the rest.

  • Persistence: the 26 tests of test_run_restart_records.py, against a PostgreSQL 17 database built by the Public API master's v2 chain (429d348ae), with its privileges reconciled. They run as the Modal worker role.

  • Stage 12 persistence: the three new tests of test_budget_window_store.py (a start refused with its own message, the statement timeout on a locked window, and a starter writing its own year on another connection during the advance), against the same database, logged in as the Modal worker role. No CI job sets these tests' database URLs, so they skip in CI.

  • Entry:

    • the 26 tests (27 cases) of test_run_restarts.py;
    • the 20 tests (22 cases) of test_run_restart_windows.py, including one comparing a window's restarted first-year call with the one submit_budget_window started;
    • the 9 tests of test_run_restart_household_reports.py, each starting from a household report submitted through HouseholdReportSubmissionService;
    • the 8 tests (13 cases) of test_run_restart_check.py;
    • the 3 new tests each in test_economy_workers.py and test_household_workers.py.
  • Executor:

    • the changed and new tests of test_stage12_run_sweep_app.py;
    • test_stage12_run_sweep.py::test_by_default_unstarted_runs_wait_until_the_restart_check_has_passed;
    • test_executor_image_dependencies.py, which checks the Docker image's build holds every main dependency from a local path;
    • test_run_restart_image.py, which imports the restart check with only the sweep image's packages, and requires every module it loads to come from the standard library, a local source or a distribution the image installs.

    A dry run of the Docker image's uv sync --frozen --no-dev, in a copy holding only its build directories, also succeeds.

  • Observability: test_stages.py::test_v2_run_restart_plan_measures_each_check_and_each_restart.

Review

Eight read-only review rounds, each by a reviewer following the review-code skill:

  • Round 1 found three bugs, each fixed with a failing test first:

    • the executor's Docker image no longer built, because the entry package was a main dependency outside the image's build;
    • a report run whose rows couldn't be read was failed without its simulation runs;
    • a restart whose own call claimed the row before the record was counted as another call's.
  • Round 2 found no bugs. Its quality findings were fixed:

    • the restart window is narrowed to 3–10 minutes, for a margin before the sweep;
    • an unexpected rebuild error defers only its own row;
    • the telemetry docs are corrected;
    • an image import test is added.
  • Round 3 found no bugs. Its quality findings were fixed: the image test checks every module file the import loads, and unexpected errors are logged with their traceback.

  • Round 4, on round 3's fixes, found one bug, fixed test-first: the unexpected-error log's run kind and ID didn't reach the observability runtime's handler.

  • Round 5, on round 4's fixes, found no bugs. Its findings were wording only.

  • Round 6, on the budget-window restart added after main was merged in, found three bugs, each fixed with a failing test first:

    • an unexpected error in one window's restart ended the whole check;
    • a database error between the window year's mark and its first start used up the window's only restart, so the sweep failed it as stalled; the mark now happens just before the first start, inside the advance;
    • a window whose first call couldn't be rebuilt was left for the sweep, with a stored error naming the wrong cause; it is now failed at once with the reason.

    Its quality findings were fixed too: the window advance's statement timeout, the outcome descriptions, Calculate budget windows on the detailed-save path, five years at a time #1054's docstrings, the window tests (now in their own module), and a type of their own for idle windows.

  • Round 7, on round 6's fixes, found two bugs, each fixed test-first: a refusal could fail a year other than the one whose call couldn't be rebuilt, and an unexpected error from the mark was treated as Modal refusing the year. Its quality findings added a PostgreSQL test of a starter writing its own year during the advance, and mark_waiting_year.

  • Round 8, on round 7's fixes and the household report restart, found no bugs. Its quality findings were fixed: a refused policy is counted under its cause's error type, the report run kinds are defined once, more household report tests, and wording.

🤖 Generated with Claude Code

anth-volk and others added 24 commits November 13, 2026 01:17
List the pending runs and household jobs with no Modal call recorded,
mark one restart of each, and read a report run's or standalone
simulation run's stored inputs back, as the Modal worker role.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart check lists the household jobs, report runs and standalone
simulation runs still pending with no Modal call that nothing has
changed since their insert, marks each one before restarting it so it
is restarted at most once, and reads back the stored rows its call is
built from, against the run's own catalog version.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ling)

The restart rebuilds each run's call from its stored rows and starts
the same call its submission started, on the worker of the run's own
.py version; a run that can't be rebuilt, or whose restart Modal
refuses, is failed with run_not_started, and each listed run is one
telemetry scope with its outcome.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart service lists the rows created 3 to 12 minutes ago that are
still pending with no call, rebuilds each call from its stored rows on
the worker of the row's own .py version, marks the row, starts the
call and records its ID. A row that can't be rebuilt, or whose restart
Modal refuses, is failed with run_not_started; one whose rows or
manifest can't be read now is left for the next check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart service keeps the check's loop and its writes; the call
builder, its stores' protocols and the errors that defer or fail a
restart move to run_restart_calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…failing)

The restart check runs every two minutes in the sweep's Modal app on
the Modal worker's database secret, with the Modal environment and the
v2 workers' telemetry settings baked into its image; each check and
each restart is an operation named by a registered stage, with the
restart's outcome logged as an event. The sweep now waits 15 minutes
before failing a pending run with no call, after the restart window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sweep's Modal app gains a scheduled function that runs the
simulation service's restart check as the Modal worker role, with the
Modal environment and the v2 workers' telemetry settings baked into its
image at deployment. Each check and each restarted run is an operation
named by a registered stage, and each restart's outcome is logged as an
event. The sweep now fails a pending run with no call at 15 minutes,
after the restart window; the entry package loads its web app lazily so
the restart code imports in the sweep's image.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ut (failing)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The executor's image builds only from the directories its Dockerfile
copies; an unreadable report run is failed with its simulation runs; a
restart is counted by whose call holds the row afterwards; a check
marks no row once its time limit has passed; and a call is rebuilt off
the event loop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The executor's Docker image builds again: the Simulation Entrypoint and
its persistence library move to the executor's dev group, which the
deploy job installs for modal deploy and the image does not. A report
run whose rows can't be rebuilt is failed with its simulation runs. A
restart is counted by whose call the row records afterwards, so a
restarted call that claims its run before the record counts as
restarted. A check marks no row after three minutes, rebuilds each call
on a thread so Modal's blocking reads don't hold its event loop, and a
restart reads its values with the submission's own reader.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A restart is recorded with a minute to spare before the sweep judges
its run, an unexpected rebuild error defers only its own run, and the
restart check imports with only the sweep image's packages (this last
test passed when first run: it guards what already holds).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart window is now 3 to 10 minutes after creation, so a row
marked at the end of a check's time limit has its call recorded minutes
before the sweep judges it; an unexpected error rebuilding one row's
call defers that row and the check goes on; and the telemetry docs say
how to count runs from restart attempts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An unexpected rebuild error is logged with its traceback, the image
test also requires every module the restart check loads to come from
the standard library, a local source or an image distribution, and the
persistence window test uses the 3-to-10-minute window.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An unexpected error rebuilding a row's call is logged with its
traceback, the descriptions of a deferred row name that case, and the
telemetry docs count the runs that needed a restart by distinct run ID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iling)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The unexpected-error log passes its run kind and ID as
policyengine_attributes, which the observability runtime's log handler
writes; the image test checks only the modules the import adds; and the
descriptions of a deferred row and of the restart counts name every
case.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wording fixes from review round 5; no behaviour changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in #1054's budget windows. The sweep's 15-minute age now also
times its settling of idle windows, and the migration document and the
sweep app's docstring say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An idle window, with no year under way and its waiting years untouched,
is listed for a restart; its earliest waiting year's call is rebuilt
from its stored rows, that year is marked, and the window is advanced
as the service starts its first years.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart check also lists the budget windows that stopped 3 to 10
minutes ago, with no year under way and waiting years untouched. It
rebuilds the earliest waiting year's call from its stored rows, marks
that year so the window restarts once each time it stops, and advances
the window with the service's own year starter, as the service starts a
new window's first years. A window whose first call can't be rebuilt is
marked and left for the sweep to fail as stalled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A window is marked only just before its first start, so a database
error before any start leaves it for the next check; an unexpected
error defers it and the check goes on; a window whose first call can't
be built again fails with the reason. The window tests move to a module
of their own, and idle windows become a type of their own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The window's earliest waiting year is now marked just before the
advance's first start, by the starter the window is advanced with, so
an advance whose lock or reads fail marks nothing and the next check
takes the window up again. An unexpected error in a window's restart is
logged with its traceback and ends only that window's restart. A window
whose first call can't be built again fails at once through the same
advance: the starter refuses that year with YearStartRefused, so the
year records run_not_started and why, and the window which year failed.

The restart's budget-window store bounds each statement with the
check's statement timeout, idle windows are a type of their own
(IdleBudgetWindow), the outcome descriptions cover windows, and the
docstrings that said only the sweep handles a stopped window now name
the restart check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 5 commits November 13, 2026 19:00
A refusal fails only the year whose call couldn't be built again; an
unexpected error from the mark defers the window and is logged; a
window's year is marked by itself, not as a stand-in row. Also covers a
refusal its year's own call overtook, a refusal that can't be written,
and a starter that writes its own year during the advance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A window whose first call can't be built again refuses only that year:
when the advance offers another first, the year read changed meanwhile,
and nothing is failed. An error from the mark other than the database's
is logged with its traceback and defers the window, rather than failing
its year as refused by Modal. A window's year is marked by itself
(mark_waiting_year), not as a stand-in row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sehold reports

The conflicts keep both sides: the persistence library's exports, its
README's household report and restart sections, the Stage 12 library's
README, the household worker tests, and the sweep's paragraph in the
Stage 12 foundation document, which keeps #1057's household report runs
and this branch's 15 minutes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ing)

A household report run is listed as its own kind, read back from its
stored rows, and started again with the call its submission started,
rebuilt from the household and policies read against the run's own
catalog version. One whose household, policy, catalog version or
worker changed is failed with why, with its simulation runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart check lists household report runs as a kind of their own,
reads back the report's household and year, the run's versions, each
side's simulation run and policy, and the catalog version the sides were
created on. It reads the household and the policies' values again with
the submission's reader, rebuilds the one household function call with
household_report_calculation on the worker of the run's .py version,
and starts it with start_report_run. The call is recorded, or the run
failed with its simulation runs, with the report run writes economy
report runs use. A run whose catalog version, worker, household or
policy changed is failed with run_not_started and why.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 3 commits November 13, 2026 20:06
A side's refused policy is counted under its cause's error type. Also
covers a policy naming parameters its catalog lacks, a payload the
contract refuses, a worker gone from the manifest, rows that no longer
describe the run, an unexpected rebuild error, and the Modal worker
role reading a household report's inputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A report side's refused policy is counted under its cause's error type,
as its stored reason already names the cause. The kinds recorded on a
report run are defined once (REPORT_RUN_KINDS), so a report run is never
recorded or failed on household_jobs. The listing's docstring names
household report runs, and the household report service names the
restart check's own module.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…runs

Conflicts kept both sides: the persistence library's README holds the
re-run section and the restart section, its exports list both sides'
names, and `_start_first_years`'s docstring names both the restart check and
`rerun`.

The merged `economy_run_store.py` was 801 lines, over the 800-line limit,
so `read_simulation_reform`, which this branch added, moves unchanged to
`policy_reform.py`, beside the policy and dynamic readers it combines.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Start a run or household job again when its Modal call never started

1 participant