[anth-volk]

Fixes #993

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

A scheduled Modal function now marks the Public API's report runs and simulation runs failed when nothing will finish them: the run is pending or running but the Modal call recorded for it has ended, for example because its container ran out of memory before it could record the outcome, or the run is still pending with no Modal call recorded ten minutes after it was created, because the simulation service stopped before starting its call. It runs every ten minutes, never fails a run whose call is still running or queued, or whose state Modal cannot report, and two sweeps at once fail each run once. The deployment workflow deploys it with the executor.

This builds on #992, which records each run's Modal call ID when the coordinator or a worker claims the run; review only the commits after it. The simulation service also records, on a pending report run, the Modal call it starts.

How it tells that a call has ended

Every ten minutes the sweep lists the report runs, and then the simulation runs, that have been pending or running for at least ten minutes, counted from their first claim (started_at) or, while pending, from their creation (created_at). A pending run with no call recorded is failed without asking Modal: run_not_started: The run stayed pending with no Modal call recorded for it. Usually the simulation service stopped before starting its call; if it started one but failed to record it, that call's claim is refused once the run has failed. A running run with no call recorded is left to the 24-hour limit. For each recorded call, on a pending or a running run, it asks Modal once for the call graph the call is part of (modal.FunctionCall.from_id(call_id).get_call_graph()) and reads the status of the inputs whose call ID is the recorded one, wherever they are in the graph: a worker spawned inside a coordinator's input is nested under it.

• any PENDING input: the call is still running, queued, or waiting to run again; Modal restarts a pre-empted call under the same call ID and reports it pending meanwhile;
• every input SUCCESS, FAILURE, TERMINATED, TIMEOUT or INIT_FAILURE: the call has ended, and the run is marked failed with modal_call_ended: The Modal call recorded for this run ended without recording the run's outcome.;
• no input with that call ID, or any error asking (an unknown call, Modal unreachable): unknown, and the run is left as it is.

A simulation run is listed only while no pending or running report run names it as its baseline or reform (report_runs.baseline_simulation_run_id or reform_simulation_run_id). That report run's coordinator starts its children's calls once Modal runs it, and records their outcomes itself; after a pre-emption it cancels them and starts them again under new calls, and a sweep while it is queued or in that gap would otherwise fail a run that is about to start. Report runs are swept first, so a dead coordinator's simulation runs are reached in the same sweep.

A run still unfinished 24 hours after its first claim, or after its creation if it was never claimed, is failed whatever Modal reports, with or without a recorded call (run_exceeded_maximum_age). No attempt of these functions can run nearly that long (the coordinator's own time limit is 90 minutes), however often Modal pre-empts and restarts it; the limit stops a run staying running for good when Modal keeps reporting a call it won't retry as pending, no longer describes a call, or no call was recorded.

The failure is one conditional UPDATE … WHERE id = … AND status = <the listed status> AND modal_call_id = <the listed call> (or IS NULL), setting only the status, error, completion time and updated_at, so a run that was claimed, finished, moved to a new call, or had its call recorded by the simulation service since it was listed is left alone, and a second sweep running at the same time changes nothing the first changed.

Where it runs

A new Modal application, policyengine-simulation-run-sweep (src/modal/run_sweep_app.py), with one function, sweep_stuck_runs, on the cron schedule */10 * * * *, at most one container at a time, with the Modal worker role's database secret (policyengine-modal-worker-database) and the locked package group the v2 images install, without the bundle's data. It is not versioned with the .py bundle, so one copy runs per Modal environment rather than one per deployed v2 application. Because it is unversioned and its schedule starts as soon as it deploys, it deploys only from main and only after the release has passed its checks: in simulation-deploy.reusable.yml's publish_stage12_v2_default job (Stage 12's own job, which runs after the integration and authenticated tests and the Cloud Run promotion, in a main deployment and in a Stage 12-only one), after the moved v2 default's reader check, with uv run --frozen modal deploy --env="${{ inputs.modal_environment }}" src/modal/run_sweep_app.py. If that deploy fails, the job fails and its last step writes the previous default back. A deployment dispatched from another branch, which may be behind main, leaves the environment's sweep as it is. Modal refuses the deployment while the Modal secret is missing; the v2 workers' deploy earlier in the run requires it too.

Not covered: household jobs, until the simulation service records their Modal calls. A run marked running with no recorded call is failed only by the 24-hour limit.

Tests

On 2026-11-09 #992's new head, which merged sim's main, was merged in (160ce40f, no conflicts), then #992's later head (585a71e8) and, after #992 merged, main (eb231f1c, no change to any file), each without conflicts. 9944590a fixes test_the_scheduled_function_sweeps_with_the_report_run_store, which failed before and after the merge because it expected the scheduled function's result without failed_without_call; it now sets that count and expects it back. After that, this pull request's 19 changed executor tests and 14 changed store tests passed by node ID, the store tests against a local database as the Modal worker role, and Ruff, the quality guards, the rules document, uv lock --check and Pyright for the Stage 12 persistence library passed. A review round on the merge found no bugs in it, but found that the deploy step, then in deploy_executor, ran before the release's checks and only in main deployments; the user chose on 2026-11-10 to move it after every check. a47c810d requires the step in publish_stage12_v2_default (seen failing), and 03cd559a moves it there and updates the changelog fragment; the job's uv sync --frozen is enough to import the sweep app, and actionlint is clean. A second review round found that a dispatch from another branch still replaced staging's sweep: cb955a3d requires the step's condition github.ref == 'refs/heads/main' (seen failing), and ca51c9d7 adds it and updates the app's docstring, the changelog fragment and the executor README's Stage 12 default section. A third review round found no bugs; its one text finding, the app docstring's account of when the sweep deploys, is fixed in 6d1b16fe. It noted that re-running an older main run after a newer one would redeploy the older sweep.

The store's listing and conditional failure run against a disposable Postgres migrated with the Public API's v2 chain, as the Modal worker role, with the worker's UPDATE limited to its columns, including two sweeps started together in threads, pending runs listed from their creation, a pending run whose call was recorded or claimed after the listing, a run with no recorded call, and simulation runs named, as a baseline or a reform, by pending, running, finished or several report runs, by a report run with no reform run, or by none; they skip when REPORT_RUN_TEST_DATABASE_URL and REPORT_RUN_TEST_SEED_DATABASE_URL are unset, as in CI. The sweep's decisions and the call-graph reading use stand-ins for Modal and the store. The Modal application's declaration is checked with the repository's fake Modal module, and the deployment step by reading the workflow.

Environment variables

• MODAL_WORKER_DATABASE_URL: read by sweep_stuck_runs. Provider: the Modal secret policyengine-modal-worker-database, created by an operator in the staging and main Modal environments (the step Record report runs and simulation runs from the v2 Modal coordinator and workers #992 waits for). Wiring: modal.Secret.from_name(..., required_keys=["MODAL_WORKER_DATABASE_URL"]) attached through secrets= to sweep_stuck_runs. Checked on 2026-11-09: the secret exists in both Modal environments (names only), and its URL logs in as policyengine_v2_modal_worker. Not verified until the first deployment after merging: modal deploy accepts its required key for this app.
• The Modal client in the container authenticates as the workspace the app is deployed in; no other variable is read.
• MODAL_TOKEN_ID, MODAL_TOKEN_SECRET: read by the new deployment step. Provider: GitHub Actions repository secrets (both exist; the beta and prod environments don't override them). Wiring: the beta and prod jobs of simulation-deploy.yml and of stage12-deploy.yml call the reusable workflow with secrets: inherit, and the step's env: reads secrets.MODAL_TOKEN_ID and secrets.MODAL_TOKEN_SECRET, as the executor's deploy step does.

What this waits for before merging

1. Done: Record report runs and simulation runs from the v2 Modal coordinator and workers #992 merged on 2026-11-09, and this pull request's base is now main. Everything else Record report runs and simulation runs from the v2 Modal coordinator and workers #992 waited for was done the same day: Record each run of a v2 simulation and report, and key results by run policyengine-api#4162 deployed, the Modal worker role and its grants in both v2 databases, and the Modal secret policyengine-modal-worker-database in the staging and main Modal environments. Without the secret, the deployment's executor job would fail.
2. An approving review.

After the first deployment, check that modal app list shows the app in each environment and that its first scheduled run returned a summary (checked, failed, running, unknown, left, failed_for_age, failed_without_call) without errors in modal app logs policyengine-simulation-run-sweep.

🤖 Generated with Claude Code