Repository navigation
Conversation
List the pending runs and household jobs with no Modal call recorded, mark one restart of each, and read a report run's or standalone simulation run's stored inputs back, as the Modal worker role. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart check lists the household jobs, report runs and standalone simulation runs still pending with no Modal call that nothing has changed since their insert, marks each one before restarting it so it is restarted at most once, and reads back the stored rows its call is built from, against the run's own catalog version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ling) The restart rebuilds each run's call from its stored rows and starts the same call its submission started, on the worker of the run's own .py version; a run that can't be rebuilt, or whose restart Modal refuses, is failed with run_not_started, and each listed run is one telemetry scope with its outcome. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart service lists the rows created 3 to 12 minutes ago that are still pending with no call, rebuilds each call from its stored rows on the worker of the row's own .py version, marks the row, starts the call and records its ID. A row that can't be rebuilt, or whose restart Modal refuses, is failed with run_not_started; one whose rows or manifest can't be read now is left for the next check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart service keeps the check's loop and its writes; the call builder, its stores' protocols and the errors that defer or fail a restart move to run_restart_calls. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…failing) The restart check runs every two minutes in the sweep's Modal app on the Modal worker's database secret, with the Modal environment and the v2 workers' telemetry settings baked into its image; each check and each restart is an operation named by a registered stage, with the restart's outcome logged as an event. The sweep now waits 15 minutes before failing a pending run with no call, after the restart window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The sweep's Modal app gains a scheduled function that runs the simulation service's restart check as the Modal worker role, with the Modal environment and the v2 workers' telemetry settings baked into its image at deployment. Each check and each restarted run is an operation named by a registered stage, and each restart's outcome is logged as an event. The sweep now fails a pending run with no call at 15 minutes, after the restart window; the entry package loads its web app lazily so the restart code imports in the sweep's image. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ut (failing) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The executor's image builds only from the directories its Dockerfile copies; an unreadable report run is failed with its simulation runs; a restart is counted by whose call holds the row afterwards; a check marks no row once its time limit has passed; and a call is rebuilt off the event loop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The executor's Docker image builds again: the Simulation Entrypoint and its persistence library move to the executor's dev group, which the deploy job installs for modal deploy and the image does not. A report run whose rows can't be rebuilt is failed with its simulation runs. A restart is counted by whose call the row records afterwards, so a restarted call that claims its run before the record counts as restarted. A check marks no row after three minutes, rebuilds each call on a thread so Modal's blocking reads don't hold its event loop, and a restart reads its values with the submission's own reader. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A restart is recorded with a minute to spare before the sweep judges its run, an unexpected rebuild error defers only its own run, and the restart check imports with only the sweep image's packages (this last test passed when first run: it guards what already holds). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart window is now 3 to 10 minutes after creation, so a row marked at the end of a check's time limit has its call recorded minutes before the sweep judges it; an unexpected error rebuilding one row's call defers that row and the check goes on; and the telemetry docs say how to count runs from restart attempts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An unexpected rebuild error is logged with its traceback, the image test also requires every module the restart check loads to come from the standard library, a local source or an image distribution, and the persistence window test uses the 3-to-10-minute window. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An unexpected error rebuilding a row's call is logged with its traceback, the descriptions of a deferred row name that case, and the telemetry docs count the runs that needed a restart by distinct run ID. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iling) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The unexpected-error log passes its run kind and ID as policyengine_attributes, which the observability runtime's log handler writes; the image test checks only the modules the import adds; and the descriptions of a deferred row and of the restart counts name every case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Wording fixes from review round 5; no behaviour changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in #1054's budget windows. The sweep's 15-minute age now also times its settling of idle windows, and the migration document and the sweep app's docstring say so. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An idle window, with no year under way and its waiting years untouched, is listed for a restart; its earliest waiting year's call is rebuilt from its stored rows, that year is marked, and the window is advanced as the service starts its first years. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart check also lists the budget windows that stopped 3 to 10 minutes ago, with no year under way and waiting years untouched. It rebuilds the earliest waiting year's call from its stored rows, marks that year so the window restarts once each time it stops, and advances the window with the service's own year starter, as the service starts a new window's first years. A window whose first call can't be rebuilt is marked and left for the sweep to fail as stalled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A window is marked only just before its first start, so a database error before any start leaves it for the next check; an unexpected error defers it and the check goes on; a window whose first call can't be built again fails with the reason. The window tests move to a module of their own, and idle windows become a type of their own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The window's earliest waiting year is now marked just before the advance's first start, by the starter the window is advanced with, so an advance whose lock or reads fail marks nothing and the next check takes the window up again. An unexpected error in a window's restart is logged with its traceback and ends only that window's restart. A window whose first call can't be built again fails at once through the same advance: the starter refuses that year with YearStartRefused, so the year records run_not_started and why, and the window which year failed. The restart's budget-window store bounds each statement with the check's statement timeout, idle windows are a type of their own (IdleBudgetWindow), the outcome descriptions cover windows, and the docstrings that said only the sweep handles a stopped window now name the restart check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A refusal fails only the year whose call couldn't be built again; an unexpected error from the mark defers the window and is logged; a window's year is marked by itself, not as a stand-in row. Also covers a refusal its year's own call overtook, a refusal that can't be written, and a starter that writes its own year during the advance. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A window whose first call can't be built again refuses only that year: when the advance offers another first, the year read changed meanwhile, and nothing is failed. An error from the mark other than the database's is logged with its traceback and defers the window, rather than failing its year as refused by Modal. A window's year is marked by itself (mark_waiting_year), not as a stand-in row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…sehold reports The conflicts keep both sides: the persistence library's exports, its README's household report and restart sections, the Stage 12 library's README, the household worker tests, and the sweep's paragraph in the Stage 12 foundation document, which keeps #1057's household report runs and this branch's 15 minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ing) A household report run is listed as its own kind, read back from its stored rows, and started again with the call its submission started, rebuilt from the household and policies read against the run's own catalog version. One whose household, policy, catalog version or worker changed is failed with why, with its simulation runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The restart check lists household report runs as a kind of their own, reads back the report's household and year, the run's versions, each side's simulation run and policy, and the catalog version the sides were created on. It reads the household and the policies' values again with the submission's reader, rebuilds the one household function call with household_report_calculation on the worker of the run's .py version, and starts it with start_report_run. The call is recorded, or the run failed with its simulation runs, with the report run writes economy report runs use. A run whose catalog version, worker, household or policy changed is failed with run_not_started and why. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A side's refused policy is counted under its cause's error type. Also covers a policy naming parameters its catalog lacks, a payload the contract refuses, a worker gone from the manifest, rows that no longer describe the run, an unexpected rebuild error, and the Modal worker role reading a household report's inputs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A report side's refused policy is counted under its cause's error type, as its stored reason already names the cause. The kinds recorded on a report run are defined once (REPORT_RUN_KINDS), so a report run is never recorded or failed on household_jobs. The listing's docstring names household report runs, and the household report service names the restart check's own module. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…runs Conflicts kept both sides: the persistence library's README holds the re-run section and the restart section, its exports list both sides' names, and `_start_first_years`'s docstring names both the restart check and `rerun`. The merged `economy_run_store.py` was 801 lines, over the 800-line limit, so `read_simulation_reform`, which this branch added, moves unchanged to `policy_reform.py`, beside the policy and dynamic readers it combines. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1058
What this does
When the Simulation Entrypoint creates a household job, economy or household report run, or standalone simulation run but stops before starting its Modal call, nothing records a call for it. Until now the stuck-run sweep failed such a run with
run_not_startedat 10 minutes. With this change, a restart check starts the run's call once more, and the sweep fails it only if that restart misses too.Where it runs. It is a second scheduled function,
restart_unstarted_runs, in the stuck-run sweep's Modal app (policyengine-simulation-run-sweep). It runs every two minutes as the Modal worker role, from the sweep's Modal secretpolicyengine-modal-worker-database. The restart code itself is the simulation service's (policyengine_simulation_entry.run_restarts,run_restart_calls,run_restart_check), onpolicyengine_simulation_persistence(run_restart_records,run_restart_store). The image adds both as local sources.What one check does:
Lists, oldest first and at most 20, the rows created 3 to 10 minutes ago that are still
pendingwith nomodal_call_idand whoseupdated_atstill equalscreated_at:household_report_run, a kind of their own: one household function call calculates both sides);A report's own simulation runs are its coordinator's to start, so they are never listed.
Reads back, on a thread, what the submission built the call from, each side's values read against the run's own catalog version with the submission's own reader. Then it rebuilds the call with the submission's own builders (
report_run_calculation,simulation_run_calculation,prepare_household_job,household_report_calculation). It uses the manifest's worker for the .py version the row records, even when that worker is no longer the default.run-restart-<uuid>).Sets the row's
updated_at(withclock_timestamp()), so the row is restarted at most once, even by two checks at once. Then it starts the call and records its ID with the submission's own conditional write.Counts the restart as done when the row then records the check's call. A row recording another call was claimed by that call first.
After three minutes a check marks no more rows and leaves the rest for the next check. A marked row then needs only its call's start and a few bounded statements, so the check ends within the function's five-minute timeout, and every row it marks gets its call recorded before the sweep's 15 minutes.
When a restart can't go ahead:
run_not_started: …. That covers a worker that left the manifest, stored rows that changed, and a policy that is gone. A report run is failed together with its simulation runs that are still pending with no call, as its submission would fail it.v2_run_restart_unexpected_error); the check goes on to the next row.Budget windows (from #1054). A window's years are never restarted one by one. A window stops when no year is under way and years still wait: the entrypoint stopped before starting its first years, or a year's coordinator stopped before starting the next. Each check also lists the windows that stopped 3 to 10 minutes ago, with their waiting years untouched. For each, it:
BudgetWindowStore.advance_window) with the entrypoint's own year starter (YearStarts), as the entrypoint starts a new window's first years. The starter derives each later year's call from the first's and shares one wait for Modal, and the advance starts at most five years under the window's row lock;A year Modal refuses fails the window, as it would at the entrypoint's start. A window whose first call can't be rebuilt is failed at once through the same advance: the starter refuses that year with
YearStartRefused, so the year and its own simulation runs recordrun_not_started: …and why, and the window records which year failed (budget_window_year_failed). An unexpected error in a window's restart is logged with its traceback and ends only that window's restart. The window advance's statements are bounded by the check's 5-second statement timeout.Idle windows are a type of their own (
IdleBudgetWindow), kept apart from the rows the check marks and records calls on.Household report runs (from #1057). The check reads back the report's household and year, the run's versions, each side's simulation run and policy, and the catalog version the sides were created on (
read_unstarted_household_report_run). It reads the household and the policies' values again with the entrypoint's reader (HouseholdReportStore.read_inputs), rebuilds the one household function call withhousehold_report_calculationon the worker of the run's .py version, and starts it withstart_report_run. The call is recorded, or the run failed with its simulation runs, with the same report run writes. A run whose .py version's catalog version is no longer the one its runs were created on, whose worker records other versions, or whose household or policy is gone or can no longer be calculated, is failed withrun_not_startedand why.The sweep now waits 15 minutes (
DEFAULT_MINIMUM_AGE, was 10) before failing a pending run with no call. That is after the restart window, so a run whose restart started is judged by its call, and a second miss is still failed withrun_not_started.Telemetry (how often runs need a restart):
v2_run_restart_checkoperation. The counts are set on it asrestart_listed,restart_restarted,restart_failed, and so on.v2_run_restartoperation withrun_kind,run_idandrun_age_seconds. A deferred row is taken up again by the next check, so the runs whose call never started are the distinctrun_idvalues.restart_outcome,restart_error_type). It is also logged as av2_run_restart_outcomeevent, at WARNING when the row was failed, couldn't be read or couldn't be recorded.Smaller changes:
ManifestEconomyWorkersandManifestHouseholdWorkersgainworker_for_version.BudgetWindowStoretakesstatement_timeout_milliseconds, set local to each transaction, and its year starters may raiseYearStartRefused(error_message)to fail a year with a message of their own. The service's and the coordinators' starters are unchanged.YearStarts(frombudget_window_submissions) is public, so the restart starts a window's years with the entrypoint's own starter.__init__loadscreate_appon first use. Before, importing any of its modules imported the FastAPI app, whose PyJWT and cryptography the sweep's image doesn't install.devgroup gains the entry and persistence packages. The deploy job'suv syncinstalls them formodal deploy, which adds them to the sweep image as local sources. The executor's own Docker image is built without that group, so it doesn't need them. The relock adds only those two.What waits for other pull requests
mainis merged in. Stopped windows are restarted as described above. Calculate budget windows on the detailed-save path, five years at a time #1054's idle-window settling uses the sweep's minimum age, so it moves to 15 minutes too, and its docs say so.mainis merged in. Household report runs are restarted as described above. Household simulations' runs that no report names are still left out; Create household reports through POST /reports and calculate them in one household function call #1057 creates none.POST /reportsandPOST /simulationsdo.Once each merges,
mainis merged into this branch and the restart covers what it adds.Deployment
The deploy step
Deploy the stuck-run sweep and the restart checkinpublish_stage12_v2_defaultdeploys the app, as before. It runs only frommain, after every check. Each variable the changed code or workflow reads or declares:MODAL_TOKEN_ID,MODAL_TOKEN_SECRETenv:(unchanged), formodal deploy.SIMULATION_V2_MODAL_ENVIRONMENTmodal_environmentinput (stagingormain, fromsimulation-deploy.yml)env:.run_sweep_app.pybakes it into the image's environment at deploy time. The restart function reads it, then maps it to the deployment label (stagingorproduction) that a call's context and artifact prefix name.OBSERVABILITY_*,OTEL_EXPORTER_OTLP_ENDPOINT,POLICYENGINE_OTEL_GOOGLE_AUDIENCEvars.OBSERVABILITY_*)env:, the same eight entries as the v2 workers' deploy step.modal_image_environment()bakes them into the image.MODAL_WORKER_DATABASE_URLpolicyengine-modal-worker-databaserequired_keys.stagingandmain.MODAL_IDENTITY_TOKEN,MODAL_TASK_ID,MODAL_REGIONTelemetry export needs an operator step (not done). The
modal-api-v1workload identity provider (projectpolicyengine-observability) admits only these app names:policyengine-simulation-gateway;policyengine-simulation-py…;policyengine-simulation-v2-py….So the sweep's app can't get Google credentials. Until its condition also admits
policyengine-simulation-run-sweep, the restart check's records reach Modal's own logs but not Cloud Logging or the collector.Tests
All run by node ID; CI runs the rest.
Persistence: the 26 tests of
test_run_restart_records.py, against a PostgreSQL 17 database built by the Public API master's v2 chain (429d348ae), with its privileges reconciled. They run as the Modal worker role.Stage 12 persistence: the three new tests of
test_budget_window_store.py(a start refused with its own message, the statement timeout on a locked window, and a starter writing its own year on another connection during the advance), against the same database, logged in as the Modal worker role. No CI job sets these tests' database URLs, so they skip in CI.Entry:
test_run_restarts.py;test_run_restart_windows.py, including one comparing a window's restarted first-year call with the onesubmit_budget_windowstarted;test_run_restart_household_reports.py, each starting from a household report submitted throughHouseholdReportSubmissionService;test_run_restart_check.py;test_economy_workers.pyandtest_household_workers.py.Executor:
test_stage12_run_sweep_app.py;test_stage12_run_sweep.py::test_by_default_unstarted_runs_wait_until_the_restart_check_has_passed;test_executor_image_dependencies.py, which checks the Docker image's build holds every main dependency from a local path;test_run_restart_image.py, which imports the restart check with only the sweep image's packages, and requires every module it loads to come from the standard library, a local source or a distribution the image installs.A dry run of the Docker image's
uv sync --frozen --no-dev, in a copy holding only its build directories, also succeeds.Observability:
test_stages.py::test_v2_run_restart_plan_measures_each_check_and_each_restart.Review
Eight read-only review rounds, each by a reviewer following the review-code skill:
Round 1 found three bugs, each fixed with a failing test first:
Round 2 found no bugs. Its quality findings were fixed:
Round 3 found no bugs. Its quality findings were fixed: the image test checks every module file the import loads, and unexpected errors are logged with their traceback.
Round 4, on round 3's fixes, found one bug, fixed test-first: the unexpected-error log's run kind and ID didn't reach the observability runtime's handler.
Round 5, on round 4's fixes, found no bugs. Its findings were wording only.
Round 6, on the budget-window restart added after
mainwas merged in, found three bugs, each fixed with a failing test first:Its quality findings were fixed too: the window advance's statement timeout, the outcome descriptions, Calculate budget windows on the detailed-save path, five years at a time #1054's docstrings, the window tests (now in their own module), and a type of their own for idle windows.
Round 7, on round 6's fixes, found two bugs, each fixed test-first: a refusal could fail a year other than the one whose call couldn't be rebuilt, and an unexpected error from the mark was treated as Modal refusing the year. Its quality findings added a PostgreSQL test of a starter writing its own year during the advance, and
mark_waiting_year.Round 8, on round 7's fixes and the household report restart, found no bugs. Its quality findings were fixed: a refused policy is counted under its cause's error type, the report run kinds are defined once, more household report tests, and wording.
🤖 Generated with Claude Code