Repository navigation
Conversation
…ailing)
The staging preemption check finds the container to stop from the Modal
call recorded on a household job or report run, so GET
/household-calculations/{id} and the run in GET /reports/{id} name it,
or null before one is recorded.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…read
GET /household-calculations/{id} and the run in GET /reports/{id} now
carry modal_call_id: the household function's or coordinator's call, or
null before the service has recorded one. Modal runs a pre-empted call
again under the same ID, so the staging preemption check can find the
container running it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (failing) The staging preemption check stops the container running a household job's or report run's call, then requires the call to finish in another container. The helper reads the call graph Modal records for the call and prints each of the call's inputs, wherever it sits in that graph, with the container (Modal's task ID) that runs or ran it, as JSON. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
describe_call_inputs reads the call graph Modal records for a call and prints each of the call's inputs, wherever it sits in that graph, with its container (Modal's task ID, which modal container stop takes) and its input status, as JSON. A failure to ask Modal is printed on standard error with exit status 1, for the caller to ask again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (failing) Every staging deploy starts a household calculation and the staging economy check's report through the candidate, stops the Modal container running each one's call with modal container stop once it is running, and requires it to succeed in another container under the same call. A calculation that finishes before its container is stopped is submitted again, up to three times; one that fails or runs out of time fails the check at once. The check runs only in Modal's staging environment and stops only a container Modal lists there. deployment_ready requires it in staging. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
staging_preemption_check runs after the staging economy check. It calculates the staging household, then creates that check's report, whose simulation runs it reuses, through the candidate. Once the household function or coordinator has claimed each one, the check finds the container running its recorded Modal call through Modal's call graph, requires Modal to list it in the staging environment, and stops it with modal container stop, which sends SIGINT and gives the input to another container under the same call ID. Each must then succeed under that call, with the call graph showing it finished in another container. A calculation that finishes before its container is stopped proves nothing, so a new one is submitted, up to three attempts. One that fails or runs past its deadline fails the check at once, and deployment_ready requires the check in staging, so production waits for it. The test fakes now give a record that never finishes a short deadline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DEPLOYMENT.md describes staging_preemption_check: what it starts, when and how it stops each container, what it requires, its attempts and deadlines, and the Modal token it needs. The entry and persistence READMEs name the Modal call the household job and report run reads now return. The changelog fragment is for #1062. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A stop Modal refuses is not tried again and starts no new submission: the check follows the job to its end, so a job left running by a container that ended for another reason fails at its deadline, and one that succeeded had finished just before the stop. - Before a new attempt, the check waits for the last attempt's container to leave Modal's listing, so the next calculation starts in a new container instead of a warm one too fast to stop. - No command, jq included, gets the client secret as an argument. - A rerun Modal records on the stopped input itself counts (this passes already: the check never compares input IDs). - The read routes' descriptions name the Modal call they return. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…m container - A stop Modal refuses, as it does for a container that has ended, is not tried again and starts no new submission. The check follows that job to its end: one that succeeds had finished just before the stop and counts as a missed attempt; one left running by a container that ended for another reason, such as running out of memory, fails the check at its deadline, naming the refusal. - Before a new attempt, the check waits up to PREEMPTION_CHECK_SCALEDOWN_SECONDS (180) for Modal to stop listing the last attempt's container, so the next calculation starts in a new container instead of the warm one, where it would finish too fast to stop. - The token request takes the client's ID, secret and audience from jq's environment rather than its arguments, so no command line holds the secret. - The read routes' descriptions and docstrings name the Modal call they return, and the deployment notes say the check uses the repository's Modal token, which production's jobs share. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tempts (failing) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its two steps can take up to 99 minutes together: three attempts of 12 minutes for the household and of 17 for the report, each new attempt after a wait of up to 3 minutes for the last attempt's container to leave. The job's limit stays past that and the install, so the script fails first. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A record that completed before its container was stopped proves nothing, even though Modal's rerun of the call then succeeds in another container: the rerun finds the record finished and returns its summary. The check compares the record's completion time with the stop, in each form the service writes, and refuses a succeeded record with no completion time. - Before a new attempt, the check also waits for a container it first saw only after it had finished the calculation. - Every Modal command runs with a 60-second time limit. - The final failure says no attempt showed the calculation finishing in another container, and a failed stop says it failed, whatever the cause. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…limits (failing) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… warm container - A stopped record must have completed after the stop: one completed before it proves nothing, because Modal's rerun of the call then finds it finished and only returns its summary, which Modal records as a success in another container. The check compares the record's completed_at (with or without fractional seconds, Z or +00:00) with the time the stop returned, and refuses a succeeded record with no completion time. - When an attempt is missed before the check found the call's container, it asks the call graph which container ran the call, so the next attempt waits for that warm container to leave too. - Every Modal command (the call-input helper, the container listing and the stop) runs with a 60-second limit, and the job's time limit is 150 minutes, past the steps' worst case of about 136. - A failed stop is reported as failed with its exit status, whatever the cause, and the final failure says no attempt showed the calculation finishing in another container. - The call-input helper's records are a TypedDict naming the keys the check reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…failing) - A record completed within 5 seconds of the stop is taken for the stopped container's own work, since that container may write the success before the signal reaches it; a rerun in a new container can't finish that soon. A fake date fixes the stop's time. - A succeeded record that names no call after the stop fails. - A container listing Modal doesn't answer is asked again: it neither marks the container as unlisted nor ends the wait for it to leave. - A stop that runs out of time is followed like a failed one (this passes already: both are a non-zero exit). - The refused-stop deadline test gets 6 seconds, enough to reach the stop with the extra fakes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gs, a lost call - A stopped record must complete at least 5 seconds after the stop. The stopped container may write the success before the signal reaches it, and Modal's rerun then only returns the record's summary; a rerun in a new container can't finish that soon, so a sooner completion counts as a missed attempt. - A container listing Modal doesn't answer, or answers unreadably, is asked again: it no longer marks the container as unlisted, and it no longer ends the wait for the last attempt's container to leave. - After the stop, a read the service answered must name the stopped call; one naming no call fails the check like one naming another call. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- After an attempt whose record completed too soon after the stop, the next attempt waits for the container Modal's rerun ran in, which is now idle and warm, not only the stopped one. The fake listing drops a stopped container, as Modal does. - A missed attempt's reason says when Modal could not list containers, and after a failed stop says the record finished after it. - Tests for paths round 3's code already handles: a listing Modal prints unreadably is asked again; a wait Modal can't answer ends at its time limit; only a listing Modal answered may confirm the container to stop. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Before a new attempt, the check waits for every container the call graph says the last attempt's call ran in, with the one it found or stopped: after a record that completed too soon after the stop, the container Modal's rerun ran in is idle and warm, and the next calculation would land there and finish too fast to stop. One listing per wait checks them all. - A missed attempt's reason says when Modal could not list containers, and after a failed stop says the record finished after it. - The 5-second rule is described as counted in whole seconds, a completion in the stop's own second reports how many seconds after the stop it came, and the edited paragraphs are re-wrapped. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Before each new attempt, and after none of the last, the check waits a fixed PREEMPTION_CHECK_SCALEDOWN_SECONDS (90) for Modal to stop the containers the last attempt left idle, instead of waiting for the containers one call-graph read names, which may not name the one Modal's rerun ran in yet. The tests of the old wait go. - An empty container listing is unreadable, not a listing without the container. - A container found after a failed listing clears the failure from the attempt's reason. - A completion in the stop's own second reports "0 seconds after" (this passes already since round 4's fix). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pty listings - Before each new attempt the check now waits a fixed PREEMPTION_CHECK_SCALEDOWN_SECONDS (90), past Modal's default 60-second idle scale-down, which the household functions and the coordinator don't change. That stops both the container the check found and the one Modal's rerun ran in, without depending on one call-graph read naming the rerun's container, which Modal records best-effort and not in real time. The listing-based wait and its call-graph read go. - container_listed answers 2 (couldn't list) for a listing that isn't a JSON list, an empty one included, rather than taking it as a listing without the container. - A container found after a failed or unlisted lookup clears those notes, so an attempt's reason names only what last happened. - The deployment notes and the job's time-limit comment describe the fixed wait. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…failing) - A listing whose entries lack Modal's "Container ID" column is unreadable, not a listing without the container; so are a JSON object and null (those pass already). - A missed attempt's reason names only the last container lookup's outcome: a listing without the container after a failed one, or a failed listing after one without the container, not both. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A container listing counts as readable only when it is a JSON list whose entries each have Modal's "Container ID" field, so a change in Modal's output reads as an unreadable listing that is asked again, not as the container having gone. - One note holds the last container listing's outcome, so a missed attempt's reason names only that outcome, not an earlier one. - The script's header and DEPLOYMENT.md no longer claim the 90-second wait always covers the container Modal ran a stopped call in again: that container goes idle only when the rerun returns, so the wait covers it only when the rerun returns within about 30 seconds of the attempt's end. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Test the listing's entry check with non-object entries ([1], [null]) and a list in which only the container's own entry has "Container ID". The [null] and one-renamed-entry cases fail against the script before round 6's fix (b6a68df); [1] already returned "couldn't list" there. - Rename the miss-reason test to say it keeps the last container listing's outcome, which is what the script promises, and say so in its docstring. - The fixed-wait test's docstring no longer says the wait always covers the container Modal ran a stopped call in again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1062
Summary
Every staging deploy now stops the Modal container running a household calculation and a small economy report, as a pre-emption does, and fails unless both still succeed. Until now only unit tests covered the simulation service's pre-emption handling: Modal runs a pre-empted call again under the same call ID, and the household function or coordinator, run again, resumes its claim on the household job or report run.
staging_preemption_check(new job insimulation-deploy.reusable.yml) runs on every staging deploy afterstaging_economy_checksucceeds, from the executor project, whose environment has the Modal client. It runs.github/scripts/simulation-staging-preemption-check.shtwice:household: calculates the staging household that thebetavariableSIMULATION_STAGING_HOUSEHOLD_IDnames, through the candidate'sPOST /household-calculations;report: creates the staging economy check's report again (current law against current law,country/wales, 2026, labelled "Staging preemption check") throughPOST /reports. Because it runs after the economy check, both its simulation runs are reused, so it costs no new simulation, and the check stops the coordinator's container.How it stops the container. Once the household job or report run reads
running(the function or coordinator has claimed it), the check takes the record'smodal_call_idand asks Modal's call graph which container runs that call. A new helper,src/modal/utils/describe_call_inputs.py, prints each of a call's inputs with its container and status. The check requiresmodal container list --env stagingto list that container, then runsmodal container stop --yes. Modal sends the container SIGINT and gives the input to another container under the same call ID.What passes. The record succeeds under the same call and completes at least 5 seconds after the stop, both counted in whole seconds. Within 120 seconds (
PREEMPTION_CHECK_RERUN_SECONDS), Modal's call graph must show that call's input finished in a container other than the stopped one. The check fails at once on any of these:Missed attempts. An attempt proves nothing in two cases:
The check then submits a new calculation, up to three attempts. Each new attempt starts 90 seconds (
PREEMPTION_CHECK_SCALEDOWN_SECONDS) after the last one ended. Modal stops a container left idle for 60 seconds by default, and these functions don't change that, so a container that finished the last calculation unstopped is gone by then. A container that Modal ran a stopped call in again goes idle only when that rerun returns. The wait covers it only when the rerun returns within about 30 seconds of the attempt's end.Refused stops. A stop Modal refuses, as it does for a container that has ended, is not tried again, and the check submits no new calculation. It follows that record to its end, so a record left running by a container that ended for another reason, such as running out of memory, fails at its deadline. Giving such a container a second calculation is left out on purpose: it would usually fail the same way.
Container listings. A listing that fails, or isn't a JSON list whose entries each have a "Container ID", counts as unreadable and is asked again. It never counts as the container having gone. Every Modal command has a 60-second time limit. The job's limit is 150 minutes, above the script's worst case of about 126 minutes.
Safety. The Modal token is the repository's, which production's jobs share, and
modal container stoptakes a container ID in any environment. So the script:MODAL_ENVIRONMENTbutstaging;The client secret reaches the token request through jq's environment, so it is on no command line.
deployment_readyrequires the check in staging (simulation-staging-preemption-ready.sh), so a failure stops the production deployment.Read answers.
GET /household-calculations/{household_job_id}and the run inGET /reports/{report_id}now returnmodal_call_id. That is the household function's or coordinator's Modal call recorded on the record, or null until the service records one.economy_run_readsselectsreport_runs.modal_call_id; the household job record already held it.Coordination
mainis merged into this branch. Neither changes the request this check sends toPOST /household-calculationsor the economyPOST /reports, or the fields it reads back.simulation-deploy.reusable.yml. Whichever PR merges second keeps both./reports. If it changes how a report is started, this check is updated after it merges.Environment variables and secrets
STAGING_HOUSEHOLD_IDSIMULATION_STAGING_HOUSEHOLD_IDin thebetaenvironmentenvironment: ${{ inputs.release_environment }}(betafromsimulation-deploy.yml), household stepenv:SIMULATION_TEST_AUTH_ISSUER,_AUDIENCE,_CLIENT_ID,_CLIENT_SECRETSIMULATION_ENTRYPOINT_AUTH_ISSUER,SIMULATION_ENTRYPOINT_AUTH_AUDIENCE,SIMULATION_ENTRYPOINT_TEST_AUTH_CLIENT_ID,SIMULATION_ENTRYPOINT_TEST_AUTH_CLIENT_SECRETin thebetaenvironmentenv:MODAL_TOKEN_ID,MODAL_TOKEN_SECRETsecrets: inheritfromsimulation-deploy.yml'sbetajob, both steps'env:; the Modal client reads themMODAL_ENVIRONMENTmodal_environment, the literalstagingfrom thebetacaller jobenv:; the script requiresstagingand passes it tomodal container list --envneeds.deploy_entrypoint.outputs.candidate_urlREQUESTED,CHECK_RESULT(ready check)inputs.deployment_environment == 'staging'andneeds.staging_preemption_check.resultdeployment_readystepenv:PREEMPTION_CHECK_ATTEMPTS,_DEADLINE_SECONDS,_POLL_SECONDS,_RERUN_SECONDS,_SCALEDOWN_SECONDSNo Secret Manager secret, Modal secret or Cloud Run variable is added or changed: the service needs nothing new to return
modal_call_id.Tests
Each change started as a failing test, committed with "(failing)" in its subject and seen failing for the expected reason, then followed by the change that makes it pass. Only the tests this PR adds or changes were run, each by node ID:
projects/policyengine-simulation-entry:test_staging_preemption_check.py. It runs the script against fakes for curl, the service, the call-input helper, the Modal CLI anddate, and covers the workflow job, its condition,deployment_readyand the ready script. Every case was run after review round 5's fix, and the cases rounds 6 and 7 add or change, with the listing tests, after theirs;test_deployment_assets.py::test_deployment_uses_gcloud_workflow_without_terraform;test_household_routes.pyandtest_economy_routes.py, the changed fixture intest_economy_submissions.py, and the newtest_openapi.pytest.projects/policyengine-simulation-executor: the six tests of the newtest_describe_call_inputs.py.libs/policyengine-simulation-persistence:test_economy_run_reads.py::test_a_report_run_is_read_with_the_modal_call_it_records. It ran against a PostgreSQL 17 database built by the Public API's v2 migration chain atmaster429d348ae, with its privileges applied and verified, as the simulation service's role.projects/policyengine-apis-integ: the two changed generated-client tests, against the client regenerated from this branch. Against a client generated frommain, both fail (nomodal_call_id).After merging
main, the read, OpenAPI, generated-client and persistence tests above were run again.Before each commit: CI's pinned Ruff, the rules registry render, the quality guards (shellcheck, file length, locked installs),
uv lock --check, and Pyright for the changed packages. CI runs the rest. The real check runs only in a staging deploy.Review
Seven read-only review rounds. The first three covered the whole branch; each later one covered the previous round's fixes. Six bugs were found, and all were fixed test-first:
Round 1: a stop Modal refused moved straight to a new attempt, so a container that had died for another reason could still let the deploy pass. The check now follows that record to its end.
Round 2:
runningread and the stop counted as proof; it must now complete after the stop.Round 3: no bugs. Its quality findings added the 5-second margin, asking again after a failed listing, failing a record that names no call after the stop, and tests for failing Modal commands.
Round 4: after a "completed too soon" miss, the wait was for the stopped container, not the container Modal re-ran the call in.
Round 5:
Both were fixed by replacing that wait with the fixed 90-second wait, and by treating a listing that isn't a JSON list as unreadable.
Round 6: no bugs. Its quality fixes:
Round 7, on round 6's fixes: no bugs. Its findings were test-only: more listing forms, and a test name and docstrings that match the script.
Not done or not verified
PREEMPTION_CHECK_SCALEDOWN_SECONDSmust change with it.jq --arg, so it appears in jq's arguments. This PR's script reads it from jq's environment instead; changing the older checks is a separate change.🤖 Generated with Claude Code