Skip to content

Check on every staging deploy that a pre-empted calculation still finishes - #1066

Draft
anth-volk wants to merge 25 commits into
mainfrom
v2plan/c1-20-staging-preemption-check
Draft

anth-volk wants to merge 25 commits into
mainfrom
v2plan/c1-20-staging-preemption-check

Conversation

@anth-volk

Copy link
Copy Markdown
Contributor

Fixes #1062

Summary

Every staging deploy now stops the Modal container running a household calculation and a small economy report, as a pre-emption does, and fails unless both still succeed. Until now only unit tests covered the simulation service's pre-emption handling: Modal runs a pre-empted call again under the same call ID, and the household function or coordinator, run again, resumes its claim on the household job or report run.

  • staging_preemption_check (new job in simulation-deploy.reusable.yml) runs on every staging deploy after staging_economy_check succeeds, from the executor project, whose environment has the Modal client. It runs .github/scripts/simulation-staging-preemption-check.sh twice:

    • household: calculates the staging household that the beta variable SIMULATION_STAGING_HOUSEHOLD_ID names, through the candidate's POST /household-calculations;
    • report: creates the staging economy check's report again (current law against current law, country/wales, 2026, labelled "Staging preemption check") through POST /reports. Because it runs after the economy check, both its simulation runs are reused, so it costs no new simulation, and the check stops the coordinator's container.
  • How it stops the container. Once the household job or report run reads running (the function or coordinator has claimed it), the check takes the record's modal_call_id and asks Modal's call graph which container runs that call. A new helper, src/modal/utils/describe_call_inputs.py, prints each of a call's inputs with its container and status. The check requires modal container list --env staging to list that container, then runs modal container stop --yes. Modal sends the container SIGINT and gives the input to another container under the same call ID.

  • What passes. The record succeeds under the same call and completes at least 5 seconds after the stop, both counted in whole seconds. Within 120 seconds (PREEMPTION_CHECK_RERUN_SECONDS), Modal's call graph must show that call's input finished in a container other than the stopped one. The check fails at once on any of these:

    • a failed record;
    • a record that names another call, or no call, after the stop;
    • a record that doesn't finish within its deadline: 10 minutes for the household, 15 for the report, per attempt.
  • Missed attempts. An attempt proves nothing in two cases:

    • the calculation finishes before its container is stopped;
    • it completes less than 5 seconds after the stop. The stopped container may have written the success before the signal reached it, and Modal's rerun then only finds the record finished.

    The check then submits a new calculation, up to three attempts. Each new attempt starts 90 seconds (PREEMPTION_CHECK_SCALEDOWN_SECONDS) after the last one ended. Modal stops a container left idle for 60 seconds by default, and these functions don't change that, so a container that finished the last calculation unstopped is gone by then. A container that Modal ran a stopped call in again goes idle only when that rerun returns. The wait covers it only when the rerun returns within about 30 seconds of the attempt's end.

  • Refused stops. A stop Modal refuses, as it does for a container that has ended, is not tried again, and the check submits no new calculation. It follows that record to its end, so a record left running by a container that ended for another reason, such as running out of memory, fails at its deadline. Giving such a container a second calculation is left out on purpose: it would usually fail the same way.

  • Container listings. A listing that fails, or isn't a JSON list whose entries each have a "Container ID", counts as unreadable and is asked again. It never counts as the container having gone. Every Modal command has a 60-second time limit. The job's limit is 150 minutes, above the script's worst case of about 126 minutes.

  • Safety. The Modal token is the repository's, which production's jobs share, and modal container stop takes a container ID in any environment. So the script:

    • refuses any MODAL_ENVIRONMENT but staging;
    • stops only a container Modal lists in staging;
    • checks every call and container ID against Modal's ID forms before passing it to a command.

    The client secret reaches the token request through jq's environment, so it is on no command line.

  • deployment_ready requires the check in staging (simulation-staging-preemption-ready.sh), so a failure stops the production deployment.

  • Read answers. GET /household-calculations/{household_job_id} and the run in GET /reports/{report_id} now return modal_call_id. That is the household function's or coordinator's Modal call recorded on the record, or null until the service records one.

    • economy_run_reads selects report_runs.modal_call_id; the household job record already held it.
    • The OpenAPI document declares the field (required, nullable string), and the regenerated client reads it.
    • The Public API doesn't read these routes.

Coordination

Environment variables and secrets

Name in the script Provider Wiring Verified
STAGING_HOUSEHOLD_ID GitHub Actions variable SIMULATION_STAGING_HOUSEHOLD_ID in the beta environment job environment: ${{ inputs.release_environment }} (beta from simulation-deploy.yml), household step env: Present.
SIMULATION_TEST_AUTH_ISSUER, _AUDIENCE, _CLIENT_ID, _CLIENT_SECRET GitHub Actions secrets SIMULATION_ENTRYPOINT_AUTH_ISSUER, SIMULATION_ENTRYPOINT_AUTH_AUDIENCE, SIMULATION_ENTRYPOINT_TEST_AUTH_CLIENT_ID, SIMULATION_ENTRYPOINT_TEST_AUTH_CLIENT_SECRET in the beta environment the job's environment, both steps' env: Present; the staging household and economy checks already use them.
MODAL_TOKEN_ID, MODAL_TOKEN_SECRET GitHub Actions repository secrets (an organization secret of the same name exists too; the repository one takes precedence) secrets: inherit from simulation-deploy.yml's beta job, both steps' env:; the Modal client reads them Present. Unverified: that this token may read call graphs, list containers and stop them.
MODAL_ENVIRONMENT workflow input modal_environment, the literal staging from the beta caller job both steps' env:; the script requires staging and passes it to modal container list --env Yes.
base URL job output needs.deploy_entrypoint.outputs.candidate_url step argument Yes.
REQUESTED, CHECK_RESULT (ready check) expressions inputs.deployment_environment == 'staging' and needs.staging_preemption_check.result deployment_ready step env: Yes.
PREEMPTION_CHECK_ATTEMPTS, _DEADLINE_SECONDS, _POLL_SECONDS, _RERUN_SECONDS, _SCALEDOWN_SECONDS optional, set nowhere script defaults (3; 600 or 900; 2; 120; 90) n/a

No Secret Manager secret, Modal secret or Cloud Run variable is added or changed: the service needs nothing new to return modal_call_id.

Tests

Each change started as a failing test, committed with "(failing)" in its subject and seen failing for the expected reason, then followed by the change that makes it pass. Only the tests this PR adds or changes were run, each by node ID:

  • projects/policyengine-simulation-entry:
    • the new test_staging_preemption_check.py. It runs the script against fakes for curl, the service, the call-input helper, the Modal CLI and date, and covers the workflow job, its condition, deployment_ready and the ready script. Every case was run after review round 5's fix, and the cases rounds 6 and 7 add or change, with the listing tests, after theirs;
    • test_deployment_assets.py::test_deployment_uses_gcloud_workflow_without_terraform;
    • the changed read tests in test_household_routes.py and test_economy_routes.py, the changed fixture in test_economy_submissions.py, and the new test_openapi.py test.
  • projects/policyengine-simulation-executor: the six tests of the new test_describe_call_inputs.py.
  • libs/policyengine-simulation-persistence: test_economy_run_reads.py::test_a_report_run_is_read_with_the_modal_call_it_records. It ran against a PostgreSQL 17 database built by the Public API's v2 migration chain at master 429d348ae, with its privileges applied and verified, as the simulation service's role.
  • projects/policyengine-apis-integ: the two changed generated-client tests, against the client regenerated from this branch. Against a client generated from main, both fail (no modal_call_id).

After merging main, the read, OpenAPI, generated-client and persistence tests above were run again.

Before each commit: CI's pinned Ruff, the rules registry render, the quality guards (shellcheck, file length, locked installs), uv lock --check, and Pyright for the changed packages. CI runs the rest. The real check runs only in a staging deploy.

Review

Seven read-only review rounds. The first three covered the whole branch; each later one covered the previous round's fixes. Six bugs were found, and all were fixed test-first:

  • Round 1: a stop Modal refused moved straight to a new attempt, so a container that had died for another reason could still let the deploy pass. The check now follows that record to its end.

  • Round 2:

    • A record completing between the running read and the stop counted as proof; it must now complete after the stop.
    • The wait before a new attempt was skipped when the check never saw the container running.
  • Round 3: no bugs. Its quality findings added the 5-second margin, asking again after a failed listing, failing a record that names no call after the stop, and tests for failing Modal commands.

  • Round 4: after a "completed too soon" miss, the wait was for the stopped container, not the container Modal re-ran the call in.

  • Round 5:

    • The wait read the call graph only once, so it could miss the re-run's container.
    • An empty listing counted as "none listed".

    Both were fixed by replacing that wait with the fixed 90-second wait, and by treating a listing that isn't a JSON list as unreadable.

  • Round 6: no bugs. Its quality fixes:

    • a missed attempt's reason names only the last listing's outcome;
    • a listing entry without "Container ID" is unreadable;
    • the documentation no longer overstates what the 90-second wait covers.
  • Round 7, on round 6's fixes: no bugs. Its findings were test-only: more listing forms, and a test name and docstrings that match the script.

Not done or not verified

  • Not run against real Modal:
    • whether a stop lands while the household function or the reused report's coordinator is still running (the coordinator may finish quickly with both simulation runs reused);
    • how Modal's call graph records the rerun. The check accepts the rerun either as a new input of the call or as the stopped input updated in place.
  • The Modal token's permission to read call graphs and to list and stop containers.
  • The 90-second wait assumes the household functions and the coordinator keep Modal's default 60-second idle time. If either sets a scale-down time later, PREEMPTION_CHECK_SCALEDOWN_SECONDS must change with it.
  • The report's simulation runs are reused, so the check doesn't cover a coordinator starting its workers again after a stop.
  • The existing staging household and economy checks pass the client secret to jq --arg, so it appears in jq's arguments. This PR's script reads it from jq's environment instead; changing the older checks is a separate change.

🤖 Generated with Claude Code

anth-volk and others added 25 commits November 13, 2026 01:24
…ailing)

The staging preemption check finds the container to stop from the Modal
call recorded on a household job or report run, so GET
/household-calculations/{id} and the run in GET /reports/{id} name it,
or null before one is recorded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…read

GET /household-calculations/{id} and the run in GET /reports/{id} now
carry modal_call_id: the household function's or coordinator's call, or
null before the service has recorded one. Modal runs a pre-empted call
again under the same ID, so the staging preemption check can find the
container running it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (failing)

The staging preemption check stops the container running a household job's
or report run's call, then requires the call to finish in another
container. The helper reads the call graph Modal records for the call and
prints each of the call's inputs, wherever it sits in that graph, with the
container (Modal's task ID) that runs or ran it, as JSON.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
describe_call_inputs reads the call graph Modal records for a call and
prints each of the call's inputs, wherever it sits in that graph, with its
container (Modal's task ID, which modal container stop takes) and its
input status, as JSON. A failure to ask Modal is printed on standard error
with exit status 1, for the caller to ask again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (failing)

Every staging deploy starts a household calculation and the staging economy
check's report through the candidate, stops the Modal container running
each one's call with modal container stop once it is running, and requires
it to succeed in another container under the same call. A calculation that
finishes before its container is stopped is submitted again, up to three
times; one that fails or runs out of time fails the check at once. The
check runs only in Modal's staging environment and stops only a container
Modal lists there. deployment_ready requires it in staging.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
staging_preemption_check runs after the staging economy check. It
calculates the staging household, then creates that check's report, whose
simulation runs it reuses, through the candidate. Once the household
function or coordinator has claimed each one, the check finds the
container running its recorded Modal call through Modal's call graph,
requires Modal to list it in the staging environment, and stops it with
modal container stop, which sends SIGINT and gives the input to another
container under the same call ID. Each must then succeed under that call,
with the call graph showing it finished in another container.

A calculation that finishes before its container is stopped proves
nothing, so a new one is submitted, up to three attempts. One that fails
or runs past its deadline fails the check at once, and deployment_ready
requires the check in staging, so production waits for it.

The test fakes now give a record that never finishes a short deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DEPLOYMENT.md describes staging_preemption_check: what it starts, when and
how it stops each container, what it requires, its attempts and deadlines,
and the Modal token it needs. The entry and persistence READMEs name the
Modal call the household job and report run reads now return. The
changelog fragment is for #1062.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A stop Modal refuses is not tried again and starts no new submission:
  the check follows the job to its end, so a job left running by a
  container that ended for another reason fails at its deadline, and one
  that succeeded had finished just before the stop.
- Before a new attempt, the check waits for the last attempt's container
  to leave Modal's listing, so the next calculation starts in a new
  container instead of a warm one too fast to stop.
- No command, jq included, gets the client secret as an argument.
- A rerun Modal records on the stopped input itself counts (this passes
  already: the check never compares input IDs).
- The read routes' descriptions name the Modal call they return.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…m container

- A stop Modal refuses, as it does for a container that has ended, is not
  tried again and starts no new submission. The check follows that job to
  its end: one that succeeds had finished just before the stop and counts
  as a missed attempt; one left running by a container that ended for
  another reason, such as running out of memory, fails the check at its
  deadline, naming the refusal.
- Before a new attempt, the check waits up to
  PREEMPTION_CHECK_SCALEDOWN_SECONDS (180) for Modal to stop listing the
  last attempt's container, so the next calculation starts in a new
  container instead of the warm one, where it would finish too fast to
  stop.
- The token request takes the client's ID, secret and audience from jq's
  environment rather than its arguments, so no command line holds the
  secret.
- The read routes' descriptions and docstrings name the Modal call they
  return, and the deployment notes say the check uses the repository's
  Modal token, which production's jobs share.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tempts (failing)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Its two steps can take up to 99 minutes together: three attempts of 12 minutes for the household and of 17 for the report, each new attempt after a wait of up to 3 minutes for the last attempt's container to leave. The job's limit stays past that and the install, so the script fails first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A record that completed before its container was stopped proves nothing,
  even though Modal's rerun of the call then succeeds in another container:
  the rerun finds the record finished and returns its summary. The check
  compares the record's completion time with the stop, in each form the
  service writes, and refuses a succeeded record with no completion time.
- Before a new attempt, the check also waits for a container it first saw
  only after it had finished the calculation.
- Every Modal command runs with a 60-second time limit.
- The final failure says no attempt showed the calculation finishing in
  another container, and a failed stop says it failed, whatever the cause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…limits (failing)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… warm container

- A stopped record must have completed after the stop: one completed
  before it proves nothing, because Modal's rerun of the call then finds it
  finished and only returns its summary, which Modal records as a success
  in another container. The check compares the record's completed_at
  (with or without fractional seconds, Z or +00:00) with the time the stop
  returned, and refuses a succeeded record with no completion time.
- When an attempt is missed before the check found the call's container,
  it asks the call graph which container ran the call, so the next
  attempt waits for that warm container to leave too.
- Every Modal command (the call-input helper, the container listing and
  the stop) runs with a 60-second limit, and the job's time limit is 150
  minutes, past the steps' worst case of about 136.
- A failed stop is reported as failed with its exit status, whatever the
  cause, and the final failure says no attempt showed the calculation
  finishing in another container.
- The call-input helper's records are a TypedDict naming the keys the
  check reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…failing)

- A record completed within 5 seconds of the stop is taken for the stopped
  container's own work, since that container may write the success before
  the signal reaches it; a rerun in a new container can't finish that
  soon. A fake date fixes the stop's time.
- A succeeded record that names no call after the stop fails.
- A container listing Modal doesn't answer is asked again: it neither
  marks the container as unlisted nor ends the wait for it to leave.
- A stop that runs out of time is followed like a failed one (this passes
  already: both are a non-zero exit).
- The refused-stop deadline test gets 6 seconds, enough to reach the stop
  with the extra fakes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…gs, a lost call

- A stopped record must complete at least 5 seconds after the stop. The
  stopped container may write the success before the signal reaches it,
  and Modal's rerun then only returns the record's summary; a rerun in a
  new container can't finish that soon, so a sooner completion counts as
  a missed attempt.
- A container listing Modal doesn't answer, or answers unreadably, is
  asked again: it no longer marks the container as unlisted, and it no
  longer ends the wait for the last attempt's container to leave.
- After the stop, a read the service answered must name the stopped call;
  one naming no call fails the check like one naming another call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- After an attempt whose record completed too soon after the stop, the
  next attempt waits for the container Modal's rerun ran in, which is now
  idle and warm, not only the stopped one. The fake listing drops a
  stopped container, as Modal does.
- A missed attempt's reason says when Modal could not list containers,
  and after a failed stop says the record finished after it.
- Tests for paths round 3's code already handles: a listing Modal prints
  unreadably is asked again; a wait Modal can't answer ends at its time
  limit; only a listing Modal answered may confirm the container to stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Before a new attempt, the check waits for every container the call
  graph says the last attempt's call ran in, with the one it found or
  stopped: after a record that completed too soon after the stop, the
  container Modal's rerun ran in is idle and warm, and the next
  calculation would land there and finish too fast to stop. One listing
  per wait checks them all.
- A missed attempt's reason says when Modal could not list containers,
  and after a failed stop says the record finished after it.
- The 5-second rule is described as counted in whole seconds, a
  completion in the stop's own second reports how many seconds after the
  stop it came, and the edited paragraphs are re-wrapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Before each new attempt, and after none of the last, the check waits a
  fixed PREEMPTION_CHECK_SCALEDOWN_SECONDS (90) for Modal to stop the
  containers the last attempt left idle, instead of waiting for the
  containers one call-graph read names, which may not name the one
  Modal's rerun ran in yet. The tests of the old wait go.
- An empty container listing is unreadable, not a listing without the
  container.
- A container found after a failed listing clears the failure from the
  attempt's reason.
- A completion in the stop's own second reports "0 seconds after" (this
  passes already since round 4's fix).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pty listings

- Before each new attempt the check now waits a fixed
  PREEMPTION_CHECK_SCALEDOWN_SECONDS (90), past Modal's default 60-second
  idle scale-down, which the household functions and the coordinator
  don't change. That stops both the container the check found and the one
  Modal's rerun ran in, without depending on one call-graph read naming
  the rerun's container, which Modal records best-effort and not in real
  time. The listing-based wait and its call-graph read go.
- container_listed answers 2 (couldn't list) for a listing that isn't a
  JSON list, an empty one included, rather than taking it as a listing
  without the container.
- A container found after a failed or unlisted lookup clears those notes,
  so an attempt's reason names only what last happened.
- The deployment notes and the job's time-limit comment describe the
  fixed wait.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…failing)

- A listing whose entries lack Modal's "Container ID" column is
  unreadable, not a listing without the container; so are a JSON object
  and null (those pass already).
- A missed attempt's reason names only the last container lookup's
  outcome: a listing without the container after a failed one, or a
  failed listing after one without the container, not both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A container listing counts as readable only when it is a JSON list
  whose entries each have Modal's "Container ID" field, so a change in
  Modal's output reads as an unreadable listing that is asked again,
  not as the container having gone.
- One note holds the last container listing's outcome, so a missed
  attempt's reason names only that outcome, not an earlier one.
- The script's header and DEPLOYMENT.md no longer claim the 90-second
  wait always covers the container Modal ran a stopped call in again:
  that container goes idle only when the rerun returns, so the wait
  covers it only when the rerun returns within about 30 seconds of the
  attempt's end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Test the listing's entry check with non-object entries ([1], [null])
  and a list in which only the container's own entry has "Container ID".
  The [null] and one-renamed-entry cases fail against the script before
  round 6's fix (b6a68df); [1] already returned "couldn't list" there.
- Rename the miss-reason test to say it keeps the last container
  listing's outcome, which is what the script promises, and say so in
  its docstring.
- The fixed-wait test's docstring no longer says the wait always covers
  the container Modal ran a stopped call in again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Check on every staging deploy that a pre-empted calculation still finishes

1 participant