[anth-volk]

Fixes #1027

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Changelog

• This PR adds a changelog fragment, changelog.d/<name>.<type>.md, whose type is breaking, added, changed, fixed or removed.

Summary

A deployment that publishes Stage 12 now moves the Stage 12 v2 manifest's default version to the new release's worker only after every check and the Cloud Run promotion pass. This is Stage 12's own mechanism, with no code shared with the v1 routing jobs.

• Before the checks: publish_stage12_v2_manifest adds the release's worker with update_v2_version_manifest --keep-default and leaves default_version unchanged. The first worker in an empty manifest still becomes the default. No check that runs after this publication is upstream of the job, so "Re-run failed jobs" never republishes the worker.
• After the checks: a new job, publish_stage12_v2_default, needs the worker publication, the integration tests, the authenticated tests and the promotion. It:
  • moves the default when the release's worker is newer (--move-default), keeping a plan of the manifest before and after the move, with their digests;
  • uploads the plan as the artifact stage12-v2-default-plan-<environment>-<attempt> whenever the move step ran;
  • checks that every reader's key holds the moved manifest and that a reader resolves the plan's default (--verify-plan);
  • if a later step fails or the run is cancelled, writes back the starting manifest, but only while the stored manifest is still what the move wrote (--restore-plan).
• deployment_ready requires that job to succeed whenever Stage 12 was requested, and to be skipped otherwise.
• The run's summary page shows each move: the manifest's digests before and after, and whether requests that pin no v2 version now reach the release's worker.
• Only a worker publication creates the manifest Dict. A move, check or restore against a Modal environment without one fails, instead of creating an empty Dict there.
• The three manifest keys: the manifest is stored as three keys, one copy per generation of reader, written one after another.
  • The move refuses to start unless the keys hold one manifest. The check reads all three.
  • The restore recognises a move or an earlier restore that stopped part-way, and writes the starting manifest back to every key.
  • Modal expires a Dict entry after 7 days without a read or write, and current readers read the oldest key only when neither newer key exists. So a missing older key is accepted and written again. The newest key is required.
  • A move that leaves the default where it is writes nothing, so the manifest digest, which Stage 12 report identities include, does not change.
• The new rule sim.deploy.stage12-default-moves-after-checks lists the contract, publisher, script and workflow tests. The executor README has a new section, "Stage 12 v2 default version", that covers the operator restore from the kept plan and the limitations listed below.

Limitations:

• The v2 worker's Modal app is named after the policyengine.py version alone. So a deployment that keeps that version redeploys the current worker's app in place, before the checks.
• If no Stage 12 request and no deployment touches the manifest for 7 days, all three keys expire. The next worker publication cannot tell that from an empty manifest: it makes its worker the default before the checks and drops the earlier workers. This was already the case before this change.
• The deployment's force_latest input does not move the v2 default.
• A release with a newer worker changes the manifest digest twice: at the worker publication and at the move. So Stage 12 reports created between the two get identities with the intermediate digest.
• If the runner is lost after the move starts writing and before the plan is uploaded, no plan is kept and no restore runs. The default stays on the release, which passed every check, and the deployment fails. "Re-run failed jobs" then finds the default already moved and succeeds. If the write stopped part-way, though, the rerun refuses the keys and an operator repairs them by hand.
• No workflow restores an earlier default after a completed deployment.
• The default job does not wait for the v1 publish_latest job. deployment_ready still fails a deployment whose v1 latest move failed.

Environment variables for the new job's steps:

Variable	Provider
MODAL_TOKEN_ID, MODAL_TOKEN_SECRET	GitHub repository secrets, passed to the reusable workflow with secrets: inherit by simulation-deploy.yml and stage12-deploy.yml
MODAL_ENVIRONMENT	the modal_environment input
POLICYENGINE_VERSION	the prepare job's policyengine_version output
REQUESTED, DEFAULT_RESULT (in deployment_ready)	the deploy_stage12_v2 input and the new job's result
RUNNER_TEMP, GITHUB_OUTPUT	the runner

Testing

Each new or changed test was written first and seen failing, then run individually by node ID:

• libs/policyengine-simulation-contract/tests/test_stage12_manifest_default.py: adding a worker, moving the default, writing the manifest copies.
• projects/policyengine-simulation-executor/tests/test_update_v2_version_manifest_default.py: the publisher modes; part-way writes; expired older keys; a manifest another publish changed; altered plans; a write that reads back differently; a manifest that changes between planning and writing; --force-default; command-line refusals.
• projects/policyengine-simulation-entry/tests/test_deployment_assets_stage12_default.py: job ordering; the job's if: across 11 result combinations; step order, conditions and environment; no shared code with v1 routing.
• projects/policyengine-simulation-entry/tests/test_stage12_default_scripts.py: each script with a fake uv.
• projects/policyengine-simulation-entry/tests/test_deployment_assets.py::test_deployment_uses_gcloud_workflow_without_terraform, for the updated count of jobs that declare the release environment.

Mutation checks. Each of these changes makes a test fail:

• changing all to any in the restore's key comparison;
• removing the plan's digest check;
• removing the read-back after a write;
• removing the recheck before the move writes;
• not passing force_default through;
• a wrong path for the plan upload;
• continue-on-error on the default job or any of its steps.

After main was merged in (bringing #898), every test this PR adds or changes was run again by node ID, and all pass.

Also run: ruff 0.15.22 format and check, shellcheck on the four scripts, python3 -m scripts.rules validate, render and guard, and Pyright on the executor package. The workflow has not run in a deployment yet.

🤖 Generated with Claude Code