[anth-volk]

Fixes #897

Same-Repository Draft PR Check

• This PR is a draft.
• This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Changelog

• This PR adds a changelog fragment, changelog.d/<name>.<type>.md, whose type is breaking, added, changed, fixed or removed: changelog.d/897.added.md.

Summary

The Simulation Entrypoint (the simulation service) connects to the Postgres database that holds the Public API's v2 tables, with its own login role, which can create simulations, reports, their runs, budget window calculations and household jobs there, and update the call state of the runs it starts. This repository gets no schema and no migration chain of its own: the Public API's migration chain creates those tables, and its privilege manifest grants the role. This replaces this PR's earlier design, which gave the entrypoint its own simulation schema, an Alembic chain and a migration job before each deploy; all of that is removed, with the rules that only served it.

• Persistence library (libs/policyengine-simulation-persistence):
  • a pooled runtime engine (psycopg 3, pre-ping, a connect timeout, no server-side prepared statements, so the same URL works through Supabase's poolers); a malformed URL is refused without repeating it;
  • the privileges the service's role needs: SELECT and INSERT on the tables it creates rows in, SELECT on the policy, dynamic, household and catalog tables a run reads, and UPDATE on five columns of simulation_runs, report_runs and household_jobs (status, modal_call_id, error_message, completed_at, updated_at), so it can record a Modal call's ID or fail a run Modal refuses; and a check of which the connected role lacks, with has_table_privilege and has_column_privilege;
  • SQLAlchemy Core models of the six tables the service writes: simulations, reports, simulation_runs, report_runs, budget_window_runs and household_jobs, matching the Public API's tables column for column, with their constraints and indexes. A report run names the baseline and reform simulation runs it compares; a report's budget window is calculated as a budget_window_runs row with one report run per year.
  • the 20 foreign keys from those tables to the Public API's own (users, policies, dynamics, households, datasets, regions and the model catalog). PostgreSQL checks each on insert, as the referenced table's owner, so the role needs no privilege on those tables for it, and a row naming a missing one fails with SQLSTATE 23503. SQLAlchemy needs a definition of each referenced table to declare a key, so referenced_metadata models each by its id alone; nothing reads or writes through them.
• Runtime URL and readiness: the deploy script mounts SIMULATION_SERVICE_DATABASE_URL as a pinned Secret Manager version on every deploy. GET /ready reports simulation_database {reachable, has_required_privileges} without changing its status: the route is public, so it says only yes or no, and the service writes the reason to its own log (simulation_database_unreachable with the error's type, or simulation_database_missing_privileges with each missing privilege). A check that reached the database answers /ready for 30 seconds, so repeated requests don't each take a database connection; a failed check isn't reused. The check runs on a thread of its own, in a transaction whose statement_timeout ends any statement that runs past two seconds. The candidate check requires both answers to be yes, asking again while the check fails (as it does for a cold candidate), and points to the revision's Cloud Run logs when it fails; the stable and rollback checks don't require it.
• Deploy script: each runtime secret is now resolved in its own assignment. Before, an array assignment took its exit status from its last substitution only, so a failure to resolve either Stage 12 Modal token secret was ignored and the revision deployed with an empty reference. The Stage 12 block uses a wrapper of its own.
• Rules: new sim.persistence.missing-privileges-reported, with a proof, in a new persistence category; sim.deploy.workflow-shell-safety, sim.deploy.container-context-minimal and sim.deploy.smoke-tests-strict are tightened, the first two with proofs.
• CI: a new Simulation database job runs the persistence tests that need PostgreSQL on postgres:17, and fails rather than skips without the server; CI complete needs it. The tests that run as the service's role against the Public API's own tables need a database the Public API's chain built; they skip here, and the stacked pull request's Public API schema job runs them.

Environment variables

Variable	Read by	Provider	How it reaches the job
SIMULATION_SERVICE_DATABASE_URL	Entrypoint runtime	Secret Manager in policyengine-simulation-entry (the project named by SIMULATION_ENTRYPOINT_GCP_PROJECT_ID)	The deploy script resolves the secret named by SIMULATION_SERVICE_DATABASE_URL_SECRET_NAME to its newest enabled version and mounts it with gcloud run deploy --set-secrets, on every deploy
SIMULATION_SERVICE_DATABASE_URL_SECRET_NAME	Deploy script	GitHub environment variable (beta, prod)	env: of the "Deploy Cloud Run entrypoint candidate" step in deploy_entrypoint, whose job declares environment: ${{ inputs.release_environment }}
REQUIRE_SIMULATION_DATABASE_ACCESS	Candidate smoke script	Literal true	env: of the "Verify the complete deployed request path" step
SIMULATION_DATABASE_READY_MAX_ATTEMPTS	Candidate smoke script	Optional, defaults to 6	Not set
SIMULATION_TEST_DATABASE_URL, SIMULATION_REQUIRE_DATABASE_TESTS	Persistence tests in CI	Literals	Job env: of Simulation database, pointing at its postgres:17 service
PUBLIC_API_SCHEMA_DATABASE_URL, SIMULATION_REQUIRE_PUBLIC_API_SCHEMA	Persistence tests marked public_api_schema	Not set in this pull request's CI (the tests skip); the stacked pull request's job sets literals	—

Checked on GitHub on 2026-11-04, and the secret-name variable again on 2026-11-06 (read only): SIMULATION_ENTRYPOINT_GCP_PROJECT_ID is policyengine-simulation-entry and SIMULATION_ENTRYPOINT_RUNTIME_SERVICE_ACCOUNT is sim-entry-beta-runtime@… / sim-entry-prod-runtime@… in beta / prod. Checked on 2026-11-09 (read only, after the operator steps): SIMULATION_SERVICE_DATABASE_URL_SECRET_NAME is simulation-service-staging-database-url in beta and simulation-service-production-database-url in prod; both secrets exist in policyengine-simulation-entry with an enabled version, and sim-entry-beta-runtime@… / sim-entry-prod-runtime@… and sim-entry-gh-deployer@… hold Secret Accessor on them, as on the stage12-*-database-url secrets. The role exists in both v2 databases, and the Public API's seed workflow granted and verified its privileges. Not verified: that the deploy service account (the secret SIMULATION_ENTRYPOINT_GCP_DEPLOY_SERVICE_ACCOUNT) is sim-entry-gh-deployer@…; the stage12-*-database-url secrets give it the same access.

What this waits for before merging

1. Done: the Public API's migration (Record each run of a v2 simulation and report, and key results by run policyengine-api#4162) merged and was deployed to staging and production on 2026-11-09, with the simulation service role's grants in its privilege manifest; Fail the v2 deploy when the simulation API's database roles are missing policyengine-api#4235 then made a missing role fail the Public API's deploy.
2. Done on 2026-11-09: the operator steps, in staging and then production. The login role policyengine_v2_simulation_service exists in each v2 database; its URL, which logs in through the pooler as policyengine_v2_simulation_service.<project ref> with sslmode=require, is in simulation-service-staging-database-url and simulation-service-production-database-url; the runtime and deploy service accounts can read each secret; and the secret-name variable is set in beta and prod. After the first deploy, curl <candidate URL>/ready | jq .simulation_database should show {"reachable": true, "has_required_privileges": true} (the candidate check requires it); if not, the revision's Cloud Run logs name the cause.
3. One approving review, which main's branch rules require from someone other than the author.

The pull request stacked on this one adds the CI check that compares the models with the Public API's tables.

Testing

Rebuilt on main at 04bf1fe on 2026-11-04 (this replaces the earlier history); main merged in at 906a64d on 2026-11-06, and again at 5a2b8e2 on 2026-11-09 (e9976b52): one conflict, in the "Verify the complete deployed request path" step of simulation-deploy.reusable.yml, where main no longer passes EXPECTED_LATEST (publish_latest checks latest after moving it) and this pull request adds REQUIRE_SIMULATION_DATABASE_ACCESS; the step keeps both changes. 3b9b6464 re-renders the rules document, whose totals the merge left stale (the rules registry guard failed on it). After the merge, the 12 deployment tests in test_simulation_database_deployment.py passed by node ID, and Ruff, the quality guards, uv lock --check and Pyright for the persistence library and the entrypoint passed. A review round on the merge found no bugs. Every commit was written test first: the new or changed tests were run by node ID and seen to fail for the expected reason, then to pass. Before each commit, Ruff (the CI-pinned version), the quality guards with the rules registry, the generated rules document, uv lock --check for the persistence library and the entrypoint, and Pyright for both passed; after review, every commit was checked again for all but Pyright.

• The persistence library's PostgreSQL tests ran on PostgreSQL 17, and the tests marked public_api_schema ran as policyengine_v2_simulation_service against a database built with the Public API's v2 migration chain and privilege manifest at the head of Record each run of a v2 simulation and report, and key results by run policyengine-api#4162 (772ca6b14): the role lacks no table or column privilege it needs, and creates and reads back a row in each of the six tables, naming catalog, dataset and household rows that exist. The models also matched those tables exactly (columns, defaults, constraints with the 20 keys, indexes and enums).
• The persistence library requires sqlalchemy>=2,<2.1, so its own lock resolves the minor version the entrypoint's image installs (2.0); a test keeps the two locks on the same minor version.
• Only the tests this pull request adds or changes were run locally; GitHub Actions ran the rest, and every check passed on 098f408b.

🤖 Generated with Claude Code

[vahid-ahmadi]

Review at 6c58a36679.

Verdict: a safe first step. It adds no tables and no data, the migration reverses cleanly, and only main can apply revisions. Two points to settle before the stacked PRs build on it. No must items.

1. should migrations/versions/0001_initial_simulation_schema.py:27-32: ALTER DEFAULT PRIVILEGES IN SCHEMA simulation without FOR ROLE applies only to objects created by the role that runs it. That is fine while every migration runs as simulation_migrator. But if an operator ever creates a table as another role (an emergency fix as the project owner, say), simulation_runtime silently gets nothing on it. Writing FOR ROLE simulation_migrator makes the intent explicit. Separately, the defaults grant no DELETE. If that is deliberate (runs and results are append-only), a line in the README says so before the later PRs need it.
2. should The "Waits for" step grants the migrator REFERENCES on Public API tables. Once a simulation table has a foreign key into public, a Public API revision that drops, renames or reshapes that table fails at deploy time, as in policyengine-api #4010 (rename user_policies) and #4014 (drop tables). ON DELETE RESTRICT there would also block the Public API's user deletes. That coupling is worth recording in policyengine-api's Alembic skill now, before the first such foreign key lands, along with which Public API tables it is meant to cover.
3. nit projects/policyengine-simulation-entry/src/policyengine_simulation_entry/simulation_database.py:38: wait_for on to_thread cannot cancel the thread, and the engine has no statement_timeout and the default 30 s pool_timeout. So while the database is slow, each /ready call leaves a thread in the default executor, which stage12_backend also uses. Only the smoke script polls /ready today, so this is minor; a short pool_timeout and a statement timeout would bound it.

Checked and fine: one transaction covers the schema creation and all revisions; lock_timeout is 10 s; offline mode is refused; URL errors never echo the URL; the migration secret is scoped to one step's env:; deploy_entrypoint needs the migration job; non-main runs only current --check-heads; the deploy serialises on simulation-deploy-main; the /ready status code does not depend on the database, so the old gateway routes are unaffected. The per-secret assignment fix in the deploy script is a real bug fix.

What I ran: fresh clone, uv run --frozen --extra test pytest in libs/policyengine-simulation-persistence (29 passed, 10 skipped because they need PostgreSQL) and in projects/policyengine-simulation-entry (797 passed). The PR is a draft and CI runs on it were cancelled, so the new Simulation database job (upgrade, alembic check, downgrade/upgrade cycle as a non-superuser) has not run on this head. I did not read the rule proof fixtures one by one.

🤖 Generated with Claude Code