Skip to content

Use one library of the Public API's table models for the simulation service and the v2 Modal runtime - #1078

Draft
anth-volk wants to merge 31 commits into
mainfrom
v2plan/c1-32-one-table-model-library
Draft

anth-volk wants to merge 31 commits into
mainfrom
v2plan/c1-32-one-table-model-library

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1077

Same-Repository Draft PR Check

  • This PR is a draft.
  • This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Changelog

  • This PR adds a changelog fragment, changelog.d/c1-32-one-table-model-library.changed.md, whose type is breaking, added, changed, fixed or removed.

Summary

The simulation service and the v2 Modal runtime now use one library of the Public API's table models, policyengine-simulation-persistence, and CI compares every run and result table sim writes with the tables the Public API's chain builds.

Builds on #1072 (shared simulation runs), which changes the same Stage 12 stores. Its branch was merged in while it was open, and it has since merged into main, so this diff shows only this PR's own changes. The migration plan has #1074 and #1070 land before this one too; their sessions agreed.

The decisions this follows

These are the migration plan's decisions of 2026-11-16 (task C1 · 31):

  1. One library of the Public API's table models for the simulation service and the v2 runtime.
  2. policyengine-simulation-persistence keeps the models.
    • The Stage 12 library's report_run_tables.py goes, and its stores import the service library's run-table models.
    • The twelve result tables move into the service library as complete models. The code that builds their rows stays in the Stage 12 library, so the service library gains no dependency.
    • The engine and URL helpers become one copy, in the service library.
    • The Stage 12 library keeps ReportRunStore, BudgetWindowStore, HouseholdReportRunStore and the comparison path.
    • The duplicated enum type declarations go. RunStatus stays as the Stage 12 code's one status for any run kind, while the service's three status enums, which name the entry's OpenAPI schemas, stay separate. This reading was confirmed with the plan's decision task.
  3. The Public API schema job compares every table sim writes exactly.
    • A new test checks the Modal worker role's grants and writes, as test_service_writes does for the service's role.
    • The Stage 12 library's database tests run in the same job, with the Modal worker role given a login there.
    • Rule sim.persistence.models-match-public-api extends to the moved models.
  4. One dependency range: the shared library's sqlalchemy>=2,<2.1 and psycopg[binary]>=3.2,<4. The Stage 12 library and the executor's modal-simulation-image group, which allowed sqlalchemy<3, now declare the same. Each lock keeps its own patch release, so there's no check that the locks are equal.
  5. Already decided: names keep "stage12" until a later rename, and the comparison path stays in the Stage 12 library until it is removed.

The plan also decided that a later task drops the four cliff columns from these models, once the Public API drops them, so they stay here while the pinned chain has them.

What changed

Run-table models (report_runs, simulation_runs, budget_window_runs, household_jobs):

  • The Stage 12 stores import them from policyengine_simulation_persistence.tables, and the partial report_run_tables.py is deleted.
  • The complete models store a Python None in household_result, result and spm_provenance as SQL NULL, as the Stage 12 models did. The service only reads those columns.

Result-table models:

  • result_tables.py models the twelve result tables in result_metadata, with every column, key and check under the Public API's names, and the v2_decile_type and v2_uk_area_type enums.
  • report_run_results.py keeps the row building and imports those tables.

Engine and URL check:

  • create_unpooled_engine sits beside create_simulation_engine in the service library's engine.py and uses the same URL check. It has no pool and a 10-second connect timeout, the Stage 12 engine's settings.
  • The Stage 12 store binds it as create_stage12_engine, and its own copies of both helpers are deleted.

Modal worker role:

  • MODAL_WORKER_TABLE_PRIVILEGES and MODAL_WORKER_UPDATABLE_COLUMNS list what the v2 Modal calls need. That is SELECT on the four run tables and INSERT on every result table. It is also UPDATE of each run's state and of the results its own row holds.
  • missing_table_privileges takes another role's lists, and still checks the service's by default.
  • test_modal_worker_writes runs as the role. It checks the Public API's grants, records a run's state in each run table and inserts a row in every result table, then reads back SQL NULL for the None JSON results.

Executor and images:

  • The executor depends on the service library.
  • The v2 app (the workers', the coordinator's and the household functions' images), the stuck-run sweep and the Stage 12 precompute copy the service library with their other sources.
  • pr-image-smoke.yml runs when the service library changes.

The v1 run_simulation code stays apart: a new test follows src/modal/app.py's imports through this repository's modules, sim's libraries and every parent package (66 modules), and reads the old gateway's source. Neither imports either persistence library.

CI: after the comparison, the Public API schema job runs public-api-schema-worker-login.sh, which gives the Modal worker role a login on the job's database. It then runs the Stage 12 library's tests as that role, with REPORT_RUN_REQUIRE_DATABASE_TESTS=1, so a missing database fails them. Before, no workflow set their database URLs, so they skipped in CI.

Dependency ranges:

  • The Stage 12 library and the executor's modal-simulation-image group declare the shared library's sqlalchemy>=2,<2.1.
  • The locks keep their releases: SQLAlchemy 2.0.49 in the executor's, which the Modal images install, and 2.0.54 in the entry's and the Stage 12 library's, which Cloud Run installs. Only the ranges they record change, so the images get the same packages.
  • A test in each package requires its database ranges to equal the shared library's.

Rules and docs:

  • sim.persistence.models-match-public-api and sim.persistence.missing-privileges-reported name the new tests.
  • Both libraries' READMEs and the database-access skill describe the one library.

What behaves the same, and what doesn't

  • The Stage 12 stores send the same writes. Compiled with psycopg's dialect against the deleted models and the shared ones, the stores' statements have the same columns, casts and bound values. Only the order of an UPDATE's SET columns differs, which PostgreSQL ignores.
  • Reads of a status now return the table's own enum member, such as ReportRunStatus.RUNNING where they returned 'running'. Every store converts with RunStatus(...) before comparing, and a new test pins RunStatus's values to each table's enum.
  • A malformed Modal worker or comparison-path database URL now raises the service's "Simulation database URL ..." error, without the parser's message, which may quote the URL.

How it combines with the open pull requests

Deployment and CI variables

No secret, Dockerfile or deploy workflow changes. The Cloud Run image already copies libs/, and Dockerfile.dockerignore lets in both libraries. The Modal images add one local source package and no third-party package.

Variable Read by Provider and wiring
MODAL_WORKER_DATABASE_URL stage12_runtime.report_run_access (unchanged) Modal secret policyengine-modal-worker-database, attached in v2_app.py and run_sweep_app.py (unchanged)
SIMULATION_SERVICE_DATABASE_URL, STAGE12_DATABASE_URL the entry's settings (unchanged) Secret Manager, wired to Cloud Run as before
PUBLIC_API_SCHEMA_DATABASE_URL the new login script, and the existing steps the job's env: literal, naming its own PostgreSQL service
MODAL_WORKER_TEST_PASSWORD the new login script a literal in the login step's env:; the password exists only in the job's disposable database
REPORT_RUN_TEST_DATABASE_URL, REPORT_RUN_TEST_SEED_DATABASE_URL, REPORT_RUN_REQUIRE_DATABASE_TESTS the Stage 12 tests' fixtures literals in the Stage 12 test step's env:

Unverified until the first CI run: that psql, which the login script runs, is on the ubuntu-latest runner. GitHub's Ubuntu images ship the PostgreSQL client. The script itself was run against a local PostgreSQL 17 database.

Testing

  • The tests this PR adds or changes, each by node ID: 22 in the service library, 11 in the Stage 12 library and 13 in the executor. The database-backed ones ran on a disposable PostgreSQL 17 database that the Public API's v2 chain built at master d7042b22, the pinned commit plus a version bump.
  • The result models' comparison was also checked to fail on a deliberate difference.
  • Each failing test was committed before the code that makes it pass, with "(failing)" in the subject.
  • Ruff, shellcheck, the rules guard, Pyright for every changed package and the lock checks pass locally.
  • Not run locally: the Stage 12 library's whole suite against the job's database. The new CI step runs it.

Review

Three read-only review rounds:

  1. Round 1 found no bugs. Its quality findings were fixed: the v1 import walk now follows parent packages and sim's libraries, which was test-first, and two README and workflow comments that had drifted were corrected.
  2. Round 2 reviewed those fixes and the dependency ranges. It found one bug: an existing executor test still required sqlalchemy>=2,<3 in the Modal image group. That test now checks the group installs both database packages, and leaves their ranges to the comparison test. Its findings on the walk were fixed test-first: the walk now covers the entry module's own package, and the real test requires that it reaches sim's libraries.
  3. Round 3 reviewed round 2's fixes and found no bugs.

Left as they are:

  • result_tables imports tables' private column helpers. Renaming them would rewrite lines Find or create simulations, reports and runs by key #1074 changes.
  • Some implementation commits also edit their red test.
  • The requirement-parsing helper is copied in two packages' test folders, which keep their helpers apart.

🤖 Generated with Claude Code

anth-volk and others added 28 commits November 16, 2026 19:16
C1 · 32 moves the Stage 12 library's table models, and #1072 changes the
Stage 12 stores that use them, so C1 · 32 builds on it. #1072 lands first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…els (failing)

The Stage 12 library will import its models of `report_runs`,
`simulation_runs`, `budget_window_runs` and `household_jobs` from
`policyengine-simulation-persistence`, whose models the CI comparison checks
against the Public API's tables. It now depends on that library, so the
entry's and executor's locks follow.

The service library's models must store a Python `None` in
`simulation_runs.household_result`, `household_jobs.result` and
`report_runs.spm_provenance` as SQL NULL, as the Stage 12 models do: the
Modal worker writes `None` to them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ReportRunStore`, `BudgetWindowStore` and `HouseholdReportRunStore` now
import `report_runs`, `simulation_runs`, `budget_window_runs` and
`household_jobs` from `policyengine_simulation_persistence.tables`, the
complete models the CI comparison checks, and `report_run_tables.py` is
gone. Every statement names the same columns with the same casts and
parameters as before; only the order of an UPDATE's SET columns follows the
complete models' column order.

The complete models store a Python `None` in `household_result`, `result`
and `spm_provenance` as SQL NULL, as the Stage 12 models did. The service
only reads those columns.

The result-table models keep their own metadata in `report_run_results.py`
until they move too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h (failing)

The twelve result tables the Modal worker role writes are to be modelled
completely in `policyengine-simulation-persistence` (`result_metadata`), so
the `Public API schema` job compares them exactly with the tables the Public
API's chain builds. The Stage 12 library then declares no table of its own
but the comparison path's, and every result row it builds goes in a shared
model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`policyengine_simulation_persistence.result_tables` models the result tables
the Modal worker role writes, in `result_metadata`: every column, the primary
and unique keys, checks and foreign keys under the Public API's names, and
the `v2_decile_type` and `v2_uk_area_type` enums. Their keys name the service
library's `report_runs` and `simulation_runs`. The comparison now covers
them, as it covers the service's own tables.

The Stage 12 library's `report_run_results.py` keeps the code that builds
result rows from the economy output, and imports the tables it inserts into
from the service library instead of declaring partial models of its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The service library is to hold the one URL check and the unpooled engine the
Modal calls' stores and the comparison path use: an engine with no pool and a
10-second connect timeout, so a long calculation holds no idle database
session. The Stage 12 library's tests of its own copies move to the service
library's engine tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ibrary

`create_unpooled_engine` in `policyengine_simulation_persistence.engine`
builds the engine the Stage 12 stores used: no pool, so each operation's
session ends with it, and a 10-second connect timeout. The Stage 12 store
binds it as `create_stage12_engine`, the name the stores and the comparison
path already use, and its own copies of the engine and URL helpers are gone.

A malformed Modal worker or comparison-path URL now raises the service's
"Simulation database URL ..." error without the parser's message, which may
quote the URL, as its cause.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The service library is to list the privileges the v2 Modal calls need as
`policyengine_v2_modal_worker`: SELECT on the four run tables, column UPDATE
of each run's state and the results its row holds, and INSERT on the twelve
result tables. `missing_table_privileges` is to check any role's lists.

In the `Public API schema` job, a new test checks that the Public API's
grants give the role those privileges, and, as the role, records a run's
state in each run table and inserts a row in every result table through the
shared models; a Python None in a JSON result is stored as SQL NULL.

The service write test's row helpers move to `tests/service_rows.py`, so both
roles' tests insert the same rows.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ants

`MODAL_WORKER_TABLE_PRIVILEGES` and `MODAL_WORKER_UPDATABLE_COLUMNS` list what
the v2 Modal calls need as `policyengine_v2_modal_worker`: SELECT on
`report_runs`, `simulation_runs`, `household_jobs` and `budget_window_runs`;
UPDATE of each run's state (status, call, failure and times) and of the
results its own row holds; and INSERT on every result table.
`missing_table_privileges` takes another role's lists, and still checks the
simulation service's by default, as the readiness check uses it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oesn't (failing)

The Stage 12 library's stores now import the service library's table models,
so every v2 Modal image (the workers', the coordinator's, the household
functions', the stuck-run sweep's and the Stage 12 precompute's) must copy
`policyengine_simulation_persistence` with the other project sources.

The v1 `run_simulation` code stays apart from both persistence libraries: a
new test follows `src/modal/app.py`'s imports through this project's modules,
and reads the old gateway's source, and finds neither library imported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The executor depends on `policyengine-simulation-persistence` directly, and
the v2 app, the stuck-run sweep and the Stage 12 precompute copy it with the
other project sources, since the Stage 12 library's stores use its table
models. The images install no new third-party package: SQLAlchemy and psycopg
are already in the `modal-simulation-image` group, and the executor's lock
keeps SQLAlchemy 2.0.49, within the service library's `<2.1`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ing)

The v2 images copy `policyengine-simulation-persistence` now, so a change to
it must build and import them again, as a change to the other libraries they
copy does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The v2 Modal images copy `policyengine-simulation-persistence`, so a pull
request that changes it builds them and imports their entry points again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rker (failing)

After the comparison, the job is to give `policyengine_v2_modal_worker` a
login on the database it built, with a password that exists only there, and
run the Stage 12 library's tests against it as that role, with the
superuser seeding the service's rows. A new script gives the login; the
Stage 12 tests are to fail instead of skipping when the job requires their
database (`REPORT_RUN_REQUIRE_DATABASE_TESTS=1`).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the comparison, the job gives `policyengine_v2_modal_worker` a login on
the database it built (`public-api-schema-worker-login.sh`, with a password
that exists only in that database), then runs the Stage 12 library's tests
as that role, the superuser seeding the service's rows. Until now no
workflow set their database URLs, so they skipped in CI; here a missing
database fails them (`REPORT_RUN_REQUIRE_DATABASE_TESTS=1`).

The new step variables are literals in `pr.yml`, naming the job's own
PostgreSQL service; the job still reads no secret.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rule `sim.persistence.models-match-public-api` now covers the result-table
models and the Modal worker role's writes, with their tests as enforcers,
and `sim.persistence.missing-privileges-reported` covers the check run for
another role's lists. The two libraries' READMEs and the database-access
skill say where the models live, who imports them, and where their tests
run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…raries (failing)

Review round 1 found that the walk from `src/modal/app.py` resolved each
imported module to its own file only. Python runs every parent package's
`__init__` first, so an import of a `stage12_runtime` submodule would load the
Stage 12 library through the package and pass unseen. The walk now takes its
source folders as arguments, follows sim's own libraries too, and two tests
on throwaway source trees check both paths; the package one fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Importing a module runs its parent packages' `__init__` first, so the walk
from `src/modal/app.py` now reaches each of them too. It walks 66 modules,
18 of them in sim's libraries, and finds no import of either table-model
library.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`RunStatus` stays the Stage 12 code's one status for any run kind; the
service library's `SimulationStatus`, `ReportRunStatus` and
`HouseholdJobStatus`, which name the entry's OpenAPI schemas, stay
separate. A read through the shared models returns the table's own member,
which the stores convert with `RunStatus(...)`, so the test requires the
same values in the same order and the conversion of each member. It fails
when `RunStatus` gains a value the tables don't have.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The service library README's pin guidance now names the comparison path's
`stage12_evaluation_*` tables as the only tables sim writes that this
comparison doesn't cover, and the `Public API schema` job's comment in the
PR workflow describes the Modal worker role's tests and the Stage 12 tests
it runs. Review round 1's documentation findings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed ranges (failing)

The migration plan decided (task C1 · 31, decision 4) that the shared table-model
library's `sqlalchemy>=2,<2.1` and `psycopg[binary]>=3.2,<4` are the one
range: the Stage 12 library and the executor's `modal-simulation-image`
group, which still allow `sqlalchemy<3`, are to declare the same, so no
declaration names a version the locks can't take.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…anges

Both now declare `sqlalchemy>=2,<2.1`, the shared table-model library's
range, beside its `psycopg[binary]>=3.2,<4`. The locks keep their own patch
releases: 2.0.49 in the executor's, which the Modal images install, and 2.0.54
in the Stage 12 library's and the entry's, which Cloud Run installs; only
their recorded ranges change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`test_v2_image_retains_required_runtime_dependencies` still required
`sqlalchemy>=2,<3` in the `modal-simulation-image` group, which now declares
the shared library's `sqlalchemy>=2,<2.1`, so it failed. Review round 2's bug.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ailing)

Importing `src.modal.app` runs `src.modal`'s `__init__` first, but the walk
reached that package only because the app happens to import another module
in it. The real v1 test also now requires the walk to reach sim's libraries.
Review round 2's quality findings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`src.modal`'s `__init__` is now reached because `src.modal.app` is in it, not
only because the app imports another module of that package.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… test

`test_v2_image_retains_required_runtime_dependencies` now checks only that
the `modal-simulation-image` group installs SQLAlchemy and psycopg;
`test_database_dependency_ranges` compares their ranges with the shared
table-model library's, so a change to that range edits one test. Review
round 3's quality finding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1072 merged `main` (4b0fce3), which this branch already has, so no file
changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`main` updated .py to 6.20.0 (#1076), which changes the executor's
`pyproject.toml` and `uv.lock` beside this branch's changes; they merged
without a conflict and the lock matches the project.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`main` removed the staging v2 route checks from the deploy workflow and the
entry's tests (#1088), none of which this branch changes; no conflict.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 2 commits November 17, 2026 12:10
#1072 (shared simulation runs) merged into `main` as 6df34f9. This branch
already had its commits, from merges of its branch, so the merge brings no
change to the Stage 12 library.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Use one library of the Public API's table models for the simulation service and the v2 Modal runtime

1 participant