Repository navigation
Use one library of the Public API's table models for the simulation service and the v2 Modal runtime - #1078
Draft
anth-volk wants to merge 31 commits into
Draft
Use one library of the Public API's table models for the simulation service and the v2 Modal runtime#1078anth-volk wants to merge 31 commits into
anth-volk wants to merge 31 commits into
Conversation
…els (failing) The Stage 12 library will import its models of `report_runs`, `simulation_runs`, `budget_window_runs` and `household_jobs` from `policyengine-simulation-persistence`, whose models the CI comparison checks against the Public API's tables. It now depends on that library, so the entry's and executor's locks follow. The service library's models must store a Python `None` in `simulation_runs.household_result`, `household_jobs.result` and `report_runs.spm_provenance` as SQL NULL, as the Stage 12 models do: the Modal worker writes `None` to them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`ReportRunStore`, `BudgetWindowStore` and `HouseholdReportRunStore` now import `report_runs`, `simulation_runs`, `budget_window_runs` and `household_jobs` from `policyengine_simulation_persistence.tables`, the complete models the CI comparison checks, and `report_run_tables.py` is gone. Every statement names the same columns with the same casts and parameters as before; only the order of an UPDATE's SET columns follows the complete models' column order. The complete models store a Python `None` in `household_result`, `result` and `spm_provenance` as SQL NULL, as the Stage 12 models did. The service only reads those columns. The result-table models keep their own metadata in `report_run_results.py` until they move too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…h (failing) The twelve result tables the Modal worker role writes are to be modelled completely in `policyengine-simulation-persistence` (`result_metadata`), so the `Public API schema` job compares them exactly with the tables the Public API's chain builds. The Stage 12 library then declares no table of its own but the comparison path's, and every result row it builds goes in a shared model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`policyengine_simulation_persistence.result_tables` models the result tables the Modal worker role writes, in `result_metadata`: every column, the primary and unique keys, checks and foreign keys under the Public API's names, and the `v2_decile_type` and `v2_uk_area_type` enums. Their keys name the service library's `report_runs` and `simulation_runs`. The comparison now covers them, as it covers the service's own tables. The Stage 12 library's `report_run_results.py` keeps the code that builds result rows from the economy output, and imports the tables it inserts into from the service library instead of declaring partial models of its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The service library is to hold the one URL check and the unpooled engine the Modal calls' stores and the comparison path use: an engine with no pool and a 10-second connect timeout, so a long calculation holds no idle database session. The Stage 12 library's tests of its own copies move to the service library's engine tests. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ibrary `create_unpooled_engine` in `policyengine_simulation_persistence.engine` builds the engine the Stage 12 stores used: no pool, so each operation's session ends with it, and a 10-second connect timeout. The Stage 12 store binds it as `create_stage12_engine`, the name the stores and the comparison path already use, and its own copies of the engine and URL helpers are gone. A malformed Modal worker or comparison-path URL now raises the service's "Simulation database URL ..." error without the parser's message, which may quote the URL, as its cause. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The service library is to list the privileges the v2 Modal calls need as `policyengine_v2_modal_worker`: SELECT on the four run tables, column UPDATE of each run's state and the results its row holds, and INSERT on the twelve result tables. `missing_table_privileges` is to check any role's lists. In the `Public API schema` job, a new test checks that the Public API's grants give the role those privileges, and, as the role, records a run's state in each run table and inserts a row in every result table through the shared models; a Python None in a JSON result is stored as SQL NULL. The service write test's row helpers move to `tests/service_rows.py`, so both roles' tests insert the same rows. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ants `MODAL_WORKER_TABLE_PRIVILEGES` and `MODAL_WORKER_UPDATABLE_COLUMNS` list what the v2 Modal calls need as `policyengine_v2_modal_worker`: SELECT on `report_runs`, `simulation_runs`, `household_jobs` and `budget_window_runs`; UPDATE of each run's state (status, call, failure and times) and of the results its own row holds; and INSERT on every result table. `missing_table_privileges` takes another role's lists, and still checks the simulation service's by default, as the readiness check uses it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…oesn't (failing) The Stage 12 library's stores now import the service library's table models, so every v2 Modal image (the workers', the coordinator's, the household functions', the stuck-run sweep's and the Stage 12 precompute's) must copy `policyengine_simulation_persistence` with the other project sources. The v1 `run_simulation` code stays apart from both persistence libraries: a new test follows `src/modal/app.py`'s imports through this project's modules, and reads the old gateway's source, and finds neither library imported. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The executor depends on `policyengine-simulation-persistence` directly, and the v2 app, the stuck-run sweep and the Stage 12 precompute copy it with the other project sources, since the Stage 12 library's stores use its table models. The images install no new third-party package: SQLAlchemy and psycopg are already in the `modal-simulation-image` group, and the executor's lock keeps SQLAlchemy 2.0.49, within the service library's `<2.1`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ing) The v2 images copy `policyengine-simulation-persistence` now, so a change to it must build and import them again, as a change to the other libraries they copy does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The v2 Modal images copy `policyengine-simulation-persistence`, so a pull request that changes it builds them and imports their entry points again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rker (failing) After the comparison, the job is to give `policyengine_v2_modal_worker` a login on the database it built, with a password that exists only there, and run the Stage 12 library's tests against it as that role, with the superuser seeding the service's rows. A new script gives the login; the Stage 12 tests are to fail instead of skipping when the job requires their database (`REPORT_RUN_REQUIRE_DATABASE_TESTS=1`). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
After the comparison, the job gives `policyengine_v2_modal_worker` a login on the database it built (`public-api-schema-worker-login.sh`, with a password that exists only in that database), then runs the Stage 12 library's tests as that role, the superuser seeding the service's rows. Until now no workflow set their database URLs, so they skipped in CI; here a missing database fails them (`REPORT_RUN_REQUIRE_DATABASE_TESTS=1`). The new step variables are literals in `pr.yml`, naming the job's own PostgreSQL service; the job still reads no secret. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rule `sim.persistence.models-match-public-api` now covers the result-table models and the Modal worker role's writes, with their tests as enforcers, and `sim.persistence.missing-privileges-reported` covers the check run for another role's lists. The two libraries' READMEs and the database-access skill say where the models live, who imports them, and where their tests run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…raries (failing) Review round 1 found that the walk from `src/modal/app.py` resolved each imported module to its own file only. Python runs every parent package's `__init__` first, so an import of a `stage12_runtime` submodule would load the Stage 12 library through the package and pass unseen. The walk now takes its source folders as arguments, follows sim's own libraries too, and two tests on throwaway source trees check both paths; the package one fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Importing a module runs its parent packages' `__init__` first, so the walk from `src/modal/app.py` now reaches each of them too. It walks 66 modules, 18 of them in sim's libraries, and finds no import of either table-model library. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`RunStatus` stays the Stage 12 code's one status for any run kind; the service library's `SimulationStatus`, `ReportRunStatus` and `HouseholdJobStatus`, which name the entry's OpenAPI schemas, stay separate. A read through the shared models returns the table's own member, which the stores convert with `RunStatus(...)`, so the test requires the same values in the same order and the conversion of each member. It fails when `RunStatus` gains a value the tables don't have. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The service library README's pin guidance now names the comparison path's `stage12_evaluation_*` tables as the only tables sim writes that this comparison doesn't cover, and the `Public API schema` job's comment in the PR workflow describes the Modal worker role's tests and the Stage 12 tests it runs. Review round 1's documentation findings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed ranges (failing) The migration plan decided (task C1 · 31, decision 4) that the shared table-model library's `sqlalchemy>=2,<2.1` and `psycopg[binary]>=3.2,<4` are the one range: the Stage 12 library and the executor's `modal-simulation-image` group, which still allow `sqlalchemy<3`, are to declare the same, so no declaration names a version the locks can't take. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…anges Both now declare `sqlalchemy>=2,<2.1`, the shared table-model library's range, beside its `psycopg[binary]>=3.2,<4`. The locks keep their own patch releases: 2.0.49 in the executor's, which the Modal images install, and 2.0.54 in the Stage 12 library's and the entry's, which Cloud Run installs; only their recorded ranges change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`test_v2_image_retains_required_runtime_dependencies` still required `sqlalchemy>=2,<3` in the `modal-simulation-image` group, which now declares the shared library's `sqlalchemy>=2,<2.1`, so it failed. Review round 2's bug. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ailing) Importing `src.modal.app` runs `src.modal`'s `__init__` first, but the walk reached that package only because the app happens to import another module in it. The real v1 test also now requires the walk to reach sim's libraries. Review round 2's quality findings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`src.modal`'s `__init__` is now reached because `src.modal.app` is in it, not only because the app imports another module of that package. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… test `test_v2_image_retains_required_runtime_dependencies` now checks only that the `modal-simulation-image` group installs SQLAlchemy and psycopg; `test_database_dependency_ranges` compares their ranges with the shared table-model library's, so a change to that range edits one test. Review round 3's quality finding. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`main` updated .py to 6.20.0 (#1076), which changes the executor's `pyproject.toml` and `uv.lock` beside this branch's changes; they merged without a conflict and the lock matches the project. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`main` removed the staging v2 route checks from the deploy workflow and the entry's tests (#1088), none of which this branch changes; no conflict. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1077
Same-Repository Draft PR Check
PolicyEngine/policyengine-sim-api, not a personal fork.Changelog
changelog.d/c1-32-one-table-model-library.changed.md, whose type is breaking, added, changed, fixed or removed.Summary
The simulation service and the v2 Modal runtime now use one library of the Public API's table models,
policyengine-simulation-persistence, and CI compares every run and result table sim writes with the tables the Public API's chain builds.Builds on #1072 (shared simulation runs), which changes the same Stage 12 stores. Its branch was merged in while it was open, and it has since merged into
main, so this diff shows only this PR's own changes. The migration plan has #1074 and #1070 land before this one too; their sessions agreed.The decisions this follows
These are the migration plan's decisions of 2026-11-16 (task C1 · 31):
policyengine-simulation-persistencekeeps the models.report_run_tables.pygoes, and its stores import the service library's run-table models.ReportRunStore,BudgetWindowStore,HouseholdReportRunStoreand the comparison path.RunStatusstays as the Stage 12 code's one status for any run kind, while the service's three status enums, which name the entry's OpenAPI schemas, stay separate. This reading was confirmed with the plan's decision task.Public API schemajob compares every table sim writes exactly.test_service_writesdoes for the service's role.sim.persistence.models-match-public-apiextends to the moved models.sqlalchemy>=2,<2.1andpsycopg[binary]>=3.2,<4. The Stage 12 library and the executor'smodal-simulation-imagegroup, which allowedsqlalchemy<3, now declare the same. Each lock keeps its own patch release, so there's no check that the locks are equal.The plan also decided that a later task drops the four cliff columns from these models, once the Public API drops them, so they stay here while the pinned chain has them.
What changed
Run-table models (
report_runs,simulation_runs,budget_window_runs,household_jobs):policyengine_simulation_persistence.tables, and the partialreport_run_tables.pyis deleted.Noneinhousehold_result,resultandspm_provenanceas SQL NULL, as the Stage 12 models did. The service only reads those columns.Result-table models:
result_tables.pymodels the twelve result tables inresult_metadata, with every column, key and check under the Public API's names, and thev2_decile_typeandv2_uk_area_typeenums.report_run_results.pykeeps the row building and imports those tables.Engine and URL check:
create_unpooled_enginesits besidecreate_simulation_enginein the service library'sengine.pyand uses the same URL check. It has no pool and a 10-second connect timeout, the Stage 12 engine's settings.create_stage12_engine, and its own copies of both helpers are deleted.Modal worker role:
MODAL_WORKER_TABLE_PRIVILEGESandMODAL_WORKER_UPDATABLE_COLUMNSlist what the v2 Modal calls need. That is SELECT on the four run tables and INSERT on every result table. It is also UPDATE of each run's state and of the results its own row holds.missing_table_privilegestakes another role's lists, and still checks the service's by default.test_modal_worker_writesruns as the role. It checks the Public API's grants, records a run's state in each run table and inserts a row in every result table, then reads back SQL NULL for theNoneJSON results.Executor and images:
pr-image-smoke.ymlruns when the service library changes.The v1
run_simulationcode stays apart: a new test followssrc/modal/app.py's imports through this repository's modules, sim's libraries and every parent package (66 modules), and reads the old gateway's source. Neither imports either persistence library.CI: after the comparison, the
Public API schemajob runspublic-api-schema-worker-login.sh, which gives the Modal worker role a login on the job's database. It then runs the Stage 12 library's tests as that role, withREPORT_RUN_REQUIRE_DATABASE_TESTS=1, so a missing database fails them. Before, no workflow set their database URLs, so they skipped in CI.Dependency ranges:
modal-simulation-imagegroup declare the shared library'ssqlalchemy>=2,<2.1.Rules and docs:
sim.persistence.models-match-public-apiandsim.persistence.missing-privileges-reportedname the new tests.What behaves the same, and what doesn't
UPDATE's SET columns differs, which PostgreSQL ignores.ReportRunStatus.RUNNINGwhere they returned'running'. Every store converts withRunStatus(...)before comparing, and a new test pinsRunStatus's values to each table's enum.How it combines with the open pull requests
main, andmainis merged in here. Its changes to the Stage 12 stores are kept as they are; only where they get their tables changed.tables.py(key columns,forced, unique partial indexes) and moves the pin. The Stage 12 stores read and write none of its new columns, andforcedhas a server default. When it lands, this branch mergesmain.Deployment and CI variables
No secret, Dockerfile or deploy workflow changes. The Cloud Run image already copies
libs/, andDockerfile.dockerignorelets in both libraries. The Modal images add one local source package and no third-party package.MODAL_WORKER_DATABASE_URLstage12_runtime.report_run_access(unchanged)policyengine-modal-worker-database, attached inv2_app.pyandrun_sweep_app.py(unchanged)SIMULATION_SERVICE_DATABASE_URL,STAGE12_DATABASE_URLPUBLIC_API_SCHEMA_DATABASE_URLenv:literal, naming its own PostgreSQL serviceMODAL_WORKER_TEST_PASSWORDenv:; the password exists only in the job's disposable databaseREPORT_RUN_TEST_DATABASE_URL,REPORT_RUN_TEST_SEED_DATABASE_URL,REPORT_RUN_REQUIRE_DATABASE_TESTSenv:Unverified until the first CI run: that
psql, which the login script runs, is on theubuntu-latestrunner. GitHub's Ubuntu images ship the PostgreSQL client. The script itself was run against a local PostgreSQL 17 database.Testing
masterd7042b22, the pinned commit plus a version bump.Review
Three read-only review rounds:
sqlalchemy>=2,<3in the Modal image group. That test now checks the group installs both database packages, and leaves their ranges to the comparison test. Its findings on the walk were fixed test-first: the walk now covers the entry module's own package, and the real test requires that it reaches sim's libraries.Left as they are:
result_tablesimportstables' private column helpers. Renaming them would rewrite lines Find or create simulations, reports and runs by key #1074 changes.🤖 Generated with Claude Code