Skip to content

Store dynamics like policies, create them through /v2/dynamics, and keep gov.simulation parameters out of policies - #4325

Merged
anth-volk merged 11 commits into
masterfrom
v2plan/c1-36-dynamics
Nov 19, 2026
Merged

anth-volk merged 11 commits into
masterfrom
v2plan/c1-36-dynamics

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4324

Summary

Dynamics hold a simulation's model assumptions: every parameter under gov.simulation., and in the US also the top-level simulation. tree. This stores them as policies are stored, creates them through /v2/dynamics, and keeps them out of policies.

  • Revision 9d1d12a7ac58 (parent f519a77ba122), autogenerated, with no corrections apart from ruff format:

    • dynamics gains policies' columns: country_id, tax_benefit_model_id and tax_benefit_model_version_id (all RESTRICT), plus canonicalization_version and content_hash, unique together. It drops name and description, which nothing wrote or read.
    • parameter_values gains a partial unique index on (dynamic_id, parameter_id, start_date) for rows that have a dynamic.
    • legacy_policy_mappings gains a nullable dynamic_id, a key (with the mapping's country) to dynamics.
    • The runtime role may read and insert dynamics.
  • policyengine_api/dynamic_parameters.py:

    • which parameters a dynamic holds, per country;
    • the UK rewrite of gov.simulation.labor_supply_responses. to labour_supply_responses. (the US name stays);
    • split_legacy_policy_json, which the one-time migration will reuse.
  • POST /v2/dynamics (country_id, optional policyengine_version) binds the values to a catalog version and stores them once per content hash, with the policies' canonical document and canonicalization version. It answers:

    • 201 when it creates a dynamic, 200 when identical content is already stored;
    • 422 with code policy_parameters, naming them, for parameters that aren't model assumptions;
    • the usual request-validation 422 for an empty dynamic: for no assumptions, send dynamic_id null.
  • GET /v2/dynamics/{dynamic_id} reads one in its country, with missing_parameters. There is no list route.

  • POST /v2/policies refuses model assumptions with 422, code dynamic_parameters, naming them.

  • /v2/simulations and /v2/reports (economy reports and budget windows) check each named dynamic after the policies:

    • one the country doesn't store answers 404;
    • one that sets parameters the running catalog lacks answers 422 missing_parameters.

    Re-runs leave a dynamic's missing parameters to the simulation service, which refuses them.

  • The v1-to-v2 policy copy splits a v1 policy's model assumptions into a dynamic, rewriting the UK's old spelling first, and keeps the rest in the policy, which is empty when nothing else remains. Its mapping records both. Copies made before this keep their mapping and have no dynamic until the one-time migration splits them. The copy's log record gains destination_dynamic_id and dynamic_created, and the failure reason stored_dynamic_integrity.

The PR adds no inline suppression (noqa or type: ignore).

Deploy order

  • The simulation API: its deployed code declares dynamics with id alone and selects only dynamics.id, so dropping name and description breaks none of its reads. Its insert_dynamic test helper inserts (id, name), which fails against tables built from this revision. In the order these tasks merge, this PR merges before Store one cliff result per simulation run in the v2 database, with grants #4327, so the first simulation API PR whose pin contains this one is C1 · 38 (Calculate each simulation run's cliff gap and share as its own cliff result policyengine-sim-api#1096), which moves the pin to Store one cliff result per simulation run in the v2 database, with grants #4327's merge; it must update the helper, and C1 · 42 and C1 · 44 inherit the change. The simulation API's weekly schema comparison against this repository's master fails from this PR's merge until then.

  • Existing rows: nothing wrote dynamics before this, and the new columns are required without a default, so the upgrade fails, changing nothing, while dynamics has a row. Check each environment first with SELECT count(*) FROM dynamics;. The downgrade adds name back as required, so it fails while dynamics has a row.

  • The index build: it reads all of parameter_values and holds back inserts into it until it finishes.

  • The previous API v1 release never reads or writes dynamics, so it keeps working at this revision.

  • Variables: the migration and grant steps (seed-v2-database.yml, from push.yml with secrets: inherit) read only:

    • V2_MIGRATION_DATABASE_URL and V2_DATA_WRITE_DATABASE_URL, GitHub environment secrets in staging and production;
    • V2_SUPABASE_PROJECT_REF and V2_SUPABASE_ENVIRONMENT, environment variables in both;
    • PRODUCTION_V2_SUPABASE_PROJECT_REF, a repository variable;
    • DEPLOYMENT_ENVIRONMENT, the workflow input.

    All exist; this PR adds none.

Other open pull requests

#4029 already conflicts with master. This PR adds text conflicts with it only in tests/contract/registry.py and tests/contract/test_app_v2_workflow_contracts.py, where both add workflow entries. It doesn't change the v1 policy routes that #4029 changes.

Testing

  • Tests run, by node ID only (new and changed tests): the unit tests, and Postgres 17 integration tests run as the runtime role:
    • test_alembic_v2_dynamics.py: the revision changes nothing else, upgrades and downgrades exactly, refuses both ways with a stored dynamic, and keeps its keys and unique rules;
    • test_v2_dynamic_persistence.py: dynamic dedupe and reads, and the copy's split and UK rewrite;
    • one creation test of each submission route.
  • Checks: ruff, ruff format, mypy (with the changed data/v2 files), lint-imports, python -m scripts.rules validate, scripts/run_quality_guards.py, alembic -c alembic-v2.ini check, and reconcile_v2_privileges.py apply --create-missing-roles then verify on a disposable database.
  • Generated docs: scripts/export_migration_contracts.py regenerated the migration contract docs, with the new dynamic_resources_v2 workflow.

🤖 Generated with Claude Code

anth-volk and others added 7 commits November 17, 2026 15:27
…ing)

Dynamics get policies' country, catalog version and content-hash columns,
lose name and description, and a dynamic's values are unique per parameter
and start date. legacy_policy_mappings names an optional dynamic of the same
country. A new pure module says which parameters a dynamic holds and splits a
v1 policy's values, rewriting the UK's old labour-supply spelling.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Revision 9d1d12a7ac58 (on f519a77ba122, autogenerated with no corrections
apart from ruff format) gives dynamics policies' country, tax-benefit model,
catalog version, canonicalization version and content hash, unique per
version and hash, and drops name and description, which nothing wrote or
read. parameter_values gains a partial unique index on a dynamic's values per
parameter and start date, and legacy_policy_mappings a nullable dynamic_id,
a RESTRICT key to a dynamic of the mapping's country. The runtime role may
read and insert dynamics.

policyengine_api.dynamic_parameters says which parameters a dynamic holds
(every gov.simulation. parameter, plus the US simulation. tree), rewrites
the UK's old labor_supply_responses spelling, and splits a v1 policy's values
into the policy's and the dynamic's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…plit copy (failing)

POST /v2/dynamics finds or creates a dynamic of gov.simulation parameters
(201 or 200), refuses an empty one and one with other parameters (422), and
GET /v2/dynamics/{id} reads one in its country. POST /v2/policies refuses
gov.simulation and US simulation parameters with 422, naming them.
/v2/simulations and /v2/reports refuse a dynamic not stored in the country
(404) or setting parameters the running catalog lacks (422). The v1-to-v2
copy splits a v1 policy's model assumptions into a dynamic, rewrites the UK's
old spelling, maps the v1 policy to both, and logs the dynamic.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…lit the copy

POST /v2/dynamics binds model assumptions (parameters under gov.simulation,
and in the US simulation) to a catalog version and stores them once per
content hash with the policies' canonical document, answering 201 or 200;
GET /v2/dynamics/{dynamic_id} reads one in its country with its
missing_parameters. A dynamic setting any other parameter answers 422 with
code policy_parameters, and an empty one fails request validation. The
dynamic service reports database failures as DynamicStorageError (503).

POST /v2/policies answers 422 with code dynamic_parameters for a policy that
sets model assumptions. Economy simulations, economy reports and budget
windows refuse a dynamic the country does not store (404) or one setting
parameters the running catalog lacks (422 missing_parameters).

The v1-to-v2 policy copy splits a v1 policy's model assumptions into a
dynamic, rewriting the UK's old labor_supply_responses spelling first, and
records the dynamic in legacy_policy_mappings.dynamic_id; its log record names
the dynamic, and classifies the dynamic's integrity failures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	policyengine_api/services/v2/simulation_service/services.py
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Dynamic gains Policy's relationships to its tax-benefit model and catalog
version, so the ORM stores a new catalog version before a dynamic in one
flush, and the unused relationships between dynamics and legacy policy
mappings are gone, so mapper configuration emits no SAWarning again. Tests
cover both, the copy losing the mapping race after creating a dynamic, and a
dynamic that cannot be written. The mapping input's country is a CountryId.
Docstrings and descriptions name the dynamic errors, the re-run description
says that the simulation service refuses a dynamic's missing parameters, and
the migration skill says that building the new index reads all of
parameter_values and holds back inserts until it finishes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.54869% with 44 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.60%. Comparing base (16f410f) to head (0229650).
⚠️ Report is 12 commits behind head on master.

Files with missing lines Patch % Lines
policyengine_api/services/v2/dynamics/services.py 69.23% 21 Missing and 3 partials ⚠️
...engine_api/services/v2/dynamics/transformations.py 72.22% 1 Missing and 4 partials ⚠️
policyengine_api/fastapi_routes/dependencies.py 33.33% 4 Missing ⚠️
.../services/v2/dynamics/database_connectors/reads.py 71.42% 3 Missing and 1 partial ⚠️
...ervices/v2/dynamics/database_connectors/creates.py 84.61% 2 Missing ⚠️
policyengine_api/dynamic_parameters.py 97.14% 0 Missing and 1 partial ⚠️
...ngine_api/services/v2/dynamics/database_session.py 92.85% 0 Missing and 1 partial ⚠️
...olicyengine_api/services/v2/dynamics/validators.py 96.29% 0 Missing and 1 partial ⚠️
...olicyengine_api/services/v2/policies/validators.py 90.90% 0 Missing and 1 partial ⚠️
...ine_api/services/v2/simulation_service/services.py 90.00% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4325      +/-   ##
==========================================
- Coverage   93.69%   93.60%   -0.10%     
==========================================
  Files         397      408      +11     
  Lines       22885    23282     +397     
  Branches     3651     3648       -3     
==========================================
+ Hits        21443    21794     +351     
- Misses        839      874      +35     
- Partials      603      614      +11     
Flag Coverage Δ
integration 57.80% <64.37%> (+0.10%) ⬆️
unit 90.93% <89.54%> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

anth-volk and others added 3 commits November 19, 2026 02:53
The new routes, table constraint and translation result changed what five
existing test files and ruff's list of runtime-evaluated model bases expect.
Text this pull request added now says API v1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review November 19, 2026 20:56
@anth-volk
anth-volk merged commit 003ee49 into master Nov 19, 2026
25 checks passed
@anth-volk
anth-volk deleted the v2plan/c1-36-dynamics branch November 19, 2026 20:56
anth-volk added a commit that referenced this pull request Nov 19, 2026
The dynamics revision 9d1d12a7ac58 (#4325) reached master first on the same
parent, so the cliff impacts revision bc334f9877b7 now revises it, and stays
the v2 head. Generating it again from 9d1d12a7ac58 gives the same operations.
The revision tests list both in chain order, the cliff impacts integration
test downgrades to the dynamics revision, and the migrations skill and the
stage 9 and stage 12a downgrade notes name the dynamics revision's step.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Nov 19, 2026
Brings in e87b0ea, which merges master at 003ee49 (C1 · 36's
#4325: stored dynamics and revision 9d1d12a7ac58). Revision
bc334f9877b7 now revises 9d1d12a7ac58 and stays the only v2 head.
master's changes to files this branch also changes are route and
protocol wording and the dynamics protocols; none touches cliffs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk added a commit that referenced this pull request Nov 19, 2026
The V2 Postgres migration lifecycle job is to run test_alembic_v2_dynamics.py,
and the cross-database integration job test_v2_dynamic_persistence.py, after
the managed roles get their privileges, since it runs as the runtime role.
Both came with #4325, and no CI job runs either yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Store dynamics like policies, create them through /v2/dynamics, and keep gov.simulation parameters out of policies

1 participant