[anth-volk]

Fixes #4045

Summary

/v2/parameters/children for the parameter tree of the app's newer workspace (v3). It is stacked on #4042, and v3 does not use it until it moves to the v2 routes.

• Tree table. The new parameter_tree_children table holds one row per node and parameter of a catalog version, under its parent's path ("" for the top level). Each row has:

  • its label;
  • its position among all its siblings;
  • its position among the siblings a reform can change (null if a reform cannot change it);
  • for nodes, its direct child counts in both views;
  • for parameters, the parameter's ID and its economy and household flags.

  Two unique indexes, (version, parent path, position) and (version, parent path, reform position), serve one level in one read.

• Revision 3869eb814a90 is chained on e86f19a6c4da, master's revision that drops the UK area impact tables' map coordinates (Drop the x and y columns from API v2's UK area impact tables #4103), which itself follows Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023's f5c1d9b4685b. It is autogenerated with no corrections and creates the table, its two listing indexes and an index on parameter_id. It writes no rows. Rows are deleted with their version or their parameter.

• Writing at publication.

  • Catalog publication writes, in its transaction and under its lock, the tree of each version in the catalog it publishes that has none. That covers the versions it publishes now, and the installed version when an earlier release stored it with catalog format 2 but without a tree.
  • An installed version stored before catalog format 2 gets no tree: its parameter rows store no economy or household flags, so a tree built with the catalog's flags would not match them. It returns the 503.
  • The rows are built from the installed packages' catalog records. Only the stored parameters' IDs are read, by name, so the rows reference them, after publication has checked that the stored rows match the catalog.
  • It is a row-only, idempotent insert of package-derived rows, which the Alembic skill's deployment-seeder passage allows.
  • Publication fails if a version in the catalog with catalog format 2 is still without a tree. The evidence adds parameter_trees_written (model, version, rows).

• Tree rules.

  • A [ starts a child as a . does, so a bracket schedule is one node whose children are its brackets, labelled "Bracket 1", "Bracket 2" and so on, in bracket order. A bracket's own children (rate, threshold) are unchanged.
  • Labels take the form v3 shows: the stored label with underscores as spaces and its first letter upper-cased. A path without a stored label, or a parameter whose stored label is one .py generated, gets the last part of its name in that form (see "Labels" below).
  • A reform can change a parameter under gov. whose economy or household flag is set and whose path names neither taxsim, gov.abolitions nor pycache, and every node above such a parameter. These are v3's rules (app/src/libs/buildParameterTree.ts).

• The route reads one level from the table, with the leaf summary the parameter routes return (including economy and household).

  • reform_changeable=true lists only what a reform can change, sorted by label, with child counts that count only those.
  • Responses get the same caching as the other metadata routes: an ETag, Cache-Control (a day for a named version, five minutes for the running one), Vary and a 304. A version's tree never changes. The 503 is not cached.

Route contract changes

/v2/parameters/children has no callers in this repository outside tests, and none in the app's main.

• Schedules: a bracket schedule is one node, instead of its brackets appearing beside its parent's groups under their raw names.
• Labels:
  • every child has its stored label in v3's display form;
  • children without a stored label, and parameters whose stored label .py generated, get the last part of their name ("Property" instead of the raw property; "Rate" instead of "Income Tax main rates (bracket 1 rate)");
  • brackets are "Bracket N".
• New query parameter: reform_changeable. The route's query goes through the shared strict models, so an unknown or repeated parameter, an unsupported country and a malformed policyengine_version return 422. An unsupported country and a malformed version returned 400 before Serve v2 metadata in parts loaded on demand, cacheable per catalog version #4042.
• 503 for a version without a tree. A catalog version without a tree returns 503 with a message that names it. The route's OpenAPI 503 says so.

Measured

Locally, against the installed catalog with format 2 fields (the .py branch containing PolicyEngine/policyengine.py#701, 6.10.0), service level, 20 requests each:

Measure	Value
Tree rows	US 131,961; UK 2,425
One level (for example gov, gov.irs, a 42-bracket schedule, the reform view of gov)	3–7 ms median
EXPLAIN ANALYZE of a reform-view level	index scan on ix_parameter_tree_children_reform_listing, 0.07 ms
Writing the US tree	12.8 s on a loaded machine; republication with the tree write 32.1 s, without it 27.6 s
Peak traced memory writing the US tree	97.6 MiB (the production-scale publication test allows 256 MiB)

Staging times need a deploy; #4042's release check records route times on each candidate.

Declared differences from v3's current tree

• A node with nothing a reform can change below it is left out of the reform view. v3 shows such nodes as empty groups, because v1's node entries carry their own flags.

• Brackets are ordered by number. v3 sorts them as text, so "Bracket 10" comes before "Bracket 2".

• Sorting by label approximates JavaScript's localeCompare with a case-folded comparison.

• Labels: generated labels are shortened in the tree only (decided). The catalog stores .py's labels, and .py generates labels for parameters without a stored one (utils/parameter_labels.py, since December 2025):

  • bracket parameters become " (bracket N field)" or "(tier N field)";
  • breakdown parameters become " ()".

  v1's /{country}/metadata instead uses the last part of the name with underscores as spaces, and "bracket N" for brackets, which is what v3 shows today. Of the three options (accept .py's labels; shorten generated labels in the tree only; have .py also publish the raw stored label, which would change Publish the model metadata the Public API can only read through a private method policyengine.py#701 and Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023), the second was chosen. 689c3bc8c implements it:

  • The tree shows a label .py generated by the last part of the parameter's name, as v3 does.
    • A bracket label counts as generated when it is exactly the schedule's stored label with the bracket number and field.
    • A breakdown label counts when it is an ancestor's stored label followed by one value per level below that ancestor: the key itself, a label the version's enum variables give the key, or a dimension label followed by the key. Each value is matched whole, so an enum label containing ", " is still one value.
  • Every other stored label is kept, in v3's display form. Other catalog reads, such as the parameter routes, keep .py's labels.
  • Over the policyengine-us 2.2.1 and policyengine-uk 2.132.6 trees, this recognises every label .py generated (58,464 US, 533 UK). The only labels the packages write themselves that it also shortens are the 10 UK LHA labels such as "Maximum LHA (category A)", shown as "A", because their form is the one .py gives US labels such as "Fee level A".

• Schedule labels. Since Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023 the Public API depends on .py 6.12.0, which contains Publish the model metadata the Public API can only read through a private method policyengine.py#701 and publishes each bracket schedule as a node with its stored label. Schedules therefore show their stored labels, or a label from their name when the package gives them none. Before 6.12.0 a schedule had no node row and always got a label from its name ("Uk", "Al").

Trees for other versions stored earlier: left out

• What a backfill would do. On each publication, find every stored catalog version other than the installed one that has parameters or nodes but no tree rows. For each, read its stored nodes and parameters, build the rows, and insert them with COPY, insert-only and idempotent, inside publication's transaction and lock.
• Why it is left out. Those rows would be derived from data that earlier releases stored. The Alembic skill's "V2 application-data migrations" section reserves that for Alembic and a human migration decision. The deployment-seeder passage covers only package-derived rows, which the installed version's tree is.
• Consequence.
  • The installed catalog version gets its tree at the first deploy that includes this PR, including the version published at Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023's .py bump, as long as it is stored with catalog format 2.
  • Every other stored version has no tree, and /v2/parameters/children returns 503 for it. So does an installed version stored before catalog format 2, for example if a .py release carrying the flags were published from master before this stack deploys.
• Decision. The backfill stays out of this PR. Trees for earlier stored versions would need their own data migration and its own approval.

Migration notes

• Route group and runtime path. The metadata route group; the route is served by FastAPI and reads Supabase (no change).
• Deploy order.
  • The seed job runs alembic upgrade head, then catalog publication, which writes the trees, before the new release takes traffic.
  • The previous release does not read the new table.
  • The revision's downgrade drops the table and keeps every catalog row.
• Privileges. The runtime role may read the table. The catalog writer may read and insert rows, and a test publishes as the catalog writer role with only its declared privileges.
• Environment variables. None are read or added, and no workflow changed.
• Suppressions and rules. No # noqa is added.
• Rule scope and statement change. rules/registry.toml adds tests/integration/test_v2_metadata_routes.py and tests/integration/test_v2_catalog_publication.py to the scope of api.v2.runtime-privileges-tested-as-runtime-role, because they now run a service as a managed role: the route as the runtime role, publication as the catalog writer. The rule's statement and the Alembic skill's privileges section now say the tests run the v2 user services and the parameter tree read as the runtime role and catalog publication as the catalog writer, and the skill's command runs the two modules. A review from PolicyEngine/core-developers is requested.

Waits for

• CI on this branch, which runs in full now that Serve v2 metadata in parts loaded on demand, cacheable per catalog version #4042 and Store the metadata v3 reads in the v2 catalog from PolicyEngine.py's public records (catalog format 2) #4023 below it are merged (stack #4155 on master). The branch includes a merge of master (below).
• Review.

The decisions on the labels and on trees for other stored versions are made (above).

Testing

The evidence below is local; each commit failed first for the expected reason and then passed, unless it only adds or moves tests. The last commit's checks were run with the whole branch ("At the head" below).

Commit	Failed first	Then passed
b7b9f393f Derive each catalog version's parameter tree	ModuleNotFoundError	11 tree tests
fb6c23230 Store each catalog version's parameter tree in its own table	KeyError: 'parameter_tree_children', the revision chain, no child revision, NoSuchTableError	148 model, revision, privilege and migration tests; alembic check clean; privilege apply and verify
b46c44aa4 Write each stored catalog version's parameter tree during publication	no tree rows; evidence field missing	57 publication tests
376bb216f Serve /v2/parameters/children from the stored parameter tree	8 route and 6 service tests; the Postgres schedule test against the old route	56 unit tests and 11 Postgres route tests, one as the runtime role
98b9c0b3d Return parameter tree labels in the form v3 shows them	stored labels returned unchanged	103 tests
bfed72372 Write the parameter tree only for the catalog versions a publication publishes	the older version was still written; the 503 message	98 tests
63e9d1936 Publish the parameter tree as the catalog writer role in a test	InsufficientPrivilege with the INSERT grant revoked	passes with the declared grant
093b4e539 Document the children route's statuses and page the reform view in a test	no route description in OpenAPI	57 tests
c6ddfa5fd Test the parameter tree with records shaped like PolicyEngine.py's output	tests only	15 unit tests and 1 Postgres test
ede52b1c3 Type a parameter tree row's child type with its two values	annotation was str	93 tests; alembic check clean
1e31c9ae5 Drop the reform rule for parameters without flags	removes code that could not run; one service test exposed unflagged fixtures, now fixed	39 unit and 68 Postgres tests
0589038d4 Declare exactly the errors the parameter tree route returns	the 503 description named no tree case	86 route tests
d4515053d Give the installed catalog version its parameter tree when it has none	the installed version stored without a tree got none	54 publication and route tests; the rebuilt tree equals a fresh publication's, parameter IDs included
1826bcfbf Test that parameter tree responses are cached like other metadata	tests only	88 route tests
55c3f6c5c Write no parameter tree for a version stored before catalog format 2	a stored format 1 installed version got a tree	49 Postgres publication and route tests
6277907a4 Test the parameter tree's link to stored parameters by name	tests only (the missing-name branch)	9 publication unit tests, 28 Postgres publication tests
1d3464c81 State the parameter tree's real rule in its documents and messages	documents and messages only	150 unit tests
4af6107b4 Name the catalog writer in the runtime-role rule and the Alembic skill	documents only	rules validate, render and guard; 22 workflow and skill tests
c4187adeb Look up a stored catalog version's ID and format in one place	refactor, no behavior change	58 publication and route tests
7e97c0a1f Add the catalog format 2 condition to the tree writer's descriptions	documents only	ruff and format
ca19fb6fd Test the tree's parameter IDs with stored IDs unlike the catalog's	tests only; both fail if the tree uses the catalog's IDs (a mismatch, and a foreign-key error)	2 tests
2475b29b9 Check that a version stored before catalog format 2 returns the tree 503	tests only; fails without the format 2 condition	1 Postgres route test
54ab346e1 Say which metadata read the runtime-role rule's tests cover	documents only	rules validate, render and guard; 22 workflow and skill tests
d4191a253 Split the v2 metadata route and service tests under the file-length limit	tests moved only	see "At the head"
689c3bc8c Show PolicyEngine.py's generated parameter labels in v3's short form	not recorded separately	see "At the head"

The commits that add a role-based test (376bb216f, 63e9d1936) also add its module to the rule's scope, so test_every_managed_role_integration_module_runs_in_ci_and_is_in_its_rule passes at each commit.

On 54ab346e1, before the lock moved to .py 6.12.0:

• make test set (without coverage), under the locked .py 6.10.0: 3,657 passed.
• Postgres tests on a schema recreated at head, grouped as the v2 integration workflow runs them after privilege apply and verify: publication and routes 50 passed; user identity and saved-policy view 20 passed; comparison runs and privilege reconciliation 9 passed; v2 migration lifecycle 10 passed.
• alembic check: no new operations.
• ruff, format, mypy, quality guards and rules checks: clean.
• Contract export: no change.
• No commit adds a .coverage file or names a private repository.

At the head, under the locked .py 6.12.0, checked before GitHub's rebases onto master (as 52c2989ea; git range-diff shows all 25 commits unchanged since):

• ruff, mypy (no issues in 187 source files), quality guards: clean.
• Contract export: no change.
• The 128 tests this branch adds or changes (117 unit): 165 cases passed.
• Its changed Postgres tests on a schema at head 3869eb814a90: 11 passed.

Merge of master

• a226f4966 merges master at a69c6e2cd. master had added revision e86f19a6c4da on f5c1d9b4685b, the same parent as 3869eb814a90, so the merge re-points 3869eb814a90 onto it, keeping one v2 head:
  • migration_target.py names 3869eb814a90 as the head;
  • the revision chain test lists e86f19a6c4da between the two, and the tree revision's test finds the tree revision as the child of e86f19a6c4da;
  • the coordinate removal's unit test no longer asserts that its revision is the head, and its lifecycle test compares that revision, not head, with its parent;
  • the tree table's lifecycle test downgrades to e86f19a6c4da.
• 3e09a92d7 regenerates 3869eb814a90 with --rev-id 3869eb814a90 on a database at e86f19a6c4da, as master regenerated e86f19a6c4da on each revision that merged before it. The generated operations are the same as before; only the creation date changed.
• ad826d366 splits test_alembic_v2_lifecycle.py, which the merge took to 1,569 lines (over the 1,500-line limit for test code). The catalog format 2 column and parameter tree lifecycle tests move unchanged to test_alembic_v2_catalog_tables.py, and three shared helpers move to tests/fixtures/integration/alembic_v2.py without their leading underscore. The v2 Alembic check's script, its workflow test and the testing guide name the new module. Serve a compact parameter search file for each catalog version at /v2/parameters/search-file #4049 makes the same split with the same names.
• Evidence, locally on the head: the five changed revision and lifecycle tests, the workflow test, and the 11 lifecycle tests the split moved or whose helper calls it renamed passed; alembic heads shows 3869eb814a90 alone; alembic check finds no new operations; ruff, mypy (187 files), the quality guards and uv lock --check pass. One review round ran on the merge and the split (below).

Review

Two review rounds were run on this branch before it was rebased onto #4042.

• Round 1 found one bug. Stored labels were returned without v3's display rule while generated labels had it. This is fixed in 98b9c0b3d. The same round's quality findings led to bfed72372 (no trees from earlier releases' stored rows), 63e9d1936, and 093b4e539.
• Round 2 reported the label difference against v1 described above. It was a product decision, now made and implemented in 689c3bc8c. Its quality findings are fixed in c6ddfa5fd, ede52b1c3, 1e31c9ae5 and 0589038d4.

After the rebase, d4515053d gives the installed version its tree, and 1826bcfbf adds the caching tests.

• Round 3, on those new commits, found one major bug. A version stored before catalog format 2 would get a tree carrying the catalog's flags, which its own parameter rows do not store. This is fixed in 55c3f6c5c: no tree for such a version, and the check that every version has a tree leaves it out. The same round's quality findings are fixed in 6277907a4, 1d3464c81 and 4af6107b4.

• Final pass, after the rebase onto Serve v2 metadata in parts loaded on demand, cacheable per catalog version #4042's final commits (3885872ae), found no substantiated bugs. Its quality findings are fixed in:

  • c4187adeb: one stored-version lookup shared with the stored-version comparison;
  • 7e97c0a1f: the format 2 condition in the remaining descriptions;
  • ca19fb6fd: tests that tell the stored parameter IDs from the catalog's;
  • 2475b29b9: a test that the format 1 case returns the 503;
  • 54ab346e1: the rule statement narrowed to what CI checks.

• Merge of master: one review round on a226f4966 and ad826d366 found no substantiated bugs. Of its quality findings, the hand-edited parent is replaced by the regenerated revision (3e09a92d7). The new module's tests start by downgrading to a named revision, so they need the lifecycle module to run first, as the v2 Alembic check's script and the testing guide run them (Serve a compact parameter search file for each catalog version at /v2/parameters/search-file #4049's split has the same order). The moved URL helper repeats the check of the disposable_v2_database_url fixture in tests/integration/conftest.py; merging the two is left for a follow-up.

🤖 Generated with Claude Code

[vahid-ahmadi]

Review at 4519b8e96e.

Verdict: the tree builder and the route look correct. One design gap to settle before the label decision lands. Stacked on #4042, so I reviewed only v2plan/1f-116-catalog-routes-on-demand...v2plan/1f-117-parameter-children. No must items.

1. should policyengine_api/data/v2/catalog/publication_parameter_tree.py:171: a version that already has a tree is skipped (_has_tree), and nothing records which tree rules built it. If the open label decision ends in option 2, or the reform rule, bracket ordering or label form changes later, every version published before that keeps its old tree and nothing fails. Serve a compact parameter search file for each catalog version at /v2/parameters/search-file #4049 avoids this for the search file with a file_format column and a hash check that fails publication when the stored file differs. A tree_format column (or the same compare-and-fail check on the installed version) would let a rule change either add new rows or stop the deploy, instead of serving two sets of rules side by side.
2. nit parameter_tree.py: _position_key orders the non-bracket siblings by raw path, while the reform view orders them by label. That matches the description, but it means /v2/parameters/children without reform_changeable lists children in a different order from v3's tree. A sentence under "Declared differences" would save a later bug report.
3. nit At about 132,000 US rows per catalog version, the table grows by that much with every published US release, and nothing prunes old versions. Fine for now; worth a line in the Stage 9 doc saying who cleans up.

Checked and fine: parent_path/bracket_index handle a.b[0].rate → a.b[0] → a.b; ancestors of reform-changeable parameters are marked changeable; the two unique listing indexes match the two ORDER BY columns; the 503 is not cached; the route is read-only, takes bound parameters, and caps parent_path at 512 characters.

What I ran: fresh clone, uv sync --frozen --extra dev, then pytest on the changed unit and contract test files: 238 passed. CI is skipped on stacked PRs here, so the integration, publication-scale and Alembic lifecycle tests have not run on this head. I did not run them (they need disposable databases and the full catalog), and I did not check the label counts in the description.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 90.00000% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.21%. Comparing base (f29abfa) to head (3e09a92).
⚠️ Report is 37 commits behind head on master.

Files with missing lines	Patch %	Lines
..._api/data/v2/catalog/publication_parameter_tree.py	77.77%	6 Missing and 8 partials ⚠️
policyengine_api/data/v2/catalog/parameter_tree.py	93.33%	3 Missing and 8 partials ⚠️
..._api/data/v2/catalog/publication_reconciliation.py	76.92%	2 Missing and 1 partial ⚠️
...tadata/database_connectors/reads_parameter_tree.py	90.90%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4046      +/-   ##
==========================================
- Coverage   88.22%   88.21%   -0.01%     
==========================================
  Files         285      288       +3     
  Lines       16843    17129     +286     
  Branches     2741     2814      +73     
==========================================
+ Hits        14859    15111     +252     
- Misses       1302     1315      +13     
- Partials      682      703      +21     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.