Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
f317711
Update PolicyEngine bundle to 6.20.0
github-actions[bot] Nov 16, 2026
fa5b1c7
Test revoking whole-table privileges declared for named columns (fail…
anth-volk Nov 16, 2026
f965ed1
Revoke whole-table privileges the manifest declares for named columns
anth-volk Nov 16, 2026
758050b
Fix review round 1's findings on the column-grant revoke
anth-volk Nov 16, 2026
46a3005
Fix review round 2's test docstring wording
anth-volk Nov 16, 2026
cbf3e42
Test recording why a v2 metadata read answered 503 or 500 (failing)
anth-volk Nov 16, 2026
19e0055
Read the metadata failure event from a test runtime (still failing)
anth-volk Nov 16, 2026
9826036
Record why a v2 metadata read answered the generic 503 or a 500
anth-volk Nov 16, 2026
3b769b6
Merge the v2 metadata read failure event into the column-grant fix
anth-volk Nov 16, 2026
ec6cee4
Test refreshing the catalog tables' statistics after publication (fai…
anth-volk Nov 16, 2026
fb8d8dd
Refresh the catalog tables' statistics after a publication
anth-volk Nov 16, 2026
0171b14
Test the metadata failure event's grouping fields (failing)
anth-volk Nov 16, 2026
a10175e
Group the metadata failure event as the error-issue job expects
anth-volk Nov 16, 2026
e20e052
Treat only a missing caller message as the generic 503
anth-volk Nov 16, 2026
a70d2ac
Test analyzing on every publication and recording a failure (failing)
anth-volk Nov 16, 2026
6a772aa
Analyze the catalog on every publication and record a failed ANALYZE
anth-volk Nov 16, 2026
50406c7
Test reporting a failed ANALYZE by its driver error and on stderr (fa…
anth-volk Nov 16, 2026
21ce1d1
Report a failed ANALYZE by its driver error, and on stderr
anth-volk Nov 16, 2026
3093dcc
Merge refreshing the catalog tables' statistics into the column-grant…
anth-volk Nov 16, 2026
e3f7cb2
Cover the two remaining branches of the failure reporting
anth-volk Nov 16, 2026
473b56c
Merge the PolicyEngine bundle update to 6.20.0 into the column-grant fix
anth-volk Nov 17, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/v2-integration-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ jobs:
- name: Verify the managed v2 roles hold their declared privileges
run: uv run --frozen --extra dev python scripts/reconcile_v2_privileges.py verify
- name: Test v2 metadata publication, resource routes, the countries list, reports and simulations
run: uv run --frozen --extra dev coverage run --branch -m pytest -q tests/integration/test_v2_catalog_publication.py tests/integration/test_v2_catalog_publication_time_periods.py tests/integration/test_v2_catalog_publication_pruning.py tests/integration/test_v2_catalog_spm_availability.py tests/integration/test_v2_metadata_routes.py tests/integration/test_v2_country_routes.py tests/integration/test_v2_report_reads.py tests/integration/test_v2_report_creation.py tests/integration/test_v2_budget_window_report_reads.py tests/integration/test_v2_budget_window_report_creation.py tests/integration/test_v2_report_runs.py
run: uv run --frozen --extra dev coverage run --branch -m pytest -q tests/integration/test_v2_catalog_publication.py tests/integration/test_v2_catalog_publication_time_periods.py tests/integration/test_v2_catalog_publication_pruning.py tests/integration/test_v2_catalog_spm_availability.py tests/integration/test_v2_catalog_statistics.py tests/integration/test_v2_metadata_routes.py tests/integration/test_v2_country_routes.py tests/integration/test_v2_report_reads.py tests/integration/test_v2_report_creation.py tests/integration/test_v2_budget_window_report_reads.py tests/integration/test_v2_budget_window_report_creation.py tests/integration/test_v2_report_runs.py
- name: Test v2 policy persistence and immediate v1 mirroring
run: uv run --frozen --extra dev coverage run -a --branch -m pytest -q tests/integration/test_v2_policy_persistence.py tests/integration/test_v1_policy_dual_write.py tests/integration/test_mysql_policy_dual_write.py tests/integration/test_v2_user_policy_mirroring.py tests/integration/test_v1_user_policy_dual_write.py
- name: Test signed-in identity provisioning, user associations, saved detailed-save records and user types
Expand Down
1 change: 1 addition & 0 deletions changelog.d/update-policyengine-bundle-6.20.0.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Update the PolicyEngine bundle to 6.20.0.
1 change: 1 addition & 0 deletions changelog.d/v2-catalog-analyze.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Catalog publication now refreshes PostgreSQL's statistics of the catalog tables after every publication, so a new version's rows no longer leave `GET /v2/parameter-values` planned as if the version had one parameter (43 to 47 seconds on staging, 4.1 ms after `ANALYZE`); a failed refresh is reported in the publication's summary and repeated by the next one. The catalog writer role gains `MAINTAIN` on those tables.
1 change: 1 addition & 0 deletions changelog.d/v2-column-update-grants.fixed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The v2 privilege reconciliation now revokes a privilege a role holds on a whole table when the manifest declares it for named columns only, such as the `UPDATE` new tables' default privileges give the runtime role on `user_type_changes`, and grants those columns again, so the deploy's privilege verification passes.
1 change: 1 addition & 0 deletions changelog.d/v2-metadata-read-failure-event.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
A v2 metadata read that answers the generic 503 or the 500 now records one runtime event, `v2_metadata_read_failed`, naming the route, the error's class and failure category, the error classes in its cause chain, the database's SQLSTATE and whether the connection was lost, without any error message.
2 changes: 1 addition & 1 deletion docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM python:3.12
# Match the API bundle in pyproject.toml; the bundle updater changes both pins.
# The models extra pins the complete certified runtime package set.
RUN pip install "policyengine[models]==6.17.1" ipython
RUN pip install "policyengine[models]==6.20.0" ipython
27 changes: 18 additions & 9 deletions docs/engineering/skills/alembic-migrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ and a revision runs once per database, so it can't repair drift.
| Role | Used by |
| --- | --- |
| `policyengine_v2_runtime` | The Public API on Cloud Run |
| `policyengine_v2_catalog_writer` | `scripts/publish_v2_metadata_catalog.py` |
| `policyengine_v2_catalog_writer` | `scripts/publish_v2_metadata_catalog.py`, which also runs `ANALYZE` on seven catalog tables after every publication, with `MAINTAIN` (PostgreSQL 17) |
| `policyengine_stage12_<environment>` | The simulation API and its Modal worker, on the temporary comparison-run tables |
| `policyengine_v2_simulation_service` | The simulation API on Cloud Run: creates simulations, reports, their runs, budget-window runs and household jobs, and reads the policy, dynamic, household and catalog rows a run needs. It updates only the status, Modal call ID, error and completion columns of the runs and household jobs whose Modal calls it starts, and the status, error and completion columns of a budget-window run, which it fails at once when Modal refuses the calls of the window's first years |
| `policyengine_v2_modal_worker` | The simulation API's Modal workers: write their runs' and household jobs' status and results, and budget-window runs' status, and read the rows a run needs. It updates only a run's status, Modal call ID, error, time and result columns, and a budget-window run's status, error and time columns: each year's coordinator starts the window's next unstarted year when its own year finishes, the last year to finish marks the window succeeded, and a failed year fails it. The workers' code, not the database, keeps a worker to its own run |
Expand Down Expand Up @@ -283,14 +283,23 @@ check the declarations:
then `verify`, before the catalog is seeded.

`apply` grants each declared privilege a role lacks, as one `GRANT` per role
and table and one per column privilege. It never revokes: it reports
privileges the file doesn't declare, on whole tables or on columns, and the
projects' default privileges, in the job summary. It grants nothing when a
declared role or table is missing. `verify` checks each column of a column
privilege with `has_column_privilege`, and fails when a role holds the same
privilege on the whole table. The roles themselves are provisioned outside
the repository; `--create-missing-roles` is accepted only for the disposable
test database.
and table and one per column privilege. It revokes one kind of privilege
only: one a role holds directly on a whole table that the file declares for
named columns only. The projects' default privileges give the runtime role
`UPDATE` on every new table, so `user_type_changes`, where it may update only
`dismissed_at`, starts with one. PostgreSQL's `REVOKE` on a table also removes
the role's column grants of that privilege, undeclared ones included, so
`apply` grants the declared columns again, in the same transaction; the job
summary lists the revoke with the undeclared column grants it removed. It
reports every other privilege the file doesn't declare, on whole tables or on
columns, and the projects' default privileges, without revoking them. It
changes nothing when a declared role or table is missing. `verify` checks each
column of a column privilege with `has_column_privilege`, and fails when a role
still holds the same privilege on the whole table: through a role it belongs
to, or through a grant another role made, which the owner's `REVOKE` leaves in
place. The roles themselves are provisioned outside the repository;
`--create-missing-roles` is accepted only for the disposable test database,
which has no default privileges.

A pull request may declare a role before an operator creates it by listing it
in `ROLES_AWAITING_PROVISIONING` in `privileges.py`. While a listed role is
Expand Down
21 changes: 21 additions & 0 deletions docs/engineering/skills/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,27 @@ own level is dropped before the runtime's handler sees it. Attribute names
must avoid the runtime's prohibited key parts (such as `authorization`,
`credential` and `token`), which it drops.

## v2 metadata read failures

A v2 metadata read that answers the generic 503 "V2 metadata catalog is
unavailable", or the 500 "V2 metadata query failed", records one runtime
event at ERROR, `v2_metadata_read_failed`
(`policyengine_api.fastapi_routes.v2.metadata.read_failure_support`). The
runtime adds the request's ID, method and matched route. The event adds
`http.response.status_code`, `error_type` (the caught error's class as
`module.QualifiedName`), `failure_category` (`v2_metadata_catalog_unavailable`
or `v2_metadata_query_failed`), `error.cause_types` (the classes of its
`__cause__` chain, outermost first), `database.sqlstate` (the first SQLSTATE in
the chain) and `database.connection_invalidated` (SQLAlchemy's lost-connection
flag). The scheduled error-issue job groups the event by route, `error_type`
and `failure_category`. A pruned catalog version, or a missing parameter tree
or search file, answers with its own message and records nothing. On the two
routes that share one read of a whole catalog version
(`response_cache_support`), only the request that ran the failed read records
it; requests that waited for it answer with a copy of its 503. The event never
carries an error's message: SQLAlchemy's and the driver's name the SQL
statement and its parameters, and can name hosts, addresses and credentials.

## Simulation client transport

The simulation HTTP client reads identifiers from request context. Its request
Expand Down
5 changes: 4 additions & 1 deletion docs/engineering/skills/v2-code-organization.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,6 +72,7 @@ user_policies/
metadata/
response_models.py
route_support.py
read_failure_support.py
query_support.py
response_cache_support.py
search_file_support.py
Expand All @@ -80,7 +81,9 @@ metadata/

`route_support.py` runs the metadata reads for the route modules, maps their
failures to HTTP, and holds the route class that gives metadata responses
their caching headers. `query_support.py` defines the typed FastAPI
their caching headers. `read_failure_support.py` records why a read answered
the generic 503 or a 500, without the error's message. `query_support.py`
defines the typed FastAPI
dependencies for the shared query models once, for every metadata route
module to import. `response_cache_support.py` keeps serialized whole-version
responses (bodies and ETags) for the process; it is HTTP-level, so it lives
Expand Down
2 changes: 1 addition & 1 deletion docs/generated/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -285,7 +285,7 @@ flowchart LR
| `api.migration.bundle-supported-by-simulation-api` | The installed PolicyEngine bundle is the latest one the simulation API has published, checked on every pull request and before each staging and production deploy. | mechanical | enforced | yes | CI job `ensure-policyengine-bundle-supported-by-simulation-api` | `pr.yml:ensure-policyengine-bundle-supported-by-simulation-api` | — |
| `api.migration.contracts-consistent` | Migration route groups and workflow contracts are internally consistent, and their generated docs are current. | mechanical | enforced | yes | quality guard `migration-contracts` | `pr.yml:quality-guards` | **unproven** |
| `api.migration.cross-database-integration` | Cross-database integration tests pass against real MySQL and Postgres. | mechanical | enforced | yes | CI job `v2-integration-check` | `pr.yml:v2-integration-check` | — |
| `api.v2.deployed-privileges-match` | Every v2 deploy grants the managed roles each privilege policyengine_api/data/v2/privileges.py declares, on the whole table or on the columns it names, right after the schema upgrade, reports privileges it doesn't declare without revoking them, and stops before the catalog is seeded if a declared privilege is still missing or a role holds on a whole table a privilege declared for only some of its columns. A declared role listed in ROLES_AWAITING_PROVISIONING that doesn't exist yet is named in the job summary and skipped instead. | mechanical | enforced | yes | CI job `seed`, `catalog-runtime-integration` | `seed-v2-database.yml:seed`, `v2-integration-check.yml:catalog-runtime-integration` | yes |
| `api.v2.deployed-privileges-match` | Every v2 deploy grants the managed roles each privilege policyengine_api/data/v2/privileges.py declares, on the whole table or on the columns it names, right after the schema upgrade, revokes a privilege a role holds on a whole table that is declared for only some of its columns, with the role's column grants of that privilege, and grants the declared columns again, reports every other privilege it doesn't declare without revoking it, and stops before the catalog is seeded if a declared privilege is still missing or a role still holds on a whole table a privilege declared for only some of its columns. A declared role listed in ROLES_AWAITING_PROVISIONING that doesn't exist yet is named in the job summary and skipped instead. | mechanical | enforced | yes | CI job `seed`, `catalog-runtime-integration` | `seed-v2-database.yml:seed`, `v2-integration-check.yml:catalog-runtime-integration` | yes |
| `api.v2.runtime-privileges-declared` | Every v2 table has an entry for each managed database role in policyengine_api/data/v2/privileges.py; a privilege declared on only some of a table's columns names columns of that table and is never also declared on the whole table; and every privilege a v2 connector module needs (from its reads, creates, updates or deletes module name and its row locks) is declared for the role that runs it, on the whole table. | mechanical | enforced | yes | pytest `tests/unit/v2/test_privilege_manifest.py` | `pr.yml:test` | yes |
| `api.v2.runtime-privileges-tested-as-runtime-role` | The cross-database integration tests run the v2 user services and the parameter tree read as policyengine_v2_runtime, catalog publication as policyengine_v2_catalog_writer, the comparison-run services as policyengine_stage12_<environment>, and the simulation API's table writes as policyengine_v2_simulation_service and policyengine_v2_modal_worker, holding only their declared privileges, so a statement those privileges don't allow fails CI. | mechanical | enforced | yes | CI job `v2-integration-check`, `catalog-runtime-integration` | `pr.yml:v2-integration-check`, `v2-integration-check.yml:catalog-runtime-integration` | — |

Expand Down
38 changes: 37 additions & 1 deletion docs/migration/stage-9-v2-metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -192,7 +192,7 @@ Use three separately managed database credentials:
| Operation | Configuration | Required database access |
| --- | --- | --- |
| Alembic schema upgrade | `V2_MIGRATION_DATABASE_URL` | Reviewed v2 schema changes; no application-runtime use |
| Catalog publication | `V2_DATA_WRITE_DATABASE_URL` | Row insertion, update, selection, deletion from `parameter_values`, `parameter_tree_children` and `parameter_search_files` (catalog pruning), temporary tables, and transaction-scoped advisory locks; no persistent schema changes |
| Catalog publication | `V2_DATA_WRITE_DATABASE_URL` | Row insertion, update, selection, deletion from `parameter_values`, `parameter_tree_children` and `parameter_search_files` (catalog pruning), `ANALYZE` of the catalog tables (`MAINTAIN`, which also allows `VACUUM`, `CLUSTER`, `REINDEX` and `LOCK TABLE` on them), temporary tables, and transaction-scoped advisory locks; no persistent schema changes |
| Preview GET requests | `V2_RUNTIME_DATABASE_URL` or `V2_RUNTIME_DATABASE_URL_SECRET_RESOURCE` | Catalog selection only during Stage 9; no migration access |

Do not place more than one database URL in the environment of either the
Expand Down Expand Up @@ -626,6 +626,39 @@ Deleted rows free space inside the tables for later publications to reuse;
PostgreSQL returns it to the operating system only after a `VACUUM FULL` or
an equivalent rewrite, which pruning doesn't run.

### Catalog table statistics

PostgreSQL's autovacuum analyzes a table again only after about a tenth of its
rows have changed. `parameters` keeps every published version (about 1.25
million rows on staging), and one release adds about 105,000, so a new
version's rows can stay out of the planner's statistics. On staging, after
6.17.1 was published on 2026-11-10, the planner estimated 1 parameter row for
US 6.17.1 instead of 102,530, chose to fetch and sort every parameter value of
that version, and `GET /v2/parameter-values?country_id=us&limit=1` took 43 to
47 seconds on every release until 2026-11-16. After a manual `ANALYZE
parameters` it walks `uq_parameters_model_version_name` and takes 4.1 ms
(UK: 4.9 ms).

Every publication therefore runs one `ANALYZE` over `datasets`,
`parameter_nodes`, `parameter_tree_children`, `parameter_values`,
`parameters`, `regions` and `variables` once it has committed and pruned, and
lists them in the evidence's `catalog_tables_analyzed`. It takes a few seconds
and blocks no reads. A failed `ANALYZE` doesn't undo the publication: the
evidence lists no tables and names the failure in `catalog_statistics_failure`
(the database driver's error class and SQLSTATE, never its message), the
command prints it to stderr and still succeeds, and the next publication
analyzes again.

`ANALYZE` needs the table's owner or the `MAINTAIN` privilege (PostgreSQL 17),
and PostgreSQL skips any other table with only a warning. The migration role
owns the tables, so the manifest grants the catalog writer `MAINTAIN` on these
seven, the deploy's `apply` step grants it before publication runs, and
publication fails before writing anything if its role lacks it. PostgreSQL 17
has no narrower privilege: `MAINTAIN` also allows `VACUUM` (`FULL` included),
`CLUSTER`, `REINDEX`, `REFRESH MATERIALIZED VIEW` and `LOCK TABLE` on those
tables, though publication runs only `ANALYZE`. The code is
`policyengine_api/data/v2/catalog/publication_statistics.py`.

## Validation evidence and retry

Successful JSON evidence contains:
Expand All @@ -650,6 +683,9 @@ Successful JSON evidence contains:
(`catalog_pruning_failures`; see "Catalog pruning");
- the number of chosen versions left for later publications
(`catalog_versions_left_to_prune`);
- the catalog tables analyzed after the publication
(`catalog_tables_analyzed`), and the class of a failed `ANALYZE`
(`catalog_statistics_failure`; see "Catalog table statistics");
- elapsed publication time; and
- a success outcome.

Expand Down
15 changes: 11 additions & 4 deletions policyengine_api/data/v2/catalog/initialization.py
Original file line number Diff line number Diff line change
Expand Up @@ -177,10 +177,10 @@ def _error_payload(error: Exception) -> dict[str, object]:
def main() -> int:
"""Run explicit initialization and return a shell-compatible status.

A failure to prune an older catalog version, or to read the simulation
service's versions to record SPM availability, leaves the status
successful, since the publication had committed; each one is in the
evidence and is also printed to stderr.
A failure to prune an older catalog version, to analyze the catalog
tables, or to read the simulation service's versions to record SPM
availability, leaves the status successful, since the publication had
committed; each one is in the evidence and is also printed to stderr.
"""
try:
evidence = initialize_catalog()
Expand All @@ -199,6 +199,13 @@ def main() -> int:
f"{failure.error_type}: {failure.message}",
file=sys.stderr,
)
if evidence.catalog_statistics_failure is not None:
print(
"catalog table statistics were not refreshed: "
f"{evidence.catalog_statistics_failure}; the next publication "
"analyzes the catalog tables again",
file=sys.stderr,
)
spm_recording = evidence.spm_availability
if (
spm_recording is not None
Expand Down
Loading
Loading