Repository navigation
Fix the v2 privilege check, record metadata read failures, and analyze the catalog after publication - #4319
Merged
Conversation
…ing) A new table's default privileges give the runtime role UPDATE on the whole table. Where the manifest declares UPDATE on named columns only, as for user_type_changes, apply reports that privilege but never revokes it, so verify fails every deploy. These tests expect apply to revoke it, grant the declared columns again, and leave every other extra privilege. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The projects' default privileges give the runtime role UPDATE on every new table, including user_type_changes, where the manifest allows it to update only dismissed_at. apply reported that privilege without revoking it, so verify failed every staging deploy after the invitations revision. apply now revokes a whole-table privilege a role holds directly when the manifest declares it for named columns only, then grants those columns again in the same transaction, since PostgreSQL's REVOKE on a table also removes the role's column grants of it. The job summary lists the revoke. Every other undeclared privilege is still reported, not revoked, and verify still fails if a role keeps the whole-table privilege another way. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The job summary now lists the undeclared column grants a revoke removes, the PostgreSQL test starts from staging's state (whole-table privileges plus a declared and an undeclared column grant), the planned-revoke test uses a state that can occur, and the privileges module, rule statement and skill document say which grants the revoke removes and that a grant another role made survives it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #4319 +/- ##
==========================================
+ Coverage 93.47% 93.60% +0.12%
==========================================
Files 369 389 +20
Lines 21455 22486 +1031
Branches 3400 3535 +135
==========================================
+ Hits 20056 21048 +992
- Misses 816 844 +28
- Partials 583 594 +11
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
On 2026-11-16 staging answered GET /v2/parameter-values with the generic 503 "V2 metadata catalog is unavailable" after the database connection was lost mid-query, and nothing recorded the error behind it. These tests expect one runtime event per generic 503 or 500 with the route, the caught error's class, its cause chain's classes, the SQLSTATE and SQLAlchemy's lost-connection flag, and no error message. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The process's runtime writes to the stream it saved at start-up, which pytest's output capture can't read, so the tests serve their requests through a runtime configured the same way but writing to a list. The route fixture takes that runtime as an optional argument. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run_metadata_read turned a lost database connection into the generic 503 "V2 metadata catalog is unavailable", and any unexpected error into the 500 "V2 metadata query failed", recording neither. Each such answer now records one runtime event, v2_metadata_read_failed, at ERROR, with the response status, the caught error's class, the classes of its cause chain, the first SQLSTATE in the chain and SQLAlchemy's lost-connection flag; the runtime adds the request's ID, method and route. It records no error message, since SQLAlchemy's and the driver's carry the statement, its parameters and connection details. A pruned version or a missing parameter tree or search file still answers with its own message and records nothing. Recording never changes the response. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The user asked for the metadata read failure event to go into this pull request too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ling) PostgreSQL re-collects a table's statistics only after about a tenth of its rows change, and parameters keeps every published version, so one release's rows never triggered it on staging: after 6.17.1 the planner estimated 1 US parameter row instead of 102,530, and the parameter-values list took 43-47 s. These tests expect publication, running as the catalog writer, to ANALYZE the catalog tables after a run that adds a version, to analyze nothing when nothing changed, and to fail before writing when its role can't analyze them. The new test file joins the v2 integration job. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PostgreSQL's autovacuum analyzes a table again only after about a tenth of its rows change, and parameters keeps every published version, so on staging the planner never saw 6.17.1's 102,530 US parameter rows and GET /v2/parameter-values?country_id=us&limit=1 took 43-47 s on every release from 2026-11-10 to 2026-11-16; after a manual ANALYZE it takes 4.1 ms. A publication that publishes, restores or prunes a version now runs one ANALYZE over datasets, parameter_nodes, parameter_tree_children, parameter_values, parameters, regions and variables after it commits and prunes, and lists them in the evidence's catalog_tables_analyzed. ANALYZE needs the owner or MAINTAIN, and PostgreSQL skips any other table with only a warning; the migration role owns the tables, so the manifest grants the catalog writer MAINTAIN on these seven (PostgreSQL 17), and publication fails before writing anything if its role lacks it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scheduled error-issue job groups an event by its route and its attributes.error_type and attributes.failure_category, and the failure event had neither, so every 503 and 500 on a route would have merged into one issue. The tests now expect error_type and a failure category that tells a catalog outage from an unexpected failure. They also use psycopg's real AdminShutdown wrapped as SQLAlchemy wraps it, keep the process runtime's settings, and cover a configuration error, the parameter tree and search file answers, the 500 with a failing runtime, and looping and long cause chains. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The event now carries error_type and failure_category, the keys the API's other events use and the scheduled error-issue job groups by, so 503s and 500s on a route, and different causes, stay apart. One helper, _caller_message, now decides both which answers have their own message and which record the event. The observability and v2 code-organization guides describe the event and its module, including that requests waiting on a shared whole-version read answer with a copy and record nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed ANALYZE after the catalog committed failed the command, and the retry, which changed nothing, analyzed nothing, leaving the statistics stale. The tests now expect every publication to analyze, a failed ANALYZE to be recorded by its class while the publication is kept, and the evidence to carry both. The catalog writer's pinned privileges and the evidence's pinned fields now include MAINTAIN and the new fields, and the manifest test checks the grant against the module's table list. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A failed ANALYZE after the catalog committed failed the command and was never repeated, because the retry changed nothing and so analyzed nothing. Every publication now analyzes the catalog tables once it has committed and pruned, which takes seconds and blocks no reads, and a failure is returned rather than raised: the evidence lists no tables and names the failure's class in catalog_statistics_failure, as a failed pruning is reported, and the next publication analyzes again. The privileges comment and the Stage 9 document now say that MAINTAIN also allows VACUUM, CLUSTER, REINDEX, REFRESH MATERIALIZED VIEW and LOCK TABLE, PostgreSQL 17 having no narrower privilege, and the evidence list names the two new fields. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iling) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
catalog_statistics_failure now names the class of the error the database driver raised and its SQLSTATE, as a failed pruning is reported, since SQLAlchemy's own class is the same for every database error, and the command prints the failure to stderr while still succeeding. Two descriptions still said the writer analyzes only the tables publication grows; it analyzes seven tables after every publication. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… fix The user asked for the statistics refresh to go into this pull request too, in commits of its own. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov reported changed lines no test ran. Locally, two of them were: an ANALYZE failure that isn't a database error, named by its class alone, and a cause chain that loops back to an error partway along. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The simulation API now serves PolicyEngine .py 6.20.0, and the check that the Public API's bundle is the simulation API's latest fails on 6.17.1. This brings in the automated update of #4315 unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk
added a commit
that referenced
this pull request
Nov 17, 2026
anth-volk
added a commit
that referenced
this pull request
Nov 17, 2026
…anch Both sides added test modules to the same v2 integration step; the step now runs master's catalog statistics module and C1 · 28's six modules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4318
Summary
Every staging deploy since #4297 fails at the privilege check. The release run for
43473bb85stopped in "Seed staging v2 database / Upgrade schema and seed v2 database", atscripts/reconcile_v2_privileges.py verify:The manifest lets the runtime role update only
dismissed_atonuser_type_changes. But the Supabase projects' default privileges givepolicyengine_v2_runtimeSELECT, INSERT, UPDATE and DELETE on every table the migration role creates.applyreported the whole-table UPDATE without revoking it, soverifyfailed.user_type_changesis the first table where the manifest limits the runtime role's UPDATE to named columns. The disposable test database has no default privileges, so CI didn't catch it.Change
applynow revokes a privilege a role holds directly on a whole table when the manifest declares it for named columns only. It does this before granting, in the same transaction:plan_reconciliationadds two fields.revocationsholds the whole-table privileges to revoke.revoked_columnsholds the undeclared column grants that PostgreSQL'sREVOKEon a table also removes. The declared columns go back intogrants, because the revoke removes them too.revoke_statementswrites oneREVOKEper role and table, using the same identifier checks as the grants.user_type_changesstays.verifyis unchanged, so it still fails when a role keeps the whole-table privilege another way: through a role it belongs to, or through a grant another role made, which the owner'sREVOKEdoesn't remove.privilegesandprivilege_reconciliationdocstrings, the statement of ruleapi.v2.deployed-privileges-match(withdocs/generated/rules.mdregenerated) and the "Table privileges" section of the Alembic migrations skill document describe the revoke.Deploy
Staging: the next release run's
applyrevokes the runtime role's whole-table UPDATE onuser_type_changesand grantsUPDATE (dismissed_at)again;verifythen passes.Production: its first upgrade to
f519a77ba122creates the table with the default privileges, and the sameapplystep revokes the whole-table UPDATE beforeverifyruns.Permission to revoke: the migration role owns every table. Grants that default privileges create, and grants a superuser makes, record the owner as grantor, so the owner's
REVOKEremoves them.Environment variables: none added; the workflows are unchanged. The script still reads these:
V2_MIGRATION_DATABASE_URL: a GitHub Actions environment secret, set in each step'senv:inseed-v2-database.yml, whichpush.ymlcalls withsecrets: inherit;DEPLOYMENT_ENVIRONMENT: the workflow input;V2_SUPABASE_PROJECT_REF,V2_SUPABASE_ENVIRONMENTandPRODUCTION_V2_SUPABASE_PROJECT_REF: GitHub Actions variables, set in the job'senv:.The failing staging run's log shows each of them set in the
applystep.Also: record why a v2 metadata read answered 503 or 500
The other staging failure after #4303: release run 43915927273 stopped at
tests/integration/test_live_v2_metadata.py::test_live_v2_resources_use_the_deployed_policyengine_version[us]. Staging answeredGET /v2/parameter-values?country_id=us&limit=1with the generic 503 "V2 metadata catalog is unavailable" (2026-11-16 21:30:13 Azerbaijan time, UTC+4).run_metadata_readturned the error into the generic 503 without recording it.The query's 45-second runtime on staging was fixed separately, by refreshing table statistics.
run_metadata_readnow records one runtime event at ERROR,v2_metadata_read_failed, for each generic 503 and each 500 ("V2 metadata query failed"). It goes throughpolicyengine_api.fastapi_routes.v2.metadata.read_failure_support, and the runtime adds the request's ID, method and matched route. The event adds:http.response.status_code;error_type: the caught error's class, asmodule.QualifiedName;failure_category:v2_metadata_catalog_unavailablefor the 503, orv2_metadata_query_failedfor the 500. The scheduled error-issue job (.github/scripts/telemetry_error_issues) groups an event by its route,error_typeandfailure_category, the keys the API's other events use, so 503s, 500s and different causes open separate issues;error.cause_types: the classes of its__cause__chain, outermost first;database.sqlstate: the first SQLSTATE in the chain;database.connection_invalidated: whether SQLAlchemy found the connection lost.What it leaves out:
_caller_message, decides both.Tests:
tests/unit/v2/test_metadata_read_failure_events.pyserves each request through a runtime with the process runtime's settings that writes its records to a list, so the tests see the event exactly as written, request context included. For that, the route fixture takes an optionalobservability_runtime. The lost connection is psycopg's realAdminShutdown(SQLSTATE 57P01), wrapped by SQLAlchemy'sDBAPIError.instance.The tests before the change:
The two that passed check behaviour the change must keep: a pruned version records nothing, and a failing runtime leaves the 503 unchanged.
Review round 1 found that the error-issue job couldn't group the event, because it had no
error_typeorfailure_category. It also asked for the cases below to be covered. The updated tests, before that fix:The new tests cover a configuration error, the parameter-tree and search-file answers, a failing runtime on the 500, and cause chains that loop or run long.
After it:
Also: refresh the catalog tables' statistics after a publication
The cause of that 503's slow query. PostgreSQL's autovacuum analyzes a table again only after about a tenth of its rows have changed.
parameterson staging holds about 1.25 million rows, because it keeps every published version, and each PolicyEngine.py release adds about 105,000. So after 6.17.1 was published on 2026-11-10, its rows never reached the statistics:GET /v2/parameter-values?country_id=us&limit=1took 43 to 47 seconds on every release from 2026-11-10 to 2026-11-16.ANALYZE parameterson staging (2026-11-17 01:28:59 Azerbaijan time, UTC+4), the same query walksuq_parameters_model_version_nameand takes 4.1 ms (UK: 4.9 ms).publish_catalognow runs oneANALYZEoverdatasets,parameter_nodes,parameter_tree_children,parameter_values,parameters,regionsandvariables(policyengine_api/data/v2/catalog/publication_statistics.py):ANALYZEis then repeated by the next publication. If it ran only after a run that changed a version, a retry that changed nothing would leave the statistics stale.catalog_tables_analyzedfield.ANALYZE: it doesn't undo or fail the publication, just as a failed pruning doesn't.catalog_tables_analyzedis then empty, and the newcatalog_statistics_failurefield names the failure's class, never its message.ANALYZEneeds the table's owner orMAINTAIN; PostgreSQL skips any other table with only a warning. PostgreSQL 17 has no narrower privilege, andMAINTAINalso allowsVACUUM(FULLincluded),CLUSTER,REINDEX,REFRESH MATERIALIZED VIEWandLOCK TABLEon those tables, though publication runs onlyANALYZE. The migration role owns these tables, and publication connects aspolicyengine_v2_catalog_writer. So the manifest now declaresMAINTAIN(PostgreSQL 17; staging runs 17.6, and both Supabase projects report engine 17) for the catalog writer on the seven tables.applystep grantsMAINTAINandverifychecks it, both before publication runs.has_table_privilegebefore it writes anything, and fails, naming the tables, when its role lacks it.Out of scope: retrying on a lost connection.
Environment variables: none added. The workflow change only adds
tests/integration/test_v2_catalog_statistics.pyto an existing step ofv2-integration-check.yml. That step'sV2_MIGRATION_DATABASE_URL,V2_ALEMBIC_DISPOSABLE_TESTandDEPLOYMENT_ENVIRONMENTare literal jobenv:values for its own PostgreSQL service container. Publication still readsV2_DATA_WRITE_DATABASE_URL, a GitHub Actions environment secret set in its step'senv:inseed-v2-database.yml.Tests, before the change:
Review round 1 found that a failed
ANALYZEwas never repeated, and that two existing unit tests pin the catalog writer's privileges and the evidence fields. The updated tests, before that fix:After it:
What the PostgreSQL tests check:
pg_stat_user_tables.last_analyzeis newer than the publication's start for each of the seven tables.ANALYZEis recorded while the publication is kept.MAINTAINfails before writing any version.Also: the PolicyEngine bundle update to 6.20.0
The simulation API now serves PolicyEngine .py 6.20.0 (PolicyEngine/policyengine-sim-api#1076, deployed once PolicyEngine/policyengine-sim-api#1088 removed the checks that had stopped it). The required check "Ensure PolicyEngine bundle is the simulation API's latest" therefore fails on 6.17.1, and CI skips the test jobs that wait on it. This pull request merges the branch of the automated update #4315 unchanged: the pin in
pyproject.tomlanddocker/Dockerfile,uv.lockand its changelog fragment, with policyengine 6.20.0, policyengine-core 3.33.3, policyengine-us 2.2.2 and policyengine-uk 2.174.1. Once this merges, #4315 (and its issue #4314) has nothing left to merge.Every test this pull request adds or changes passes on 6.20.0, and the Hugging Face credential check still resolves the default UK dataset to
microcosm_uk_2024_25.h5atmicrocosm-uk-2024-25-national-20261107T020434Z-eeb6093e.Testing
f519a77ba122). I gavepolicyengine_v2_runtimewhat the default privileges give onuser_type_changes.applythenverifyfailed as in staging.applyrevoked the whole-table UPDATE and grantedUPDATE (dismissed_at)again, a secondapplychanged nothing, andverifypassed. The runtime role can updatedismissed_atand no other column.test_apply_revokes_a_new_tables_whole_table_update_and_verify_passesstarts from staging's state: whole-table privileges, the declared column grant and an undeclared one.test_column_and_whole_table_privileges_beyond_the_manifest_are_reportednow uses an extra whole-table DELETE, becauseapplynow revokes the whole-table UPDATE that test used.lint-imports,python -m scripts.rules validate,python scripts/run_quality_guards.pyand mypy all passed.ANALYZEwas never repeated, and two pinned unit tests broke. Neither second round found a bug. The privilege fix had two rounds of review, neither of which found a bug. Their fixes are included:🤖 Generated with Claude Code