Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/v2-integration-check.yml
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ jobs:
- name: Verify the managed v2 roles hold their declared privileges
run: uv run --frozen --extra dev python scripts/reconcile_v2_privileges.py verify
- name: Test v2 metadata publication, resource routes and the countries list
run: uv run --frozen --extra dev coverage run --branch -m pytest -q tests/integration/test_v2_catalog_publication.py tests/integration/test_v2_metadata_routes.py tests/integration/test_v2_country_routes.py
run: uv run --frozen --extra dev coverage run --branch -m pytest -q tests/integration/test_v2_catalog_publication.py tests/integration/test_v2_catalog_publication_time_periods.py tests/integration/test_v2_metadata_routes.py tests/integration/test_v2_country_routes.py
- name: Test v2 policy persistence and immediate v1 mirroring
run: uv run --frozen --extra dev coverage run -a --branch -m pytest -q tests/integration/test_v2_policy_persistence.py tests/integration/test_v1_policy_dual_write.py tests/integration/test_mysql_policy_dual_write.py tests/integration/test_v2_user_policy_mirroring.py tests/integration/test_v1_user_policy_dual_write.py
- name: Test signed-in identity provisioning and user associations
Expand Down
1 change: 1 addition & 0 deletions changelog.d/catalog-time-periods.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Catalog publication gives an already-stored catalog version the time-period options (`metadata_time_periods`) of the catalog it stages, instead of failing when they differ, prints a warning naming the country, the version and both lists, and reports them under `updated_time_periods` in its evidence. Every other column and table must still match.
1 change: 1 addition & 0 deletions changelog.d/us-years-from-2024.removed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
US 2022 and 2023 are no longer offered for society-wide calculations: `economy_options.time_period` in `/us/metadata` and the v2 catalog list US 2024 to 2035, and the UK years are 2024 to 2032 in both, each defined once in `policyengine_api/economy_years.py`. A society-wide request for a year outside its country's years (an annual `time_period`, or a budget window's `start_year`) returns 400 with the code `DATASET_YEAR_UNAVAILABLE` in `errors`, even when a result for that year is stored. Household calculations are unchanged.
8 changes: 7 additions & 1 deletion docs/canonical-spm.md
Original file line number Diff line number Diff line change
Expand Up @@ -327,7 +327,13 @@ HTTP 400. An omitted `spm` inherits the certified default measurement, while a
misspelled parameter is rejected. `region` is required;
annual requests require `time_period`, while budget-window requests require
`start_year` and `window_size` (1–75, ending no later than 2099). Years use four
digits. Optional fields are `version` (installed country model version),
digits. An annual `time_period`, and a budget window's `start_year`, must be
one of the country's years in `/{country}/metadata`'s
`economy_options.time_period` (US 2024 to 2035, UK 2024 to 2032, from
`policyengine_api/economy_years.py`). Any other year returns HTTP 400 with the
code `DATASET_YEAR_UNAVAILABLE` in `errors`, before anything is read from the
cache or submitted, even when a result for that year is stored. A budget
window that starts in those years may end after them. Optional fields are `version` (installed country model version),
`target` (`general`, or annual-only `cliff`) and the annual-only `breakdown`.
Omitted `spm` inherits certified bundle defaults. The Flask parser's typed
models define these query rules; the legacy `GET /specification` document
Expand Down
2 changes: 1 addition & 1 deletion docs/generated/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ Every rule that governs `policyengine-api` is listed here with how it is enforce

### Ratcheted (existing violations still tolerated)

- `api.dependencies.declared-and-used`: 15 remaining in `rules/baselines/deptry.txt`
- `api.dependencies.declared-and-used`: 14 remaining in `rules/baselines/deptry.txt`
- `api.quality.suppressions-explained`: 29 remaining in `rules/baselines/suppressions.txt`
- `api.quality.tests-assert`: 21 remaining in `rules/baselines/tests-without-checks.txt`
- `api.v2.connectors-no-cross-calls`: 7 remaining in `rules/baselines/import-linter-api-v2-connectors-independent.txt`
Expand Down
54 changes: 42 additions & 12 deletions docs/migration/stage-9-v2-metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,9 @@ transaction is needed, and the database receives no `BEGIN` before the reads
and no `ROLLBACK` when the session closes. The sessions refuse to flush added
or changed rows (`ReadOnlySessionError`); a statement passed directly to
`execute` is not checked, so the metadata connectors keep to `SELECT`. Each process also keeps every catalog version a request has
selected (`SelectedCatalogStore`), since a published version never changes, so
selected (`SelectedCatalogStore`), since a published version changes only
when a release's publication replaces its time-period options, before that
release's processes start (see "Time-period options"), so
later requests for the same country and version skip the selection query;
failed selections are not kept. Measured with statement logging in a local
database, a page of 500 parameters or one parameter's values took five
Expand All @@ -109,7 +111,8 @@ statement per read after the liveness check. The pool's liveness check
Every successful metadata response carries a strong `ETag`, a hash of its
exact body, and `Cache-Control: public`. A request that names its catalog with
`policyengine_version` may be reused for a day (`max-age=86400`): a published
catalog version never changes, so such a response changes only with the
catalog version changes only when a release replaces its time-period options
(see "Time-period options"), so such a response changes only then, with the
route's response format or with the day it is read on, and the `ETag` lets a
client revalidate after the day. A request without it selects the running
version, which changes on deploy, and may be reused for five minutes
Expand Down Expand Up @@ -244,10 +247,10 @@ compares every format 2 field for the US and the UK with what
variable and parameter of the installed models, with their values, and builds
each country's v1 metadata; set `RUN_V2_CATALOG_METADATA_PARITY=1` to run it.

Publication never rewrites a stored version. A version published before
format 2 has a null `catalog_format` and null format 2 fields, and later
publications of the same PolicyEngine.py version compare it without those
fields. The first PolicyEngine.py version published after format 2 deploys
Publication never fills in a stored version's format 2 fields. A version
published before format 2 has a null `catalog_format` and null format 2
fields, and later publications of the same PolicyEngine.py version compare it
without those fields. The first PolicyEngine.py version published after format 2 deploys
carries them.

Parameter extraction requires the PolicyEngine.py public history contract from
Expand Down Expand Up @@ -295,8 +298,9 @@ Review the fallback and omitted-region summaries on every new PolicyEngine.py
release. A changed count may identify an upstream change to the release's
datasets even when publication otherwise succeeds.

A catalog version is identified by its country and PolicyEngine.py version and
never changes once published. When the command republishes a stored version
A catalog version is identified by its country and PolicyEngine.py version.
Once published, its rows change only in its time-period options (see
"Time-period options" below). When the command republishes a stored version
whose staged regions differ from the stored ones, it keeps the stored regions
unchanged, prints a warning with the country, the version and the counts of
staged region rows the stored version lacks and of stored rows not staged, and
Expand All @@ -307,6 +311,28 @@ since they describe staged regions that were not published. Every other table
must still match exactly. Region reference changes therefore take effect in the
next catalog version that is published.

### Time-period options

A catalog version's time-period options (`metadata_time_periods`), the years
`/v2/economy-options` offers for it, come from the Public API's code, not from
PolicyEngine.py: each country's years are defined once in
`policyengine_api/economy_years.py`, which `/{country}/metadata` offers in
`economy_options.time_period` and the economy routes accept (US 2024 to 2035,
newest first; UK 2024 to 2032, oldest first). They can therefore change while
the PolicyEngine.py version stays the same, so republication does not compare
them. After every other table and column of a stored version has been
compared, publication replaces its time-period options with the staged ones
in the same transaction when the two lists differ, order included. It prints a
warning with the country, the version and the previous and new years, and the
evidence's `updated_time_periods` lists the same. Publication stages only the
installed PolicyEngine.py version, so other stored versions keep the years
they were published with.

A release publishes its catalog before its own processes start. Processes of
the previous release keep the years they already read until they stop, and an
HTTP cache can keep a response with the previous years until it expires (five
minutes, or a day for a request naming `policyengine_version`).

### Parameter tree

Publication also writes parameter trees into `parameter_tree_children`, which
Expand Down Expand Up @@ -423,6 +449,8 @@ Successful JSON evidence contains:
- fallback counts and omitted-region counts by country and region type, and
the stored and staged region counts of any stored version whose regions were
kept;
- the country, version, and previous and new years of any stored version whose
time-period options were replaced (`updated_time_periods`);
- the model, version and row count of each catalog version whose parameter
tree was written;
- the model, version and uncompressed and compressed sizes of each parameter
Expand All @@ -446,10 +474,12 @@ A failed attempt rolls back all catalog changes. Retry the same artifact only
after correcting the external failure. A matching retry preserves identifiers,
content, and row counts. Different normalized content under the same
PolicyEngine.py version fails for operator review, with an error naming the
first table that differs; it is never silently replaced. Regions are the
exception: they come from the Public API's reference files, so a stored version
keeps the regions it was published with (see "Source and regional dataset
behavior"). A later PolicyEngine.py version adds a catalog version without
first table that differs; it is never silently replaced. Regions and
time-period options are the exceptions, because both come from the Public API
rather than PolicyEngine.py: a stored version keeps the regions it was
published with (see "Source and regional dataset behavior"), and takes the
staged time-period options, with a warning and an evidence entry (see
"Time-period options"). A later PolicyEngine.py version adds a catalog version without
deleting or modifying earlier versioned rows, including their logical datasets,
regions, model descriptions, current-law IDs, and time-period options.

Expand Down
23 changes: 5 additions & 18 deletions policyengine_api/country.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
build_congressional_district_metadata,
)
from policyengine_api.data.uk_regions import uk_areas
from policyengine_api.economy_years import uk_economy_years
from policyengine_api.economy_years import ECONOMY_YEARS
from policyengine_api.services.household_calculation_service import CalculationResult
from policyengine_api.spm import (
SPM_INPUT_ERROR_CODES,
Expand Down Expand Up @@ -206,7 +206,8 @@ def _json_safe(
def build_microsimulation_options(self) -> dict:
"""Build the regions, time periods and datasets offered for economy runs."""
# Shape: { region: [{ name: "uk", label: "the UK" }],
# time_period: [{ name: 2022, label: "2022", ... }] }
# time_period: [{ name: 2024, label: "2024", ... }] }
# The years are each country's ECONOMY_YEARS, in their order.
options = {}
if self.country_id == "uk":
region = [
Expand All @@ -229,8 +230,7 @@ def build_microsimulation_options(self) -> dict:
for area in sorted(uk_areas(area_type), key=attrgetter("name"))
)
time_period = [
{"name": year, "label": str(year)}
for year in uk_economy_years(self.tax_benefit_system.parameters)
{"name": year, "label": str(year)} for year in ECONOMY_YEARS["uk"]
]
datasets = [get_bundle_default_dataset_option("uk")]
options["region"] = region
Expand Down Expand Up @@ -294,20 +294,7 @@ def build_microsimulation_options(self) -> dict:
# Add all 436 congressional districts (435 voting + DC)
region.extend(build_congressional_district_metadata())
time_period = [
{"name": 2035, "label": "2035"},
{"name": 2034, "label": "2034"},
{"name": 2033, "label": "2033"},
{"name": 2032, "label": "2032"},
{"name": 2031, "label": "2031"},
{"name": 2030, "label": "2030"},
{"name": 2029, "label": "2029"},
{"name": 2028, "label": "2028"},
{"name": 2027, "label": "2027"},
{"name": 2026, "label": "2026"},
{"name": 2025, "label": "2025"},
{"name": 2024, "label": "2024"},
{"name": 2023, "label": "2023"},
{"name": 2022, "label": "2022"},
{"name": year, "label": str(year)} for year in ECONOMY_YEARS["us"]
]
datasets = [get_bundle_default_dataset_option("us")]
options["region"] = region
Expand Down
8 changes: 4 additions & 4 deletions policyengine_api/data/v2/catalog/extraction.py
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@
filtered_artifacts,
scope_regions,
)
from policyengine_api.economy_years import ECONOMY_YEARS
from policyengine_api.supported_countries import SUPPORTED_COUNTRY_IDS

if TYPE_CHECKING:
Expand Down Expand Up @@ -83,10 +84,9 @@
*(f"policyengine-{country_id}" for country_id in SUPPORTED_COUNTRY_IDS),
)
CURRENT_LAW_IDS = {"us": 2, "uk": 1}
METADATA_TIME_PERIODS = {
"us": tuple(range(2035, 2021, -1)),
"uk": tuple(range(2024, 2031)),
}
# The years the Public API offers for society-wide calculations, defined once
# for /{country}/metadata, these catalog versions and the economy routes.
METADATA_TIME_PERIODS = ECONOMY_YEARS
PLACEHOLDER_VERSIONS = frozenset({"", "0", "0.0", "0.0.0", "unknown"})


Expand Down
Loading
Loading