Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
064adf9
Test that economy reports take an optional local-area breakdown (fail…
anth-volk Nov 13, 2026
68f1167
Accept breakdown "local_areas" on UK-wide and nation economy reports
anth-volk Nov 13, 2026
d4957ed
Test that the local-area dataset is read from the catalog version (fa…
anth-volk Nov 13, 2026
b4e95c1
Read a catalog version's local-area dataset
anth-volk Nov 13, 2026
a2dce1b
Merge C1 · 23's local-area report profile and manifest declaration
anth-volk Nov 13, 2026
a2e23a8
Test that local areas follow the worker's declaration and the catalog…
anth-volk Nov 13, 2026
5fe191f
Decide a report's local areas by the default worker's declaration
anth-volk Nov 13, 2026
dd6ae8a
Test that a report's local-area breakdown is never stored (failing)
anth-volk Nov 13, 2026
fa71c6c
Never store a report's local-area breakdown
anth-volk Nov 13, 2026
d0f8efb
Merge C1 · 23's local-area report runs up to 30c9ebea
anth-volk Nov 13, 2026
caa9668
Test that a UK-wide or nation report runs on the local-area dataset w…
anth-volk Nov 13, 2026
9a3c9ec
Run a UK-wide or nation report on the local-area dataset with data_so…
anth-volk Nov 13, 2026
787b47c
Test that no report, simulation or budget-window request names a data…
anth-volk Nov 13, 2026
f176fec
Take no dataset_name on reports, simulations or budget windows
anth-volk Nov 13, 2026
2ca53c7
Test that the staging economy check's Wales report runs on the local-…
anth-volk Nov 13, 2026
2a3bbc6
Run the staging economy check's Wales report on the local-area dataset
anth-volk Nov 13, 2026
7d681d0
Merge C1 · 23's local-area report runs up to b383db26
anth-volk Nov 13, 2026
4d167a1
Merge C1 · 23's local-area report runs up to 45ffb33e
anth-volk Nov 13, 2026
0855c61
Test review round 1's findings for data_source (one failing)
anth-volk Nov 13, 2026
ed55d35
Fix review round 1's findings for data_source
anth-volk Nov 13, 2026
b14ce85
Fix review round 2's findings for data_source
anth-volk Nov 13, 2026
a25e064
Describe the local-area 400 codes' conditions exactly
anth-volk Nov 13, 2026
417a39c
Merge main (#1068 and its release) into the data_source branch
anth-volk Nov 16, 2026
85c036f
Merge main (#1072, #1088 and their releases) into the data_source branch
anth-volk Nov 17, 2026
abcbfd1
Merge main (#1074, C1 · 27's find-or-create by key) into the data_sou…
anth-volk Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 9 additions & 5 deletions .github/scripts/simulation-staging-economy-check.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,13 @@
# Send a small economy report request through a deployed Simulation
# Entrypoint, force a new run of the report, and require that run to succeed:
# current law against current law for Wales in 2026, which needs no stored
# policy. The request reuses the simulation runs an earlier check on the same
# versions calculated (and, once reports are found by key, ends on that
# check's report run), so only a forced run, which nothing joins or reuses,
# calculates both simulations on every deploy.
# policy, on the bundle's local-area dataset (data_source), so the run also
# calculates Wales's constituencies and local authorities. The coordinator
# fails a run of such a report unless both area levels have rows or withheld
# codes, so its success covers them. The request reuses the simulation runs
# an earlier check on the same versions calculated (and, once reports are
# found by key, ends on that check's report run), so only a forced run, which
# nothing joins or reuses, calculates both simulations on every deploy.
# Usage: simulation-staging-economy-check.sh <base-url>
# Required env vars: the test client's SIMULATION_TEST_AUTH_ISSUER,
# SIMULATION_TEST_AUTH_AUDIENCE, SIMULATION_TEST_AUTH_CLIENT_ID and
Expand Down Expand Up @@ -80,7 +83,8 @@ submission_status="$(
region_code: "country/wales",
baseline: {policy_id: null, dynamic_id: null},
reform: {policy_id: null, dynamic_id: null},
include_cliffs: false
include_cliffs: false,
data_source: "local_area_dataset"
}' \
| send submission "${submission_timeout_seconds}" --request POST --header "@${authorization_header}" \
--header 'Content-Type: application/json' --data @- \
Expand Down
1 change: 1 addition & 0 deletions changelog.d/c1-29a-data-source.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`POST /reports` takes `data_source: "local_area_dataset"` on UK-wide and nation economy reports: the report runs on the bundle's local-area dataset (the one constituency and local-authority regions use) and returns its full results with its constituency and local-authority results. Another value, another country or region, and a release that routes those areas to no dataset of their own (`no_local_area_dataset`) are refused with 422. The staging economy check's Wales report runs on that dataset.
1 change: 1 addition & 0 deletions changelog.d/c1-29a-no-dataset-name.removed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
`POST /reports` (economy reports and budget windows) and `POST /simulations` no longer take `dataset_name`, and refuse it with 422: a report or simulation runs on the dataset the catalog gives its region, or on the local-area dataset with `data_source`. The Public API stopped sending it in PolicyEngine/policyengine-api#4307, which must be deployed first.
28 changes: 19 additions & 9 deletions libs/policyengine-simulation-persistence/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,12 +87,15 @@ runs too long and the limit never stays on a pooled connection.
run writes for an economy report, or a standalone economy simulation. It
reads the country's model (`read_model_id`) and a catalog version's region
by code (`read_region`) and dataset by name or ID (`read_dataset`,
`read_dataset_by_id`), and builds a simulation run `pending` with no Modal
call ID (`simulation_run_row`, `insert_simulation_run`). The Simulation
Entrypoint creates every simulation, report and run by key ("Shared
simulations, reports and runs" below). A row naming one that does not exist
is refused by its foreign key with SQLSTATE `23503`;
`EconomyRecordReferenceError` names the table and column.
`read_dataset_by_id`), and its local-area dataset
(`read_local_area_dataset`): the one default dataset every constituency and
local-authority region of the version uses, or none. It builds a simulation
run `pending` with no Modal call ID (`simulation_run_row`,
`insert_simulation_run`). The Simulation Entrypoint creates every
simulation, report and run by key ("Shared simulations, reports and runs"
below). A row naming one that does not exist is refused by its foreign key
with SQLSTATE `23503`; `EconomyRecordReferenceError` names the table and
column.

`record_report_run_call` and `record_simulation_run_call` record a Modal
call's ID, and `fail_unstarted_report_run` and
Expand All @@ -106,9 +109,16 @@ a transaction of its own: `read_report_inputs` reads the catalog version,
model, region and dataset, and each side's policy and dynamic values
together; `read_simulation_inputs` does the same for one simulation, and
`read_budget_window_inputs` for a budget window's two sides; the writes and
reads each take their own. A missing region or dataset raises
`UnknownRegionError` or `UnknownDatasetError`, and a report side's refused
policy or dynamic raises `ReportSideError`, naming the side. A create whose
reads each take their own. The catalog rows come from `economy_inputs.py`
(`read_catalog_rows`): the dataset is the region's default, a re-run's
stored dataset by name (`EconomyRequest`'s `dataset_name`), or, for a
report created with `data_source` (`read_report_inputs`'s
`local_area_dataset_name`), the catalog version's local-area dataset; a
request naming two of them is refused with `ValueError` before any read. A
missing region or dataset raises `UnknownRegionError` or
`UnknownDatasetError`, a local-area dataset the catalog doesn't give both
area types raises `LocalAreaDatasetMismatchError`, and a report side's
refused policy or dynamic raises `ReportSideError`, naming the side. A create whose
commit fails after its rows were inserted raises
`EconomyRecordOutcomeUnknownError`: PostgreSQL may have applied the commit,
so the rows may exist.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@
fail_unstarted_simulation_run,
read_dataset,
read_dataset_by_id,
read_local_area_dataset,
read_model_id,
read_region,
record_report_run_call,
Expand All @@ -73,6 +74,7 @@
EconomyRequest,
EconomyRunStore,
EconomySimulationInputs,
LocalAreaDatasetMismatchError,
ReportSide,
ReportSideError,
SideRefusal,
Expand Down Expand Up @@ -221,6 +223,7 @@
"KeyedEconomySimulation",
"KeyedHouseholdReport",
"KeyedWindowYear",
"LocalAreaDatasetMismatchError",
"MissingPolicyParametersError",
"NewHouseholdJob",
"NewHouseholdReportRerun",
Expand Down Expand Up @@ -296,6 +299,7 @@
"read_economy_report",
"read_economy_simulation",
"read_household_job",
"read_local_area_dataset",
"read_missing_table_privileges",
"read_model_id",
"read_policy_reform",
Expand Down
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
"""What an economy request is read from: its catalog rows and its sides' values.

A request names a region code, a dataset name (or the region's default) and
a year, for the .py version of the default worker, and each side's stored
policy and dynamic. ``read_catalog_rows`` reads the catalog version's model,
region and dataset; ``read_report_sides`` and ``read_window_sides`` read
each side's values on that catalog version. ``EconomyRunStore`` runs these
in its transactions.
A request names a region code, a dataset name (or the region's default, or
the catalog version's local-area dataset for a report created with
``data_source``) and a year, for the .py version of the default worker, and
each side's stored policy and dynamic. ``read_catalog_rows`` reads the
catalog version's model, region and dataset; ``read_report_sides`` and
``read_window_sides`` read each side's values on that catalog version.
``EconomyRunStore`` runs these in its transactions.
"""

from __future__ import annotations
Expand All @@ -20,6 +21,7 @@
SimulationChoice,
read_dataset,
read_dataset_by_id,
read_local_area_dataset,
read_model_id,
read_region,
)
Expand Down Expand Up @@ -67,6 +69,23 @@ class UnknownDatasetError(LookupError):
"""The catalog version has no dataset with the requested name."""


class LocalAreaDatasetMismatchError(LookupError):
"""The catalog doesn't give both area types the dataset the worker routes them to.

``declared_dataset`` is the dataset the default worker declares it
calculates a report's local areas on. The catalog version's
constituency and local-authority regions use another, or one each.
"""

def __init__(self, declared_dataset: str) -> None:
"""Keep the declared dataset."""
super().__init__(
"the catalog version's constituency and local-authority regions do "
f"not all use {declared_dataset}"
)
self.declared_dataset = declared_dataset


class ReportSideError(Exception):
"""A report side's policy or dynamic is refused.

Expand Down Expand Up @@ -156,13 +175,26 @@ class BudgetWindowInputs:


def read_catalog_rows(
connection: Connection, request: EconomyRequest
connection: Connection,
request: EconomyRequest,
*,
local_area_dataset_name: str | None = None,
) -> EconomyCatalogRows:
"""Read the catalog version, model, region and dataset a request names.

Raise UnknownCatalogVersionError, UnknownRegionError or
UnknownDatasetError.
The dataset is the region's default, or the one ``request`` names (a
re-run's stored dataset), or, with ``local_area_dataset_name``, the
catalog version's local-area dataset, which must have that name (a
report created with ``data_source``). Raise ValueError, before any read,
when both name a dataset; otherwise UnknownCatalogVersionError,
UnknownRegionError, UnknownDatasetError or LocalAreaDatasetMismatchError.
"""
if local_area_dataset_name is not None and request.dataset_name is not None:
msg = (
"a catalog read takes either a dataset name or the local-area "
"dataset, not both"
)
raise ValueError(msg)
country = request.country
catalog_version_id = read_catalog_version_id(
connection,
Expand All @@ -184,19 +216,22 @@ def read_catalog_rows(
f"has no region {request.region_code}"
)
raise UnknownRegionError(msg)
dataset = (
read_dataset_by_id(
if local_area_dataset_name is not None:
dataset = read_local_area_dataset(
connection, catalog_version_id=catalog_version_id
)
if dataset is None or dataset.name != local_area_dataset_name:
raise LocalAreaDatasetMismatchError(local_area_dataset_name)
elif request.dataset_name is None:
dataset = read_dataset_by_id(
connection,
catalog_version_id=catalog_version_id,
dataset_id=region.default_dataset_id,
)
if request.dataset_name is None
else read_dataset(
connection,
catalog_version_id=catalog_version_id,
name=request.dataset_name,
else:
dataset = read_dataset(
connection, catalog_version_id=catalog_version_id, name=request.dataset_name
)
)
if dataset is None:
msg = (
f"{country}'s catalog version for .py {request.policyengine_version} "
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -30,11 +30,19 @@
from dataclasses import dataclass
from typing import TYPE_CHECKING, Any, Final

from sqlalchemy import func, insert, select, update
from sqlalchemy import (
String,
cast,
func,
insert,
select,
update,
)
from sqlalchemy.exc import IntegrityError

from policyengine_simulation_persistence.input_tables import (
MODEL_NAME_PREFIX,
UK_AREA_REGION_TYPES,
datasets,
regions,
tax_benefit_models,
Expand Down Expand Up @@ -182,6 +190,46 @@ def read_dataset_by_id(
return _dataset(connection, catalog_version_id, datasets.c.id == dataset_id)


def read_local_area_dataset(
connection: Connection, *, catalog_version_id: UUID
) -> CatalogDataset | None:
"""Return the catalog version's local-area dataset, or None when it has none.

It is the one default dataset every constituency and local-authority
region of the version uses. The Public API's catalog lists those regions
only when the .py release routes them to a dataset of their own, and
gives them that dataset, so a version with no such regions, with only
one of the two types, or with a dataset for each type has none: its
national reports calculate both area levels themselves. A report created
with ``data_source`` runs on this dataset, which may be the national
file: Stage 12 leaves the areas out of a UK-wide or nation run whenever
the release routes both types to one dataset
(``uk_area_breakdowns_omitted``).
"""
region_type = cast(regions.c.region_type, String)
rows = connection.execute(
select(region_type.label("region_type"), datasets.c.id, datasets.c.name)
.distinct()
.join(
datasets,
(datasets.c.id == regions.c.default_dataset_id)
& (
datasets.c.tax_benefit_model_version_id
== regions.c.tax_benefit_model_version_id
),
)
.where(
regions.c.tax_benefit_model_version_id == catalog_version_id,
region_type.in_(UK_AREA_REGION_TYPES),
)
).all()
region_types = {row.region_type for row in rows}
found = {CatalogDataset(id=row.id, name=row.name) for row in rows}
if region_types != set(UK_AREA_REGION_TYPES) or len(found) != 1:
return None
return found.pop()


def _dataset(
connection: Connection, catalog_version_id: UUID, condition: ColumnElement[bool]
) -> CatalogDataset | None:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
EconomyReportInputs,
EconomyRequest,
EconomySimulationInputs,
LocalAreaDatasetMismatchError,
ReportSide,
ReportSideError,
SideRefusal,
Expand Down Expand Up @@ -152,15 +153,23 @@ def read_report_inputs(
*,
baseline: SimulationChoice,
reform: SimulationChoice,
local_area_dataset_name: str | None = None,
) -> EconomyReportInputs:
"""Read a report's catalog rows and both sides' values together.

Raise UnknownCatalogVersionError, UnknownRegionError or
UnknownDatasetError, or ReportSideError naming the side whose policy
or dynamic is refused.
With ``local_area_dataset_name``, the dataset the default worker
calculates local-area reports on, the report is read on the catalog
version's local-area dataset (``read_local_area_dataset``) in place
of the region's default. Raise ValueError when ``request`` also names
a dataset; otherwise UnknownCatalogVersionError, UnknownRegionError
or UnknownDatasetError, ReportSideError naming the side whose policy
or dynamic is refused, or LocalAreaDatasetMismatchError when the
catalog version doesn't give both area types that dataset.
"""
with self._transaction() as connection:
catalog = read_catalog_rows(connection, request)
catalog = read_catalog_rows(
connection, request, local_area_dataset_name=local_area_dataset_name
)
return read_report_sides(
connection, request, catalog, baseline=baseline, reform=reform
)
Expand Down Expand Up @@ -381,6 +390,7 @@ def read_simulation(self, simulation_id: UUID) -> EconomySimulationState | None:
"EconomyRequest",
"EconomyRunStore",
"EconomySimulationInputs",
"LocalAreaDatasetMismatchError",
"ReportSide",
"ReportSideError",
"SideRefusal",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,16 @@

from typing import Final

from sqlalchemy import Column, DateTime, Integer, MetaData, String, Table, Uuid
from sqlalchemy import (
Column,
DateTime,
Enum,
Integer,
MetaData,
String,
Table,
Uuid,
)
from sqlalchemy.dialects.postgresql import JSONB

from policyengine_simulation_persistence.tables import TABLE_SCHEMA
Expand All @@ -24,6 +33,20 @@

# A country's tax-benefit model is named "policyengine-<country>".
MODEL_NAME_PREFIX: Final = "policyengine-"
# The kinds of geography a region covers, as the Public API's
# ``v2_region_type`` lists them.
REGION_TYPES: Final = (
"national",
"country",
"state",
"congressional_district",
"constituency",
"local_authority",
"city",
"place",
)
# The UK region types a report on the local-area dataset calculates.
UK_AREA_REGION_TYPES: Final = ("constituency", "local_authority")

tax_benefit_models = Table(
"tax_benefit_models",
Expand All @@ -49,6 +72,12 @@
input_metadata,
Column("id", Uuid, nullable=False),
Column("code", String(255), nullable=False),
# The enum names its schema itself, as the service tables' enums do.
Column(
"region_type",
Enum(*REGION_TYPES, name="v2_region_type", schema=TABLE_SCHEMA),
nullable=False,
),
Column("tax_benefit_model_version_id", Uuid, nullable=False),
Column("default_dataset_id", Uuid, nullable=False),
)
Expand Down
Loading
Loading