Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/c1-44-household-dynamics.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Household calculations (`POST /household-calculations`) take a `dynamic_id`, and each side of a household report takes one, as economy simulations do: the dynamic's values are applied with the policy's, the job stores it in `household_jobs.dynamic_id` and the job's state returns it, each household simulation's key names it, and a stored household report whose side names a dynamic is re-run with it.
1 change: 1 addition & 0 deletions changelog.d/c1-44-household-dynamics.changed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Every simulation, economy and household alike, refuses a dynamic stored for another country as `unknown_dynamic`, as it refuses another country's policy, and the refusal's detail names the country. The persistence models compare with API v1's tables at its commit `003ee4996`, where `dynamics` has a country.
44 changes: 26 additions & 18 deletions libs/policyengine-simulation-persistence/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,18 +114,20 @@ and they are kept apart from the models of the tables the service writes
for a household calculation, one `household_jobs` row. The service finds the
catalog version by country and .py version (`read_catalog_version_id`) and
the stored household in its country (`read_stored_household`), creates the
job `pending` with no Modal call ID and no dynamic (`insert_household_job`),
job `pending` with no Modal call ID, naming its policy and dynamic
(`insert_household_job`),
starts the household function's call, then records the call's ID
(`record_household_call`), or fails the job at once when Modal refuses the
call (`fail_unstarted_household_job`). Both updates apply only while the job
is still `pending` with no call ID, and set `updated_at` themselves. A job
naming a household, policy or catalog version that does not exist is refused
by its foreign key with SQLSTATE `23503`; `HouseholdJobReferenceError` names
the column. Each function takes an ORM session, so the caller chooses the
transactions. `HouseholdJobStore` runs them as the Simulation Entrypoint
does, each step in a session and transaction of its own: `read_inputs` reads
the catalog version, the household and the policy's reform values together,
and `create_job`, `record_call`, `fail_unstarted` and `read` each write or
naming a household, policy, dynamic or catalog version that does not exist
is refused by its foreign key with SQLSTATE `23503`;
`HouseholdJobReferenceError` names the column. Each function takes an ORM
session, so the caller chooses the transactions. `HouseholdJobStore` runs
them as the Simulation Entrypoint does, each step in a session and
transaction of its own: `read_inputs` reads the catalog version, the
household, and the policy's and dynamic's values combined
(`read_simulation_reform`) together, and `create_job`, `record_call`, `fail_unstarted` and `read` each write or
read the job. `insert_household_job` flushes the new row, so its INSERT runs
before the commit. A `create_job` whose commit fails after that flush raises
`EconomyRecordOutcomeUnknownError`, as the economy and household report
Expand Down Expand Up @@ -243,7 +245,8 @@ report and run rows (C1 · 24). These modules do it:
A household report calculates one stored household, for one year, under a
baseline and, optionally, a reform. Its report has type `household`
(`HOUSEHOLD_REPORT_TYPE`) and names the household, and each side is a
household simulation (no dataset or region) naming the side's policy.
household simulation (no dataset or region) naming the side's policy and
dynamic.
`keyed_household_records.py` finds or creates the report, its simulations
and its run by key, as an economy report's are (C1 · 24's answer 8);
`household_report_records.py` builds each side's run, `pending` with no
Expand All @@ -254,12 +257,13 @@ is refused as `EconomyRecordReferenceError`.
`HouseholdReportStore` runs the steps in ORM sessions and transactions of
their own, opened by `from_session_factory` with `bounded_session`:
`read_inputs` reads the catalog version for the .py version, the country's
model, the stored household and each side's policy values together,
raising `UnknownCatalogVersionError`, `UnknownHouseholdError` or
`ReportSideError` for the side whose policy is missing or sets parameters
the catalog version lacks; `find_or_create_report` raises
`EconomyRecordOutcomeUnknownError` when the commit fails after the rows were
inserted; and `record_report_call` and `fail_unstarted_report` use the
model, the stored household and each side's policy and dynamic values,
combined (`read_simulation_reform`), together, raising
`UnknownCatalogVersionError`, `UnknownHouseholdError` or `ReportSideError`
for the side whose policy or dynamic is missing in the country, sets
parameters the catalog version lacks, or sets one the other also sets;
`find_or_create_report` raises `EconomyRecordOutcomeUnknownError` when the
commit fails after the rows were inserted; and `record_report_call` and `fail_unstarted_report` use the
economy report run writes.

## Re-running a report or simulation
Expand Down Expand Up @@ -335,9 +339,13 @@ the parameters of the catalog version it was saved on, so a calculation on
any catalog version applies them by name; a parameter the calculation's
version lacks raises `MissingPolicyParametersError`, naming each one, and a
policy the country doesn't have raises `UnknownPolicyError`.
`read_dynamic_reform` reads a stored dynamic the same way (`UnknownDynamicError`
for none), and `combine_reforms` joins a simulation's policy and dynamic
values, refusing a parameter both set (`OverlappingReformError`).
`read_dynamic_reform` reads a stored dynamic the same way: a dynamic, like a
policy, belongs to one country, and one the country doesn't have, another
country's included, raises `UnknownDynamicError`. `combine_reforms` joins a
simulation's policy and dynamic values, refusing a parameter both set
(`OverlappingReformError`), and `read_simulation_reform` reads both and
joins them, as every simulation does: economy (`read_simulation_values`),
household job and household report side.

Each period is a range, `YYYY-MM-DD.YYYY-MM-DD`, with the stored dates read
in UTC. A value with no end date follows .py's own rule
Expand Down
2 changes: 1 addition & 1 deletion libs/policyengine-simulation-persistence/public-api-commit
Original file line number Diff line number Diff line change
@@ -1 +1 @@
63cce8273e6a2c3659392d7b1f9dfeb917672645
003ee49965a2190e0e5c78f7e946b4bb100c0f16
Original file line number Diff line number Diff line change
Expand Up @@ -162,11 +162,13 @@
MissingPolicyParametersError,
OverlappingReformError,
PolicyReform,
SimulationReform,
UnknownDynamicError,
UnknownPolicyError,
combine_reforms,
read_dynamic_reform,
read_policy_reform,
read_simulation_reform,
)
from policyengine_simulation_persistence.referenced_tables import referenced_metadata
from policyengine_simulation_persistence.result_tables import (
Expand Down Expand Up @@ -319,6 +321,7 @@
"SimulationChoice",
"SimulationDefinition",
"SimulationInputs",
"SimulationReform",
"SimulationRun",
"SimulationRunState",
"SimulationStatus",
Expand Down Expand Up @@ -383,6 +386,7 @@
"read_region",
"read_report_definition",
"read_simulation_definition",
"read_simulation_reform",
"read_stored_household",
"record_household_call",
"record_report_run_call",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,9 +36,7 @@
OverlappingReformError,
UnknownDynamicError,
UnknownPolicyError,
combine_reforms,
read_dynamic_reform,
read_policy_reform,
read_simulation_reform,
)

if TYPE_CHECKING:
Expand Down Expand Up @@ -253,28 +251,14 @@ def read_simulation_values(
choice: SimulationChoice,
) -> SimulationInputs:
"""Read one simulation's policy and dynamic values, combined."""
policy_reform = (
{}
if choice.policy_id is None
else read_policy_reform(
session,
country=country,
policy_id=choice.policy_id,
catalog_version_id=catalog_version_id,
)
)
dynamic_reform = (
{}
if choice.dynamic_id is None
else read_dynamic_reform(
session,
dynamic_id=choice.dynamic_id,
catalog_version_id=catalog_version_id,
)
)
return SimulationInputs(
choice=choice, reform=combine_reforms(policy_reform, dynamic_reform)
reform = read_simulation_reform(
session,
country=country,
policy_id=choice.policy_id,
dynamic_id=choice.dynamic_id,
catalog_version_id=catalog_version_id,
)
return SimulationInputs(choice=choice, reform=reform.values)


def read_side_values(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@
A household calculation is one ``household_jobs`` row, which the service
creates and a household function on Modal calculates. The service reads the
stored household and the catalog version from API v1's tables, creates
the job ``pending`` with no Modal call ID, starts the call, then records the
the job ``pending`` with no Modal call ID, naming the stored policy and
dynamic it applies, starts the call, then records the
call's ID, or fails the job at once when Modal refuses the call. Both updates
apply only while the job is still ``pending`` with no call ID: the household
function may already have claimed it, and the stuck-run sweep fails a pending
Expand Down Expand Up @@ -74,13 +75,16 @@ class StoredHousehold:
class NewHouseholdJob:
"""A household job to create, ``pending`` and with no Modal call yet.

``request_data`` holds the request as the service received it.
``policy_id`` and ``dynamic_id`` name the stored policy and dynamic it
applies, each None for none. ``request_data`` holds the request as the
service received it.
"""

id: UUID
country: str
household_id: UUID
policy_id: UUID | None
dynamic_id: UUID | None
year: int
request_data: dict[str, Any]
tax_benefit_model_version_id: UUID
Expand All @@ -102,6 +106,7 @@ class HouseholdJobRecord:
country: str
household_id: UUID
policy_id: UUID | None
dynamic_id: UUID | None
year: int
request_data: dict[str, Any]
tax_benefit_model_version_id: UUID
Expand Down Expand Up @@ -164,7 +169,7 @@ def read_stored_household(


def insert_household_job(session: Session, job: NewHouseholdJob) -> None:
"""Create the job ``pending``, with no Modal call ID and no dynamic.
"""Create the job ``pending``, with no Modal call ID.

The row is flushed here, so a failure of its INSERT is raised here and
never at the commit. Raise HouseholdJobReferenceError when the database
Expand All @@ -177,7 +182,7 @@ def insert_household_job(session: Session, job: NewHouseholdJob) -> None:
country=job.country,
household_id=job.household_id,
policy_id=job.policy_id,
dynamic_id=None,
dynamic_id=job.dynamic_id,
year=job.year,
request_data=job.request_data,
tax_benefit_model_version_id=job.tax_benefit_model_version_id,
Expand Down Expand Up @@ -264,6 +269,7 @@ def read_household_job(session: Session, job_id: UUID) -> HouseholdJobRecord | N
HouseholdJob.country,
HouseholdJob.household_id,
HouseholdJob.policy_id,
HouseholdJob.dynamic_id,
HouseholdJob.year,
HouseholdJob.request_data,
HouseholdJob.tax_benefit_model_version_id,
Expand All @@ -285,6 +291,7 @@ def read_household_job(session: Session, job_id: UUID) -> HouseholdJobRecord | N
country=row.country,
household_id=row.household_id,
policy_id=row.policy_id,
dynamic_id=row.dynamic_id,
year=row.year,
request_data=row.request_data,
tax_benefit_model_version_id=row.tax_benefit_model_version_id,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@
read_stored_household,
record_household_call,
)
from policyengine_simulation_persistence.policy_reform import read_policy_reform
from policyengine_simulation_persistence.policy_reform import read_simulation_reform
from policyengine_simulation_persistence.store_transactions import (
TransactionFactory,
bounded_transactions,
Expand All @@ -38,7 +38,7 @@
NewHouseholdJob,
StoredHousehold,
)
from policyengine_simulation_persistence.policy_reform import PolicyReform
from policyengine_simulation_persistence.policy_reform import SimulationReform
from policyengine_simulation_persistence.tables import HouseholdJobStatus


Expand All @@ -55,13 +55,13 @@ class HouseholdCalculationInputs:
"""What a household calculation is read from.

``catalog_version_id`` is the country's catalog version for the .py
version the calculation runs on; ``reform`` holds the policy's values by
parameter name and period, empty for current law.
version the calculation runs on. ``reform`` holds the policy's and the
dynamic's values together, and names the parameters the dynamic sets.
"""

catalog_version_id: UUID
household: StoredHousehold
reform: PolicyReform
reform: SimulationReform


class HouseholdJobStore:
Expand Down Expand Up @@ -92,12 +92,17 @@ def read_inputs(
country: str,
household_id: UUID,
policy_id: UUID | None,
dynamic_id: UUID | None,
policyengine_version: str,
) -> HouseholdCalculationInputs:
"""Read the catalog version, household and policy in one transaction.

Raise UnknownCatalogVersionError, UnknownHouseholdError, or the
policy reader's UnknownPolicyError or MissingPolicyParametersError.
"""Read the catalog version, household, policy and dynamic together.

The policy's and the dynamic's values are combined, as an economy
simulation's are (``read_simulation_reform``). Raise
UnknownCatalogVersionError, UnknownHouseholdError, or the readers'
UnknownPolicyError, UnknownDynamicError (a dynamic of another
country included), MissingPolicyParametersError or
OverlappingReformError.
"""
with self._transaction() as session:
catalog_version_id = read_catalog_version_id(
Expand All @@ -112,15 +117,12 @@ def read_inputs(
if household is None:
msg = f"{country} has no household {household_id}"
raise UnknownHouseholdError(msg)
reform = (
{}
if policy_id is None
else read_policy_reform(
session,
country=country,
policy_id=policy_id,
catalog_version_id=catalog_version_id,
)
reform = read_simulation_reform(
session,
country=country,
policy_id=policy_id,
dynamic_id=dynamic_id,
catalog_version_id=catalog_version_id,
)
return HouseholdCalculationInputs(
catalog_version_id=catalog_version_id, household=household, reform=reform
Expand Down
Loading