Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/c1-42-drop-report-columns.removed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The simulation service and the v2 Modal runtime no longer map `reports.label`, `description` and `user_id`, `report_runs.trigger` and its four cliff columns, or `budget_window_runs.trigger`, which API v1 drops. They work on a database with or without those columns.
5 changes: 2 additions & 3 deletions docs/migration/stage-12-modal-worker-foundation.md
Original file line number Diff line number Diff line change
Expand Up @@ -351,7 +351,7 @@ report's run on the current versions the same way, or else creates the
report's new run naming each side's simulation run on those versions, found
or inserted (C1 · 24's answers 3 and 8); a forced re-run, and every re-run
of a report created before keys, creates an extra run on a new run of each
side, all marked forced. No new run names a trigger (answer 6). The new runs record the
side, all marked forced. The new runs record the
catalog version, dataset and region of the current .py version's catalog;
the `simulations` and `reports` rows are unchanged. A budget-window report's
re-run joins or creates a `budget_window_runs` row with a report run per
Expand Down Expand Up @@ -414,8 +414,7 @@ output reads from either side (`self_employment_income`,
its region-group split. Cliffs are a calculation of each simulation run of
their own (above). A report run asking for them (`include_cliffs: true`) still
fails before any worker starts, with `cliff_impacts_unavailable`, and touches
none of its simulation runs, and its completion never writes `report_runs`'
four cliff columns. The
none of its simulation runs. The
coordinator gives each worker its simulation's
stored plan, and keeps the report's own plan, from its pair, for what the
aggregation reads. A reused or waited-for file must hold what the report reads
Expand Down
27 changes: 12 additions & 15 deletions libs/policyengine-simulation-persistence/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,8 @@ or `report_runs.spm_provenance` is stored as SQL `NULL`, not JSON `null`, so
a query for `IS NULL` finds a run with no result.

Each service writes only its own tables, and the foreign keys between them
stay: 20 keys name rows the Public API writes in its users, policies,
dynamics, households, datasets, regions and model catalog tables. PostgreSQL
stay: 19 keys name rows API v1 writes in its policies, dynamics, households,
datasets, regions and model catalog tables. PostgreSQL
checks each of them on insert, as the referenced table's owner, so the
service's role needs no privilege on those tables for it, and a row naming
one that doesn't exist fails with SQLSTATE `23503`. `referenced_tables.py`
Expand Down Expand Up @@ -242,9 +242,7 @@ report and run rows (C1 · 24). These modules do it:
runs, found or inserted in simulation ID order, and each year's report
run (`insert_window_years`, which the window re-runs share).
`EconomyRunStore.find_or_create_budget_window` runs it in a transaction
of its own. No keyed create writes a label, description, user or trigger
(C1 · 24's answer 6): each user's saved report and report-run record hold
them, in the Public API.
of its own.

## Household reports

Expand Down Expand Up @@ -284,8 +282,8 @@ version, dataset and region it calculates on.
each simulation's policy and dynamic.
- `insert_report_rerun` joins the report's run on the re-run's versions
that hasn't failed and wasn't forced (pending, running or succeeded),
creating nothing, or else creates the report's new run, naming no
trigger (answer 6), each side joining its simulation's run on those versions that hasn't
creating nothing, or else creates the report's new run, each side joining
its simulation's run on those versions that hasn't
failed and wasn't forced, in any state, or inserting one
(`find_or_insert_side_runs`, as a keyed create does). A report run another
request inserts after the lookup is joined, and a savepoint takes back
Expand All @@ -310,8 +308,8 @@ re-run's versions that hasn't failed and isn't a recalculation, with its
year runs, or else inserts the window run and each year's report run on
its sides' runs, found or inserted. One that recalculates is a forced
re-run, never refused, an extra window run whose year runs and simulation
runs are marked `forced`. Either records the window run's .py version and
no trigger, and `EconomyRunStore.create_window_rerun` returns the window
runs are marked `forced`. Either records the window run's .py version, and
`EconomyRunStore.create_window_rerun` returns the window
run and years it ended on.

A household report's definition also names its household (`household_id`).
Expand All @@ -320,7 +318,7 @@ A household report's definition also names its household (`household_id`).
report's re-run is, it takes the same lock and joins the report's run on
the re-run's versions, or else each side joins its own household
simulation's run on those versions (no dataset or region, the report's
year) or inserts one, under a new report run (no trigger). A forced
year) or inserts one, under a new report run. A forced
one, as every re-run of a household report without keys is (interim answer
B), takes no lock, is never refused, and marks its report run and a new run
of each side `forced`. `HouseholdReportStore.create_report_rerun` runs it
Expand Down Expand Up @@ -392,8 +390,8 @@ the service writes, as the service's role; `test_modal_worker_writes`, as the
Modal worker role, checks the role's privileges, records a run's state in
each run table, inserts a row in every result table and reads back that a
`None` JSON result is SQL `NULL`. Another test, outside the rule, requires the modelled
tables' foreign keys to the Public API's users, policies, dynamics,
households, datasets, regions and model catalog tables to be exactly the 20
tables' foreign keys to API v1's policies, dynamics, households, datasets,
regions and model catalog tables to be exactly the 19
in `tests/public_api_keys.py`, each by its own table, its name and the table
it names, so a key dropped, added, renamed, moved or pointed at another table
shows even if the models follow it. The job then gives the Modal worker role
Expand Down Expand Up @@ -464,9 +462,8 @@ run's and cliff calculation's state (`status`, `modal_call_id`,
run has no call of its own) and of the results its own row holds: a report
run's `spm_provenance` and `data_version`, a simulation run's output file and
`household_result`, a household job's `result`, and a cliff calculation's
`cliff_gap` and `cliff_share`. It no longer writes a report run's four cliff
columns, which API v1 is to drop: a report run's cliff impact is its
two simulation runs' cliff results. The same privilege manifest grants them;
`cliff_gap` and `cliff_share`. A report run's cliff impact is its two
simulation runs' cliff results. API v1's manifest grants these privileges;
`test_modal_worker_writes` checks that in CI.

`missing_table_privileges` returns each one the connected role lacks, ordered
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -209,7 +209,6 @@
Report,
ReportRun,
ReportRunStatus,
ReportRunTrigger,
Simulation,
SimulationRun,
SimulationStatus,
Expand Down Expand Up @@ -306,7 +305,6 @@
"ReportRun",
"ReportRunState",
"ReportRunStatus",
"ReportRunTrigger",
"ReportSide",
"ReportSideError",
"RerunRecordDeletedError",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -112,8 +112,6 @@
# failure and times.
MODAL_WORKER_UPDATABLE_COLUMNS: Final[Mapping[str, frozenset[str]]] = MappingProxyType(
{
# A report run's cliff impact is its simulation runs' cliff results,
# so the role never writes its four cliff columns.
"report_runs": _RUN_STATE | {"spm_provenance", "data_version"},
"simulation_runs": _RUN_STATE
| {"output_uri", "output_sha256", "household_result"},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,7 @@
- creates, in one transaction, the new run and each report side's run of
its own simulation, all ``pending`` with no Modal call ID; an unforced
economy or household report re-run's side joins its simulation's run on
the current versions instead, when there is one. No new run names a
trigger (answer 6): each user's report-run record holds it. A budget
the current versions instead, when there is one. A budget
window's new run is a ``budget_window_runs``
row with a report run per year, each year on the simulations its report
run in the report's newest window run names. A household report's new
Expand Down Expand Up @@ -348,10 +347,7 @@ def insert_report_rerun(session: Session, rerun: NewReportRerun) -> FoundReportR
def _rerun_report_run_row(
*, run_id: UUID, report_id: UUID, versions: RunVersions
) -> dict[str, Any]:
"""Return a re-run's report run as inserted, without its sides' runs.

It names no trigger: each user's report-run record holds it (answer 6).
"""
"""Return a re-run's report run as inserted, without its sides' runs."""
return {
"id": run_id,
"report_id": report_id,
Expand Down Expand Up @@ -404,8 +400,8 @@ def insert_window_rerun(session: Session, rerun: NewWindowRerun) -> FoundBudgetW
window's create does (C1 · 24's answer 8). A forced re-run
(``recalculate``) takes no lock, is never refused, and inserts an extra
window run on new runs of every year, all marked forced. Every new run
is pending with no Modal call; the window run records its .py version
and no trigger (answer 6). Return the window run and its years, which
is pending with no Modal call; the window run records its .py version.
Return the window run and its years, which
only when this call inserted the window run need their first years
started. Raise as ``insert_report_rerun`` does.
"""
Expand Down Expand Up @@ -495,7 +491,7 @@ def insert_household_report_rerun(
) -> FoundReportRun:
"""Join the household report's run on the re-run's versions, or create it.

The report run names no budget-window run and no trigger (answer 6);
The report run names no budget-window run;
each side's run is of the report's own household simulation, with no
Modal call: the service then starts one household function call for a
report run it created. Unforced (a keyed report's re-run), take the
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -88,8 +88,6 @@ class KeyedEconomyReport:
current catalog version's rows with the definition's dataset name and
region code. ``inputs`` is what ``reports.inputs`` stores: the SPM
selection when the request gave one, and no other option (answer 4).
No label, description, user or trigger is stored on the shared rows
(answer 6): each user's saved report and report-run record hold them.
"""

definition: EconomyReportDefinition
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -73,8 +73,7 @@ class KeyedHouseholdReport:

``reform`` is None exactly when the definition has no reform (a report
of the baseline alone). The runs calculate on the catalog version
``tax_benefit_model_version_id``. No label, description, user or
trigger is stored on the shared rows (answer 6).
``tax_benefit_model_version_id``.
"""

definition: HouseholdReportDefinition
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,7 @@

A year's simulation run may be a single-year report's, or another window's,
pending, running or succeeded (answers 4 and 5). Rows keep random IDs: the
IDs a request proposes are used only for the rows it inserts. No per-user
column is written (answer 6).
IDs a request proposes are used only for the rows it inserts.
"""

from __future__ import annotations
Expand Down Expand Up @@ -247,8 +246,7 @@ def window_run_row(
) -> dict[str, Any]:
"""Return a window's new run as inserted, pending, with its .py version.

``recalculate`` marks a forced re-run's, which nothing joins. It names
no trigger: each user's report-run record holds it (answer 6).
``recalculate`` marks a forced re-run's, which nothing joins.
"""
return {
"id": run_id,
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
"""API v1's own tables that the simulation service's rows name, by ``id`` alone.

Some keys of the service's tables name rows API v1 writes: users, policies,
dynamics, households, datasets, regions and the model catalog. PostgreSQL checks
Some keys of the service's tables name rows API v1 writes: policies, dynamics,
households, datasets, regions and the model catalog. PostgreSQL checks
each of those keys when a row is inserted, as the referenced table's owner, so
the service's role needs no privilege for it, and a row naming one that doesn't
exist fails with SQLSTATE 23503. These classes give SQLAlchemy each key's
Expand Down Expand Up @@ -86,13 +86,4 @@ class ReferencedTaxBenefitModel(ReferencedBase):
id: Mapped[UUID] = mapped_column(Uuid)


class ReferencedUser(ReferencedBase):
"""The ``id`` of a row of ``users``."""

__tablename__ = "users"
__table_args__: ClassVar = (PrimaryKeyConstraint("id", name="pk_users"),)

id: Mapped[UUID] = mapped_column(Uuid)


referenced_metadata = ReferencedBase.metadata
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,6 @@
ReferencedRegion,
ReferencedTaxBenefitModel,
ReferencedTaxBenefitModelVersion,
ReferencedUser,
)

if TYPE_CHECKING:
Expand Down Expand Up @@ -91,14 +90,6 @@ class ReportRunStatus(StrEnum):
FAILED = "failed"


class ReportRunTrigger(StrEnum):
"""What started a report run or a budget window's calculation."""

INITIAL = "initial"
MANUAL = "manual"
SYSTEM = "system"


class HouseholdJobStatus(StrEnum):
"""Lifecycle state of a household job."""

Expand Down Expand Up @@ -276,12 +267,6 @@ class Report(ServiceBase):
__tablename__ = "reports"
__table_args__: ClassVar = (
PrimaryKeyConstraint("id", name="pk_reports"),
_api_v1_key(
"user_id",
ReferencedUser.id,
ondelete="SET NULL",
name="fk_reports_user_id_users",
),
_api_v1_key(
"tax_benefit_model_id",
ReferencedTaxBenefitModel.id,
Expand Down Expand Up @@ -348,20 +333,13 @@ class Report(ServiceBase):
Index("ix_reports_dataset_independent_key", "dataset_independent_key"),
Index("ix_reports_country_type_created_at", "country", "type", "created_at"),
Index("ix_reports_tax_benefit_model_id", "tax_benefit_model_id"),
# Deleting a user sets its reports' user_id to null.
Index("ix_reports_user_id", "user_id"),
)

id: Mapped[UUID] = identifier_column()
created_at: Mapped[datetime] = server_timestamp_column()
updated_at: Mapped[datetime] = server_timestamp_column()
# Per-user columns the service no longer writes (C1 · 24's answer 6): a
# report's name and description live on each user's saved report.
label: Mapped[str | None] = mapped_column(String(255))
description: Mapped[str | None] = mapped_column(String)
country: Mapped[str] = mapped_column(String(16))
type: Mapped[str | None] = mapped_column(String(128))
user_id: Mapped[UUID | None] = mapped_column(Uuid)
tax_benefit_model_id: Mapped[UUID] = mapped_column(Uuid)
policy_id: Mapped[UUID | None] = mapped_column(Uuid)
baseline_simulation_id: Mapped[UUID | None] = mapped_column(Uuid)
Expand Down Expand Up @@ -528,10 +506,6 @@ class BudgetWindowRun(ServiceBase):
# The .py version the window calculates on, so a window run is found by
# its report and version set (answer 8); null on runs before C1 · 27.
policyengine_version: Mapped[str | None] = mapped_column(String(128))
# Per-request, so the service no longer writes it (answer 6).
trigger: Mapped[ReportRunTrigger | None] = mapped_column(
enum_type(ReportRunTrigger, "v2_report_run_trigger")
)
# The window's forced marker: a recalculating run is never joined.
recalculate: Mapped[bool] = mapped_column(Boolean)
status: Mapped[ReportRunStatus] = mapped_column(
Expand Down Expand Up @@ -591,12 +565,6 @@ class ReportRun(ServiceBase):
"budget_window_run_id IS NULL OR year IS NOT NULL",
name="ck_report_runs_budget_window_year",
),
CheckConstraint(
"(baseline_cliff_gap IS NULL) = (baseline_cliff_share IS NULL) AND "
"(baseline_cliff_gap IS NULL) = (reform_cliff_gap IS NULL) AND "
"(baseline_cliff_gap IS NULL) = (reform_cliff_share IS NULL)",
name="ck_report_runs_cliff_impact_complete",
),
_completion_required("report_runs"),
_year_between("report_runs", nullable=True),
Index(
Expand Down Expand Up @@ -638,10 +606,6 @@ class ReportRun(ServiceBase):
status: Mapped[ReportRunStatus] = mapped_column(
enum_type(ReportRunStatus, "v2_report_run_status")
)
# Per-request, so the service no longer writes it (answer 6).
trigger: Mapped[ReportRunTrigger | None] = mapped_column(
enum_type(ReportRunTrigger, "v2_report_run_trigger")
)
idempotency_key: Mapped[UUID | None] = mapped_column(Uuid)
started_at: Mapped[datetime | None] = optional_timestamp_column()
completed_at: Mapped[datetime | None] = optional_timestamp_column()
Expand All @@ -655,11 +619,6 @@ class ReportRun(ServiceBase):
budget_window_run_id: Mapped[UUID | None] = mapped_column(Uuid)
year: Mapped[int | None] = mapped_column(Integer)
modal_call_id: Mapped[str | None] = mapped_column(String(MODAL_CALL_ID_LENGTH))
# Cliff impact: all four values or none.
baseline_cliff_gap: Mapped[float | None] = mapped_column(Float)
baseline_cliff_share: Mapped[float | None] = mapped_column(Float)
reform_cliff_gap: Mapped[float | None] = mapped_column(Float)
reform_cliff_share: Mapped[float | None] = mapped_column(Float)
spm_provenance: Mapped[dict[str, Any] | None] = mapped_column(
JSONB(none_as_null=True)
)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,6 @@
Report,
ReportRun,
ReportRunStatus,
ReportRunTrigger,
RunVersions,
Simulation,
SimulationChoice,
Expand Down Expand Up @@ -243,13 +242,12 @@ def insert_run_of(

@dataclass(frozen=True)
class ReportRunOutcome:
"""How a stored report run ended, its versions and trigger, and its options."""
"""How a stored report run ended, its versions, and its options."""

status: ReportRunStatus = ReportRunStatus.SUCCEEDED
inputs: dict[str, Any] | None = None
error_message: str | None = None
versions: RunVersions = VERSIONS
trigger: ReportRunTrigger = ReportRunTrigger.INITIAL


def insert_report_run_naming(
Expand Down Expand Up @@ -287,7 +285,6 @@ def insert_report_of( # noqa: PLR0913 # the report's sides and what else it ma
connection.execute(
insert(Report).values(
id=report_id,
label="An earlier report",
country=economy.catalog.country,
type="economy",
tax_benefit_model_id=economy.catalog.model_id,
Expand Down Expand Up @@ -327,7 +324,6 @@ def insert_report_run_of(
country_package_version=outcome.versions.country_package_version,
policyengine_version=outcome.versions.policyengine_version,
status=outcome.status,
trigger=outcome.trigger,
completed_at=COMPLETED_AT if finished else None,
error_message=outcome.error_message,
)
Expand Down
Loading