Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/scripts/test_alembic_v2_lifecycle.sh
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,5 @@ uv run --frozen --extra dev pytest -q \
tests/integration/test_alembic_v2_user_types.py \
tests/integration/test_alembic_v2_withheld_area_codes.py \
tests/integration/test_alembic_v2_shared_report_keys.py \
tests/integration/test_alembic_v2_cliff_impacts.py
tests/integration/test_alembic_v2_cliff_impacts.py \
tests/integration/test_alembic_v2_report_column_drops.py
1 change: 1 addition & 0 deletions changelog.d/drop-report-per-user-columns.removed.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
The v2 database no longer stores a label, description or user on reports, a trigger on report runs or budget-window runs, or cliff results on report runs. A report's name and description are on each user's saved report, each user's trigger is on their record of the run, and cliffs are each simulation run's cliff results.
80 changes: 75 additions & 5 deletions docs/engineering/skills/alembic-migrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -457,15 +457,18 @@ it publishes that version, with nothing to restore it. Downgrading past the
revision is therefore safe only while no version is pruned (no
`tax_benefit_model_versions` row has `catalog_values_pruned_at` set).
From the head, such a downgrade first runs the later revisions' downgrades:
`4e30076d4b9f`'s adds the dropped report columns back empty (see "API v2 report
column drops"),
`bc334f9877b7`'s drops `cliff_impacts` with every simulation run's cliff
results (see "API v2 simulation-run cliff impacts"),
`9d1d12a7ac58`'s needs `dynamics` empty (see "API v2 dynamics stored like
policies"),
`f519a77ba122`'s drops `user_invitations` and needs no recorded change with
source `invitation` (see "API v2 user invitations"),
`94dc43079186`'s drops every user's report-run record, the simulation and
report keys and the window runs' .py versions, and needs every report to have
a label and every run a trigger (see "API v2 shared reports and user report-run records"),
report keys and the window runs' .py versions, and needs no report or report
run to exist, since `4e30076d4b9f`'s restored their labels and triggers empty
(see "API v2 shared reports and user report-run records"),
`970be5d29283`'s drops `user_type_changes` with every recorded change (see
"API v2 user type changes"), `617f531f1506`'s needs no user to be an author
or an admin (see "API v2 user types"), and `1356f658347d`'s needs
Expand Down Expand Up @@ -1128,9 +1131,8 @@ them and update their state and result columns, never insert them; the
runtime role may read them; neither simulation role may change the run a row
names or delete a row. The reconcile step after each schema upgrade grants
them. Until the simulation API writes the table, no run has a cliff
calculation, and its cliffs stay refused. `report_runs`' four cliff columns
stay until a later revision drops them, once API v1 reads cliffs from
here.
calculation, and its cliffs stay refused. Revision `4e30076d4b9f` drops
`report_runs`' four cliff columns.

It has one post-generation correction, commented in the revision: `status`
reuses the baseline's `v2_simulation_status` type
Expand All @@ -1148,3 +1150,71 @@ else changes, that the downgrade leaves the type and an upgrade works again,
and the index, checks and delete rule;
`tests/integration/test_v2_simulation_role_privileges.py` runs the writes
and reads as the roles that do them.

### API v2 report column drops

Revision `4e30076d4b9f` (parent `bc334f9877b7`) was autogenerated, with its ID
given by `--rev-id`, after these columns left the models. Reports, report runs
and budget-window runs are shared by every request that joins them, so they
hold nothing about one caller, and a report run's cliffs are its two simulation
runs' results in `cliff_impacts`. It drops:

- `reports.label`, `description` and `user_id`, with
`fk_reports_user_id_users` and `ix_reports_user_id`. A report's name and
description are on each user's saved report, and deleting a user no longer
touches `reports`.
- `report_runs.trigger` and `budget_window_runs.trigger`. Each user's trigger
is on their `user_report_run_associations` row, which still uses the
`v2_report_run_trigger` type, and `/v2/reports` run items no longer have a
`trigger` field.
- `report_runs`' `baseline_cliff_gap`, `baseline_cliff_share`,
`reform_cliff_gap` and `reform_cliff_share`, with
`ck_report_runs_cliff_impact_complete`. The Modal worker's column privileges
on them go too.

It has one post-generation correction, commented in the revision: autogenerate
strips the first and last parenthesis of the reflected check condition, which
leaves the downgrade's `create_check_constraint` with invalid SQL.

**What must be in production first.** The deploy upgrades the schema before the
new Cloud Run revision takes traffic (see "CI/CD behavior"), so the previous
release serves on this schema, for longer if promotion fails:

- An API v1 release whose `Report`, `ReportRun` and `BudgetWindowRun` no longer
map these columns. Report reads load whole rows, and a whole-row load names
every mapped column, so otherwise every `/v2/reports` read fails.
- A simulation API release whose v2 Modal runtime no longer writes the four
cliff columns. Otherwise every economy report run's completion fails.

The simulation API's table models must then lose the same columns. Until they
do, its comparison with API v1's tables fails; requests are not affected.

**The dropped values are not kept.** Nothing copies them, since these rows name
no user to copy them to, and the downgrade can't restore them. One merge takes
the revision through staging and production in a single release run, so before
merging, an operator exports what those rows hold in both environments with
these read-only queries, and keeps the result outside the repository, because
labels and descriptions are users' text:

```sql
SELECT id, created_at, label, description, user_id FROM reports
WHERE label IS NOT NULL OR description IS NOT NULL OR user_id IS NOT NULL;
SELECT id, report_id, created_at, trigger, baseline_cliff_gap,
baseline_cliff_share, reform_cliff_gap, reform_cliff_share
FROM report_runs
WHERE trigger IS NOT NULL OR baseline_cliff_gap IS NOT NULL;
SELECT id, report_id, created_at, trigger FROM budget_window_runs
WHERE trigger IS NOT NULL;
```

The downgrade adds the columns, key, index and check back with every value
null. The Modal worker's `UPDATE` on the four cliff columns doesn't come back
with them: it returns only when `reconcile_v2_privileges.py apply` runs from a
`privileges.py` that names them. Continuing below `bc334f9877b7` to
`94dc43079186`'s parent fails while any report or run exists, because that
downgrade needs every report to have a label and every run a trigger.

`tests/integration/test_alembic_v2_report_column_drops.py` compares the revision
and its downgrade with the parent schema built from the base, and checks that
rows survive the upgrade, that the downgrade restores no values, and that the
deploy's privilege steps pass.
6 changes: 4 additions & 2 deletions docs/migration/stage-12a-user-login.md
Original file line number Diff line number Diff line change
Expand Up @@ -770,15 +770,17 @@ admin; `scripts/set_v2_user_type.py` makes another admin afterwards.
`users.primary_country`, so it fails while legacy-only rows, any simulation
or report associations, or any unauthenticated user exist. Removing them
first deletes users' saved work. From the head, the later revisions'
downgrades run first: the cliff impacts revision (`bc334f9877b7`) drops
downgrades run first: the report column drops revision (`4e30076d4b9f`) adds
the dropped report columns back empty, the cliff impacts revision (`bc334f9877b7`) drops
every simulation run's cliff results, the dynamics revision
(`9d1d12a7ac58`) needs `dynamics` empty, the user invitations revision
(`f519a77ba122`) drops
every invitation and dismissal and needs no change with source
`invitation`, the shared reports revision (`94dc43079186`) drops
every user's report-run record, every simulation and report key, the runs'
`forced` markers, the window runs' .py versions and the one-unfailed-run
indexes, and needs every report to have a label and every run a trigger, the
indexes, and needs no report or report run to exist, since the report column
drops revision restored their labels and triggers empty, the
user type changes revision (`970be5d29283`) drops every recorded change,
the user types revision (`617f531f1506`) needs no
author or admin, and the simulation runs revision (`1356f658347d`) needs
Expand Down
8 changes: 5 additions & 3 deletions docs/migration/stage-9-v2-metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -835,16 +835,18 @@ pruning").

A schema downgrade past revision `334859ef4fec` drops
`catalog_last_published_at` and `catalog_values_pruned_at`. From the head,
it first runs the later revisions' downgrades: `bc334f9877b7`'s drops
it first runs the later revisions' downgrades: `4e30076d4b9f`'s adds the
dropped report columns back empty, `bc334f9877b7`'s drops
`cliff_impacts` with every simulation run's cliff results, `9d1d12a7ac58`'s
needs `dynamics` empty, `f519a77ba122`'s
drops `user_invitations` and needs no recorded change with source
`invitation`,
`94dc43079186`'s drops
`user_report_run_associations` with every user's report-run record, the
simulation and report keys, the runs' `forced` markers, the window runs' .py
versions and the one-unfailed-run indexes, and needs every report to have a
label and every run a trigger, `970be5d29283`'s drops `user_type_changes` with
versions and the one-unfailed-run indexes, and needs no report or report run
to exist, since `4e30076d4b9f`'s restored their labels and triggers empty,
`970be5d29283`'s drops `user_type_changes` with
every recorded change, `617f531f1506`'s needs no
user to be an author or an admin, and `1356f658347d`'s needs `simulations`
empty. A pruned version
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
"""drop per-user and cliff columns from report rows

Revision ID: 4e30076d4b9f
Revises: bc334f9877b7
Create Date: 2026-11-19 02:25:37.425693
Generation: uv run alembic -c alembic-v2.ini revision --autogenerate --rev-id 4e30076d4b9f

One post-generation correction, commented where it is made: the downgrade's
check condition gets back the first and last parenthesis autogenerate strips.

Reports, report runs and budget-window runs are shared by every request that
joins them, so they hold nothing about one caller, and a report run's cliffs
are its simulation runs' results in cliff_impacts. This drops reports.label,
description and user_id, report_runs.trigger and its four cliff columns, and
budget_window_runs.trigger, with their foreign key, index and check.

The dropped values are not recoverable: the downgrade adds the columns back
empty. Export them first if they are wanted. Every release still serving when
this runs must already leave these columns out of its models and statements.
"""

from typing import Sequence, Union

from alembic import op
import sqlalchemy as sa
import sqlmodel
from sqlalchemy.dialects import postgresql

revision: str = '4e30076d4b9f'
down_revision: Union[str, None] = 'bc334f9877b7'
branch_labels: Union[str, Sequence[str], None] = None
depends_on: Union[str, Sequence[str], None] = None


def upgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.drop_column('budget_window_runs', 'trigger')
op.drop_constraint(op.f('ck_report_runs_cliff_impact_complete'), 'report_runs', type_='check')
op.drop_column('report_runs', 'baseline_cliff_share')
op.drop_column('report_runs', 'trigger')
op.drop_column('report_runs', 'reform_cliff_gap')
op.drop_column('report_runs', 'reform_cliff_share')
op.drop_column('report_runs', 'baseline_cliff_gap')
op.drop_index(op.f('ix_reports_user_id'), table_name='reports')
op.drop_constraint(op.f('fk_reports_user_id_users'), 'reports', type_='foreignkey')
op.drop_column('reports', 'description')
op.drop_column('reports', 'label')
op.drop_column('reports', 'user_id')
# ### end Alembic commands ###


def downgrade() -> None:
# ### commands auto generated by Alembic - please adjust! ###
op.add_column('reports', sa.Column('user_id', sa.UUID(), autoincrement=False, nullable=True))
op.add_column('reports', sa.Column('label', sa.VARCHAR(length=255), autoincrement=False, nullable=True))
op.add_column('reports', sa.Column('description', sa.VARCHAR(), autoincrement=False, nullable=True))
op.create_foreign_key(op.f('fk_reports_user_id_users'), 'reports', 'users', ['user_id'], ['id'], ondelete='SET NULL')
op.create_index(op.f('ix_reports_user_id'), 'reports', ['user_id'], unique=False)
op.add_column('report_runs', sa.Column('baseline_cliff_gap', sa.DOUBLE_PRECISION(precision=53), autoincrement=False, nullable=True))
op.add_column('report_runs', sa.Column('reform_cliff_share', sa.DOUBLE_PRECISION(precision=53), autoincrement=False, nullable=True))
op.add_column('report_runs', sa.Column('reform_cliff_gap', sa.DOUBLE_PRECISION(precision=53), autoincrement=False, nullable=True))
op.add_column('report_runs', sa.Column('trigger', postgresql.ENUM('initial', 'manual', 'system', name='v2_report_run_trigger'), autoincrement=False, nullable=True))
op.add_column('report_runs', sa.Column('baseline_cliff_share', sa.DOUBLE_PRECISION(precision=53), autoincrement=False, nullable=True))
# Post-generation correction: autogenerate strips the first and last
# parenthesis of the reflected condition, which leaves invalid SQL.
op.create_check_constraint(op.f('ck_report_runs_cliff_impact_complete'), 'report_runs', '(baseline_cliff_gap IS NULL) = (baseline_cliff_share IS NULL) AND (baseline_cliff_gap IS NULL) = (reform_cliff_gap IS NULL) AND (baseline_cliff_gap IS NULL) = (reform_cliff_share IS NULL)')
op.add_column('budget_window_runs', sa.Column('trigger', postgresql.ENUM('initial', 'manual', 'system', name='v2_report_run_trigger'), autoincrement=False, nullable=True))
# ### end Alembic commands ###
2 changes: 1 addition & 1 deletion policyengine_api/data/v2/migration_target.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
from collections.abc import Mapping

V2_ALEMBIC_DISPOSABLE_TEST = "V2_ALEMBIC_DISPOSABLE_TEST"
V2_ALEMBIC_HEAD_REVISION = "bc334f9877b7"
V2_ALEMBIC_HEAD_REVISION = "4e30076d4b9f"
DISPOSABLE_DATABASE_NAME = "policyengine_v2_alembic_test"
DISPOSABLE_HOSTS = frozenset({"localhost", "127.0.0.1", "::1", "postgres"})
MIGRATION_ROLE = "policyengine_v2_migrator"
Expand Down
8 changes: 1 addition & 7 deletions policyengine_api/data/v2/models/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,7 @@

# These are the complete direct-SQLAlchemy categories permitted in the v2
# table layer. Each remains underneath canonical SQLModel table classes and
# Field/Relationship declarations, apart from table-only columns awaiting
# removal; no parallel declarative model exists.
# Field/Relationship declarations; no parallel declarative model exists.
DIRECT_SQLALCHEMY_EXCEPTIONS = MappingProxyType(
{
"timezone_aware_timestamps": (
Expand Down Expand Up @@ -40,11 +39,6 @@
"Concurrent report idempotency requires a savepoint and bounded "
"IntegrityError recovery around the database uniqueness constraint."
),
"table_only_columns_awaiting_removal": (
"A column API v1 no longer uses stays in its table, outside the "
"SQLModel fields and the mapper, until other writers have stopped "
"and a later revision drops it; API v1 doesn't load it meanwhile."
),
}
)

Expand Down
Loading
Loading