Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/scripts/deploy_cloud_run_candidate.sh
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,16 @@ if [[ -n "${AUTH0_USER_PROFILES_DOMAIN:-}" ]]; then
)
fi

# Optional: invitation and type-change emails go through Postmark only when
# its settings are set. Validation requires all three values or none.
if [[ -n "${POSTMARK_SERVER_TOKEN_SECRET_RESOURCE:-}" ]]; then
env_vars+=(
"POSTMARK_SERVER_TOKEN_SECRET_RESOURCE=${POSTMARK_SERVER_TOKEN_SECRET_RESOURCE}"
"POSTMARK_SENDER_ADDRESS=${POSTMARK_SENDER_ADDRESS}"
"POLICYENGINE_APP_URL=${POLICYENGINE_APP_URL}"
)
fi

if [[ -n "${OLD_SIMULATION_GATEWAY_URL:-}" ]]; then
env_vars+=("OLD_SIMULATION_GATEWAY_URL=${OLD_SIMULATION_GATEWAY_URL}")
fi
Expand Down
1 change: 1 addition & 0 deletions .github/scripts/test_alembic_v2_lifecycle.sh
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ set -euo pipefail

uv run --frozen --extra dev pytest -q \
tests/unit/v2/test_alembic_v2.py \
tests/unit/v2/test_alembic_v2_revisions.py \
tests/integration/test_alembic_v2_lifecycle.py \
tests/integration/test_alembic_v2_catalog_tables.py \
tests/integration/test_alembic_v2_simulation_runs.py \
Expand Down
25 changes: 25 additions & 0 deletions .github/scripts/validate_cloud_run_deploy_env.sh
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,31 @@ if (( auth0_user_profiles_count != 0 )) && [[ -z "${PUBLIC_API_AUTH_ISSUER:-}" ]
exit 1
fi

# Optional: invitation and type-change emails go through Postmark only when
# its server token's Secret Manager version, the sender address and the app's
# address the emails link to are all set. Staging and production may both
# configure it, each with its own Postmark server.
postmark_count=0
for name in \
POSTMARK_SERVER_TOKEN_SECRET_RESOURCE \
POSTMARK_SENDER_ADDRESS \
POLICYENGINE_APP_URL; do
if [[ -n "${!name:-}" ]]; then
postmark_count=$((postmark_count + 1))
fi
done
if (( postmark_count != 0 && postmark_count != 3 )); then
echo "Set POSTMARK_SERVER_TOKEN_SECRET_RESOURCE, POSTMARK_SENDER_ADDRESS, and POLICYENGINE_APP_URL together, or none" >&2
exit 1
fi
# The emails' links open the app, so its address is HTTPS, and has no query
# or fragment for the links' paths to follow.
app_url_pattern='^https://[^/?#[:space:]]+(/[^?#[:space:]]*)?$'
if [[ -n "${POLICYENGINE_APP_URL:-}" && ! "${POLICYENGINE_APP_URL}" =~ ${app_url_pattern} ]]; then
echo "POLICYENGINE_APP_URL must be an https address with no query or fragment" >&2
exit 1
fi

selected_url_env="$(
simulation_entrypoint_url_env_name "${SIM_ENTRYPOINT}"
)"
Expand Down
9 changes: 9 additions & 0 deletions .github/workflows/push.yml
Original file line number Diff line number Diff line change
Expand Up @@ -524,6 +524,12 @@ jobs:
AUTH0_USER_PROFILES_DOMAIN: ${{ vars.AUTH0_USER_PROFILES_DOMAIN }}
AUTH0_USER_PROFILES_CLIENT_ID: ${{ vars.AUTH0_USER_PROFILES_CLIENT_ID }}
AUTH0_USER_PROFILES_CLIENT_SECRET_RESOURCE: ${{ secrets.AUTH0_USER_PROFILES_CLIENT_SECRET_RESOURCE }}
# Invitation and type-change emails through the environment's Postmark
# server: its token's Secret Manager version, the sender address, and
# the app's address the emails link to. All three, or none.
POSTMARK_SERVER_TOKEN_SECRET_RESOURCE: ${{ secrets.POSTMARK_SERVER_TOKEN_SECRET_RESOURCE }}
POSTMARK_SENDER_ADDRESS: ${{ vars.POSTMARK_SENDER_ADDRESS }}
POLICYENGINE_APP_URL: ${{ vars.POLICYENGINE_APP_URL }}
CLOUD_RUN_SERVICE: policyengine-api-staging
CLOUD_RUN_RUNTIME_CACHE_ENVIRONMENT: staging
CLOUD_RUN_RUNTIME_CACHE_URL_SECRET: policyengine-api-staging-runtime-cache-url:latest
Expand Down Expand Up @@ -868,6 +874,9 @@ jobs:
AUTH0_USER_PROFILES_DOMAIN: ${{ vars.AUTH0_USER_PROFILES_DOMAIN }}
AUTH0_USER_PROFILES_CLIENT_ID: ${{ vars.AUTH0_USER_PROFILES_CLIENT_ID }}
AUTH0_USER_PROFILES_CLIENT_SECRET_RESOURCE: ${{ secrets.AUTH0_USER_PROFILES_CLIENT_SECRET_RESOURCE }}
POSTMARK_SERVER_TOKEN_SECRET_RESOURCE: ${{ secrets.POSTMARK_SERVER_TOKEN_SECRET_RESOURCE }}
POSTMARK_SENDER_ADDRESS: ${{ vars.POSTMARK_SENDER_ADDRESS }}
POLICYENGINE_APP_URL: ${{ vars.POLICYENGINE_APP_URL }}
CLOUD_RUN_SERVICE: policyengine-api
CLOUD_RUN_RUNTIME_CACHE_ENVIRONMENT: production
CLOUD_RUN_RUNTIME_CACHE_URL_SECRET: policyengine-api-prod-runtime-cache-url:latest
Expand Down
1 change: 1 addition & 0 deletions changelog.d/user-invitations.added.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Admins invite an email address to become an author or admin, and a signed-in user with that verified Auth0 email accepts the invitation; every type change emails the user through Postmark, and `GET /v2/me` reports the latest type change until the user dismisses it.
8 changes: 5 additions & 3 deletions docs/engineering/migration-contracts.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi
| --- | ---: |
| route group count | 9 |
| workflow count | 18 |
| request count | 99 |
| request count | 101 |
| db entity count | 6 |
| sim flow count | 3 |

Expand Down Expand Up @@ -160,12 +160,14 @@ Generated from `policyengine_api/migration_registry.py` and `tests/contract/regi

| Method | Path | Status | Route group | Required headers | Stable response fields |
| --- | --- | ---: | --- | --- | --- |
| `GET` | `/v2/me?country_id=us` | 200 | `user_profile` | `Authorization` | `status`, `message`, `result.item.id`, `result.item.user_type`, `result.item.primary_country`, `result.item.first_name`, `result.item.last_name`, `result.item.email`, `result.item.created_at` |
| `GET` | `/v2/me?country_id=us` | 200 | `user_profile` | `PolicyEngine-Unauthenticated-User` | `status`, `message`, `result.item.id`, `result.item.user_type`, `result.item.primary_country`, `result.item.first_name`, `result.item.last_name`, `result.item.email`, `result.item.created_at` |
| `GET` | `/v2/me?country_id=us` | 200 | `user_profile` | `Authorization` | `status`, `message`, `result.item.id`, `result.item.user_type`, `result.item.primary_country`, `result.item.first_name`, `result.item.last_name`, `result.item.email`, `result.item.created_at`, `result.latest_type_change` |
| `GET` | `/v2/me?country_id=us` | 200 | `user_profile` | `PolicyEngine-Unauthenticated-User` | `status`, `message`, `result.item.id`, `result.item.user_type`, `result.item.primary_country`, `result.item.first_name`, `result.item.last_name`, `result.item.email`, `result.item.created_at`, `result.latest_type_change` |
| `PATCH` | `/v2/me?country_id=us` | 200 | `user_profile` | `Authorization` | `status`, `message`, `result.item.id`, `result.item.primary_country`, `result.item.first_name`, `result.item.last_name` |
| `GET` | `/v2/me/export?country_id=us` | 200 | `user_profile` | `Authorization` | `status`, `message`, `result.user`, `result.policies`, `result.households`, `result.simulations`, `result.reports` |
| `GET` | `/v2/me/export?country_id=us` | 200 | `user_profile` | `PolicyEngine-Unauthenticated-User` | `status`, `message`, `result.user`, `result.policies`, `result.households`, `result.simulations`, `result.reports` |
| `POST` | `/v2/me/upgrade?country_id=us` | 200 | `user_profile` | `Authorization`, `PolicyEngine-Unauthenticated-User` | `status`, `message`, `result.item.id`, `result.item.user_type`, `result.item.primary_country`, `result.outcome` |
| `POST` | `/v2/me/invitations/accept?country_id=us` | 200 | `user_profile` | `Authorization` | `status`, `message`, `result.item.user_type`, `result.item.previous_user_type`, `result.item.type_changed` |
| `POST` | `/v2/me/type-changes/{type_change_id}/dismiss?country_id=us` | 204 | `user_profile` | `Authorization` | |
| `DELETE` | `/v2/me?country_id=us` | 204 | `user_profile` | `Authorization` | |
| `DELETE` | `/v2/me?country_id=us` | 204 | `user_profile` | `PolicyEngine-Unauthenticated-User` | |

Expand Down
45 changes: 42 additions & 3 deletions docs/engineering/skills/alembic-migrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -447,6 +447,8 @@ it publishes that version, with nothing to restore it. Downgrading past the
revision is therefore safe only while no version is pruned (no
`tax_benefit_model_versions` row has `catalog_values_pruned_at` set).
From the head, such a downgrade first runs the later revisions' downgrades:
`f519a77ba122`'s drops `user_invitations` and needs no recorded change with
source `invitation` (see "API v2 user invitations"),
`94dc43079186`'s drops every user's report-run record, the simulation and
report keys and the window runs' .py versions, and needs every report to have
a label and every run a trigger (see "API v2 shared reports and user report-run records"),
Expand Down Expand Up @@ -799,9 +801,9 @@ script), the previous and new types (each `standard`, `author` or `admin`,
and different), and the `source`, `admin_route` or `script`. An index on
`(user_id, created_at)` serves a user's history, and one on
`changed_by_user_id` the `SET NULL` that every user deletion runs. The runtime role may read and
insert its rows, never update or delete them; a user's deletion removes them
through the foreign key, which PostgreSQL runs with the table owner's
privileges.
insert its rows, and since the user invitations revision update only their
`dismissed_at`, never delete them; a user's deletion removes them through
the foreign key, which PostgreSQL runs with the table owner's privileges.

The previous Public API revision never uses the table. The downgrade drops it
with every recorded change. `test_user_type_changes_revision_adds_only_its_table`
Expand Down Expand Up @@ -964,3 +966,40 @@ fails while any report has no label or any run no trigger.
revision changes nothing else and downgrades exactly, and
`tests/integration/test_v2_shared_report_rows.py` runs the finding, joining
and recording as the roles that do them.

### API v2 user invitations

Revision `f519a77ba122` (parent `94dc43079186`) was autogenerated, with no
post-generation corrections, after the `UserInvitation` model was added,
`UserTypeChange` gained `dismissed_at`, and `UserTypeChangeSource` gained
`invitation`. It was generated on `0c136a278f29`; when `master` gained the
withheld area codes revision on the same parent, its `down_revision` moved to
`1e1e3f079f2c`, and when `master` then gained the shared reports revision on
that one, to `94dc43079186`. Generating it again from that revision gives the
same operations. It adds `user_invitations`, one row per admin's invitation for
an email address to become an author or admin: the address, the type
(`author` or `admin`), the SHA-256 digest of its token in hexadecimal
(unique; the token itself is never stored), the inviting admin, its expiry,
its last send's time or failure, and when it was accepted, and by whom, or
revoked (never both). Deleting the inviting admin keeps the invitation and
sets that column to NULL; deleting the user who accepted it deletes it, since
its address was theirs (`CASCADE`). An index on each serves what every user
deletion runs. It adds the nullable `user_type_changes.dismissed_at`, and
replaces the source check: Alembic compares checks by name only, so the
check that also allows `invitation` is named
`ck_user_type_changes_source_allowed`, and autogenerate drops
`ck_user_type_changes_source` and adds it, as for the user types revision.
The runtime role reads, inserts and updates invitations, never deletes them,
and updates only `dismissed_at` of `user_type_changes`
(`update_columns("dismissed_at")`).

The previous Public API revision never uses `user_invitations` and doesn't
map `dismissed_at`, but it reads every recorded change's source as
`admin_route` or `script`: there, an admin's read of the type history of a
user who accepted an invitation fails with 500. The downgrade drops
`user_invitations` and `dismissed_at` with their values, and restores the
check that allows only `admin_route` and `script`, so it fails while any
change's source is `invitation`.
`test_user_invitations_revision_adds_invitations_and_dismissals_only` and
`test_user_invitations_revision_records_invitation_changes_until_downgraded`
in `tests/integration/test_alembic_v2_user_types.py` check both.
8 changes: 7 additions & 1 deletion docs/engineering/skills/testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -302,7 +302,8 @@ uv run pytest tests/unit/v2/test_models.py tests/unit/v2/test_model_persistence.
The generated-only v2 Alembic chain and disposable Postgres lifecycle:

```bash
uv run pytest tests/unit/v2/test_alembic_v2.py tests/unit/test_alembic_workflows.py -q
uv run pytest tests/unit/v2/test_alembic_v2.py tests/unit/v2/test_alembic_v2_revisions.py \
tests/unit/test_alembic_workflows.py -q
V2_ALEMBIC_DISPOSABLE_TEST=1 V2_MIGRATION_DATABASE_URL="postgresql+psycopg://.../policyengine_v2_alembic_test" \
uv run pytest tests/integration/test_alembic_v2_lifecycle.py tests/integration/test_alembic_v2_catalog_tables.py -q
```
Expand Down Expand Up @@ -613,6 +614,7 @@ uv run --frozen --extra dev mypy
uv run --frozen pytest \
tests/unit/libs/test_user_auth.py \
tests/unit/libs/test_auth0_management.py \
tests/unit/libs/test_postmark_email.py \
tests/unit/libs/test_rate_limit_wait_budget.py \
tests/unit/v2/test_user_route_auth.py \
tests/unit/v2/test_me_routes.py \
Expand All @@ -622,6 +624,9 @@ uv run --frozen pytest \
tests/unit/v2/test_admin_user_profile_service.py \
tests/unit/v2/test_admin_user_deletion.py \
tests/unit/v2/test_admin_user_saved_items_service.py \
tests/unit/v2/test_admin_invitation_routes.py \
tests/unit/v2/test_user_invitation_service.py \
tests/unit/v2/test_user_email_service.py \
tests/unit/test_set_v2_user_type_script.py \
tests/unit/v2/test_user_policy_routes.py \
tests/unit/v2/test_user_household_routes.py \
Expand All @@ -634,6 +639,7 @@ uv run --frozen pytest \
tests/unit/v2/test_user_association_migration_qualification.py \
tests/unit/v2/test_models.py \
tests/unit/v2/test_alembic_v2.py \
tests/unit/v2/test_alembic_v2_revisions.py \
tests/unit/test_readiness.py \
tests/unit/test_cloud_run_deploy_scripts.py \
tests/unit/test_migration_contract_artifacts.py \
Expand Down
34 changes: 31 additions & 3 deletions docs/generated/migration_contracts.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"metadata": {
"db_entity_count": 6,
"request_count": 99,
"request_count": 101,
"route_group_count": 9,
"sim_flow_count": 3,
"workflow_count": 18
Expand Down Expand Up @@ -1185,7 +1185,8 @@
"result.item.first_name",
"result.item.last_name",
"result.item.email",
"result.item.created_at"
"result.item.created_at",
"result.latest_type_change"
]
},
{
Expand All @@ -1205,7 +1206,8 @@
"result.item.first_name",
"result.item.last_name",
"result.item.email",
"result.item.created_at"
"result.item.created_at",
"result.latest_type_change"
]
},
{
Expand Down Expand Up @@ -1279,6 +1281,32 @@
"result.outcome"
]
},
{
"expected_status": 200,
"headers": [
"Authorization"
],
"method": "POST",
"path": "/v2/me/invitations/accept?country_id=us",
"route_group": "user_profile",
"stable_response_fields": [
"status",
"message",
"result.item.user_type",
"result.item.previous_user_type",
"result.item.type_changed"
]
},
{
"expected_status": 204,
"headers": [
"Authorization"
],
"method": "POST",
"path": "/v2/me/type-changes/{type_change_id}/dismiss?country_id=us",
"route_group": "user_profile",
"stable_response_fields": []
},
{
"expected_status": 204,
"headers": [
Expand Down
2 changes: 1 addition & 1 deletion docs/generated/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -376,7 +376,7 @@ flowchart LR
| `api.quality.environment-variables-load` | Required environment variables and credentials load in a deployed-like environment. | mechanical | enforced | yes | CI job `test_env_vars` | `pr.yml:test_env_vars` | — |
| `api.quality.file-length` | A tracked Python file has at most 800 physical lines, or at most 1,500 if it is test code (a file under a tests or fixtures directory, or a conftest.py); a longer file is split by responsibility into modules, and keeps its public import path. | metric | enforced | yes | quality guard `file-length` | `pr.yml:quality-guards` | yes |
| `api.quality.make-quality` | make quality is the command for the quality checks, locally and in CI: it applies ruff check --fix and ruff format, then runs make quality-check, which runs uv lock --check, ruff check, ruff format --check, Import Linter, mypy and the quality guards without changing files. Run with a stand-in uv, make quality runs each of them and fails whenever one fails, and make -n prints the same commands. pr.yml's quality-guards job runs make quality and then git diff --exit-code HEAD, with neither step skipped, allowed to fail, or run in another shell or directory, no MAKEFLAGS, GNUMAKEFLAGS or MAKEFILES set for them, and no earlier step writing to GITHUB_ENV or GITHUB_PATH; the Makefile uses no syntax the rules registry doesn't follow and nothing that hides a failing command; and no other pr.yml job's commands run ruff, mypy, lint-imports or the quality guards, directly or through a make target. | mechanical | enforced | yes | pytest `tests/unit/test_make_quality.py` | `pr.yml:test` | yes |
| `api.quality.mypy-v2` | Shared query models, FastAPI dependencies, v2 routes and services, the gzip negotiation, the user sign-in and Auth0 Management libraries with their rate-limit wait budget, the country list and its startup check pass mypy with untyped definitions disallowed. | mechanical | enforced | yes | mypy | `pr.yml:quality-guards` | — |
| `api.quality.mypy-v2` | Shared query models, FastAPI dependencies, v2 routes and services, the gzip negotiation, the user sign-in and Auth0 Management libraries with their settings and rate-limit wait budget, the Postmark email library, the country list and its startup check pass mypy with untyped definitions disallowed. | mechanical | enforced | yes | mypy | `pr.yml:quality-guards` | — |
| `api.quality.pre-commit-hook` | make install turns on the git hooks in .githooks through core.hooksPath, a setting of the clone that every worktree shares and resolves against its own checkout, unless that would change an enclosing repository, replace another core.hooksPath, or stop the clone's own hooks (files or symbolic links), which .githooks/turn-on.sh says, always exiting 0. pre-commit runs make quality from the root of the working tree before git commit, and pre-merge-commit runs it before a merge commits without conflicts, with make's variables from the environment cleared. Because make quality checks the working tree, the hook stops the commit, naming the files or what to install, when the working tree holds unstaged changes to tracked files (case-only renames included), skip-worktree or assume-unchanged files, untracked files outside the ignore files, Python files or tool settings that only local or global excludes hide (outside virtual environments), or tracked Python files those excludes match; when make quality fails; when ruff's fixes changed a file; or when make, uv or the dev environment is missing. | mechanical | enforced | yes | pytest `tests/unit/test_pre_commit_hook.py` | `pr.yml:test` | yes |
| `api.quality.ruff-format` | All Python files are formatted with ruff format. | mechanical | enforced | yes | ruff format | `pr.yml:quality-guards` | — |
| `api.quality.ruff-lint` | All Python code passes ruff check with every ruff rule selected (ALL), with no ratchet, except the ignores in pyproject.toml: COM812 and ISC001, which conflict with ruff format, and D203 and D213, which conflict with the kept D211 and D212. | mechanical | enforced | yes | ruff check (ALL) | `pr.yml:quality-guards` | — |
Expand Down
Loading
Loading