Skip to content

Run reports again and list their runs in /v2/reports - #4298

Merged
anth-volk merged 23 commits into
masterfrom
v2plan/c1-18-re-runs
Nov 13, 2026
Merged

anth-volk merged 23 commits into
masterfrom
v2plan/c1-18-re-runs

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4294

Summary

Runs a stored report (an economy report, a household report or a budget window) again and lists its runs in /v2/reports, with each run's package versions beside the current ones.

  • POST /v2/reports/{report_id}/runs?country_id=, body {"force": false} (a JSON boolean; {} is the default, unknown fields refused, 16 KiB limit):

    • reads the report in its country (404 otherwise);
    • checks the report as a new report of its kind is checked, through SimulationSubmissionService.rerun_report: the country's economy flag for an economy report or budget window, or, for a household report, that the country has household calculations (a household-only country passes), as Serve household reports through /v2/reports #4293's create checks; then the policies of the report's simulations, baseline first, including Refuse /v2/reports and /v2/simulations submissions whose policies set missing parameters #4286's 422 missing_parameters. A household report's household isn't read again: it stays stored while a report names it;
    • has the simulation service create and start the report's new run (POST /reports/{report_id}/runs with {"force": …}), with the database session closed while it is asked;
    • answers 201 with the report read back from the tables, its new run as the newest. Every earlier run is kept.
  • GET /v2/reports/{report_id}/runs?country_id= answers {report_id, kind, items, current_versions}, newest first (created last, ties by the larger ID):

    • an economy report (kind: economy) or a household report (kind: household): its report runs, each with its status, trigger, country_package_version, policyengine_version, simulation run IDs, error and times (reform_simulation_run_id is null for a household report of the baseline alone);
    • a budget window (kind: budget_window): its budget-window runs, each with recalculate, its versions and its years, as GET /v2/reports/{id} answers a window's newest run.

    current_versions are the versions the Public API runs, as GET /v2/reports/{id} already answers, so v3 can mark a report whose newest run used an older .py version as out of date and offer a re-run.

  • How the simulation service's re-run answers are read (policyengine_api/libs/simulation_report_reruns.py):

    • 201 or 202 naming the report: started. One that can't be read, or names another report: 502.
    • 409 (run_in_progress with run_id, or rerun_not_supported): 409 with the service's code and reason and run_id. The service answers 409 only before writing, so one that can't be read is 503.
    • 422, and 400 unknown_region, unknown_dataset or unknown_dynamic for a row the stored report names: 422 with code and reason. Any other 400: 503.
    • 401, 403, 404, 405, 413, 429: 503. The service's 404 is its own state, since the Public API has just read the report.
    • 503 naming no record, or naming this report and the run it failed after Modal refused the call: 503, naming the failed run. A 503 that can't be read or names another record: 502.
    • 500, a lost answer, a proxy's 502 or 504, and any other status: 502. Each message asks the caller to read the report to see whether it has a new run before running it again.
    • A report that can't be read back once its run started: 502 with report_id.
  • The status and code sets the create client and the re-run client share move from simulation_reports into simulation_service_answers.

  • The migration contract documents gain the two requests, and v2-integration-check.yml runs the new PostgreSQL tests.

Waits for other PRs

Known limit: current_versions are the versions the Public API runs, as GET /v2/reports/{id} already answers, while a re-run records the simulation service's default worker's versions. During a release, while the two differ, a report run again a moment ago can still compare as out of date until both deploy the same .py version.

Environment variables and secrets

No new ones. The re-run uses the create's client and configuration:

  • SIMULATION_ENTRYPOINT_URL: GitHub environment secret (staging, production) → push.yml job env: → deploy_cloud_run_candidate.sh → Cloud Run. Unverified: that it names a simulation service revision with the re-run route (not yet released).
  • GATEWAY_AUTH_ISSUER, GATEWAY_AUTH_AUDIENCE, GATEWAY_AUTH_CLIENT_ID, GATEWAY_AUTH_CLIENT_SECRET_RESOURCE: organization secrets → the deploy step's env: → the deploy script → Cloud Run, the secret read from Secret Manager at runtime. Unverified: the Auth0 client's simulation:submit permission on the new route (the same permission as POST /reports).
  • V2_RUNTIME_DATABASE_URL_SECRET_RESOURCE: existing wiring; the runtime role already has SELECT on reports and report_runs.
  • CI: the new test file joins an existing step of v2-integration-check.yml, which uses the job's literal throwaway database values.

Testing

  • Red-green throughout: each change's tests were committed first and seen failing for the expected reason.
  • New tests, each run by node ID (71 node IDs, 123 cases, all passing):
    • the re-run client's request and every answer of the simulation service (tests/unit/libs/test_simulation_report_reruns.py);
    • the checks before a re-run, of each kind (tests/unit/v2/test_report_rerun_submissions.py): a household report's in a household-only country, of the baseline alone, without household calculations, with a policy setting missing parameters, and with countries that can't be read;
    • both routes over a stand-in service: bodies, queries, the 409, 413, 422 and 502 answers, error mapping and the OpenAPI document (tests/unit/v2/test_report_run_routes.py);
    • the report service with a database that can't be opened (tests/unit/v2/test_report_rerun_service.py);
    • PostgreSQL 17 as the runtime role (tests/integration/test_v2_report_runs.py): run lists newest first with versions, other reports' runs left out, another country's report or an unknown one (404), a household report's runs listed as kind household, a household report run again in a household-only country, a re-run with and without force answered with its new run, no report session open while the simulation service is asked, the service's 409 and 503 answered as its own, and a report that can't be read back (502).
  • Modified tests: the OpenAPI path list, the contract request count (99), the integration workflow's file list and the contract registry check. Serve household reports through /v2/reports #4293's household report integration tests now set the UK household-only through the same helper as the run tests; those 9 tests were run by node ID too.
  • Checks: alembic -c alembic-v2.ini check on a fresh PostgreSQL 17 database (no operations), lint-imports, python -m scripts.rules validate, python scripts/run_quality_guards.py, ruff, ruff format, mypy, and mypy on the changed libs modules. The migration contract documents are regenerated.
  • Review: three adversarial review rounds on the economy routes, one on the budget-window extension after Serve budget windows through /v2/reports #4290 merged, one on its fixes, one on the household extension after Serve household reports through /v2/reports #4293 merged, and one on its fixes; none found a bug. Their quality findings are fixed:
    • an unreachable 409 mapping is removed;
    • the 503 description lists every 503;
    • docstrings are updated;
    • tests are added for the service with a database that can't be opened, for the sessions open during the call, for household reports and for a window run with no year;
    • the run list matches on the report's kind with no default, and so does the choice of a re-run's country check, so a kind added later fails mypy until both handle it;
    • a household re-run goes through the same submission check as the others, with the country check of its kind, and the HTTP run list shares the service's kind type;
    • the household-only country setup of the integration tests is one helper, and a household report of the baseline alone is tested.
  • Left as they are, with reasons:
    • The run ID in the simulation service's 201 isn't compared with the run read back. No other run can start while the new one is pending or running, so the newest run read back is the new one.
    • The 503 after Modal refused the call names the failed run only in its message. That run is also the report's newest run.
    • The re-run client's helpers are not shared with the create client's yet. Serve budget windows through /v2/reports #4290 and Serve household reports through /v2/reports #4293 have both merged, so they could be; that would change the create client, which this PR otherwise leaves alone.
    • The run tests store household reports with their own small helper. Serve household reports through /v2/reports #4293's helper stores household simulations with calculated households, which the run tests don't need.

🤖 Generated with Claude Code

anth-volk and others added 10 commits November 13, 2026 01:07
…iling)

The Public API's client of the simulation service gains rerun_report,
which posts {"force": …} to POST /reports/{report_id}/runs and reads
each answer the service's re-run route gives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SimulationServiceReportClient.rerun_report posts {"force": …} to the
simulation service's POST /reports/{report_id}/runs. Its answers are read
in the new policyengine_api.libs.simulation_report_reruns: a started run
naming the report is accepted; a 409 (a run in progress, or a report the
service does not run again) is SimulationServiceConflictError, which the
report routes answer 409; refusals, unavailability and failures follow
the create's rules, with messages that ask the caller to read the report
before running it again. The status and code sets both clients use move
into simulation_service_answers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GET /v2/reports/{report_id}/runs lists an economy report's runs, newest
first, each with its package versions, beside the versions the Public
API runs. POST /v2/reports/{report_id}/runs checks the report's country
and policies, has the simulation service start its new run, with or
without force, and answers the report with that run; a run in progress
answers 409.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /v2/reports/{report_id}/runs?country_id= reads the economy report in
its country, checks the country and the policies its simulations name as
a new report is checked, has the simulation service create and start the
report's new run (body {"force": …}), and answers 201 with the report
and that run read back from the tables. A run in progress, or a report
the service does not run again, answers 409 naming the run; a report that
cannot be read back once its new run started answers 502 with its ID.

GET /v2/reports/{report_id}/runs?country_id= lists the report's runs,
newest first, each with its country package and PolicyEngine.py
versions, beside the versions the Public API runs. Household reports and
budget windows answer 422 on both routes until their own PRs add them.

The migration contract documents gain the two requests, and the
cross-database integration workflow runs the new PostgreSQL tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ns (failing)

Review round 1's quality findings: the run routes refuse household
reports and budget windows with their own message, rather than the read
route's, so the read route's check can change on its own. New tests
cover what was untested: no report session is open while the simulation
service is asked, the simulation service's refusal is answered as its
own, and a report that cannot be read, or an unconfigured service, asks
the simulation service nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- The run routes refuse household reports and budget windows through
  their own check, require_economy_report_runs, saying their runs are not
  started or listed here yet, so the read routes' kind check can change
  without changing them.
- The 409 mapping for SimulationServiceConflictError is removed: the
  re-run route answers it itself, with run_id.
- The re-run's 503 description lists every answer that gives a 503, and
  its description says why the newest run read back is the new one.
- Docstrings that described only creation, or said re-runs should call
  the policy check, now describe the re-run, and the unavailable error
  lists a re-run's unreadable 409.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- require_economy_report_runs refuses every kind but the ones the run
  routes take (RUN_ROUTE_KINDS, economy for now), so a kind added later is
  refused by default; REPORT_KIND_NAMES names each kind for the message.
  The behavior is unchanged, so the existing tests cover it.
- The database test now stores a household report as well as a budget
  window, and checks that neither run route lists or starts its runs,
  answering 422 without asking the simulation service; round 2 found the
  household refusal tested only through the validator. It passes before
  the change, since the behavior exists.
- Docstrings no longer describe the run routes' kind refusal as the read
  route's, ReportKindUnsupportedError covers both kinds of route, the
  client protocol lists a re-run's unreadable 409, and the report service
  says a re-run also needs the simulation service.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
RUN_ROUTE_KINDS is typed as a set of report kinds, so mypy refuses a
misspelled one, and a test checks that REPORT_KIND_NAMES names every
report kind, so a kind added later is refused with its name (422) rather
than failing on a missing key (500). The test passes as is: every kind is
named today.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.09886% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.45%. Comparing base (fd52ced) to head (bdc0d68).
⚠️ Report is 1 commits behind head on master.

Files with missing lines Patch % Lines
...engine_api/fastapi_routes/v2/reports/run_models.py 95.00% 0 Missing and 2 partials ⚠️
...ine_api/services/v2/simulation_service/services.py 90.90% 0 Missing and 1 partial ⚠️
...ces/v2/stored_reports/database_connectors/reads.py 50.00% 0 Missing and 1 partial ⚠️
...yengine_api/services/v2/stored_reports/services.py 96.96% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4298      +/-   ##
==========================================
+ Coverage   93.38%   93.45%   +0.06%     
==========================================
  Files         366      369       +3     
  Lines       21194    21442     +248     
  Branches     3366     3400      +34     
==========================================
+ Hits        19793    20038     +245     
- Misses        819      822       +3     
  Partials      582      582              
Flag Coverage Δ
integration 56.59% <58.17%> (+0.03%) ⬆️
unit 91.00% <89.73%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

# Conflicts:
#	.github/workflows/v2-integration-check.yml
#	policyengine_api/fastapi_routes/resource_service_protocols.py
#	policyengine_api/libs/simulation_reports.py
#	policyengine_api/services/v2/simulation_service/services.py
#	policyengine_api/services/v2/simulation_service/types.py
#	policyengine_api/services/v2/stored_reports/services.py
#	policyengine_api/services/v2/stored_reports/types.py
#	policyengine_api/services/v2/stored_reports/validators.py
#	tests/unit/test_alembic_workflows.py
anth-volk and others added 5 commits November 13, 2026 03:09
With #4290 merged, the run routes take budget windows too. GET
/v2/reports/{report_id}/runs lists a window's budget-window runs, newest
first, each with its years, as kind budget_window; POST answers the
window with its new run. Only household reports are refused now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With #4290 merged, the run routes take budget windows.
require_run_route_kind returns the report's kind and refuses only
household reports now. GET /v2/reports/{report_id}/runs answers a budget
window's budget-window runs, newest first, each with its years
(BudgetWindowRunListResult, kind budget_window); ReportRunListResponse's
result is told apart by kind. POST runs a budget window again through
the same simulation service route and answers the window with its new
run. reads.read_budget_window_years is now public, for the run list.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- require_run_route_kind returns RunRouteKind (economy or budget_window),
  and list_report_runs matches on it with a case for each, so a kind the
  run routes take later fails mypy (a missing return) until its list is
  added, rather than being listed as an economy report.
- A test lists a window run with no year: no years and no versions. It
  passes as is, since the behavior exists.
- The unit window re-run's years carry the current versions, as a window
  run's first year always does.
- ReportRunsRead and the reads module say what they now hold.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The accepted-kind test now runs for every kind in RunRouteKind, from a
table of the columns that make each kind, so the validator and the type
can't drift apart unseen; the window re-run in the unit stand-in has
simulation runs of its own.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk anth-volk changed the title Run economy reports again and list their runs in /v2/reports Run economy reports and budget windows again and list their runs in /v2/reports Nov 12, 2026
anth-volk and others added 6 commits November 13, 2026 18:40
Brings in household reports (#4293). The run routes still refuse household
reports with 422, so the merge keeps ReportKindUnsupportedError, its 422
mapping and require_run_route_kind, which #4293 removed with the read
routes' refusal. The re-run tests pass the submission service's new
households source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A household report's runs are listed as kind household, newest first, and
its re-run is checked as a new household report is: the country must have
household calculations, then the policies of its simulations. Each re-run
names its report's kind. The run routes' household refusal tests go.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The run routes take every kind of report. A re-run names its report's kind,
and a household report's is checked as a new household report is: its
country must have household calculations (a household-only country passes),
then the policies of its simulations. Its household is not read again; it
stays stored while a report names it. A household report's runs are its
report runs, listed as kind household beside the current versions.

The run routes' refusal of other kinds, its 422 answer and the kind names
go, as #4293 removed the read routes' refusal. The simulation service
answers 409 rerun_not_supported for a kind it does not run again yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A re-run goes through the shared submission check with the country check a
new report of its kind makes, chosen by a match with a case per kind, so a
kind added later fails mypy until it has one. The HTTP run list takes its
kind type from the stored reports types. The household-only country setup
of both integration test files is one helper beside store_catalog. Tests
cover a household report of the baseline alone, re-run and listed. The
protocol's re-run docstring and the changelog say what a household re-run
checks and that the simulation service answers 409 for it for now.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…list

Review round 7's finding: the route test now asserts the listed run's
baseline simulation run ID and that the list is a household report's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk anth-volk changed the title Run economy reports and budget windows again and list their runs in /v2/reports Run reports again and list their runs in /v2/reports Nov 13, 2026
The simulation service's re-run of a household report (PolicyEngine/policyengine-sim-api#1059)
gives each side a new run of its own household simulation, so force changes
nothing for it. The route, request model, service and re-run input texts say
so, and the changelog no longer says the simulation service refuses
household re-runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review November 13, 2026 19:04
@anth-volk
anth-volk merged commit 4b16976 into master Nov 13, 2026
23 of 24 checks passed
@anth-volk
anth-volk deleted the v2plan/c1-18-re-runs branch November 13, 2026 19:04

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run economy reports again and list their runs in /v2/reports

1 participant