Skip to content

Find or create simulations, reports and runs by key - #1074

Merged
anth-volk merged 110 commits into
mainfrom
v2plan/c1-27-find-or-create-by-key
Nov 17, 2026
Merged

anth-volk merged 110 commits into
mainfrom
v2plan/c1-27-find-or-create-by-key

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #1073
Also fixes #1081

Identical requests now end on the same simulation, report and run rows, together or apart, and the Simulation Entrypoint starts a Modal call only for a run its own transaction inserted (tracker C1 · 27, following C1 · 24's decisions in the shared-reports proposal).

What changes

  • Keys. definition_keys builds each record's key: the lowercase SHA-256 of a definition's canonical JSON, with key_version 1. An economy simulation is keyed by its country, policy, dynamic, dataset name, region code, year, a null filter and the SPM selection; an economy report or budget window by both sides, the dataset, region, year and SPM selection they share, and the window size, with no other option; a household simulation by its country, policy, dynamic, stored household and year; a household report by its sides, household and year. Each report also stores a dataset_independent_key.
  • Find or create. shared_rows does one INSERT … ON CONFLICT DO NOTHING per row against the Public API's unique indexes (simulations and reports by key; simulation runs, report runs and budget-window runs among runs that haven't failed and weren't forced), then reads the row that won. keyed_economy_records, keyed_household_records and keyed_window_records run a create in one transaction: simulations by key in key order, the report by key, the report's (or window's) run on the current versions joined when there is one, otherwise each side's simulation run found or inserted and the report run (or window run and each year's report run) naming them. A competing request's report run is joined and the savepoint takes back this request's simulation runs. A budget-window year and a single-year report of that year share simulations and runs.
  • Creates (POST /reports for economy, budget-window and household reports; POST /simulations) find or create by key, and start a coordinator, household function or simulation function only for a run they inserted; a joined run is answered as it is, 201.
  • Re-runs. A re-run of a report created by key joins the report's run on the default worker's versions, or each side joins its simulation's run, the same way. {"force": true}, and every re-run of a report created before keys (interim answer B), is an extra run marked forced (a window's recalculate), which nothing joins and the run indexes leave out. A simulation's re-run joins its run. No re-run is refused while a run is under way, so 409 run_in_progress is gone from the API.
  • Cliffs (interim answer A): include_cliffs: true answers 422 cliffs_not_available, and so does a re-run of a stored report whose options hold it, until C1 · 39 calculates cliffs per simulation run.
  • Per-user columns (answer 6): no shared row gets reports.label, description or user_id, or a report_runs / budget_window_runs trigger; the requests accept label and description as optional and don't store them. tables.py models those columns as nullable, and budget_window_runs.policyengine_version with its index, matching Share detailed-save reports and runs by key, and record each user's report runs policyengine-api#4303.
  • One simulation on both sides (current law against current law, as the staging check sends) names one simulation and one run on both sides; this branch carries C1 · 26's contract changes that allow it (economy ReportRunIds and ReportExecutionInput, household HouseholdReportRunCalculation).
  • The staging economy check script (.github/scripts/simulation-staging-economy-check.sh) creates its report, then re-runs it with {"force": true} and follows that forced run, so each run of it calculates. Remove the staging v2 route checks that need the Public API's catalog first #1088 has since removed the deploy job that ran it, because a new .py version has no Public API catalog rows until the Public API deploys it; main keeps the script, and so does this branch, with the forced re-run.

C1 · 24's answers it follows

1a to 1c (keys, dynamics by ID, key columns with random row IDs), revised 1a (dataset_independent_key without the dataset), 2 (a failed run never counts), 3 (force makes an extra run), 4 (SPM in the simulation key, no options in the report key, each run storing its own simulation's plan), 5 (a coordinator waits for a run another report run is calculating), 6 (no per-user columns on shared rows), 8 (household reports and budget windows by key), 10, who may send force, and the interim answers A and B.

Deploy first

Deployment code and environment variables

.github/scripts/simulation-staging-economy-check.sh changes. The Simulation Entrypoint's deploy step in .github/workflows/simulation-deploy.reusable.yml and .github/scripts/deploy-cloud-run-simulation-entry.sh stop passing, checking and setting V2_SIMULATION_OUTPUT_BUCKET (below). No Dockerfile, Terraform or Modal change, and no variable is added.

The staging economy check script reads SIMULATION_TEST_AUTH_ISSUER, SIMULATION_TEST_AUTH_AUDIENCE, SIMULATION_TEST_AUTH_CLIENT_ID and SIMULATION_TEST_AUTH_CLIENT_SECRET, and optionally ECONOMY_CHECK_DEADLINE_SECONDS and ECONOMY_CHECK_POLL_SECONDS (defaults 1800 and 15). Since #1088 no workflow job runs the script, so nothing in CI provides those variables; the job that did mapped them from the beta environment secrets SIMULATION_ENTRYPOINT_AUTH_ISSUER, SIMULATION_ENTRYPOINT_AUTH_AUDIENCE, SIMULATION_ENTRYPOINT_TEST_AUTH_CLIENT_ID and SIMULATION_ENTRYPOINT_TEST_AUTH_CLIENT_SECRET. Its forced re-run needs the same simulation:submit permission as POST /reports; not verified, as only a staging run of the script would show it.

V2_SIMULATION_OUTPUT_BUCKET leaves the Simulation Entrypoint, which read it into a setting and used it for nothing once reports join their sides' runs by version set:

Variable Provider Before After
V2_SIMULATION_OUTPUT_BUCKET (Simulation Entrypoint) GitHub Actions environment variable vars.V2_SIMULATION_OUTPUT_BUCKET, set in beta and prod; no repository-level variable of that name exists Job deploy_entrypoint runs in environment: ${{ inputs.release_environment }} (beta, then prod, from simulation-deploy.yml). Its step "Deploy Cloud Run entrypoint candidate" passed the variable as V2_SIMULATION_OUTPUT_BUCKET_VALUE to deploy-cloud-run-simulation-entry.sh, which required it, checked its format and wrote V2_SIMULATION_OUTPUT_BUCKET into the --env-vars-file of gcloud run deploy; Settings.from_env read it The step and the script don't read it, and the service doesn't either. --env-vars-file replaces all of a new revision's variables, so revisions deployed from this commit on have no V2_SIMULATION_OUTPUT_BUCKET; revisions deployed earlier keep theirs
V2_SIMULATION_OUTPUT_BUCKET (Stage 12) The same GitHub Actions environment variable Job deploy_stage12_v2, in the same environment, passes it to stage12-validate-infrastructure.sh and stage12-sync-modal-secrets.sh; the sync script writes it into the stage12-evaluation-runtime Modal secret, which src/modal/v2_app.py attaches and stage12_runtime/dependencies.py reads Unchanged: the Modal coordinator and workers still write simulation output files to that bucket

Not verified: the runtime variables of the Cloud Run revisions now serving, which this change doesn't read; --env-vars-file sets each new revision's variables in full.

Meeting other open PRs

The code this leaves unused, removed (#1081)

With creates and re-runs by key, nothing the service runs calls the code that created those rows without keys, or the search for an earlier succeeded run to reuse. This PR removes it (first built as #1082, now folded in here). Requests, answers and the rows the service writes don't change for it.

  • Persistence library: the create functions that use no keys (insert_economy_report, insert_economy_simulation, insert_household_report, insert_budget_window_report, and insert_simulation_with_run, which only they called), their input types, and the store methods that wrapped them (EconomyRunStore.create_report, create_simulation and create_budget_window_report, HouseholdReportStore.create_report); the run-reuse search (find_reusable_run, find_reusable_run_of, ReusableRun, RunReuse, search_by_definition, search_own_runs, REFUSED_REUSE_PREFIX), the reuse parameter of the input reads (always None), the window reads' years parameter, which only fed the search, and the reusable_run and reusable_runs fields; RerunSide.reused and the checks that refused it, with the reused member of the WindowSide protocol (year_created takes the FoundWindowYear it is always called with).
  • Simulation Entrypoint: the reuse and years arguments in the store protocols and their callers, and the output-bucket setting (above).
  • Tests: tests that store a record created before keys insert its rows directly (tests/economy_rows.py gains insert_pending_report; the new tests/household_report_rows.py; a window helper in test_economy_window_reruns.py). The tests of only the removed code are deleted. Behaviour they checked that remains is now tested on the keyed creators: a refused region, dataset, dynamic, policy or household reference names its column (which the Simulation Entrypoint turns into its unknown_* refusals), and a refused window writes nothing. The economy store's commit-failure tests use a forced report re-run, whose create only inserts rows.

Tests

Eight read-only review rounds (the last four found no bugs; their quality findings are fixed). Every new or changed test was seen failing first, then run by node ID only, the PostgreSQL ones against a database built from PolicyEngine/policyengine-api#4303's head 63cce8273; Ruff format and lint, Pyright and the quality guards passed for each changed package before each commit. A test sends two identical requests at once and checks they end on the same rows with exactly one Modal call. public-api-commit pins PolicyEngine/policyengine-api#4303's head 63cce8273, whose files are identical to its merge commit e96e52ffb, so the Public API schema job runs every schema test against the tables this PR needs. That job's last step requires the pinned commit on the Public API's master and fails until the pin moves to e96e52ffb, which waits until production has revision 94dc43079186, so this PR can't merge before the tables it writes exist. For the removal: its changed and added tests (68 persistence tests against a database built from e96e52ffb, and 56 entry tests) pass by node ID, the same checks pass, and its one review round found no bugs.

🤖 Generated with Claude Code

anth-volk and others added 30 commits November 14, 2026 00:12
… (failing)

The Public API's next revision (C1 · 25) gives simulations and reports a
key_version and definition_key, unique together, reports a
dataset_independent_key, and both run tables a forced marker and a
partial unique index allowing one unfailed, unforced run per definition
and version set. Find-or-create names these as its conflict targets, so
the models must declare them under the same names and conditions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sim's copies of the columns C1 · 25's revision adds: key_version and
definition_key on simulations and reports (with dataset_independent_key
on reports), each nullable, with the Public API's checks, the unique
constraint find-or-create targets and the dataset-independent index;
forced on simulation_runs and report_runs, false unless a forced re-run
inserts it; and the partial unique indexes allowing one unfailed, unforced
run per definition and version set. The models now match the Public API's
tables at revision 94dc43079186.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A key is the lowercase SHA-256 of a definition's canonical JSON: keys
sorted, no spaces, UTF-8, UUIDs as lowercase strings, missing values null,
and no value JSON can't write exactly. An economy report's object names
its key version, record type, country, each side's policy and dynamic,
dataset name, region code, year, window size and options, as C1 · 24's
answer 1a sets out; the proposal's worked example gives three reports'
keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
definition_keys hashes a definition's canonical JSON with SHA-256, the
format key_version 1 names, and builds an economy report's key object:
each side's policy and dynamic, the dataset name, region code and year
both sides share, the window size and the stored options. Names and
codes, not catalog IDs, so a report keeps its key across releases. The
dataset-independent key waits for C1 · 24's revised answer 1a.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ling)

Each lookup inserts against the Public API's unique index for the row's
definition or version set and, when that does nothing, reads the row that
won: a simulation or report by its key, a simulation run by simulation,
year, catalog version and versions, a report run by report and versions.
A failed run is no longer joined, a forced run is never joined and never
takes a run's place, a transaction inserting a run another has inserted
waits for it and joins it, and a run that fails between the insert and
the read is inserted again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
shared_rows' four lookups each insert with ON CONFLICT DO NOTHING against
the Public API's unique index for the row (a simulation's or report's
definition key; a run's simulation or report, catalog version and versions,
among runs neither failed nor forced) and, when that does nothing, read the
row that won. A run that fails between the two is inserted again, up to
three times. The partial indexes' predicates come from the models, so the
conflict targets match the indexes. Forced runs and a budget window's year
runs are never looked up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
C1 · 24's answer 3: a forced re-run is an extra run. Its report run, or
each year's report run of a budget window, and the simulation runs it
calculates are marked forced, so the Public API's one-unfailed-run indexes
leave them out: a forced re-run is inserted beside a pending, running or
succeeded run on the same versions, and is never refused while one is
under way. A forced report re-run calculates both sides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A forced report re-run's report run and new simulation runs, and a
recalculating budget-window re-run's year runs and simulation runs, are
inserted with forced set, so the Public API's one-unfailed-run indexes
leave them out. Such a re-run takes no re-run lock and is never refused
while a run is under way; a forced report re-run reuses no run.
Unforced re-runs are unchanged until C1 · 24's answers 4 and 8.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The service passes a re-run's force to its new run, which the store then
inserts as an extra run marked forced, never refused while a run is under
way. A budget window's recalculate already did. The README and the
module's steps say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Once identical requests join existing runs, the check's report request
ends on a run a check on the same versions already calculated. The check
then forces a new run with POST /reports/{report_id}/runs and
{"force": true}, asked once, and waits for that run; a refused or
unanswered re-run, or one naming no run, fails the check before any read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The check finds or creates its Wales report as before, then sends
POST /reports/{report_id}/runs with {"force": true}, asked once, and
waits for that run. Once identical requests join existing runs, the
report request alone would end on a run an earlier check on the same
versions calculated; a forced run, which nothing joins, calculates on
every deploy (C1 · 24's answer 3). The re-run needs the same
simulation:submit permission and reads no new variable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ling)

C1 · 24's answer 3: a re-run joins a run under way, so #1059's 409
run_in_progress goes, and a re-run on unchanged versions joins the
succeeded run there. Answer 2: a failed run is never joined, so the next
re-run inserts a new one. The re-run answers the run it ended on, whether
it inserted it, and its status.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A simulation re-run now looks up the simulation's run that hasn't failed
and wasn't forced on the same year, catalog version and versions, pending,
running or succeeded, and creates one only when there is none (C1 · 24's
answers 2 and 3). It takes no re-run lock and raises no RunInProgressError:
the Public API's index makes concurrent re-runs end on one run. It returns
the run, its status and whether it was created, so the service starts a
Modal call only for a run it created. The store's creates return what
their insert returns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Once the store joins the simulation's run on the re-run's versions, the
service answers that run and its status, and starts and records no Modal
call for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /simulations/{simulation_id}/runs now answers the run the store
joined, pending, running or succeeded, with its status and no Modal call,
and starts and records a call only for a run the store created. The
service's create helpers return what the store's create returns. The
route's description, the module's steps and the README say so, and that
409 run_in_progress is a report's only.

The checks of which stored records the service re-runs, and what each
calculates on, move unchanged from economy_reruns into
rerunnable_records, which keeps economy_reruns under the 800-line limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
C1 · 24's answer 3: a re-run while the report has a run pending, running
or succeeded on the current versions joins that run, with the simulation
runs it names, and creates nothing, so #1059's 409 run_in_progress goes.
A run still under way on older versions is no run to join: the re-run
creates its own on the current versions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iling)

Another request may create the report's run on the re-run's versions
between the re-run's lookup and its insert. The re-run then joins that
run and takes back the simulation runs it had inserted, so no simulation
run is left that no report run names. The fixtures committing a catalog
for tests of two transactions move to conftest, the deletion of its rows
to committed_rows, so both test modules use them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An unforced report re-run, under the report's re-run lock, joins the
report's run on the current versions that hasn't failed and wasn't
forced, pending, running or succeeded, with the simulation runs it names,
and creates nothing (C1 · 24's answers 2 and 3). Only a budget window's
run under way still refuses it. Otherwise it creates its run inside a
savepoint, inserting the report run against the Public API's index: one
another request created after the lookup is joined, and the savepoint
takes back this re-run's simulation runs. It returns the run, its status
and pair, and whether it created it. shared_rows gains the lookup of a
report's joinable run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Once the store joins the report's run on the re-run's versions, the
service answers that run, its status and the simulation runs it names,
with both sides reused, and starts and records no coordinator call.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
POST /reports/{report_id}/runs now answers the run the store joined,
pending, running or succeeded on the current versions, with its status,
the simulation runs it names and both sides reused, and starts no
coordinator. Only a run the store created gets its call. The 409
run_in_progress is left to budget windows, whose re-runs C1 · 24's answer
8 decides; the README, the route's notes and the module's steps say so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 1's bug: the reuse search behind find_reusable_run and
find_reusable_run_of has no condition on forced, so a forced re-run's
succeeded simulation run is offered to the next identical report, and to
an unforced re-run of the same report. C1 · 24's answer 3 says nothing
joins a forced run. The test rows' run shape gains the forced marker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reuse search behind find_reusable_run and find_reusable_run_of now
leaves out runs marked forced, so a forced re-run's simulation runs are
never named by another report run, as C1 · 24's answer 3 requires. Fixes
review round 1's bug.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The report re-run join test counts report runs for the re-run's report
run ID, and simulation runs for its simulation run IDs, rather than
looking for the report run among simulation runs. The two-transaction
test commits the first transaction only once a backend waits on a lock
that transaction holds (pg_blocking_pids), not on any lock in the
database, and re-raises a thread's error instead of failing on a missing
result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 1: SharedRowNotSettledError, raised when the run a lookup
lost to keeps failing before it can be read, was answered as a 500.
Nothing was written, so the re-run should be answered unavailable and
asked again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The service's create answers SharedRowNotSettledError, raised when the
run a lookup lost to keeps failing before it can be read, as the 503
database_unavailable it gives a database error: its transaction wrote
nothing, so the caller may ask again. Fixes a review round 1 finding.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 1: the OpenAPI document still says a re-run's 201 is a new
run, created and started, and that a simulation's re-run answers 409
run_in_progress, which it no longer can: it joins its run. The 201 must
say the run may have been joined, and only the report route's 409 names
run_in_progress.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…und 1)

Review round 1's documentation findings:
- The re-run routes' 201 says the run may have been joined, and only the
  report route's 409 names run_in_progress, for a budget-window or
  household report with a run under way; a simulation's re-run joins.
- The report route's description, ReportRerunRequest, the conflict codes,
  and the created answers' status and reused fields describe joining.
- The READMEs say which re-runs take the advisory lock, that a household
  report's re-run is still refused while a run is under way, that an
  unforced re-run whose side has an unreused run on the current versions
  answers 500 until sides are found by version set, and that the GET
  routes read the newest run, which may be later than the one answered.
- The staging check's script, guide, tests and workflow comment say its
  report request reuses simulation runs (and, once reports are found by
  key, the report run), so the forced run is what calculates.
- insert_report_rerun says when its sides' plain inserts are refused, and
  the budget-window refusal shares one query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seven tests stored two runs that haven't failed and weren't forced on one
version set, which the Public API's next revision refuses; each failed
with a unique violation against it and passes now, its intent kept:
- the test rows' report run takes its versions, and the later run the
  re-run tests reuse is used by another report's run on the later
  versions, not by a run of the stored report, which a re-run would now
  join (five re-run tests);
- the re-run after a failed re-run fails that re-run's simulation runs
  too, as its runs have ended;
- the newest-run read's two later runs are forced re-runs' runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 14 commits November 14, 2026 03:46
…ice (failing)

C1 · 24's answer 8, in the entry: a window request reads no run to reuse,
finds or creates its window by key (its definition holding its first year,
size and SPM selection, its inputs the selection alone), starts its first
years only for a window run it created, on the year runs the store names
(a year may name another report's run), and answers a joined window run as
it is, starting no year. A keyed window's unforced re-run joins its window
run the same way; force recalculates. The fakes find or create windows and
can join.

Also the pinned formatter's one-line change to the household builder's
test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…indow re-runs

C1 · 24's answer 8, in the entry. A window request reads no run to reuse,
checks its first year's calculation on the run IDs it proposes, and finds
or creates its window by key (KeyedBudgetWindow: the definition holds its
first year, size and SPM selection; the report stores the selection
alone); only for a window run it created does it start the first years,
on the year runs the store names, which may be another report's. A joined
window run is answered as it is. A keyed window's unforced re-run joins
its window run the same way, and starts its first years only for a window
run it created; force, or a window without keys, recalculates.

The economy service's reusable-side helper, which only the window create
used, goes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… (failing)

C1 · 24's answer 6: the window's report stores neither, as an economy or
household report's doesn't.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
C1 · 24's answer 6, as for economy and household reports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… force

The routes, ReportRerunRequest, the re-run modules and both READMEs say a
budget window, its years' simulations and its window run are found or
created by key, a window year sharing a single-year report's runs; that
only a window run the request created has its first years started; that a
keyed window's re-run joins its window run, force or a window created
before keys recalculating; and that no keyed create writes a per-user
column.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 5's finding 4 and 6. insert_report_rerun no longer refuses while a
window run of the report is under way: the service re-runs a window with
insert_window_rerun, and nothing maps the refusal any more, so
RunInProgressError, its query and the test of it go. The unused
ServiceSettings.reuse_prefix and the test-only
FoundEconomyReport.inserted_simulation_run_ids go too (the tests check each
side's run_inserted instead). find_or_insert_report_run_on_sides returns no
side runs after a lost race, since the savepoint took back what it inserted.
The output bucket setting stays, read by nothing, until the reuse search
is removed with its deployment setting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 5's findings 1 to 3: no re-run writes a trigger (answer 6); a keyed
window's re-run joins after the re-run lock, as a report's does; a
household report's re-run joins or forces as an economy one's does; window
answers name the window run a request ended on, joined or created, and a
side reused when this request didn't insert its run; request options are
what the calculation reads, the report storing the SPM selection alone;
include_cliffs: true is refused until C1 · 39; the old create functions
without keys are no longer the service's, kept for tests; the persistence
README lists the household keys. The window options test is renamed for
what it checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Round 5's finding 5. Both find the stored window, whose earlier window run
failed, so neither finds a run to join; the first inserts a window run and
holds its transaction open until the second waits for it at the window
run's index, then the second joins it with its year runs, and only the
first starts its years. It passes against the code already there. The
two-transaction helpers moved from the economy test module to
committed_rows, which both use.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…er, test isolation

- Docs: the persistence README no longer names the removed run-in-progress
  refusal; the entry README's economy re-run names no trigger; the
  deployment notes say the service no longer reads the output bucket; the
  Stage 12 document's re-run and household re-run passages describe joining
  and forced runs with no trigger.
- budget_window_records' year run row, used only by the window insert
  without keys, is private and writes initial and unforced itself; the
  module docstring says such a window could name a succeeded run on a
  year's side.
- The window two-transaction test keys its window under the committed
  catalog's own country, and asserts that the second request waited at the
  window run's insert (wait_until_another_backend_waits_for now returns the
  waiting statement); test_shared_rows uses committed_rows' helpers instead
  of its own copies. The pinned formatter's changes to three test files
  come with them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… a race in a test helper

- The entry README names the Public API's per-user report-run record, not
  API v1's.
- The deployment notes say the deploy still sets and checks
  V2_SIMULATION_OUTPUT_BUCKET, the service reads it into its settings and
  uses it for nothing, and both go with the run-reuse search later.
- The Stage 12 document's re-run passage says a simulation's re-run joins
  or creates its run whether or not the simulation has a key, and takes no
  force.
- budget_window_records' year run row docstring says it builds each year's
  run of a window's first window run.
- test_shared_rows loses a comment its moved constant left behind.
- wait_until_another_backend_waits_for reads the waiting backend's
  statement again in a transaction of its own once it has found it, so the
  statement is the one the backend waits in, not the one before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The settings comment, the Cloud Run deploy script's comment above the
bucket check, the economy deployment test's module docstring and
ServiceSettings now say the same as the deployment notes: every deploy sets
and checks V2_SIMULATION_OUTPUT_BUCKET, the service reads it into its
settings and uses it for nothing, and both go with the run-reuse search.
The two-transaction helper's docstring says the statement is read once the
waiting backend's pid is found, and test_shared_rows is formatted again.
Comments only in the deploy script: no variable or check changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d until it merges

The Public API schema job built the tables at 429d348ae, before #4303, so
every test needing the key columns, forced markers and nullable per-user
columns failed. 63cce8273 is #4303's pushed head, which this branch's
models match (B1 · 5's comparison passes locally against it). The job's
last step requires a commit on the Public API's master, so it fails until
#4303 merges; then the pin moves to #4303's merge commit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 2 commits November 14, 2026 05:18
…un test

The Test integration job generates the client from the entry's OpenAPI
document, which no longer lists run_in_progress, so the test's sample 409
with that code failed to parse. No re-run is refused while a run is under
way; the 409 left is rerun_not_supported. Run by node ID against the
regenerated client, passing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in #1068 (local-area reports). The only conflict was
`public-api-commit`: main pins the Public API's ca4d396eb (#4305's merge,
which adds `withheld_area_codes`), and this branch pins
PolicyEngine/policyengine-api#4303's head 63cce8273, which already contains
ca4d396eb. This branch's pin stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 2 commits November 17, 2026 02:07
…e entrypoint's output-bucket setting

After #1074 the Simulation Entrypoint finds or creates every simulation,
report and run by its definition key, so nothing it runs calls the code
that created those rows without keys or searched for an earlier succeeded
simulation run to reuse. Remove that code, with no change in behaviour:

- the persistence library's insert_economy_report,
  insert_economy_simulation, insert_household_report and
  insert_budget_window_report, their input types, and the store methods
  that wrapped them (EconomyRunStore.create_report, create_simulation and
  create_budget_window_report, HouseholdReportStore.create_report);
- find_reusable_run, find_reusable_run_of, ReusableRun, RunReuse and the
  searches built on them, the store's `reuse` parameters (and the window
  reads' `years`, which only fed the searches), and the reusable_run and
  reusable_runs fields;
- RerunSide.reused and the checks that refused it;
- ServiceSettings.output_bucket, Settings.v2_simulation_output_bucket, and
  the entrypoint deploy's V2_SIMULATION_OUTPUT_BUCKET format check, runtime
  variable and workflow input. The Stage 12 steps, scripts and Modal
  workers keep the bucket.

Tests that store a record created before keys now insert its rows
directly (economy_rows.insert_pending_report, household_report_rows and a
window helper in the window re-run tests). The economy store's
commit-failure tests use a forced report re-run, whose create only
inserts rows.

Fixes #1081

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…orrect the Stage 12 creation passages

Review round 1: the tests deleted with the create functions without keys
were the only database tests that a refused region, dataset, dynamic,
policy or household reference names its column, which the Simulation
Entrypoint turns into its unknown_* refusals. The same checks now call
find_or_create_economy_report, find_or_create_household_report and
find_or_create_budget_window, and pass, as the behaviour is unchanged.
The Stage 12 migration doc's standalone-simulation and budget-window
passages now say the service finds or creates the rows by key and starts
a call only for a run it created.

Refs #1081

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in policyengine.py 6.20.0 (#1076) and #1088, which removes the
staging household and economy check jobs from the deploy workflow,
because a new .py version has no Public API catalog rows until the
Public API deploys it.

Two conflicts, resolved to main's side:
- `simulation-deploy.reusable.yml`: both check jobs are removed, which
  also drops this branch's renamed economy check step and its comment.
  This branch's removal of `V2_SIMULATION_OUTPUT_BUCKET_VALUE` from the
  entrypoint's deploy step stays.
- `tests/test_staging_economy_check.py`: deleted with its job, as main
  deletes it. The script `.github/scripts/simulation-staging-economy-check.sh`
  keeps this branch's forced re-run; main keeps the script too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in #1072 (C1 · 26: the v2 coordinator and household function
wait for a simulation run another call holds, and each run stores its
own simulation's plan). This branch already carried copies of its
contract changes, which match main's.

One conflict, in the Stage 12 migration document's coordinator passage:
main's sentence on waiting for a run another call holds (pointing to
"Shared simulation runs") is kept, with this branch's pointer to the
joined runs in place of the removed reuse of succeeded runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review November 17, 2026 09:03
@anth-volk
anth-volk merged commit 9015adb into main Nov 17, 2026
30 checks passed
@anth-volk
anth-volk deleted the v2plan/c1-27-find-or-create-by-key branch November 17, 2026 09:03
anth-volk added a commit that referenced this pull request Nov 17, 2026
…rce branch

Main's reports and simulations are now found or created by key, with no
run reuse, and it refuses include_cliffs: true. This branch's changes are
kept on that design:

- read_catalog_rows lives in main's economy_inputs.py; its local-area
  dataset branch and LocalAreaDatasetMismatchError move there from this
  branch's economy_catalog_rows.py, which is removed. The store's
  read_report_inputs keeps local_area_dataset_name, without the reuse
  argument main removed.
- economy_request_for and calculation_scope, now in economy_requests.py,
  name no dataset, as before.
- The local-area tests send include_cliffs: false, and expect the report
  to store no option and its key to name the local-area dataset.
- The staging check's script comment and DEPLOYMENT.md describe both the
  forced re-run and data_source.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant