Repository navigation
Conversation
When the commit fails after the job's row was inserted, PostgreSQL may already have applied it. The household job store is to raise EconomyRecordOutcomeUnknownError then, the service HouseholdCalculationOutcomeUnknownError with the job's ID and the commit error's type, and the route a JSON 500 internal_error, logged as household_job_outcome_unknown. Guard tests: a failed insert statement and a refused reference keep their own errors, and a failed insert still answers database_unavailable and logs household_job_not_created. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the database fails after the household job's row was inserted, PostgreSQL may already have applied the commit, so the job may exist. The household job store now tells a failed commit from a failed insert and raises the existing EconomyRecordOutcomeUnknownError, as the economy and household report creates do. The service starts no call and raises HouseholdCalculationOutcomeUnknownError, and POST /household-calculations answers the JSON 500 internal_error, which the Public API reads as a job that may exist, and logs household_job_outcome_unknown with the commit error's type. 503 database_unavailable now means no job was written. economy_run_store imports household_job_store, so create_job imports the error inside the method rather than at module level. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e emits Review round 1's quality findings. The household job store's tests take their commit-failing and statement-failing transactions from a new support module instead of adding another copy. The insert-failure service test now checks only that the service logs household_job_not_created, the event the unconfirmed-commit test checks is absent. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1089
Summary
When the database fails after a household job's row was inserted, PostgreSQL may already have applied the commit, so the job may exist.
POST /household-calculationsnow answers that case with the JSON 500internal_errorinstead of 503database_unavailable, the way the economy and household report routes already do (#1052).HouseholdJobStore.create_jobtells a failed commit from a failed insert. An error while the insert runs propagates as before (nothing written; a refused reference is stillHouseholdJobReferenceError). An error once the insert has run is the commit's: it raises the existingEconomyRecordOutcomeUnknownError, caused by the commit's error, ashousehold_report_storeandeconomy_run_storedo.economy_run_storeimportshousehold_job_store(forUnknownCatalogVersionError), socreate_jobimports the error inside the method, with the reason on thePLC0415suppression.economy_run_store.pyis unchanged.HouseholdCalculationService._createturns that error into the newHouseholdCalculationOutcomeUnknownError(the commit error's type), before any Modal call is started. Every otherSQLAlchemyErrorstill answers 503database_unavailableand logshousehold_job_not_created.{"code": "internal_error", "detail": "The service may have created the household job, but could not confirm it was written."}with status 500 and noRetry-After, and logs the fixed eventhousehold_job_outcome_unknownat error level witherror_typeinpolicyengine_attributes, aseconomy_record_outcome_unknowndoes. The record names no job ID: the observability runtime drops every attribute whose key containshousehold, and the observability guide reservesjob_idfor a Modal invocation. The runtime adds the submission'sobservability_id.fixedchangelog fragment.If the job was written, it stays
pendingwith no Modal call ID, and the stuck-run sweep fails it.How the Public API reads the 500
No Public API change is needed.
policyengine_api/libs/simulation_household_calculations.pyonmaster(_accepted_job) reads sim's 500 asSimulationServiceInternalErrorwith the "may have created the household job" message, which asks the caller to submit again and says this creates a second job if the first request went through. It reads sim's own coded 503 asSimulationServiceUnavailableError, "no job of the submission will run". So with this change, a lost commit is no longer reported as nothing created.Tests
Written first and seen failing (
2d32be72), then passing, each run by node ID:libs/policyengine-simulation-persistence/tests/test_household_job_store.pytest_a_job_whose_commit_fails_may_have_been_createdtest_a_job_whose_insert_fails_was_not_createdtest_a_job_naming_missing_rows_is_refused_before_its_commit, against PostgreSQL 17 with the Public API's v2 schema frommaster, in a real top-level transaction: the keys refuse the insert statement, so a refusal is never taken for a lost commit.projects/policyengine-simulation-entry/tests/test_household_calculations.pytest_a_job_whose_commit_is_unconfirmed_may_exist_and_starts_no_calltest_a_job_whose_insert_fails_is_logged_by_the_serviceprojects/policyengine-simulation-entry/tests/test_household_routes.pytest_a_job_whose_commit_is_unconfirmed_answers_a_500_and_is_loggedThe refused-reference (400
missing_reference), failed-insert route (503) and normal-create cases are covered by existing tests, which this PR doesn't change. Ruff, the rules rendering, the quality guards, the lock checks and Pyright on both packages pass.Review
One read-only review round found no bugs. Two of its quality findings are fixed (
5d145807): the store tests take their failing-transaction fakes from a new support module,tests/transaction_fakes.py, instead of adding another copy; and the insert-failure service test checks only the event the service emits. The older copies of those fakes intest_economy_run_store.py,test_household_report_store.pyandtest_household_report_reruns.py, and the third copy of the insert-then-wrap create logic, are left for after #1074 and #1070, which change those files andeconomy_run_store.py.No Dockerfile, workflow, image or Terraform changes, and no new environment variable or secret.
🤖 Generated with Claude Code