Skip to content

Answer 500, not 503, when a household job's create may have been committed - #1090

Draft
anth-volk wants to merge 3 commits into
mainfrom
v2plan/c1-33-household-create-outcome-unknown
Draft

anth-volk wants to merge 3 commits into
mainfrom
v2plan/c1-33-household-create-outcome-unknown

Conversation

@anth-volk

Copy link
Copy Markdown
Contributor

Fixes #1089

Summary

When the database fails after a household job's row was inserted, PostgreSQL may already have applied the commit, so the job may exist. POST /household-calculations now answers that case with the JSON 500 internal_error instead of 503 database_unavailable, the way the economy and household report routes already do (#1052).

  • Persistence: HouseholdJobStore.create_job tells a failed commit from a failed insert. An error while the insert runs propagates as before (nothing written; a refused reference is still HouseholdJobReferenceError). An error once the insert has run is the commit's: it raises the existing EconomyRecordOutcomeUnknownError, caused by the commit's error, as household_report_store and economy_run_store do.
    • economy_run_store imports household_job_store (for UnknownCatalogVersionError), so create_job imports the error inside the method, with the reason on the PLC0415 suppression. economy_run_store.py is unchanged.
  • Service: HouseholdCalculationService._create turns that error into the new HouseholdCalculationOutcomeUnknownError (the commit error's type), before any Modal call is started. Every other SQLAlchemyError still answers 503 database_unavailable and logs household_job_not_created.
  • Route: answers {"code": "internal_error", "detail": "The service may have created the household job, but could not confirm it was written."} with status 500 and no Retry-After, and logs the fixed event household_job_outcome_unknown at error level with error_type in policyengine_attributes, as economy_record_outcome_unknown does. The record names no job ID: the observability runtime drops every attribute whose key contains household, and the observability guide reserves job_id for a Modal invocation. The runtime adds the submission's observability_id.
  • Docs: the 500 and 503 OpenAPI descriptions, the error-code comments, the module docstrings and both READMEs now say that a submission's 503 means no job will run and that its 500 may follow a write. A fixed changelog fragment.

If the job was written, it stays pending with no Modal call ID, and the stuck-run sweep fails it.

How the Public API reads the 500

No Public API change is needed. policyengine_api/libs/simulation_household_calculations.py on master (_accepted_job) reads sim's 500 as SimulationServiceInternalError with the "may have created the household job" message, which asks the caller to submit again and says this creates a second job if the first request went through. It reads sim's own coded 503 as SimulationServiceUnavailableError, "no job of the submission will run". So with this change, a lost commit is no longer reported as nothing created.

Tests

Written first and seen failing (2d32be72), then passing, each run by node ID:

  • libs/policyengine-simulation-persistence/tests/test_household_job_store.py
    • test_a_job_whose_commit_fails_may_have_been_created
    • test_a_job_whose_insert_fails_was_not_created
    • test_a_job_naming_missing_rows_is_refused_before_its_commit, against PostgreSQL 17 with the Public API's v2 schema from master, in a real top-level transaction: the keys refuse the insert statement, so a refusal is never taken for a lost commit.
  • projects/policyengine-simulation-entry/tests/test_household_calculations.py
    • test_a_job_whose_commit_is_unconfirmed_may_exist_and_starts_no_call
    • test_a_job_whose_insert_fails_is_logged_by_the_service
  • projects/policyengine-simulation-entry/tests/test_household_routes.py
    • test_a_job_whose_commit_is_unconfirmed_answers_a_500_and_is_logged

The refused-reference (400 missing_reference), failed-insert route (503) and normal-create cases are covered by existing tests, which this PR doesn't change. Ruff, the rules rendering, the quality guards, the lock checks and Pyright on both packages pass.

Review

One read-only review round found no bugs. Two of its quality findings are fixed (5d145807): the store tests take their failing-transaction fakes from a new support module, tests/transaction_fakes.py, instead of adding another copy; and the insert-failure service test checks only the event the service emits. The older copies of those fakes in test_economy_run_store.py, test_household_report_store.py and test_household_report_reruns.py, and the third copy of the insert-then-wrap create logic, are left for after #1074 and #1070, which change those files and economy_run_store.py.

No Dockerfile, workflow, image or Terraform changes, and no new environment variable or secret.

🤖 Generated with Claude Code

anth-volk and others added 3 commits November 17, 2026 11:50
When the commit fails after the job's row was inserted, PostgreSQL may
already have applied it. The household job store is to raise
EconomyRecordOutcomeUnknownError then, the service
HouseholdCalculationOutcomeUnknownError with the job's ID and the commit
error's type, and the route a JSON 500 internal_error, logged as
household_job_outcome_unknown. Guard tests: a failed insert statement and a
refused reference keep their own errors, and a failed insert still answers
database_unavailable and logs household_job_not_created.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When the database fails after the household job's row was inserted,
PostgreSQL may already have applied the commit, so the job may exist. The
household job store now tells a failed commit from a failed insert and
raises the existing EconomyRecordOutcomeUnknownError, as the economy and
household report creates do. The service starts no call and raises
HouseholdCalculationOutcomeUnknownError, and POST /household-calculations
answers the JSON 500 internal_error, which the Public API reads as a job
that may exist, and logs household_job_outcome_unknown with the commit
error's type. 503 database_unavailable now means no job was written.

economy_run_store imports household_job_store, so create_job imports the
error inside the method rather than at module level.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e emits

Review round 1's quality findings. The household job store's tests take
their commit-failing and statement-failing transactions from a new support
module instead of adding another copy. The insert-failure service test now
checks only that the service logs household_job_not_created, the event the
unconfirmed-commit test checks is absent.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Answer 500, not 503, when a household job's create may have been committed

1 participant