Skip to content

Answer 422 for a stored policy whose values simulations can't read - #4251

Merged
anth-volk merged 7 commits into
masterfrom
fix/stored-policy-values-422
Nov 9, 2026
Merged

anth-volk merged 7 commits into
masterfrom
fix/stored-policy-values-422

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4244

Summary

  • Before the annual or budget-window economy route starts a job, and before it replays a stored failure, the service checks the stored reform and baseline policies for entries economy simulations refuse (policyengine_api/services/economy_policy_values.py):
    • every value must pass the value check Refuse new policies whose period keys or values simulations cannot read #4142 added for new policies (is_readable_value in policyengine_api/utils/policy_data_validation.py): a finite number, a boolean, null, a list, or "Infinity" or "-Infinity". That includes a parameter given one value without periods;
    • every period key must be one the simulation service can parse: one date or two joined by ., each read with strptime and %Y-%m-%d, with the end no earlier than the start.
  • When any entry fails, the route answers 422 in the Public API's usual error body. Nothing is claimed, submitted, stored or replayed, and the refusal is logged at WARNING (Refused economic impact request for a stored policy that simulations can't read, with policy_ids and invalid_entries), so it forms no error group in the daily report. The shape (period keys illustrative):
    {
      "status": "error",
      "message": "Policy #29093 holds period keys or values that simulations can't read, so no economic impact can be calculated for it: Value at `data.gov.irs.credits.ctc.amount.base[0].amount.2023-01-01` must be a finite number, a boolean, null, a list, or the string \"Infinity\" or \"-Infinity\", but it is \"3000\". …",
      "result": null,
      "errors": [
        {"type": "policy_value", "name": "gov.irs.credits.ctc.amount.base[0].amount", "path": "data.gov.irs.credits.ctc.amount.base[0].amount.2023-01-01", "message": "…", "policy_id": 29093}
      ]
    }
    Each entry in errors has the keys the policy route's 400 uses for a new policy, plus policy_id, reform first. A policy compared with itself is named once.
  • Policies whose values pass are submitted, polled and replayed as before.
  • Where the check runs:
    • annual: _create_economic_impact (after both policies load, so a missing policy still answers 404 first) and the stored-failure branch of _answer_from_stored_impact;
    • budget window: _submit_budget_window and the failed branch of _budget_window_result_from_state.
    • A completed or computing record is answered as before. A job already running for such a policy, started before this deploy or by an older revision, is polled until it fails, and that one poll answers 502 (annual) or 200 with status: error (budget window) and logs at ERROR as today. Later requests get the 422.
  • Before a replay the check reads both policies again, two lookups by primary key, only when a failure is stored. A policy that no longer exists, or whose stored JSON isn't an object, is left out. If the lookup or parsing fails, as during a database outage, the check is skipped, logged at WARNING (Could not check the stored policies before replaying a stored economic impact failure, with error_type), and the failure is replayed as before; during an outage the replay can wait for the database to fail first.
  • The service returns the refusal as a result (unreadable_policies on EconomicImpactResult and BudgetWindowEconomicImpactResult), and the routes answer 422 only for an UnreadablePolicies refusal, so results that tests stub with a Mock or another class pass through unchanged.
  • describe_value in policy_data_validation.py is now public, for the economy check's messages. The economy routes' docstrings and docs/canonical-spm.md describe the 422.
  • Repairing stored policies, for example converting numeric text to numbers, is out of scope.

Why the check is narrower than the checks for new policies

The checks #4142 added for new policies also cover household calculations, which read only zero-padded dates and only objects of periods. Economy simulations read a stored policy through the simulation service and .py, which also read unpadded months and days in period keys (policyengine-sim-api's request_policy.py parses with strptime "to accept what it accepted"), and apply a value given without periods from 1 January of the simulated year (.py's compile_reform). A stored policy written either way runs today, so refusing it would break it. The value check itself is #4142's.

Open question: "Infinity" text

Review round 2 found that "Infinity" and "-Infinity" pass #4142's value check (the Public API writes infinite values that way), but in the simulation executor's environment .py keeps them as text and policyengine-core then raises the same ParameterParsingError as for numeric text; the household path converts them in country.py, the economy path doesn't. If that holds for deployed jobs, a stored or new policy with an infinite value fails every economy job. That isn't confirmed end to end, and refusing these values would refuse what new policies may save, so this PR leaves them as they are; it needs a decision, either refusing them for economy jobs or converting them before the simulation.

Inline suppression

policyengine_api/services/economy_simulation_requests.py: except Exception as error: # noqa: BLE001 # the check must not make a stored failure's replay fail; logged below. The check added before a replay must never make the replay fail, whatever the policy lookup raises; a narrower except would let an unexpected exception from the lookup turn a replay that works today into a 500. The handler logs the exception's type at WARNING.

How this interacts with #4220

#4220 (draft, issue #4219) changes the except Exception blocks of get_economic_impact and get_budget_window_economic_impact so that a refused request is logged at WARNING and every other exception at ERROR.

Testing

  • tests/unit/routes/test_economy_unreadable_stored_policy.py (new) serves both routes from a real service over in-memory caches, through a real observability runtime with the API's logging settings that keeps every record, including request.completed. Each test runs for the annual and the budget-window route; 30 cases.
    • A stored reform, baseline, both, or one policy compared with itself, with text values or an impossible date: 422 with the exact body, nothing submitted or claimed, nothing in the cache, no record at ERROR or above, and one WARNING refusal record. Also a value without periods stored as text, and period keys the simulation service can't parse (a reversed range, three dates, a bare year).
    • A job submitted while the policy still read as valid, then failed: the next request gets the 422 instead of the replay, with no record at ERROR or above and no further poll or submission.
    • Readable policies are submitted unchanged, including unpadded dates in a key or a range and a value without periods, and a readable policy's stored failure is replayed as before (502 annual, 200 with status: error for the budget window), also when the policy lookup fails before the replay.
  • Red-green: the first commit added the first 14 cases (10 refusals failed, 4 for readable policies passed), the second added the check. After review round 1, 10 new cases were committed failing and then fixed. The 6 period-key cases added after round 2 passed at once: they pin behaviour the round 1 fix already had.
  • tests/unit/services/test_economy_service_annual_simulation_requests.py::TestEconomyService::TestGetEconomicImpact::test__given_policies_created_through_orm__submits_decoded_json (changed): CI found that its reform's period key "2026", a bare year, is now refused. The simulation service refuses it too (strptime with %Y-%m-%d), so a real job for it would fail; the test is about submitting ORM-decoded JSON, so it now uses 2026-01-01.2100-12-31. It failed before that change and passes after, run by node ID.
  • Only the tests this branch adds or modifies were run locally, by node ID. make quality passes.

Adversarial review

Three rounds, each by a fresh review agent given the diff and the issue.

  • Round 1 found:
    • The routes read unreadable_policies from whatever the service returned, which breaks about 18 existing tests that stub the result with a Mock, a SimpleNamespace or a fake class. Fixed: only an UnreadablePolicies refusal answers 422.
    • The check refused stored policies economy jobs read (unpadded dates in period keys, a value without periods). Fixed: the narrower check above.
    • The check before a replay turned a failed policy lookup into a 500 at ERROR. Fixed: the check is skipped, logged at WARNING, and the failure replayed.
  • Round 2 found no defect in what the PR implements. It confirmed that the period-key check gives the simulation service's answer on 25 keys and that the refused values fail in .py and core. It raised the "Infinity" question above, asked for this suppression to be listed and the description brought up to date, moved the new WARNING into a module the logger fixture lists, and asked for more period-key tests; all done.
  • Round 3, after CI found the existing test above, found no defect. It confirmed that the simulation service refuses "2026" on every economy path, that the branch's period-key check and the simulation service's agree on 33 keys, and, by reading, that no other existing test's fixture policy would be refused. It noted that the readable-policy replay test would also pass if a stored failure polled the job again; the test now checks that it doesn't.

🤖 Generated with Claude Code

anth-volk and others added 2 commits November 9, 2026 22:04
…failing)

An annual or budget-window economy request whose stored reform or baseline
policy holds period keys or values simulations can't read must answer 422
naming the policy, parameters and periods, submit and claim nothing, and
write nothing at ERROR, also when a failure is stored for the request.
Readable policies are submitted and their failures replayed as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before an annual or budget-window economy request starts a job, and before
it replays a stored failure, the service checks the stored reform and
baseline policies with the checks new policies get. When any period key or
value fails, the route answers 422 in the usual error body, with errors
naming each entry's parameter, path and policy ID, and nothing is claimed,
submitted or replayed. The refusal is logged at WARNING.

The service returns the refusal as a result (unreadable_policies on
EconomicImpactResult and BudgetWindowEconomicImpactResult) instead of
raising, so the request methods' except blocks, which log every exception
at ERROR, never see it. Readable policies are submitted, and their stored
failures replayed, as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk and others added 2 commits November 9, 2026 22:32
…ailing)

Review round 1 found that the check refused stored policies the
simulation service and .py read (unpadded dates in period keys, a value
without periods), and that a failed policy lookup before a replay turned
the stored failure's replay into a 500. The new tests submit those
policies unchanged, refuse a value without periods that is text, and
replay a stored failure as before when the policies can't be read.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 1 found three problems:
- The routes read unreadable_policies from whatever the service returned,
  so tests that stub the result with a Mock, a SimpleNamespace or a fake
  class got a 422 they can't serialize or an AttributeError. The routes
  now answer 422 only for an UnreadablePolicies refusal.
- The checks for new policies also cover household calculations, so they
  refused stored policies economy jobs read: the simulation service reads
  unpadded dates in period keys, and .py applies a value without periods
  from 1 January. The economy check keeps the value check for every value,
  including one without periods, and refuses a period key only when the
  simulation service can't parse it either.
- The check before a replay looked both policies up, so a failed lookup
  turned the replay into a 500 at ERROR. A lookup or parse failure now
  skips the check, logged at WARNING, and the failure is replayed as
  before.

describe_value in policy_data_validation is now public, for the economy
check's messages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.30508% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.31%. Comparing base (8819e66) to head (6dcaad3).
⚠️ Report is 36 commits behind head on master.

Files with missing lines Patch % Lines
policyengine_api/services/economy_policy_values.py 96.87% 0 Missing and 2 partials ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##           master    #4251    +/-   ##
========================================
  Coverage   92.30%   92.31%            
========================================
  Files         300      301     +1     
  Lines       17116    17233   +117     
  Branches     2851     2883    +32     
========================================
+ Hits        15799    15908   +109     
- Misses        791      797     +6     
- Partials      526      528     +2     
Flag Coverage Δ
integration 53.13% <0.84%> (-0.37%) ⬇️
unit 89.96% <98.30%> (+0.02%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…keys

Review round 2 noted that economy_simulation_requests now logged through
its own logger, which ECONOMY_SERVICE_LOGGER_MODULES doesn't list. The
WARNING for a skipped check now comes from economy_impact_types, which
the list already holds. New tests refuse a reversed range, a three-date
key and a bare year, which the simulation service can't parse either.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI found that test__given_policies_created_through_orm__submits_decoded_json
now gets a refusal: its reform's period key "2026" is a bare year, which
the simulation service's period parsing refuses too, so a real job for it
would fail. The test is about submitting ORM-decoded JSON, so it now uses
a date range the simulation service reads.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 3 showed the replay test would also pass if a stored failure
polled the job again instead of being replayed. It now checks that the
replay makes no further poll.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review November 9, 2026 21:00
@anth-volk
anth-volk merged commit 4612b77 into master Nov 9, 2026
24 checks passed
@anth-volk
anth-volk deleted the fix/stored-policy-values-422 branch November 9, 2026 21:01

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Refuse economy requests for a stored policy whose values simulations can't read, with a 422

1 participant