Repository navigation
Answer 422 for a stored policy whose values simulations can't read - #4251
Merged
Merged
Conversation
…failing) An annual or budget-window economy request whose stored reform or baseline policy holds period keys or values simulations can't read must answer 422 naming the policy, parameters and periods, submit and claim nothing, and write nothing at ERROR, also when a failure is stored for the request. Readable policies are submitted and their failures replayed as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before an annual or budget-window economy request starts a job, and before it replays a stored failure, the service checks the stored reform and baseline policies with the checks new policies get. When any period key or value fails, the route answers 422 in the usual error body, with errors naming each entry's parameter, path and policy ID, and nothing is claimed, submitted or replayed. The refusal is logged at WARNING. The service returns the refusal as a result (unreadable_policies on EconomicImpactResult and BudgetWindowEconomicImpactResult) instead of raising, so the request methods' except blocks, which log every exception at ERROR, never see it. Readable policies are submitted, and their stored failures replayed, as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ailing) Review round 1 found that the check refused stored policies the simulation service and .py read (unpadded dates in period keys, a value without periods), and that a failed policy lookup before a replay turned the stored failure's replay into a 500. The new tests submit those policies unchanged, refuse a value without periods that is text, and replay a stored failure as before when the policies can't be read. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 1 found three problems: - The routes read unreadable_policies from whatever the service returned, so tests that stub the result with a Mock, a SimpleNamespace or a fake class got a 422 they can't serialize or an AttributeError. The routes now answer 422 only for an UnreadablePolicies refusal. - The checks for new policies also cover household calculations, so they refused stored policies economy jobs read: the simulation service reads unpadded dates in period keys, and .py applies a value without periods from 1 January. The economy check keeps the value check for every value, including one without periods, and refuses a period key only when the simulation service can't parse it either. - The check before a replay looked both policies up, so a failed lookup turned the replay into a 500 at ERROR. A lookup or parse failure now skips the check, logged at WARNING, and the failure is replayed as before. describe_value in policy_data_validation is now public, for the economy check's messages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4251 +/- ##
========================================
Coverage 92.30% 92.31%
========================================
Files 300 301 +1
Lines 17116 17233 +117
Branches 2851 2883 +32
========================================
+ Hits 15799 15908 +109
- Misses 791 797 +6
- Partials 526 528 +2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…keys Review round 2 noted that economy_simulation_requests now logged through its own logger, which ECONOMY_SERVICE_LOGGER_MODULES doesn't list. The WARNING for a skipped check now comes from economy_impact_types, which the list already holds. New tests refuse a reversed range, a three-date key and a bare year, which the simulation service can't parse either. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI found that test__given_policies_created_through_orm__submits_decoded_json now gets a refusal: its reform's period key "2026" is a bare year, which the simulation service's period parsing refuses too, so a real job for it would fail. The test is about submitting ORM-decoded JSON, so it now uses a date range the simulation service reads. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 3 showed the replay test would also pass if a stored failure polled the job again instead of being replayed. It now checks that the replay makes no further poll. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4244
Summary
policyengine_api/services/economy_policy_values.py):is_readable_valueinpolicyengine_api/utils/policy_data_validation.py): a finite number, a boolean, null, a list, or"Infinity"or"-Infinity". That includes a parameter given one value without periods;., each read withstrptimeand%Y-%m-%d, with the end no earlier than the start.Refused economic impact request for a stored policy that simulations can't read, withpolicy_idsandinvalid_entries), so it forms no error group in the daily report. The shape (period keys illustrative):{ "status": "error", "message": "Policy #29093 holds period keys or values that simulations can't read, so no economic impact can be calculated for it: Value at `data.gov.irs.credits.ctc.amount.base[0].amount.2023-01-01` must be a finite number, a boolean, null, a list, or the string \"Infinity\" or \"-Infinity\", but it is \"3000\". …", "result": null, "errors": [ {"type": "policy_value", "name": "gov.irs.credits.ctc.amount.base[0].amount", "path": "data.gov.irs.credits.ctc.amount.base[0].amount.2023-01-01", "message": "…", "policy_id": 29093} ] }errorshas the keys the policy route's 400 uses for a new policy, pluspolicy_id, reform first. A policy compared with itself is named once._create_economic_impact(after both policies load, so a missing policy still answers 404 first) and the stored-failure branch of_answer_from_stored_impact;_submit_budget_windowand thefailedbranch of_budget_window_result_from_state.status: error(budget window) and logs at ERROR as today. Later requests get the 422.Could not check the stored policies before replaying a stored economic impact failure, witherror_type), and the failure is replayed as before; during an outage the replay can wait for the database to fail first.unreadable_policiesonEconomicImpactResultandBudgetWindowEconomicImpactResult), and the routes answer 422 only for anUnreadablePoliciesrefusal, so results that tests stub with aMockor another class pass through unchanged.describe_valueinpolicy_data_validation.pyis now public, for the economy check's messages. The economy routes' docstrings anddocs/canonical-spm.mddescribe the 422.Why the check is narrower than the checks for new policies
The checks #4142 added for new policies also cover household calculations, which read only zero-padded dates and only objects of periods. Economy simulations read a stored policy through the simulation service and .py, which also read unpadded months and days in period keys (policyengine-sim-api's
request_policy.pyparses withstrptime"to accept what it accepted"), and apply a value given without periods from 1 January of the simulated year (.py'scompile_reform). A stored policy written either way runs today, so refusing it would break it. The value check itself is #4142's.Open question:
"Infinity"textReview round 2 found that
"Infinity"and"-Infinity"pass #4142's value check (the Public API writes infinite values that way), but in the simulation executor's environment .py keeps them as text and policyengine-core then raises the sameParameterParsingErroras for numeric text; the household path converts them incountry.py, the economy path doesn't. If that holds for deployed jobs, a stored or new policy with an infinite value fails every economy job. That isn't confirmed end to end, and refusing these values would refuse what new policies may save, so this PR leaves them as they are; it needs a decision, either refusing them for economy jobs or converting them before the simulation.Inline suppression
policyengine_api/services/economy_simulation_requests.py:except Exception as error: # noqa: BLE001 # the check must not make a stored failure's replay fail; logged below. The check added before a replay must never make the replay fail, whatever the policy lookup raises; a narrowerexceptwould let an unexpected exception from the lookup turn a replay that works today into a 500. The handler logs the exception's type at WARNING.How this interacts with #4220
#4220 (draft, issue #4219) changes the
except Exceptionblocks ofget_economic_impactandget_budget_window_economic_impactso that a refused request is logged at WARNING and every other exception at ERROR.economy_impact_types, whichECONOMY_SERVICE_LOGGER_MODULESalready lists, so it doesn't change the test fixture Log refused economy requests at WARNING instead of ERROR #4220 adds a module to.git merge-treeof the two branches reports no conflict, and this PR's tests pass on the merged tree.Testing
tests/unit/routes/test_economy_unreadable_stored_policy.py(new) serves both routes from a real service over in-memory caches, through a real observability runtime with the API's logging settings that keeps every record, includingrequest.completed. Each test runs for the annual and the budget-window route; 30 cases.status: errorfor the budget window), also when the policy lookup fails before the replay.tests/unit/services/test_economy_service_annual_simulation_requests.py::TestEconomyService::TestGetEconomicImpact::test__given_policies_created_through_orm__submits_decoded_json(changed): CI found that its reform's period key"2026", a bare year, is now refused. The simulation service refuses it too (strptimewith%Y-%m-%d), so a real job for it would fail; the test is about submitting ORM-decoded JSON, so it now uses2026-01-01.2100-12-31. It failed before that change and passes after, run by node ID.make qualitypasses.Adversarial review
Three rounds, each by a fresh review agent given the diff and the issue.
unreadable_policiesfrom whatever the service returned, which breaks about 18 existing tests that stub the result with aMock, aSimpleNamespaceor a fake class. Fixed: only anUnreadablePoliciesrefusal answers 422."Infinity"question above, asked for this suppression to be listed and the description brought up to date, moved the new WARNING into a module the logger fixture lists, and asked for more period-key tests; all done."2026"on every economy path, that the branch's period-key check and the simulation service's agree on 33 keys, and, by reading, that no other existing test's fixture policy would be refused. It noted that the readable-policy replay test would also pass if a stored failure polled the job again; the test now checks that it doesn't.🤖 Generated with Claude Code