Repository navigation
Answer 400 for a household calculation whose policy can't be read - #4279
Merged
Merged
Conversation
Telemetry issue #4275: POST /<country_id>/calculate answered 500, with an ERROR record, for a policy that isn't an object, a parameter set to true instead of an object of period keys, and values such as "abc", null or a list. Each failed while the calculation converted the policy's values, after the model's tax-benefit system had been copied. These tests expect a 400 listing each problem, with no ERROR record and before the situation is parsed, for /calculate and /calculate-full, and pin which period keys and values the check accepts, against the installed US and UK models' own value conversion. They fail until the check exists. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before parsing copies the model's tax-benefit system, prepare_household_calculation now checks the request's policy (find_unreadable_policy_inputs, in utils/household_policy_validation.py) and refuses, as InvalidHouseholdInputsError's 400 with an errors list: - a policy that isn't an object (an empty or null one still means no policy, as the calculation reads it); - a parameter whose value isn't an object of period keys; - a period key neither the US reader (policyengine-core's Reform.from_dict) nor the UK reader (policyengine-uk's dated_reforms) reads, and a range that ends before it starts; - a value the parameter's type can't take. _normalize_reform_values converts to the type of the parameter's earliest value; it now shares that conversion (conversion_type, convert_reform_value) with the check, which reads the earliest value from the country metadata. The saved-household route keeps its behaviour: a stored policy isn't checked this way. Fixes #4275 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4279 +/- ##
==========================================
+ Coverage 92.52% 92.61% +0.09%
==========================================
Files 318 319 +1
Lines 18137 18273 +136
Branches 3033 3056 +23
==========================================
+ Hits 16781 16924 +143
+ Misses 813 803 -10
- Partials 543 546 +3
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Review round 1 of #4279 found period keys the check accepted that the calculated country's model then failed on, answering 500 as before: - a range or period whose last day is 9999-12-31, such as 2026-01-01.9999-12-31 or year:9999:1: both models write a change up to the day after its last, which is in year 10000; - ETERNITY on /uk/, and an unpadded single date such as 2026-1 on /us/: the check accepted any key either model read. find_unreadable_policy_inputs now takes the country ID and checks keys as that country's reader does: policyengine-uk's dated_reforms for the UK, policyengine-core's Reform.from_dict for the US. Range ends are read with policyengine-core's instant(), as both readers read them. A range that ends before it starts is refused for the UK, which refuses it, and accepted for the US, which calculates it today. The UK reader's bare years, with spaces around them, are accepted for the UK. The tests now check each country's key lists against its reader: the US lists against Reform.from_dict applied to a parameter of their own, the UK lists against read_reform_dict and the date arithmetic of writing the change. The route tests' stand-in country reads period keys with Reform.from_dict too, so a key the check lets through fails there as it does in production. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 2 of #4279 found two period keys the check got wrong for /uk/: - a range whose last day doesn't exist, such as 2026-01-01.2026-02-30: the check tested the day after it, which policyengine-core rolls over to a real day, so it accepted the key and the UK model then failed with a 500. The UK check now also needs a real last day. The US model applies such a key, so it stays accepted there; - a key such as "²", for which isdigit() is true but int() fails: the check itself raised ValueError, which answered 500. The UK bare-year test now reads such a key as no year, and the key is refused. A route test now posts the same ETERNITY key to /uk/calculate, which refuses it, and /us/calculate, which accepts it, so the service is shown to pass the calculated country to the check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 3 of #4279 found that a key such as month:2026-01:<size> made the check slow in proportion to the size: policyengine-core's Period.stop finds a month period's last day a year at a time, so a billion months took about two seconds and a size with thousands of digits never finished. Master spent the same time in the model, after copying its tax-benefit system. A month period longer than the 9999 * 12 months from year 1 ends after 9999-12-31, which neither model reads, so the check now refuses it before finding its last day. No decision changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 4 of #4279 noted that the module docstring claimed a policy the check lets through fails in the stand-in as in production, which holds only on /us/: the stand-in reads period keys with policyengine-core's Reform.from_dict, not policyengine-uk's reader. The UK case in the file is one the check refuses. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #4275
Summary
POST /<country_id>/calculateand/calculate-fullanswered 500, with a record at ERROR, for apolicythe calculation can't read. Telemetry issue #4275 and two smaller groups (aTypeErrorand aValueError) came from one outside client: 9 requests on 10 and 11 November sent a number aspolicy,trueas a parameter's value, and values"abc",nulland a list. Each failed inPolicyEngineCountry._normalize_reform_values, after_create_simulationhad copied the model's tax-benefit system, so each took 5–17 s before failing;_parse_prepared_household_calculationmaps every exception other thanSituationParsingErrorto a 500.HouseholdCalculationService.prepare_household_calculationnow also runsfind_unreadable_policy_inputs(new,policyengine_api/utils/household_policy_validation.py), next tofind_invalid_inputs. It reads only the request and the country metadata, so a refusal comes before parsing copies the tax-benefit system. Problems raiseInvalidHouseholdInputsError, which answers the existing 400 with anerrorslist, before the request's observability ID starts and before the HTTP cache is read. Nothing is logged at ERROR. It refuses:policythat isn't an object:type: policy, pathpolicy. An empty ornullpolicy ({},null,0,false,"",[]) still means no policy, as the calculation reads it today;type: policy_period, pathpolicy.<parameter>. That includestrue, a number,nulland a list; an empty object still changes nothing;type: policy_period, pathpolicy.<parameter>.<period key>;type: policy_value, pathpolicy.<parameter>.<period key>.type,name,path,message). With any of them, the message startsInvalid calculate input(s):(the existing wording when not every problem is an unrecognized name). For example,{"policy": {"gov.irs.credits.ctc.amount.base[0].amount": {"2026-01-01.2026-12-31": null}}}:{ "status": "error", "message": "Invalid calculate input(s): Value at `policy.gov.irs.credits.ctc.amount.base[0].amount.2026-01-01.2026-12-31` must be a number, or text that reads as one such as \"Infinity\", but it is null.", "result": null, "errors": [ { "type": "policy_value", "name": "gov.irs.credits.ctc.amount.base[0].amount", "path": "policy.gov.irs.credits.ctc.amount.base[0].amount.2026-01-01.2026-12-31", "message": "Value at `policy.gov.irs.credits.ctc.amount.base[0].amount.2026-01-01.2026-12-31` must be a number, or text that reads as one such as \"Infinity\", but it is null." } ] }_normalize_reform_valuesnow converts withconversion_typeandconvert_reform_valuefrom the new module, the same code the check runs, so the two can't drift. The conversion itself is unchanged.What the calculation reads
I ran period keys and values through
PolicyEngineCountry.prepare_calculationandcalculatewith the installed US and UK models (the versionsuv.lockpins).Path.
_simulation_accepts_tax_benefit_system()is false for both countries, so the policy goes toSimulation(reform=...), not_apply_reform_to_system. The US model reads it with policyengine-core'sReform.from_dict, the UK model with its own reader (policyengine_uk/utils/dated_reforms.py).create_policy_reformhas no callers.Values.
_normalize_reform_valuestriesfloat()on a value, then converts it to the type of the parameter's earliest value, reading an int as a float. The check reads that type from the metadata, which lists each parameter's values latest first; a test compares it with the loaded model's for every US and UK parameter.float()reads, such as"2500","Infinity"and"-Infinity". It fails onnull, lists, objects, other text and integers too large for a float.list()reads as characters and keys. It fails on numbers, booleans andnull.bool()converts anything, so the check refuses none."false"and"abc"are both read as true, as today.Period keys. The check reads each key as the calculated country's model does (
is_readable_period_key(period_key, country_id)): range ends with policyengine-core'sinstant(),:periods with itsperiod(), and the UK model's bare years asdated_reforms.bare_yeardoes. Both models then write the change from its first day to the day after its last, so that day must exist.2026-01-01.2026-12-31,2026.2027,2026-01.2026-06,2026-01-01,2026,2026-01,year:2026:5,month:2026-01:3ETERNITY,9999-12-31(one date: from then on)2026-12-31.2026-01-01(ends before it starts)2026-01-01.2026-02-30(last day doesn't exist)2026-1-1,2026-1," 2026"(unpadded, or with spaces, no range)2026-01-01.9999-12-31,year:9999:1(last day 9999-12-31)2026-1-1.2026-12-31,2026-02-30.2026-12-31,2026-13-01.2026-12-31,0000-01-01.2026-12-31, three dates, an empty end,"",abc,eternity,²,2026-02-30,2026-00-01,year:abc:1,day:2026-02-30:1, times of day, spaces in a rangeThe check accepts exactly what the calculated country's model reads, so a reversed range or a non-existent last day still calculates on
/us/…as it does today.Where this departs from the plan for #4275
2026-01-01to fail because_apply_reform_to_systemandcreate_policy_reformsplit each key on., but neither runs for the US or UK: both models read single dates. Refusing them would break requests that work today, so the period-key tests use an impossible date and a range ending 9999-12-31 instead."2500"converts today, so this check accepts it, unlike the check Refuse new policies whose period keys or values simulations cannot read #4142 added for saving a policy.find_invalid_periods_and_valuesisn't reused. It refuses keys and values this path reads (a bare year,year:2026:5, numeric text) and accepts values this path can't convert (nulland lists for a number parameter). The new module reusesdescribe_valueandPERIOD_OBJECT_REQUIREMENTfrom it and leaves what it accepts for saved policies unchanged.Left as they are
"false"and"abc"set a boolean parameter to true, text sets a list parameter to its characters, and a reversed or non-existent range end on/us/…applies as described above. None of these fails today, so none is refused." 2026","02026") is accepted on/uk/…. The UK model reads it as that fiscal year, except on the two parameters it writes straight onto the processed values (gov.economic_assumptions.yoy_growth.capital_gainsandgov.economic_assumptions.indices.capital_gains), where it fails as before. Refusing it would break it for every other parameter./uk/…, some keys whose dates reach into year 9999 still fail, because the UK model can't convert them to fiscal or calendar years. Examples are a range ending9999-12-30ongov.hmrc.income_tax.rates.uk[0].rate, and9999-01-01on a parameter it reads as one value per calendar year, such asgov.economic_assumptions.energy_prices.consumption_shares.electricity.q1. Where the UK model stops depends on the parameter: on the income tax rate, a range ending9999-04-06and the single date9999-01-01both calculate. So a single limit would refuse keys that calculate today. These fail on master too.errorslists unrecognized names (fromfind_invalid_inputs) before the new entries, so it isn't in policy order across the two, and a parameter the model lacks that is also malformed gets one entry for each problem.gov.hmrc.income_tax.rates.scotland.rates[0].threshold._normalize_reform_valuesthen converts withtype(None), which takes no argument, so every policy that sets one answers 500 today. The check doesn't refuse them, because the request isn't at fault; that conversion needs its own fix.Saved-household route
GET /<country_id>/household/<household_id>/policy/<policy_id>keeps its behaviour. It runsfind_invalid_inputson the stored policy, which this PR doesn't change, and not the new check, so a stored policy with such values answers as before: a 500, for example fornullon a number parameter, which saving a policy accepts. The new check's 400 says the caller sent a malformed policy, which is wrong for a stored one; #4251 answers 422 for stored policies simulations can't read, and doing the same here would be a separate change. No telemetry group has come from this route.Testing
tests/unit/routes/test_calculate_malformed_policy.py(new) posts to/us/calculateand/us/calculate-full. The stand-in country has the loaded US model's metadata. It parses a situation by running the loaded model's_normalize_reform_valueson its tax-benefit system, without copying it or building a simulation, and reads each parameter's period keys with policyengine-core'sReform.from_dict, so a policy the check lets through fails as in production.policy,trueas a parameter's value,"abc",null, a list, a period key that isn't a date (2026-02-30.2026-12-31) and a range ending 9999-12-31. Each expects a 400 with the exacterrorsentry and message, no record at ERROR, no situation parsed and no observability ID started.year:2030:2,ETERNITY, numeric text,"Infinity","-Infinity", a boolean and a list; an empty policy; anullpolicy. Each calculates (200), with the policy passed on unchanged.ETERNITYto/uk/calculate(400) and/us/calculate(200), which shows the service passes the calculated country to the check.tests/unit/utils/test_household_policy_validation.py(new) pins each rule: falsy policies, non-object policies and parameter values, values for number, boolean and list parameters, a parameter without recorded values, the UK parameter with a null earliest value, the order of entries, and each country's period-key lists. Checks against the installed packages:_normalize_reform_valuesconverts to, for every US and UK parameter;convert_reform_value, so this checks the type read from metadata;Reform.from_dictapplies it to a parameter;read_reform_dictreads it and the dates it names pass the date arithmetic the UK model does when it writes the change. That arithmetic came from household calculations with the installed UK model, which the test doesn't run.Error calculating household under policyfromcountry.py, as in [telemetry] policyengine-api: AttributeError on/<country_id>/calculate(47418e783c1209c4) #4275; the 2 period-key cases with 200), the 6 accepted cases passed, and the rule tests failed to import the module. The second commit adds the check, and all 106 pass. Each review round's fixes were also written test-first (below); all 174 tests pass on the last commit.make qualitypasses.Review
Four rounds of adversarial self-review ran, each by a new reviewer given only the diff and #4275. Rounds 1–3 were each followed by fixes written test-first.
Round 1 found:
ETERNITYon/uk/…and an unpadded single date on/us/…still answered 500;/us/…, where it calculates today;/uk/…were read differently from the models' readers;Changes: keys are checked as the calculated country's reader reads them (
find_unreadable_policy_inputstakes the country ID), range ends with policyengine-core'sinstant(), and the day after the last day must exist. The tests compare each country's keys with its reader, and the route stand-in reads keys withReform.from_dict. Red before the fix: the range ending 9999-12-31 answered 500 (year 10000 is out of range) in the route tests, and the rule tests failed.Round 2 found:
2026-01-01.2026-02-30) passed, and the UK model failed;², for whichisdigit()is true butint()fails, made the check itself raise.Changes: the UK range check needs a real last day, and the UK bare-year test reads such a key as no year. A route test shows the service checks keys for the calculated country. Red before the fix: 4 UK rule cases.
Round 3 found no regression, and no input from [telemetry] policyengine-api: AttributeError on
/<country_id>/calculate(47418e783c1209c4) #4275 that still fails. It found that amonth:period with a huge size kept the check, like master's model, busy for as long as the size: policyengine-core took 1.6 s to find the last day of a billion months, and a size with thousands of digits never finishes. Change: a month period longer than 9999 × 12 months is refused before its last day is computed. Red before the fix: 2 cases over the test's 0.5 s limit. It also found the spaced UK bare years listed above.Round 4 reviewed every change above, round 3's included. It found no regression: on the keys it ran through the real models, the check refused nothing that calculates on master, and agreed with
Reform.from_dicton 68 keys. It found nothing that makes the check raise or run long. Its one new finding was the late UK dates listed above, which can't be refused without refusing keys that calculate. Its one change: the route tests' docstring now says the stand-in reads period keys only as the US model does. Round 4 found nothing else substantive, so the rounds stopped there.Each round also raised the UK parameters with a null earliest value, the order of
errors, and that the UK reader test repeats the UK model's write-step date arithmetic instead of running the write. These are left as described above.🤖 Generated with Claude Code