Skip to content

Answer 400 for a household calculation whose policy can't be read - #4279

Merged
anth-volk merged 6 commits into
masterfrom
fix/calculate-policy-400
Nov 13, 2026
Merged

anth-volk merged 6 commits into
masterfrom
fix/calculate-policy-400

Conversation

@anth-volk

@anth-volk anth-volk commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #4275

Summary

POST /<country_id>/calculate and /calculate-full answered 500, with a record at ERROR, for a policy the calculation can't read. Telemetry issue #4275 and two smaller groups (a TypeError and a ValueError) came from one outside client: 9 requests on 10 and 11 November sent a number as policy, true as a parameter's value, and values "abc", null and a list. Each failed in PolicyEngineCountry._normalize_reform_values, after _create_simulation had copied the model's tax-benefit system, so each took 5–17 s before failing; _parse_prepared_household_calculation maps every exception other than SituationParsingError to a 500.

  • HouseholdCalculationService.prepare_household_calculation now also runs find_unreadable_policy_inputs (new, policyengine_api/utils/household_policy_validation.py), next to find_invalid_inputs. It reads only the request and the country metadata, so a refusal comes before parsing copies the tax-benefit system. Problems raise InvalidHouseholdInputsError, which answers the existing 400 with an errors list, before the request's observability ID starts and before the HTTP cache is read. Nothing is logged at ERROR. It refuses:
    • a policy that isn't an object: type: policy, path policy. An empty or null policy ({}, null, 0, false, "", []) still means no policy, as the calculation reads it today;
    • a parameter whose value isn't an object of period keys: type: policy_period, path policy.<parameter>. That includes true, a number, null and a list; an empty object still changes nothing;
    • a period key the calculated country's model can't read or apply: type: policy_period, path policy.<parameter>.<period key>;
    • a value the parameter's type can't take: type: policy_value, path policy.<parameter>.<period key>.
  • Each entry has the keys of an unrecognized input's entry (type, name, path, message). With any of them, the message starts Invalid calculate input(s): (the existing wording when not every problem is an unrecognized name). For example, {"policy": {"gov.irs.credits.ctc.amount.base[0].amount": {"2026-01-01.2026-12-31": null}}}:
    {
      "status": "error",
      "message": "Invalid calculate input(s): Value at `policy.gov.irs.credits.ctc.amount.base[0].amount.2026-01-01.2026-12-31` must be a number, or text that reads as one such as \"Infinity\", but it is null.",
      "result": null,
      "errors": [
        {
          "type": "policy_value",
          "name": "gov.irs.credits.ctc.amount.base[0].amount",
          "path": "policy.gov.irs.credits.ctc.amount.base[0].amount.2026-01-01.2026-12-31",
          "message": "Value at `policy.gov.irs.credits.ctc.amount.base[0].amount.2026-01-01.2026-12-31` must be a number, or text that reads as one such as \"Infinity\", but it is null."
        }
      ]
    }
  • _normalize_reform_values now converts with conversion_type and convert_reform_value from the new module, the same code the check runs, so the two can't drift. The conversion itself is unchanged.

What the calculation reads

I ran period keys and values through PolicyEngineCountry.prepare_calculation and calculate with the installed US and UK models (the versions uv.lock pins).

  • Path. _simulation_accepts_tax_benefit_system() is false for both countries, so the policy goes to Simulation(reform=...), not _apply_reform_to_system. The US model reads it with policyengine-core's Reform.from_dict, the UK model with its own reader (policyengine_uk/utils/dated_reforms.py). create_policy_reform has no callers.

  • Values. _normalize_reform_values tries float() on a value, then converts it to the type of the parameter's earliest value, reading an int as a float. The check reads that type from the metadata, which lists each parameter's values latest first; a test compares it with the loaded model's for every US and UK parameter.

    • A number parameter takes numbers, booleans and text float() reads, such as "2500", "Infinity" and "-Infinity". It fails on null, lists, objects, other text and integers too large for a float.
    • A list parameter takes lists, and also text and objects, which list() reads as characters and keys. It fails on numbers, booleans and null.
    • A boolean parameter takes any value, since bool() converts anything, so the check refuses none. "false" and "abc" are both read as true, as today.
  • Period keys. The check reads each key as the calculated country's model does (is_readable_period_key(period_key, country_id)): range ends with policyengine-core's instant(), : periods with its period(), and the UK model's bare years as dated_reforms.bare_year does. Both models then write the change from its first day to the day after its last, so that day must exist.

    Key US model UK model
    2026-01-01.2026-12-31, 2026.2027, 2026-01.2026-06, 2026-01-01, 2026, 2026-01, year:2026:5, month:2026-01:3 reads reads
    ETERNITY, 9999-12-31 (one date: from then on) reads fails
    2026-12-31.2026-01-01 (ends before it starts) applies it, writing the values out of date order fails
    2026-01-01.2026-02-30 (last day doesn't exist) applies it through 2026-03-02: policyengine-core rolls the day after 2026-02-30 over to 2026-03-03 fails
    2026-1-1, 2026-1, " 2026" (unpadded, or with spaces, no range) fails reads
    2026-01-01.9999-12-31, year:9999:1 (last day 9999-12-31) fails fails
    2026-1-1.2026-12-31, 2026-02-30.2026-12-31, 2026-13-01.2026-12-31, 0000-01-01.2026-12-31, three dates, an empty end, "", abc, eternity, ², 2026-02-30, 2026-00-01, year:abc:1, day:2026-02-30:1, times of day, spaces in a range fails fails

    The check accepts exactly what the calculated country's model reads, so a reversed range or a non-existent last day still calculates on /us/… as it does today.

Where this departs from the plan for #4275

  • A single-date period key is accepted. The plan expected a key such as 2026-01-01 to fail because _apply_reform_to_system and create_policy_reform split each key on ., but neither runs for the US or UK: both models read single dates. Refusing them would break requests that work today, so the period-key tests use an impossible date and a range ending 9999-12-31 instead.
  • Numeric text is accepted. "2500" converts today, so this check accepts it, unlike the check Refuse new policies whose period keys or values simulations cannot read #4142 added for saving a policy.
  • find_invalid_periods_and_values isn't reused. It refuses keys and values this path reads (a bare year, year:2026:5, numeric text) and accepts values this path can't convert (null and lists for a number parameter). The new module reuses describe_value and PERIOD_OBJECT_REQUIREMENT from it and leaves what it accepts for saved policies unchanged.

Left as they are

  • Boolean and list parameters keep their silent readings: "false" and "abc" set a boolean parameter to true, text sets a list parameter to its characters, and a reversed or non-existent range end on /us/… applies as described above. None of these fails today, so none is refused.
  • A UK bare year with spaces around it or a leading zero (" 2026", "02026") is accepted on /uk/…. The UK model reads it as that fiscal year, except on the two parameters it writes straight onto the processed values (gov.economic_assumptions.yoy_growth.capital_gains and gov.economic_assumptions.indices.capital_gains), where it fails as before. Refusing it would break it for every other parameter.
  • On /uk/…, some keys whose dates reach into year 9999 still fail, because the UK model can't convert them to fiscal or calendar years. Examples are a range ending 9999-12-30 on gov.hmrc.income_tax.rates.uk[0].rate, and 9999-01-01 on a parameter it reads as one value per calendar year, such as gov.economic_assumptions.energy_prices.consumption_shares.electricity.q1. Where the UK model stops depends on the parameter: on the income tax rate, a range ending 9999-04-06 and the single date 9999-01-01 both calculate. So a single limit would refuse keys that calculate today. These fail on master too.
  • errors lists unrecognized names (from find_invalid_inputs) before the new entries, so it isn't in policy order across the two, and a parameter the model lacks that is also malformed gets one entry for each problem.
  • 9 UK parameters have a null earliest value, such as gov.hmrc.income_tax.rates.scotland.rates[0].threshold. _normalize_reform_values then converts with type(None), which takes no argument, so every policy that sets one answers 500 today. The check doesn't refuse them, because the request isn't at fault; that conversion needs its own fix.

Saved-household route

GET /<country_id>/household/<household_id>/policy/<policy_id> keeps its behaviour. It runs find_invalid_inputs on the stored policy, which this PR doesn't change, and not the new check, so a stored policy with such values answers as before: a 500, for example for null on a number parameter, which saving a policy accepts. The new check's 400 says the caller sent a malformed policy, which is wrong for a stored one; #4251 answers 422 for stored policies simulations can't read, and doing the same here would be a separate change. No telemetry group has come from this route.

Testing

  • tests/unit/routes/test_calculate_malformed_policy.py (new) posts to /us/calculate and /us/calculate-full. The stand-in country has the loaded US model's metadata. It parses a situation by running the loaded model's _normalize_reform_values on its tax-benefit system, without copying it or building a simulation, and reads each parameter's period keys with policyengine-core's Reform.from_dict, so a policy the check lets through fails as in production.
    • Refused (one case each, for both routes): a number as policy, true as a parameter's value, "abc", null, a list, a period key that isn't a date (2026-02-30.2026-12-31) and a range ending 9999-12-31. Each expects a 400 with the exact errors entry and message, no record at ERROR, no situation parsed and no observability ID started.
    • Accepted: a policy with a range, a single date, a bare year, year:2030:2, ETERNITY, numeric text, "Infinity", "-Infinity", a boolean and a list; an empty policy; a null policy. Each calculates (200), with the policy passed on unchanged.
  • The route tests also post ETERNITY to /uk/calculate (400) and /us/calculate (200), which shows the service passes the calculated country to the check.
  • tests/unit/utils/test_household_policy_validation.py (new) pins each rule: falsy policies, non-object policies and parameter values, values for number, boolean and list parameters, a parameter without recorded values, the UK parameter with a null earliest value, the order of entries, and each country's period-key lists. Checks against the installed packages:
    • the type read from metadata equals the type _normalize_reform_values converts to, for every US and UK parameter;
    • for parameters of each type in each model, 22 sample values are refused exactly when the conversion fails. The model now converts with the check's own convert_reform_value, so this checks the type read from metadata;
    • every US key in the lists is accepted exactly when policyengine-core's Reform.from_dict applies it to a parameter;
    • every UK key is accepted exactly when policyengine-uk's read_reform_dict reads it and the dates it names pass the date arithmetic the UK model does when it writes the change. That arithmetic came from household calculations with the installed UK model, which the test doesn't run.
  • Red-green: the first commit adds the tests. On it, the 12 refused route cases failed (10 with 500 and the ERROR record Error calculating household under policy from country.py, as in [telemetry] policyengine-api: AttributeError on /<country_id>/calculate (47418e783c1209c4) #4275; the 2 period-key cases with 200), the 6 accepted cases passed, and the rule tests failed to import the module. The second commit adds the check, and all 106 pass. Each review round's fixes were also written test-first (below); all 174 tests pass on the last commit.
  • Only these tests were run, by node ID. make quality passes.

Review

Four rounds of adversarial self-review ran, each by a new reviewer given only the diff and #4275. Rounds 1–3 were each followed by fixes written test-first.

  • Round 1 found:

    • a range or period ending 9999-12-31 passed the check, and then both models failed, because the day after it is in year 10000;
    • the check accepted any key either model read, so ETERNITY on /uk/… and an unpadded single date on /us/… still answered 500;
    • a reversed range was refused on /us/…, where it calculates today;
    • rare keys such as Unicode digits, a trailing newline and a spaced bare year on /uk/… were read differently from the models' readers;
    • the period-key tests compared the check with a list, not with the readers.

    Changes: keys are checked as the calculated country's reader reads them (find_unreadable_policy_inputs takes the country ID), range ends with policyengine-core's instant(), and the day after the last day must exist. The tests compare each country's keys with its reader, and the route stand-in reads keys with Reform.from_dict. Red before the fix: the range ending 9999-12-31 answered 500 (year 10000 is out of range) in the route tests, and the rule tests failed.

  • Round 2 found:

    • a UK range whose last day doesn't exist (2026-01-01.2026-02-30) passed, and the UK model failed;
    • a key such as ², for which isdigit() is true but int() fails, made the check itself raise.

    Changes: the UK range check needs a real last day, and the UK bare-year test reads such a key as no year. A route test shows the service checks keys for the calculated country. Red before the fix: 4 UK rule cases.

  • Round 3 found no regression, and no input from [telemetry] policyengine-api: AttributeError on /<country_id>/calculate (47418e783c1209c4) #4275 that still fails. It found that a month: period with a huge size kept the check, like master's model, busy for as long as the size: policyengine-core took 1.6 s to find the last day of a billion months, and a size with thousands of digits never finishes. Change: a month period longer than 9999 × 12 months is refused before its last day is computed. Red before the fix: 2 cases over the test's 0.5 s limit. It also found the spaced UK bare years listed above.

  • Round 4 reviewed every change above, round 3's included. It found no regression: on the keys it ran through the real models, the check refused nothing that calculates on master, and agreed with Reform.from_dict on 68 keys. It found nothing that makes the check raise or run long. Its one new finding was the late UK dates listed above, which can't be refused without refusing keys that calculate. Its one change: the route tests' docstring now says the stand-in reads period keys only as the US model does. Round 4 found nothing else substantive, so the rounds stopped there.

  • Each round also raised the UK parameters with a null earliest value, the order of errors, and that the UK reader test repeats the UK model's write-step date arithmetic instead of running the write. These are left as described above.

🤖 Generated with Claude Code

anth-volk and others added 2 commits November 11, 2026 19:11
Telemetry issue #4275: POST /<country_id>/calculate answered 500, with an
ERROR record, for a policy that isn't an object, a parameter set to true
instead of an object of period keys, and values such as "abc", null or a
list. Each failed while the calculation converted the policy's values,
after the model's tax-benefit system had been copied.

These tests expect a 400 listing each problem, with no ERROR record and
before the situation is parsed, for /calculate and /calculate-full, and
pin which period keys and values the check accepts, against the installed
US and UK models' own value conversion. They fail until the check exists.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Before parsing copies the model's tax-benefit system,
prepare_household_calculation now checks the request's policy
(find_unreadable_policy_inputs, in utils/household_policy_validation.py)
and refuses, as InvalidHouseholdInputsError's 400 with an errors list:

- a policy that isn't an object (an empty or null one still means no
  policy, as the calculation reads it);
- a parameter whose value isn't an object of period keys;
- a period key neither the US reader (policyengine-core's
  Reform.from_dict) nor the UK reader (policyengine-uk's dated_reforms)
  reads, and a range that ends before it starts;
- a value the parameter's type can't take. _normalize_reform_values
  converts to the type of the parameter's earliest value; it now shares
  that conversion (conversion_type, convert_reform_value) with the check,
  which reads the earliest value from the country metadata.

The saved-household route keeps its behaviour: a stored policy isn't
checked this way.

Fixes #4275

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@codecov

codecov Bot commented Nov 11, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 97.97297% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.61%. Comparing base (df394c5) to head (344e430).
⚠️ Report is 265 commits behind head on master.

Files with missing lines Patch % Lines
...icyengine_api/utils/household_policy_validation.py 98.58% 1 Missing and 1 partial ⚠️
policyengine_api/country.py 66.66% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #4279      +/-   ##
==========================================
+ Coverage   92.52%   92.61%   +0.09%     
==========================================
  Files         318      319       +1     
  Lines       18137    18273     +136     
  Branches     3033     3056      +23     
==========================================
+ Hits        16781    16924     +143     
+ Misses        813      803      -10     
- Partials      543      546       +3     
Flag Coverage Δ
integration 53.47% <28.37%> (-0.18%) ⬇️
unit 90.38% <97.97%> (+0.11%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Review round 1 of #4279 found period keys the check accepted that the
calculated country's model then failed on, answering 500 as before:

- a range or period whose last day is 9999-12-31, such as
  2026-01-01.9999-12-31 or year:9999:1: both models write a change up to
  the day after its last, which is in year 10000;
- ETERNITY on /uk/, and an unpadded single date such as 2026-1 on /us/:
  the check accepted any key either model read.

find_unreadable_policy_inputs now takes the country ID and checks keys as
that country's reader does: policyengine-uk's dated_reforms for the UK,
policyengine-core's Reform.from_dict for the US. Range ends are read with
policyengine-core's instant(), as both readers read them. A range that
ends before it starts is refused for the UK, which refuses it, and
accepted for the US, which calculates it today. The UK reader's bare
years, with spaces around them, are accepted for the UK.

The tests now check each country's key lists against its reader: the US
lists against Reform.from_dict applied to a parameter of their own, the
UK lists against read_reform_dict and the date arithmetic of writing the
change. The route tests' stand-in country reads period keys with
Reform.from_dict too, so a key the check lets through fails there as it
does in production.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 2 of #4279 found two period keys the check got wrong for
/uk/:

- a range whose last day doesn't exist, such as 2026-01-01.2026-02-30:
  the check tested the day after it, which policyengine-core rolls over
  to a real day, so it accepted the key and the UK model then failed
  with a 500. The UK check now also needs a real last day. The US model
  applies such a key, so it stays accepted there;
- a key such as "²", for which isdigit() is true but int() fails: the
  check itself raised ValueError, which answered 500. The UK bare-year
  test now reads such a key as no year, and the key is refused.

A route test now posts the same ETERNITY key to /uk/calculate, which
refuses it, and /us/calculate, which accepts it, so the service is shown
to pass the calculated country to the check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 3 of #4279 found that a key such as month:2026-01:<size>
made the check slow in proportion to the size: policyengine-core's
Period.stop finds a month period's last day a year at a time, so a
billion months took about two seconds and a size with thousands of
digits never finished. Master spent the same time in the model, after
copying its tax-benefit system.

A month period longer than the 9999 * 12 months from year 1 ends after
9999-12-31, which neither model reads, so the check now refuses it
before finding its last day. No decision changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Review round 4 of #4279 noted that the module docstring claimed a policy
the check lets through fails in the stand-in as in production, which
holds only on /us/: the stand-in reads period keys with
policyengine-core's Reform.from_dict, not policyengine-uk's reader. The
UK case in the file is one the check refuses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@anth-volk
anth-volk marked this pull request as ready for review November 13, 2026 14:58
@anth-volk
anth-volk merged commit cc51235 into master Nov 13, 2026
23 of 24 checks passed
@anth-volk
anth-volk deleted the fix/calculate-policy-400 branch November 13, 2026 14:58

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[telemetry] policyengine-api: AttributeError on /<country_id>/calculate (47418e783c1209c4)

1 participant