[MaxGhenis]

Draft. Head cb9abe89b634deaf555df0e7b4f1001faa25f3b2, with master (c86af926) merged in. Measurements and targeted tests below are at this head; CI is the evidence for the full suite. The PR stays a draft while #621 (rules for Core's parameter caches) is open; see "Open" below.

Problem

A policyengine-core test <~1,500 policyengine-us baseline YAML files> -c policyengine_us process grew without bound (two runs reached 118 GB and 111 GB on a 128 GB machine on 2026-10-02). Two causes are in tools/test_runner.py:

1. pytest keeps every collected item until the session ends, and each YamlItem kept its simulation and tax_benefit_system after its case ran, so every simulation of the run stayed alive. Simulation.__init__ also sets tax_benefit_system.simulation = self, so a cached system kept the last simulation built on it.
2. _tax_benefit_system_cache kept every system it built, one per distinct combination of reforms, extensions and dotted parameter inputs, and never evicted. Each is a full clone of the baseline.

A third cause, a copy of the whole parameter tree per instant, is #570's subject; the two PRs are independent.

Change

• YamlItem.teardown drops the case's simulation and system, restores the system's simulation back-reference to what it was before the case (also when building the simulation failed), and unmarks a parameter tree that outside code marked with this case's tracer.
• The cache keeps each baseline's reform-free system for the baseline's lifetime, plus an LRU of reform_cache_size reform systems (default 2; policyengine-core test --reform-cache-size N or options["reform_cache_size"]; 0 caches none). Each call's bound first trims what earlier calls cached. A baseline's entry still iterates, indexes and deletes as a mapping of reform systems, so test hooks that walk it keep working.
• Keys: reforms as a tuple, inline parameter values by repr (0.2 and "0.2" differ), and a class reform with no reform_key by the class itself.
• Tracing wrappers are built on each read and never cached, in ParameterNode._get_at_instant and in TaxBenefitSystem.get_parameters_at_instant, so a cached wrapper no longer keeps the first traced simulation's tracer alive.
• Merged with Pause automatic garbage collection while the YAML runner builds a reform system #643's collection pause: a cache miss on either path (the reform-free system, or a reform system, including one evicted earlier or any build with the cache disabled) builds with automatic collection paused; a hit does not touch collection. The full-collection count is each baseline's reform-free system plus its reform systems, across every baseline, after the call's bound trims and before the new system goes in. The caller's collection setting is restored in finally, so it survives any BaseException, including KeyboardInterrupt and a gc.collect() that raises.

Invariants

• After a case, the item holds no simulation or system, and the system's simulation back-reference is what it was before the case, including after a failed build. No simulation or tracer created by a probed run stays alive, and the number of live systems does not grow with the number of cases (test_runner_memory.py).
• In a 300-case run, traced memory at every sample after case 100 stays within 2 MB of its value after case 100, with plain cases and with every third case a reform (test_traced_memory_stays_flat_over_hundreds_of_cases).
• Against a reference LRU, for any request sequence with per-call sizes 0 to 3: the cache holds at most the call's size of reform systems, returns a cached object exactly when the reference would, always returns a system built for exactly the requested reforms and extensions, and keeps one reform-free system (test_reform_cache_property.py; also test_gc_pause.py's property).
• A miss forces a full collection only when the caller had collection on, the build returned, and the count above is under 3. While the cache only grows, a run forces at most 3. Eviction can keep the count under 3 for good: no reform-free case, a reform_cache_size of 0 or 1, or a hook that deletes entries. Then every miss forces one.
• Outcomes are unchanged: the cache only decides when a system is rebuilt.

Measurements at this head

Each run is one policyengine-core test <file> -c <package> process: CPython 3.14 (US) and 3.13 (UK), Linux, 4 vCPU, policyengine-us 6145ad45, policyengine-uk 9b456bd. Peak RSS is polled from /proc and checked against ru_maxrss. Builds count the distinct system objects _get_tax_benefit_system returned. Master and head runs were interleaved, three rounds each, and every run passed every case.

File	Core	Cases	Builds	Peak RSS	Wall
UK tests/policy/baseline/gov/hmrc/private_school_vat.yaml	master c86af926	15	10	1,084–1,085 MB	11.8–14.7 s
	this PR	15	12	785 MB (all 3)	12.9–13.1 s
US baseline/gov/states/az/tax/income/credits/az_property_tax_credit_schedules.yaml (no reforms)	master	142	1	2,851 MB (all 3)	99.7–103.2 s
	this PR	142	1	2,533 MB (all 3)	84.4–86.6 s

At reform_cache_size 2 the UK file rebuilds 2 systems that master kept, and still peaks 300 MB lower.

No policyengine-us file that builds a reform system fits under 4 GiB. The 4 GiB limit was a constraint of this measurement session.

• A baseline-only run peaks at 2.15 GB.
• contrib/aca/ptc_700_fpl_cliff.yaml builds one reform system. Master and this PR both passed 4 GiB in about 61–64 s, all three rounds.
• contrib/states/hi/dependent_exemption_reform_test.yaml builds 3 reform combinations over 17 cases. At master it passed 4 GiB within 55 s. One run each, outside that limit:

HI file, one run each	Builds	Peak RSS	Wall
master	3	7,109 MB	205 s
this PR, reform_cache_size 2 (default)	8 (5 rebuilds)	8,369 MB	276 s
this PR, --reform-cache-size 3	3	7,101 MB	194 s

On this file, the default of 2 costs 5 rebuilds and 1.26 GB more peak than master. The file alternates between 3 combinations, one more than the cache holds. A miss builds a third reform system while two are cached. The evicted system is dropped only after that paused build. Its reference cycles then wait for a later collection, so up to four reform systems can be alive at once. With size 3, the PR matches master. The bound pays off on runs with more combinations than the cache holds plus one, such as the multi-file batches measured on October 3 (linked below), where master's cache grows without limit. The peak bound is about reform_cache_size + 2 reform systems, not reform_cache_size.

CI

Run 41401698001 at 17fc0fd1ce cancelled three Test jobs (windows 3.12, ubuntu 3.11, windows 3.11) at pr.yaml's 45-minute timeout-minutes. They didn't hang: the other five took 33–44 min, against 7–18 min for the same jobs on master-based PRs (runs 41548423680, 41636046932, 41706941813, 41714952193). The jobs' logs put 19–26 min of that in test_reform_cache_property.py. After the #643 merge, each of its cache misses below three cached systems ran a real full gc.collect(). That was 1,453 collections over the pytest heap, made dearer by coverage --branch. The property now replaces gc.collect with a no-op: 72 s to 0.6 s locally. test_gc_pause.py checks collection with a recording spy.

Tests run at this head

One file per process, uv run pytest -q -p no:cacheprovider <file>:

• tests/core/tools/test_runner/: test_gc_pause.py 44 passed, test_reform_cache_property.py 1, test_runner_memory.py 27, test_yaml_runner.py 14
• test_parameter_tracing_cache.py 5, test_parameter_tracing_cache_property.py 1, test_tracing_parameter_isolation.py 49, test_tax_benefit_system_cache_identity.py 2, test_yaml.py 15
• ruff format --check . (376 files) and ruff check . pass.

Mutants of test_runner.py, each in its own process and restored after, all killed:

• Keep the simulation on the item after teardown: 8 tests in test_runner_memory.py.
• Drop the back-reference reset: 11 tests in test_runner_memory.py.
• Evict nothing: 4 in test_gc_pause.py, the property in test_reform_cache_property.py, 5 in test_runner_memory.py.
• Pause collection on a cache hit: 6 in test_gc_pause.py.
• Restore collection on success and except Exception instead of finally: 6 KeyboardInterrupt cases in test_gc_pause.py.

Not run: the full suite, the country template YAML suite, any full country suite, and microsimulations.

Open

• Define how Core's parameter caches are owned, invalidated, copied and shared #621. On October 15, review asked that this PR's parameter-cache changes wait for Define how Core's parameter caches are owned, invalidated, copied and shared #621's rules. Define how Core's parameter caches are owned, invalidated, copied and shared #621 is still open. Its design 3 proposes the same two rules this PR applies: "Never cache tracing wrappers. Wrap on each read." and a bounded runner cache "as Stop the YAML test runner keeping every case's simulation and reform system #569 proposes". Whether this lands before Define how Core's parameter caches are owned, invalidated, copied and shared #621 settles is a maintainer decision.
• Peak with eviction. See the HI measurement above. A possible follow-up is to drop an evicted reform system and collect it before building its replacement.

Downstream impact

Earlier measurements on two policyengine-us CI batches (October 3, core 760a001a, alone and with #570) are of an earlier snapshot, not this head. No country-policy or partner baseline files change. Whichever of #569 and #570 lands second must rerun the overlapping parameter-cache and runner-memory tests.

https://claude.ai/code/session_01LwQq6JNXsLbdg36pd2zkMe

---

Generated by Claude Code

[MaxGhenis]

Measurements of this PR, alone and combined with #570, on the two policyengine-us CI batches that have been dying on 16 GB runners since 3.32.12 (PolicyEngine/policyengine-us#9757). Each row is one policyengine-core test <paths> -c policyengine_us process on policyengine-us main 615a0f2d. Setup: CPython 3.14.7 on macOS arm64, CI's itemization setting, core imported from a snapshot of each commit. Peak memory is the process's peak physical footprint (/usr/bin/time -l). CPU is user plus system time; the host was shared and loaded, so wall time is not comparable.

Core	NY/OH/OK batch (1,328 cases): peak	CPU	USDA batch (1,373 cases): peak	CPU
3.32.11 (7950c01)	16,282 MB	213 s	17,076 MB	272 s
3.32.12 (b78b0ba)	18,156 MB	302 s	18,935 MB	321 s
#578 on 3.32.12	16,287 MB	243 s	17,055 MB	260 s
#569 (760a001)	9,636 MB	236 s	13,912 MB	237 s
#569 + #570 (760a001 merged with 4e6bc31)	7,033 MB	149 s	6,399 MB	137 s

All runs passed every case.

• On their own, both batches are already above the 16 GB runner limit on 3.32.11 and 3.32.12. In CI, NY/OH/OK is one of four sequential batches and USDA follows the SSA batches.
• Give a storage a set of shared keys only while it shares an array #578 (issue 3.32.12 gives every holder's storage an empty set: about 1.3 MB more per policyengine-us simulation #577) only removes the 3.32.12 regression, an empty set in every holder's storage.
• Stop the YAML test runner keeping every case's simulation and reform system #569 with Resolve parameter nodes at instant on first read #570 is what gives these jobs headroom: under half the 3.32.11 peak on both batches, and less CPU.

Scripts, logs and raw results: ~/reviews/ci-shutdown-core-3-32-12-2026-10-02/ on Max's machine (scripts/measure.sh, results/summary.tsv).

[anth-volk]

Rebased onto master (b909d5a6) and force-pushed: 91744d3e → 20269775. There were no conflicts, and the tree is the same as merging master into the old head. The full suite, lint and the country template YAML suite pass at the new head, both on its own and combined with #570. Details are under "October 14 rebase" in the description. #616 is stacked on this branch.

[anth-volk]

I've dropped the commits I added to this branch and restored Max's description.

• Dropped: 3089c343, which rebuilt the system's parameters at an instant after the tree changes; 5b415423, a merge of master; and two review fixes that were never pushed.
• Now on the branch: only Max's 9 commits, rebased onto master (4adfea70) and force-pushed: 5b415423 → ebef3bfe. The rebase changes only the base. Since the October 14 rebase, master has moved only by Pin the country smoke bundle to policyengine-us 2.42.13 #620's CI pin and a version bump.

Review kept finding edge cases in how Core's parameter caches are filled, invalidated, copied and shared, mostly in behaviour that predates this PR. #621 proposes rules for those caches, and this PR's changes to parameter caching should wait for them.

[MaxGhenis]

Picking this up for the US hub. I'll post a hand-off comment here when it's ready.

---

Generated by Claude Code

[MaxGhenis]

Picking this up for the US hub. I'll post a hand-off comment here when it's ready.

---

Generated by Claude Code

[MaxGhenis]

Picking this up for the US hub. I'll post a hand-off comment here when it's ready.

---

Generated by Claude Code

[MaxGhenis]

Ready for hub review
RESULT: FIXED
HEAD: cb9abe89b634deaf555df0e7b4f1001faa25f3b2

The branch was fast-forwarded from 17fc0fd1ce. It has two new commits: 67b7a30b merges master c86af926, and cb9abe89 carries the fix. The PR stays a draft.

1. Review findings

The PR has no formal reviews and no inline comments. Its findings are in comments:

• Still open: anth-volk, October 15. The PR's parameter-cache changes should wait for Define how Core's parameter caches are owned, invalidated, copied and shared #621. Define how Core's parameter caches are owned, invalidated, copied and shared #621 is still open, and its design 3 already adopts this PR's two rules: wrap tracing wrappers on each read and never cache them, and bound the runner cache "as Stop the YAML test runner keeping every case's simulation and reform system #569 proposes". No code fix applies. Landing order is the hub's call, so the PR stays a draft.
• The October 7 repair items (the deepcopy regression and pytest-rerunfailures retention) are in the branch. Their tests pass at this head.

2. Master merge in _get_tax_benefit_system

I checked fd4fece8b5 and 17fc0fd1ce against each part of step 2. Each part was already done, apart from the comment qualification below.

• Both paths paused, hits outside the pause. The reform-free miss is at policyengine_core/tools/test_runner.py:615-621 and the reform miss at :644-660. A reform hit returns at :638-639 and a reform-free hit at :622, both outside the pause.
• Count. _cached_system_count (:570) counts each baseline's reform-free slot plus its reform entries across all baselines. It runs after the call's bound trims (:611-612) and before insertion (:662-665).
• Kept from this PR: the keys, LRU, configured capacity, disabled cache (cache_size 0 builds and never inserts) and teardown (:321).
• Restoration in finally: :532.
• Tests, all in tests/core/tools/test_runner/test_gc_pause.py:
  • count of 2 against 3: :334;
  • miss on an evicted system: :354;
  • disabled-cache misses: :387;
  • ordinary exception and KeyboardInterrupt on both paths: :431, and on the pause alone: :448;
  • gc.collect() raising an Exception or a KeyboardInterrupt: :467.
• Changed now: the sequence invariant. The comment at :489-496 said the cache "reaches 3 after its third build and stays there". It now says a run forces at most 3 collections while the cache only grows. It also names when eviction keeps the count below 3, and then every miss collects: no reform-free case, a reform_cache_size of 0 or 1, or a hook deleting entries. The module docstring of test_gc_pause.py already said this.
• The model's ranking moves with heap shape: already at :486-487.
• Master had moved since fd4fece8b5 by Publish a release only after its tests pass #591, Guard the release tests against masked test failures and other PyPI uploads #648 and version bumps. I merged it as merge commit 67b7a30b, with no conflicts.
• uv.lock. The branch had rewritten it to lock revision 3 and a stale version. I restored master's copy, so the PR no longer touches it.

3. The PR's invariants at the merged head

These are one process per file:

• test_runner_memory.py: 27 passed
• test_reform_cache_property.py: 1 passed
• test_gc_pause.py: 44 passed, including the reference-LRU differential property
• test_yaml_runner.py: 14
• test_parameter_tracing_cache.py: 5
• test_parameter_tracing_cache_property.py: 1
• test_tracing_parameter_isolation.py: 49
• test_tax_benefit_system_cache_identity.py: 2
• test_yaml.py: 15
• ruff format --check . (376 files) and ruff check . are clean.

4. Measurements on real YAML

Setup: policyengine-us 6145ad45 and policyengine-uk 9b456bd, on a 4 vCPU Linux VM. Each run is one policyengine-core test <file> -c <pkg> process. Master runs used a c86af926 worktree on PYTHONPATH, and every log's policyengine_core.__file__ confirms which core ran. Peak RSS was polled from /proc and checked against ru_maxrss. Builds count the distinct systems _get_tax_benefit_system returned. Rounds were interleaved, master first, three each.

File	Core	Builds	Peak RSS (3 rounds)	Wall (3 rounds)
UK baseline/gov/hmrc/private_school_vat.yaml, 15 cases, all passed	master	10	1085, 1085, 1084 MB	14.7, 13.3, 11.8 s
	head	12 at size 2	785, 785, 785 MB	13.1, 12.9, 13.1 s
US baseline/gov/states/az/tax/income/credits/az_property_tax_credit_schedules.yaml, 142 cases, all passed, no reforms	master	1	2851 ×3 MB	102.7, 99.7, 103.2 s
	head	1	2533 ×3 MB	84.4, 86.6, 85.1 s

The US contrib files didn't fit under 4 GiB at master.

• contrib/states/hi/dependent_exemption_reform_test.yaml passed 4 GiB at master within 55 s, while its first case ran. The log shows only MEASURE rc=-9 wall=54.7s peak_rss=4105MB killed_over_4096MB=True.
• The one-reform contrib/aca/ptc_700_fpl_cliff.yaml also passed 4 GiB, at master and head, in all three rounds (killed at about 61-64 s).
• A US baseline plus its reform-free system already peaks at 2.15 GB. No US file that builds a reform system fits, so I used the reform-free baseline file above.

Rebuild count at reform_cache_size 2 on a US reform file. I ran the HI file once per configuration, with a 12 GB ceiling, outside the 4 GiB protocol. All passed 17 cases.

• master: 3 builds, 7,109 MB, 205 s
• head at size 2: 8 builds (5 rebuilds), 8,369 MB, 276 s
• head with --reform-cache-size 3: 3 builds, 7,101 MB, 194 s

On that file the default is worse than master. The file alternates among 3 combinations, one more than the cache holds. Each miss builds a third system while two are cached. The evicted one is dropped only after the paused build, and its cycles wait for a later collection. The PR body now says the bound is about reform_cache_size + 2 reform systems at peak, not reform_cache_size.

Mutants

Mutants were applied by exact string replacement to test_runner.py, one at a time. Each ran against the three test files, one process per file, and was restored with git checkout.

Mutant	Result
Keep the simulation on the item after teardown	killed: 8 in test_runner_memory.py (e.g. test_finished_cases_release_their_simulations)
Drop the system's back-reference reset	killed: 11 in test_runner_memory.py (e.g. test_cases_whose_simulation_fails_to_build_release_it[*])
Evict nothing (both popitem loops)	killed: 4 in test_gc_pause.py, test_cache_matches_reference_lru, and 5 in test_runner_memory.py
Pause collection on a cache hit (both paths)	killed: 6 in test_gc_pause.py (e.g. test_a_cache_hit_leaves_collection_alone)
finally replaced by success restoration plus except Exception	killed: 6 KeyboardInterrupt cases in test_gc_pause.py, e.g. test_the_pause_restores_the_callers_state_after_a_keyboard_interrupt[True]. It survives test_runner_memory.py and test_reform_cache_property.py.

CI

Pull request run 41731108677 at cb9abe89 concluded success. All 19 jobs succeeded and none was cancelled. The 8 Test jobs took 15-27 min (ubuntu 3.14 15, ubuntu 3.12 18, ubuntu 3.13 19, ubuntu 3.11 20, windows 3.14 18, windows 3.13 23, windows 3.12 27, windows 3.11 27), against the 45-minute timeout. The 8 country-package smoke jobs took 1-3 min.

Why run 41401698001 had cancelled jobs. The branch made the Test jobs slow; nothing hung.

• pr.yaml sets timeout-minutes: 45 on Test. At 17fc0fd1ce the five jobs that finished took 33-44 min. The same jobs on master-based PRs took 7-18 min (runs 41548423680, 41636046932, 41706941813 and 41714952193).
• The logs of the cancelled ubuntu 3.11 job and the passing ubuntu 3.14 job put 1,578 s and 1,143 s in test_reform_cache_property.py. The cancelled job was in test_runner_memory.py's last six tests when the timeout fired, at 86% of the suite.
• The Pause automatic garbage collection while the YAML runner builds a reform system #643 merge made each miss in that property test run a real full gc.collect(), because the cache stays below 3 systems. That came to 1,453 collections over the whole pytest heap, under coverage --branch.
• Fix: tests/core/tools/test_runner/test_reform_cache_property.py:67 stubs gc.collect per example (72 s to 0.6 s locally). Collection is checked by test_gc_pause.py's spy.
• Still heavier than master: test_runner_memory.py (about 4 min on CI) and test_parameter_tracing_cache_property.py (about 2.5-3 min). Both collect on purpose to count live objects.

Judgment calls

• No code change for anth-volk's Define how Core's parameter caches are owned, invalidated, copied and shared #621 finding. It's about landing order, not a defect.
• Step 2's "after any required trimming" read as the call's own bound, not making room for the new entry. Evicting before the build would cut the count to 2 with the default cache, so every miss would force a collection.
• I didn't redesign the eviction or collection order to fix the HI peak. That change belongs with Pause automatic garbage collection while the YAML runner builds a reform system #643's design, so I've proposed it as a follow-up.
• uv.lock restored to master's copy, rather than kept at the branch's older lock revision.
• The PR body can't keep the exact "🤖 Generated with [Claude Code]" line: the GitHub proxy replaces it with its own Claude Code attribution footer on every PATCH.

Proposed follow-up issue

Collect an evicted reform system before building its replacement in the YAML runner
When a miss evicts, the evicted system is dropped only after the paused build, so a 3-combination policyengine-us file peaks at 8.4 GB with reform_cache_size 2 against 7.1 GB on master. Evict first and collect before the build when something was evicted, then re-measure #643's collection-CPU model.

---

Generated by Claude Code