[derek73]

Summary

PR 2 of the PatentsView plan (#931 added the corpus). name_usage.py --boxes adds a row for each fielded local corpus whose extract keeps every record uncapped: orcid-researchers and patentsview-inventors. The row gives:

• the people whose given box and whose family box write the word, counted over the whole extract, not the capped corpus;
• the --countries N countries with the most of them in each box, each as a share of that country's people, with the records stating no country last;
• any country asked for with --country CC (repeatable; implies --boxes), listed even outside the top N.

PatentsView counts the inventors filing under a name; ORCID counts one holder per record. A country is where they work or live, so the counts describe researchers and US-patent inventors, not a country's population. The docstring and README say so.

uv run python tools/name_usage.py --boxes del --country PH

For del (ORCID, 2026-10-28):

• Given box: Peru 4,910 (1.10% of its holders), Mexico 2,063 (1.07%), Spain 1,753 (0.69%), as in María del Carmen.
• Family box: Spain 1,870 (0.74%), Peru 1,711 (0.38%).

So del is written in the given box more often than the family box in the Spanish-speaking countries, a measurement a Spanish pack's particle rule can start from.

How

• local_core.Source gains extract and boxes: the extract's path, and a reader yielding (given, family, country, weight). local_orcid.orcid_boxes and local_patentsview.pv_boxes read the existing extracts and refuse one of another release.
• local_corpora.cached_box_index builds a BoxIndex: per country the people in all, and per word and country the people whose box writes it, once per record per box, with the word index's own fold. It is cached as boxindex.json.gz beside the corpus.
  • The key is the extract's sha256, taken from the bytes as they are read and checked again after the read, plus a fingerprint of the reader and the fold. A test holds that fingerprint's hand-kept list to what box_index calls.
• Measured 2026-10-28:
  • ORCID's index took 2 min 19 s to build: 3,780,309 words in 6,660,756 word-and-country cells, 41 MB cached. PatentsView's is 8.5 MB.
  • A lookup with both cached took 10.4 s with --boxes against 5.3 s without, nearly all of the difference spent reading the two indexes. That's why the rows are opt-in.

Review

A code review found five problems, all fixed:

• A damaged extract crashed the whole run. A truncated gzip, corrupt deflate bytes, an empty file, a bad header or a missing field raised exceptions _attempt does not catch. It now fails only its row, naming the refresh.
• A corrupt cached index crashed the run too (zlib.error). It is now rebuilt.
• Every lookup paid for both indexes, and the first one after a fetch silently started a 2-minute build. The rows are now opt-in with --boxes.
• My README timing claim was wrong. It blamed hashing; the cost is reading the index.
• --country could list a country twice. It now drops duplicates and blanks.

Negative controls: each of seven behaviours made its test fail when switched off. That includes the once-per-record count, whose first test fixture couldn't detect it and was strengthened.

Notes

• JSON output: --json now carries empty given_shares and family_shares on every row (asdict).
• Exit status: a corpus fetched with its extract deleted fails its box row, so the run exits 2, but only when --boxes is asked.
• Checks: the full suite (32,634), ruff and mypy pass. The changed tests pass on 3.15.

🤖 Generated with Claude Code