Add patentsview-inventors, a local corpus of US patents' inventors

docs/design/mechanisms.md

@@ -426,7 +426,7 @@ print(len(MAIDEN_MARKERS), 'members ->', len(cls), 'structural classes')"
 
 **How it works.** WATCH asks whether a change moved a name: a committed corpus, `snapshot.jsonl` and a test that fails, or a local corpus read with `snapshot.py --local`, which no CI run sees. SCORE asks whether the parser agrees with someone who knows the fields: a committed fields file and agreement record, read by `cldr_agreement.py` from the repository alone, or a local corpus whose records carry the fields their filers, or the bearers themselves, typed, with its record kept in the cache beside it (`local_agreement.py`). The local score tells a change's author what moved among real filed names, by field, and gives agreement by group and how often each disagreement shape occurs. It cannot guard anything: no CI run and no reviewer sees it, and no test fails when it moves (the tool itself exits 1). Its truth is the filer's, so a box filled wrongly (a job typed as a title) reads as a miss unless the scorer sets it aside, and the scorer has to say what it sets aside; where the bearer typed the boxes, a miss is a disagreement with the person, though one who typed a family-first name into the given box reads as a miss too. COUNT asks how one word is used: a cached source behind a row of `tools/name_usage.py`, with only the counts cited. DERIVE asks what a shipped list should hold: a cached source and a script writing the vocabulary (`tools/zh_surnames.py`). SURVEY asks how people write whole names, and how OFTEN they write each shape: a cached local corpus (`tools/differential/local_corpora.py`).
 
-A second axis cuts across the jobs: whether the names are real or invented, labelled with fields or bare text, chosen or unselected. Every committed corpus is CHOSEN: invented to fill fields (CLDR, the conventions table, the rules examples), cataloged (GND), filtered to names whose every word aligns with a stated part (both Wikidata corpora, a filter that drops credentials and typos), or picked by someone testing or reporting a case (v1's test names in `corpus.jsonl`, the case-table rows in `corpus_cjk.jsonl` and `corpus_shapes.jsonl`, and the strings reporters typed into `corpus_issues.jsonl`). The last group does hold hand-typed shapes — a job title in the name (`Bob Jones, author`), a credential before the comma (`Washington Jr. MD, Franklin`) — but one instance of each, as chosen. What SURVEY adds is the unselected population and its frequencies: ClinicalTrials.gov's study officials (local corpus `ctgov-officials`, measured 2026-10-21) are typed into a free-text field by sponsors' staff, so they show how often each shape occurs, and hold some no committed corpus does, such as `MBBS,FCPS` with no space. The FEC's individual contributors (local corpus `fec-individuals`, measured 2026-10-22) are unselected too, but mostly a filer's software writing a form's fields in a fixed order, so nearly all of its 1.89 million names share one shape (capitals, family, comma, given) and the hand-typed part is what goes inside the fields (a credential run or a suffix after a second comma): the population for that shape's variants, and blind to case. The UK Electoral Commission's individual donors (local corpus `ukec-individuals`, measured 2026-10-22) are the opposite shape: names as a donee's return gave them, mixed case and in spoken order, `parse()` reading a title in 8,219 of 15,214 (`Mr`, `Lord`, `Sir`) and only 2 holding a comma. Brazil's individual donors to candidates (local corpus `tse-donors`, measured 2026-10-22) are the FEC's comma-less counterpart: 848,351 filed names, nearly all in capitals and in spoken order, full of particles and agnomens (`DE`, `DA`, `DOS`; `FILHO`, `NETO`, `JUNIOR`), each beside the tax registry's name for the same donor, so a typed form can be read against its reference form (205,925 of 854,531 pairs differ). A source that pairs two spellings of one person scores without fields: the parse of each spelling should find the same surname, so where the spellings differ only in what cannot move a surname (accents, case), a disagreement is the parser reading one name two ways, found with no one labelling either (`local_corpora.py pairs --agreement`; on `tse-donors` on 2026-10-22, 1,304 of 43,552 accents-only pairs, 1,121 of them from `JÚNIOR`/`JUNIOR` and `SÁ`/`SA`). Read it per shape: a pair that leaves a word out can change the surname for real. JRC-Names (local corpus `jrc-names`, measured 2026-10-24) widens the pair to a group: one person's spellings harvested from news text across scripts, 218,822 persons with two or more. Words in two scripts cannot be compared, so `local_corpora.py groups` compares the readings' structure instead (title, suffix, given, family, the family's length and where it stands), counting persons. Read it per pair of scripts, since a structure can differ by convention: a script read family first stands its family elsewhere, and a writing variant moves the reading for real (the Arabic comma, katakana with and without `・`). Companies House's people with significant control (local corpus `psc-individuals`, measured 2026-10-22) are unselected and labelled at once: each filer types the name into four boxes (title, forename, middle name, surname) and states a nationality, so the corpus is the one real population a local SCORE reads by nationality. It is filed through a form's boxes, so it holds almost no comma (838 of 1,040,708 sampled records) or credential. ORCID's researchers (local corpus `orcid-researchers`, measured 2026-10-27) are labelled by the bearer rather than a filer: each holder typed their own name into two boxes (given names, family name), from 250 countries. Two things keep it from being a SURVEY. It keeps at most 20,000 records a country, so it shows which shapes each country holds and how the parser reads them, never how often a shape occurs across countries. And its country is where a holder works or lives, most often taken from an employment, so a country's row is its institutions' researchers, not its nationals. It is the SCORE to read for a language pack: 80.5% of its 1,831,533 records agreed on every field, as few as 13.4% in Peru, and the other 17 countries of at least 1,000 records under half were Brazil and 16 Spanish-speaking ones (Argentina read 80.3%); in the two commonest disagreement shapes, 249,526 of the 357,517 records that disagreed, a two-word family box was read as one word (measured on master at 598167f4). One source may do several jobs, each through its own code: Wikidata does COUNT live and WATCH and SCORE through its builders. A live source has no fixed file to pin, so its cached copy is the pin, carrying the source's own data date and a sha256 checked on every load (a truncated file otherwise reads as a run of false zeros, #775).
+A second axis cuts across the jobs: whether the names are real or invented, labelled with fields or bare text, chosen or unselected. Every committed corpus is CHOSEN: invented to fill fields (CLDR, the conventions table, the rules examples), cataloged (GND), filtered to names whose every word aligns with a stated part (both Wikidata corpora, a filter that drops credentials and typos), or picked by someone testing or reporting a case (v1's test names in `corpus.jsonl`, the case-table rows in `corpus_cjk.jsonl` and `corpus_shapes.jsonl`, and the strings reporters typed into `corpus_issues.jsonl`). The last group does hold hand-typed shapes — a job title in the name (`Bob Jones, author`), a credential before the comma (`Washington Jr. MD, Franklin`) — but one instance of each, as chosen. What SURVEY adds is the unselected population and its frequencies: ClinicalTrials.gov's study officials (local corpus `ctgov-officials`, measured 2026-10-21) are typed into a free-text field by sponsors' staff, so they show how often each shape occurs, and hold some no committed corpus does, such as `MBBS,FCPS` with no space. The FEC's individual contributors (local corpus `fec-individuals`, measured 2026-10-22) are unselected too, but mostly a filer's software writing a form's fields in a fixed order, so nearly all of its 1.89 million names share one shape (capitals, family, comma, given) and the hand-typed part is what goes inside the fields (a credential run or a suffix after a second comma): the population for that shape's variants, and blind to case. The UK Electoral Commission's individual donors (local corpus `ukec-individuals`, measured 2026-10-22) are the opposite shape: names as a donee's return gave them, mixed case and in spoken order, `parse()` reading a title in 8,219 of 15,214 (`Mr`, `Lord`, `Sir`) and only 2 holding a comma. Brazil's individual donors to candidates (local corpus `tse-donors`, measured 2026-10-22) are the FEC's comma-less counterpart: 848,351 filed names, nearly all in capitals and in spoken order, full of particles and agnomens (`DE`, `DA`, `DOS`; `FILHO`, `NETO`, `JUNIOR`), each beside the tax registry's name for the same donor, so a typed form can be read against its reference form (205,925 of 854,531 pairs differ). A source that pairs two spellings of one person scores without fields: the parse of each spelling should find the same surname, so where the spellings differ only in what cannot move a surname (accents, case), a disagreement is the parser reading one name two ways, found with no one labelling either (`local_corpora.py pairs --agreement`; on `tse-donors` on 2026-10-22, 1,304 of 43,552 accents-only pairs, 1,121 of them from `JÚNIOR`/`JUNIOR` and `SÁ`/`SA`). Read it per shape: a pair that leaves a word out can change the surname for real. JRC-Names (local corpus `jrc-names`, measured 2026-10-24) widens the pair to a group: one person's spellings harvested from news text across scripts, 218,822 persons with two or more. Words in two scripts cannot be compared, so `local_corpora.py groups` compares the readings' structure instead (title, suffix, given, family, the family's length and where it stands), counting persons. Read it per pair of scripts, since a structure can differ by convention: a script read family first stands its family elsewhere, and a writing variant moves the reading for real (the Arabic comma, katakana with and without `・`). Companies House's people with significant control (local corpus `psc-individuals`, measured 2026-10-22) are unselected and labelled at once: each filer types the name into four boxes (title, forename, middle name, surname) and states a nationality, so the corpus is the one real population a local SCORE reads by nationality. It is filed through a form's boxes, so it holds almost no comma (838 of 1,040,708 sampled records) or credential. ORCID's researchers (local corpus `orcid-researchers`, measured 2026-10-27) are labelled by the bearer rather than a filer: each holder typed their own name into two boxes (given names, family name), from 250 countries. Two things keep it from being a SURVEY. It keeps at most 20,000 records a country, so it shows which shapes each country holds and how the parser reads them, never how often a shape occurs across countries. And its country is where a holder works or lives, most often taken from an employment, so a country's row is its institutions' researchers, not its nationals. It is the SCORE to read for a language pack: 80.5% of its 1,831,533 records agreed on every field, as few as 13.4% in Peru, and the other 17 countries of at least 1,000 records under half were Brazil and 16 Spanish-speaking ones (Argentina read 80.3%); in the two commonest disagreement shapes, 249,526 of the 357,517 records that disagreed, a two-word family box was read as one word (measured on master at 598167f4). PatentsView's inventors (local corpus `patentsview-inventors`, measured 2026-10-28) are labelled by a professional filer, the applicant's attorney, and carry the number of people who filed under each name in each country, kept uncapped in the extract: the one fielded source that counts the people under each name, so a word's frequency among a country's US-patent inventors (not its population) can be read from it. Its agreement reads beside ORCID's as how a name is written against how its bearer types it: 99.8% in Japan, 99.5% in China and 99.4% in Korea, where ORCID read 95.5%, 84.5% and 92.4%, while Spain reads about as low in both (45.6% here, 48.0% in ORCID). One source may do several jobs, each through its own code: Wikidata does COUNT live and WATCH and SCORE through its builders. A live source has no fixed file to pin, so its cached copy is the pin, carrying the source's own data date and a sha256 checked on every load (a truncated file otherwise reads as a run of false zeros, #775).
 
 **Reach for it when.** A source is proposed, before choosing between a builder and a tool column; and when a corpus would hold real people's names nobody vetted for publication, which is the case for a local corpus rather than a committed one.
 

tests/v2/test_local_agreement.py

@@ -266,26 +266,35 @@ def test_a_record_written_by_another_scorer_is_refused(
         monkeypatch: pytest.MonkeyPatch) -> None:
     _corpus(monkeypatch, *_ALL)
     assert agreement.main([_PSC, "--record"]) == 0
-    before = agreement.scorer_fingerprint()
+    before = agreement.scorer_fingerprint(_PSC)
     # Each part of the code that decides a record's entry moves the
     # fingerprint: how a field is compared, which records get an entry,
-    # and how an entry is keyed.
+    # how an entry is keyed, and the source's own boxes; another source's
+    # boxes do not, so adding a source leaves this one's record standing.
+    other = next(s for s in agreement.FIELDED if s != _PSC)
+    with monkeypatch.context() as patch:
+        patch.setitem(agreement.FIELDED, other,
+                      agreement.Fielded({"forename": "given", "surname": "family"}, group="x"))
+        assert agreement.scorer_fingerprint(_PSC) == before, other
+        patch.setitem(agreement.FIELDED, _PSC,
+                      agreement.Fielded({"forename": "given", "surname": "family"}, group="x"))
+        assert agreement.scorer_fingerprint(_PSC) != before, "its own boxes"
     for name, changed in (("fold", lambda text: text.casefold()),
                           ("score", lambda rows, fielded, parser: None),
                           ("key", lambda record: str(record))):
         with monkeypatch.context() as patch:
             patch.setattr(agreement, name, changed)
-            assert agreement.scorer_fingerprint() != before, name
+            assert agreement.scorer_fingerprint(_PSC) != before, name
     with monkeypatch.context() as patch:
         patch.setattr(agreement.corpus_build, "jsonl_line",
                       lambda obj, sort_keys=False: str(obj))
-        assert agreement.scorer_fingerprint() != before, "jsonl_line"
+        assert agreement.scorer_fingerprint(_PSC) != before, "jsonl_line"
     # fold() casefolds, so a Python whose Unicode database moves it reads
     # names differently; its moves would be the database's, not the parser's.
     with monkeypatch.context() as patch:
         patch.setattr(unicodedata, "unidata_version", "0.0.0")
-        assert agreement.scorer_fingerprint() != before, "unidata_version"
-    assert agreement.scorer_fingerprint() == before
+        assert agreement.scorer_fingerprint(_PSC) != before, "unidata_version"
+    assert agreement.scorer_fingerprint(_PSC) == before
     monkeypatch.setattr(agreement, "fold", lambda text: text.casefold())
     with pytest.raises(SystemExit, match="another version of this scorer"):
         agreement.main([_PSC])

tests/v2/test_local_corpora.py

@@ -49,9 +49,10 @@
 # the fetches read, so a patch reaches the code only when it is set on
 # the module holding the name (local_core.RETRIES, local_fec.FEC_MAX_BAD).
 # A second load by path (load_tool) would be a copy nothing reads.
-core, ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid = (
+core, ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid, patentsview = (
     importlib.import_module(f"local_{key}")
-    for key in ("core", "ctgov", "fec", "tse", "tw", "ukec", "psc", "jrc", "nepal", "orcid"))
+    for key in ("core", "ctgov", "fec", "tse", "tw", "ukec", "psc", "jrc", "nepal", "orcid",
+                "patentsview"))
 
 _SOURCE = "ctgov-officials"
 _DATE = "2026-10-21T09:00:05"
@@ -3346,10 +3347,11 @@ def test_no_module_imports_a_sibling_constant() -> None:
     # module. A `from local_x import CONSTANT` would bind a second name a
     # patch on local_x cannot reach, and a test could pass with its guard
     # off. The modules are SOURCES' own, so a new source is covered. PSC,
-    # JRC and ORCID are sources' names, keys of SOURCES, and no fetch's setting.
+    # JRC, ORCID and PV are sources' names, keys of SOURCES, and no fetch's setting.
     import ast
     sources = {source.fetch.__module__ for source in local.SOURCES.values()}
-    assert sources == {m.__name__ for m in (ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid)}
+    assert sources == {m.__name__ for m in (ctgov, fec, tse, tw, ukec, psc, jrc, nepal, orcid,
+                                            patentsview)}
     siblings = {"local_corpora", "local_core", *sources}
     imported = {(path.stem, alias.name)
                 for path in (Path(str(local.__file__)).parent / f"{s}.py"
@@ -3358,7 +3360,7 @@ def test_no_module_imports_a_sibling_constant() -> None:
                 if isinstance(node, ast.ImportFrom) and node.module in siblings
                 for alias in node.names if alias.name.lstrip("_").isupper()}
     assert imported == {("local_corpora", "PSC"), ("local_corpora", "JRC"),
-                        ("local_corpora", "ORCID")}
+                        ("local_corpora", "ORCID"), ("local_corpora", "PV")}
 
 
 def test_the_fold_lists_everything_its_functions_call_and_read() -> None: