Read a comma in a box as a space when scoring local agreement

tests/v2/test_local_agreement.py

@@ -65,6 +65,19 @@ def test_a_box_agrees_whatever_its_case_periods_and_spacing(
     assert _read(upper)[0] == {"given": "ANN"}
 
 
+@pytest.mark.parametrize("surname", ["Roe, Jr.", "Roe,Jr.", "Roe, III", "Roe, Jr., PhD"])
+def test_a_suffix_after_a_comma_in_the_surname_box_agrees(
+        monkeypatch: pytest.MonkeyPatch, surname: str) -> None:
+    # The parser reads "Bob Roe, Jr." as family Roe and suffix Jr., and
+    # the surname box holds the two with the filer's comma between them.
+    filed = _filed("", "Bob", surname=surname)
+    assert _read(filed)[0] == {}
+    # The control: the fold before commas were read as spaces misses it.
+    monkeypatch.setattr(agreement, "fold",
+                        lambda text: " ".join(text.replace(".", "").casefold().split()))
+    assert list(_read(filed)[0]) == ["family"]
+
+
 def test_a_two_word_surname_read_with_a_middle_name_is_two_misses() -> None:
     got = _read(_DOUBLE_SURNAME)[0]
     assert got == {"middle": "García", "family": "López"}

tools/differential/README.md

@@ -295,7 +295,7 @@ A live source has no fixed snapshot to pin, so the cached copy is the pin: `meta
 
 `nepal-candidates` reads each local body's file through the site's own JSON handler, which asks for the site's double-submit check: the `CsrfToken` cookie a GET of the site sets, echoed in an `X-CSRF-Token` header, with the `Referer` its pages send. The site refuses a User-Agent holding `python` or `curl` with a 403 (2026-10-24), so this source sends a browser's form naming the tool (`NEPAL_USER_AGENT`) where the others send the repository's URL. A 403 mid-fetch is what a lapsed token gets, and renews the session once before it is raised; a 404 counts the file as missing, its id kept in `meta.json`'s detail, and with the empty files (a list of no candidates, as a site that blanked its files would answer) past `NEPAL_MAX_MISSING` of the `NEPAL_BODIES` bodies refuses the fetch, the empty ones' ids kept too. The election is over and its files final, so the candidates are pinned as the bodies are: a fetch reading every file must read exactly `NEPAL_CANDIDATES`, so a site that trimmed its files (to the winners, or to one post) is refused however well formed what is left; with files missing or empty the pin cannot be exact, and a shortfall of up to `NEPAL_MAX_SHORT` of it is accepted, so a few candidates trimmed elsewhere are not seen then. Every listed file answered on 2026-10-24: the 18 a survey on 2026-10-23 saw refused, among them ids 5551 to 5554, answered when asked again. A file is JSON, so a body cut short does not parse, and is retried as one cut in transit rather than read as a body with fewer candidates. The fetch refuses a lookup listing other than `NEPAL_BODIES` bodies (the election is over, so its files are final), a file that is not a list of candidates, too many candidates with no romanized or no Devanagari name (`NEPAL_MAX_NAMELESS`) or with a name in the wrong script (`NEPAL_MAX_WRONG_SCRIPT`: two columns swapped, or a romanization dropped), and too few distinct names in either column (`NEPAL_MIN_DISTINCT`). A record is a distinct pair, as `tse-donors`' are, not a candidate row: a body lists a candidate once per post, and a common name recurs across bodies. The `pairs` view refuses it, its records carrying no reference name: that view compares the letters of two spellings, and a romanization shares none with its Devanagari, so the Devanagari is kept under a key of its own. `meta.json` records the keys the corpus was built from (`corpus_keys`), and a corpus written under other keys is refused on load. A fetch took 28 minutes on 2026-10-24 at `NEPAL_DELAY` (0.3 s) a request.
 
-`local_agreement.py` does the SCORE job for a local corpus. It parses each filed display name and compares the parser's roles with the boxes the source states (title, given, middle and family for `psc-individuals`; given and middle joined against the given-names box, then family, for `orcid-researchers`), folding case, periods and spacing. A suffix the parser peels is compared inside the surname box: the display name joins the surname last, so a trailing suffix was typed into it whenever it is filled. Where a source has a title box, a title the parser reads otherwise, holding a word the vocabulary lists as neither a title nor a connective (a job typed as a title), is set aside as filer error rather than scored, and the report lists the commonest such titles as leads for the vocabulary. The report gives agreement overall and by the group the source states (nationality, country), misses per field, and the commonest disagreement shapes, as counts, word counts and the commonest set-aside titles; `--list N` prints moved records, names and all, for reading locally. Its record, `agreement.jsonl` beside the corpus, holds every record that did not wholly agree, so a re-run reports what moved. As with the local snapshot, check it from the branch against a base you name (`--base <checkout>`, whose record is kept under `agreements/` beside the corpus, keyed by the fetch, the base's `nameparser/` files, the interpreter's minor version and the scorer, the ten most recent kept), or, alone, record the shared one from the base checkout (`--record --tree <checkout>`) and run it from the branch; it refuses a record from another fetch, or one written by another version of the scorer, whose own change would otherwise read as the parser's moves. No CI run sees it.
+`local_agreement.py` does the SCORE job for a local corpus. It parses each filed display name and compares the parser's roles with the boxes the source states (title, given, middle and family for `psc-individuals`; given and middle joined against the given-names box, then family, for `orcid-researchers`), folding case, periods, commas and spacing. A suffix the parser peels is compared inside the surname box: the display name joins the surname last, so a trailing suffix was typed into it whenever it is filled, with or without a comma (`Smith, Jr.`); the fold read the comma as a miss until 2026-10-28, which moved 83 `psc-individuals` records and 561 `orcid-researchers` records to agreeing and none away. Where a source has a title box, a title the parser reads otherwise, holding a word the vocabulary lists as neither a title nor a connective (a job typed as a title), is set aside as filer error rather than scored, and the report lists the commonest such titles as leads for the vocabulary. The report gives agreement overall and by the group the source states (nationality, country), misses per field, and the commonest disagreement shapes, as counts, word counts and the commonest set-aside titles; `--list N` prints moved records, names and all, for reading locally. Its record, `agreement.jsonl` beside the corpus, holds every record that did not wholly agree, so a re-run reports what moved. As with the local snapshot, check it from the branch against a base you name (`--base <checkout>`, whose record is kept under `agreements/` beside the corpus, keyed by the fetch, the base's `nameparser/` files, the interpreter's minor version and the scorer, the ten most recent kept), or, alone, record the shared one from the base checkout (`--record --tree <checkout>`) and run it from the branch; it refuses a record from another fetch, or one written by another version of the scorer, whose own change would otherwise read as the parser's moves. No CI run sees it.
 
 `pairs` reads a paired source (`tse-donors`, whose records carry the registry's name beside the filed one) as two spellings of one person, and sorts each pair by how the filed name departs from the reference: the first of its shapes that fits, from case, accents (case unchanged) and case and accents together, through words left out, shortened or misspelled, to few words in common. A word left out or added counts once per pair, and is listed only when ten pairs share it. `--agreement` also parses both spellings of every pair with a reference name that differs, and counts the pairs whose two readings end the family name in the same word (two readings with no family name share none). It lists the accents-only pairs read with two surnames by the two spellings' last words, the same ten-pair floor applied.
 

tools/differential/local_agreement.py

@@ -38,10 +38,12 @@
 
 What counts:
 
-- Each field is compared case-folded, with periods dropped and runs of
-  spaces read as one. The display name is the boxes joined, so the
-  parser sees the filer's own spelling, and the fold only keeps a box's
-  case, periods or spacing from reading as a miss.
+- Each field is compared case-folded, with periods dropped, commas read
+  as spaces and runs of spaces read as one. The display name is the
+  boxes joined, so the parser sees the filer's own spelling, and the
+  fold only keeps a box's case, periods, commas or spacing from reading
+  as a miss: a suffix typed into the surname box after a comma
+  (`Smith, Jr.`) is the parser's family and suffix rejoined.
 - There is no suffix box. A suffix the parser peels (`Jr`, `OBE`) stood
   last in the display name, so it was typed into the last filled box,
   normally the surname: the parser's family is compared with its suffix appended,
@@ -130,7 +132,7 @@ def __post_init__(self) -> None:
 
 
 def fold(text: str) -> str:
-    return " ".join(text.replace(".", "").casefold().split())
+    return " ".join(text.replace(".", "").replace(",", " ").casefold().split())
 
 
 def _title_words(lexicon: Lexicon) -> frozenset[str]: