Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/design/AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,6 @@ Skip what the suite already enforces. `tests/v2/test_rules_doc.py`, `tests/v2/te
10. **Re-run every capability claim.** Axis 1 recomputes numbers; this one recomputes CAN and CANNOT. A statement that the parser never or cannot produce some reading is measured by a parse run written now, never derived from a mechanism — most of all when it justifies building something. (#291 derived "a multi-word credential is unparseable" from "vocabulary is matched one token at a time" — which `given_name_titles` had been an exception to since 2026-07-19, a month before that arc, and `maiden_markers` is a second since #434 — and specified a vocabulary set, a matching unit, a predicate and a `Lexicon` field on it; `parse("John Smith, MD PhD").suffix` had been `'MD PhD'` since 1.4.0.)
11. **State the invariant that relates two DIFFERENT inputs.** Axes 1–10 compare a document to another document, to its own examples, or to the implementation, so every one of them can only find code that DISAGREES with a rule — none of them can find a rule that is simply wrong. When a rule implies that two spellings of the same name must parse alike, write that as an invariant over both inputs and run it: it consults no rule statement, so it is not blind that way. Distinct from an invariant over two VIEWS of one parse, like `test_the_family_partitions_into_particles_and_base` in tests/v2/test_cases.py — that one holds however wrong the rule is. (Two review rounds a month apart found real defects and missed the premise under them: #461 was backed out after the rule turned out wrong, #466 closed unmerged for the same reason, and in both, every reviewer checked the code against the rule and THE RULE was at fault. The worked invariant: the family-comma writing under the default order and the comma-less writing under FAMILY_FIRST must parse alike. Measured, it holds for the great majority of the particle vocabulary and fails exactly on the words that are also suffix vocabulary — recovering P6's contested-membership set from behavior, having read no rule. Composition, measurement and recompute recipe: decisions/two-input-invariants.)
12. **Name the signal each new clause reads.** For every clause the diff adds to a rule's statement — in a skeleton rule, for every Except bullet, since each is a place where another reading takes over — say what in the written name it reads, and check it against the rule's `Signal:` line; a diff that moves tests/v2/rules_budget.json is where to look first. A signal the rule did not already read is a new rule wanting its own ID and a line in the hub's map ("One rule per piece of evidence" above), and a clause that restates another rule's condition instead of citing it is the same finding from the other side. This is a structural finding, not a style one: report it under the usual format, the consequence being the next split and its hand retargeting. (S2 reached seven rules' worth of clauses and C1 ten, one sound clause at a time, every review passing, until #682 and #728 split them after the fact.)
13. **TRANSITIONAL: a rule migrated from the block format to the skeleton (#761). Delete this axis in the PR that retires the block format.** List the distinct facts in the old statement and its Accepted clauses (`git show master:docs/design/rules/<file>`) and find each one in the new head, an Except, Precedence or Accepted bullet, or a dated decisions entry; a fact found nowhere was dropped, and a code-comment quote of the rule left with no verbatim home is the usual sign of one. Account for every ID the old `interacts:` named, in the PR body: kept, as an ID in Then, Except or Precedence, or dropped as noise, with a word on why. Check that every old example still runs or that the PR says why it went. A migration changes no rule's meaning: one that finds a rule wrong records it (a `deviates:` marker or an issue) rather than fixing it in the same diff.
13. **TRANSITIONAL: a rule migrated from the block format to the skeleton (#761). Delete this axis in the PR that retires the block format.** List the distinct facts in the old statement and its Accepted clauses (`git show master:docs/design/rules/<file>`) and find each one in the new head, an Except, Precedence or Accepted bullet, or a dated decisions entry; a fact found nowhere was dropped, and a code-comment quote of the rule left with no verbatim home is the usual sign of one. Account for every ID the old `interacts:` named, in the PR body: kept, as an ID in Then, Except or Precedence, or dropped as noise, with a word on why. Check that every old example still runs or that the PR says why it went. A migration changes no rule's meaning: one that finds a rule wrong records it (a `deviates:` marker or an issue) rather than fixing it in the same diff. A block rule scopes many sentences by the PARAGRAPH they stand in rather than by their words, and a skeleton field reads rule-wide: before moving a sentence into Except, Precedence, Then's head or a bullet lead, ask which site or order its paragraph was about, and write that scope into the lead (#777: four P6 clauses about the comma site and one about the family-first site read as covering both sites, each falsified by the other site's input, `Berg Jan vd` under family-first and `van, Jan de`).

Also: before reporting "N names move", report the size of the population that COULD move — a small count over a corpus blind to the shape is evidence about the corpus. For rules/P6 the population is the comma-bearing names that END in a particle — a small fraction of the comma-bearing names, which are themselves under a third of the corpus; decisions/P6 carries the dated measurement and the recompute. Do not restate a corpus count here. Axis 1 over this very sentence once caught it conflating the population with the movers, and the corrected figures that replaced it were superseded within the month as the corpus grew — a standing count in this file is wrong twice over, once for the class and once for the digits.
1 change: 1 addition & 0 deletions docs/design/decisions/P6-trailing-orphan-particle.md
Original file line number Diff line number Diff line change
Expand Up @@ -62,3 +62,4 @@ Decided 2026-08-19 (was Open: [#380](https://github.com/derek73/python-nameparse
RENDERING FOLLOWED THE VOCABULARY, found in review: R4's `capitalized()` split a token at its mark and asked the particle test of the stem, so `IVOI, PAUL D'` repaired to `Paul D' Ivoi` — 30 of the 32 first-listed spellings, all but `al-` and `de’`, whose stems are particles themselves. The whole token is now asked the particle arm's question first, and lowercased only where that arm would lowercase it; wherever the arm declines (an all-particle part, the given or suffix role, a recorded casing) the token goes to the per-word walk as before — a first version sent every whole-token particle down the arm's other clauses instead, which moved `smith, john, ed-` from 'ED-' to 'Ed-' and a configured `van-der` from 'Van-Der' to 'Van-der' (the fix commit's own review). The one shipped word outside #700 it reaches is `'t`, whose stem is `t`: `van 't Hoff` repairs to 'van 't Hoff', where 1.4.0 through 2.4.2 wrote 'van 'T Hoff' — R4's reading, and the right Dutch one. A casing configured for a stem (`capitalization_exceptions` `al` → `AL`) no longer reaches the marked spelling, which is now a particle in its own right: `al-` is lowercased. The snapshot cannot see a render view (decisions/R4); measured instead by `capitalized()` and `capitalized(force=True)` against `00dd2759`, over every name in `de6c045d`'s `tools/differential/corpus*.jsonl` as written, lowercased and uppercased (28 names move, every move lowercasing one token ending in `'` or `-`), and over every particle in eight positions under all three orders (adding `'T` → `'t`, which no corpus name held until R4's `van 't hoff` example joined corpus_rules.jsonl). The boundary `Ivoi, d'` was taken out of rules/P6: P6 does not attach there, and the reading it gets instead (family `Ivoi d'`, as `Jong, de` has read since 2.0) is the never-given vocabulary's family-comma reading, which P6's statement does not describe — a P6 example would have pinned another rule's output. It moves from given `d'` (1.4.0 through 2.4.2, the word being no particle there), and so does every new entry standing alone after a comma (`Smith, al-`); it stays pinned as a case row.
Measured 2026-10-17 against the parent commit `b4a004c3`: 39 corpus names move, every one the same shape — a trailing `d'` or article leaves the middle name and joins the family (38 in corpus_gnd.jsonl, `Orval, Pierre d'` in corpus_conventions.jsonl), and no ambiguity kind changes. Recompute: `uv run python tools/differential/snapshot.py --since b4a004c3`. That is evidence about the corpus as much as the change: these are the comma-bearing names ending in one of the new forms, and GND's index forms are where they occur.
- 2026-10-19 — RULE SPLIT: the third exception (a lone word of both the particle and the unambiguous suffix vocabulary inside a run of post-nominals) left rules/P6 for rules/P9, one rule per piece of evidence (docs/design/AGENTS.md), with rules/P2's clause on the same word before a family comma; #573 decided both. The #573 entry moved to decisions/P9 and a cross-reference stands in its place. Where an entry here says P6's third exception or `_inside_a_credential_run` (the #613 entry), that piece is rules/P9's now.
- 2026-10-20 #761 (skeleton migration) — three passages of the block-format statement left the rule as argument or history rather than statement. The draft that tested the family-first slot alone and fired on 52 default-order names and 366 FAMILY_FIRST_GIVEN_LAST middles, and the FAMILY_FIRST line for `Nguyen Thi Van` removed rather than updated (#470), are the 2026-08-30 #467 entry above. The third is moved here: the comma site and the no-comma family-first site are checked against each other directly, as a pair, rather than only against their own examples — tests/v2/test_order_correspondence.py generates both writings and asserts they parse alike, plus a negative-control sweep pinning the disagreeing set the precedence over S2 names, so a change that breaks one side of that pair should expect that test, not the rule file, to say so first (decisions/two-input-invariants).
127 changes: 60 additions & 67 deletions docs/design/rules/P2-particles-join-forward.md
Original file line number Diff line number Diff line change
@@ -1,70 +1,63 @@
# P2 — particles join forward

P2. Rationale: a particle is written as part of the surname it
precedes, and a title stands outside the name entirely.
A particle joins the words after it into one name part, the
join running until the next particle starts a group of its own,
a trailing suffix begins — the trailing run S2 reads, once,
before the chain is made and counting the chain's own run as the
one name word it will be, so the chain cannot change what it
counts: a trailing roman numeral S10 reads as the generation, or
a bare acronym with words to spare, ends the chain as a suffix
word does, and so does a bare ambiguous acronym written in capitals in
a mixed-case name (S6), which needs no words to spare; the same
acronym written Title-case in a mixed-case name ends nothing and
joins the chain as any name word does, words to spare or not,
and the fork the chain called is reported there exactly as it is
where no particle stands (S4) —
a maiden
marker takes the
words after it (M2), or the name ends. The final group reads as
the family name;
earlier groups read by position. The chain begins wherever the
name begins, and a preceding title does not move that point.
Where P1's fold has claimed the opening, the fold decides the
family instead — and may take only PART of the final group,
since it counts name words and the group is one part.
"John van der Berg" → family="van der Berg"
"John van der Berg Smith" → family="van der Berg Smith"
"Vincent van Gogh van Beethoven" → middle="van Gogh"
"Dr. John van der Berg" → family="van der Berg"
"Juan de" → family="de" · boundary
"de la Cruz Juan Carlos" family-first → family="de la Cruz"
"John van der Berg PhD" → family="van der Berg"
"John van der Berg V" → family="van der Berg"
"John van der Berg V" → suffix="V"
"John van der Berg Ma" → family="van der Berg Ma"
"John van der Berg Ma" → ambiguities=("suffix-or-name",)
"john van der berg ma" → suffix="ma"
"John van der J. V" → family="van der J. V" · boundary
"Freiherr von Berg MA" → family="von Berg"
"Freiherr von Berg MA" → suffix="MA"
"Freiherr von Richthofen V" → family="von Richthofen V" · boundary
"John van der Berg née Jones" → family="van der Berg"
Accepted: a particle of the unambiguous suffix vocabulary too
(vd, mc) is a suffix piece to the peel, so where it opens the
trailing run the chain stops before it as before any suffix
word, and the peel takes it; where it continues a prefix run,
the run takes it as a particle, as P6 reads it after a comma —
and there the chain reports nothing, the particle reading being
P6's fork rather than S4's.
The same carve-out covers `do`, the one word that is both a
particle and an AMBIGUOUS credential acronym: where a particle
run takes it, the word is the run's and the chain reports
nothing, whatever case it is written in — so `Anh van Do` is
silent where `Anh Do`, with no particle standing, reports.
A word that is both belongs to the particle run and to P6's
fork, not to S4's.
"John Smith Mc V" → suffix="Mc V"
"John van Mc" → family="van Mc"
"anh van do" → family="van do" · boundary
"anh van do" → ambiguities=() · boundary
"Anh van Do" → family="van Do" · boundary
"Anh van Do" → ambiguities=() · boundary
Accepted: a caller wanting the combined double-surname reading
(#132's ask) has it as the surnames view rather than the
family field.
"Vincent van Gogh van Beethoven" → surnames="van Gogh van Beethoven"
Signal: a neighbor: the words following a particle, up to the next particle, suffix run or marker
history: decisions/P2 · interacts: P1, P4, H5, M2, S2, C2, S4, S10, S6, P9 · implemented: nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_post_rules.py
**When:** a particle stands inside a name with name words after it, as in "John van der Berg".

**Then:** a particle joins the words after it into one name part, the join running until the next particle starts a group of its own, a trailing suffix begins, a maiden marker takes the words after it (M2), or the name ends; the final group reads as the family name, and earlier groups read by position.

- **The chain begins wherever the name begins**, and a preceding title does not move that point (P4).
```
"Dr. John van der Berg" → family="van der Berg"
```
- **Where a trailing suffix begins** is the trailing run S2 reads, once, before the chain is made and counting the chain's own run as the one name word it will be, so the chain cannot change what it counts: a trailing roman numeral S10 reads as the generation, or a bare acronym with words to spare, ends the chain as a suffix word does, and so does a bare ambiguous acronym written in capitals in a mixed-case name (S6), which needs no words to spare.
```
"John van der Berg PhD" → family="van der Berg"
"John van der Berg V" → family="van der Berg"
"John van der Berg V" → suffix="V"
"john van der berg ma" → suffix="ma"
"John van der J. V" → family="van der J. V" · boundary
"Freiherr von Berg MA" → family="von Berg"
"Freiherr von Berg MA" → suffix="MA"
"Freiherr von Richthofen V" → family="von Richthofen V" · boundary
```
- **The same acronym written Title-case in a mixed-case name ends nothing** and joins the chain as any name word does, words to spare or not, and the fork the chain called is reported there exactly as it is where no particle stands (S4).
```
"John van der Berg Ma" → family="van der Berg Ma"
"John van der Berg Ma" → ambiguities=("suffix-or-name",)
```
- **Where P1's fold has claimed the opening, the fold decides the family instead** — and may take only PART of the final group, since it counts name words and the group is one part.
```
"de la Cruz Juan Carlos" family-first → family="de la Cruz"
```

**Because:** a particle is written as part of the surname it precedes, and a title stands outside the name entirely.

```
"John van der Berg" → family="van der Berg"
"John van der Berg Smith" → family="van der Berg Smith"
"Vincent van Gogh van Beethoven" → middle="van Gogh"
"Juan de" → family="de" · boundary
"John van der Berg née Jones" → family="van der Berg"
```

**Accepted:**

- A particle of the unambiguous suffix vocabulary too (vd, mc) is a suffix piece to the peel, so where it opens the trailing run the chain stops before it as before any suffix word, and the peel takes it; where it continues a prefix run, the run takes it as a particle, as P6 reads it after a comma — and there the chain reports nothing, the particle reading being P6's fork rather than S4's.
```
"John Smith Mc V" → suffix="Mc V"
"John van Mc" → family="van Mc"
```
- The same carve-out covers `do`, the one word that is both a particle and an AMBIGUOUS credential acronym: where a particle run takes it, the word is the run's and the chain reports nothing, whatever case it is written in — so `Anh van Do` is silent where `Anh Do`, with no particle standing, reports. A word that is both belongs to the particle run and to P6's fork, not to S4's.
```
"anh van do" → family="van do" · boundary
"anh van do" → ambiguities=() · boundary
"Anh van Do" → family="van Do" · boundary
"Anh van Do" → ambiguities=() · boundary
```
- A caller wanting the combined double-surname reading (#132's ask) has it as the surnames view rather than the family field.
```
"Vincent van Gogh van Beethoven" → surnames="van Gogh van Beethoven"
```

---
**Signal:** neighbor: the words following a particle, up to the next particle, suffix run or marker
**history:** decisions/P2 · **implemented:** nameparser/_pipeline/_group.py, nameparser/_pipeline/_pieces.py, nameparser/_pipeline/_post_rules.py
Loading
Loading