Merge pull request #444 from derek73/fix/410-h1-name-words

fix(post_rules): H1 counts name words, not annotations (#410)

Author: derek73

docs/design/decisions.md

@@ -29,18 +29,22 @@ Background: an honorific title precedes a name and is not itself part of it; it
 
 H1. Rationale: a title normally addresses by surname, so a title
     followed by a single name word usually names the family; but a
-    given-name title addresses by given name.
-    A title followed by exactly one name word and nothing else makes
-    that word the family name, unless the title is a given-name
-    title, which keeps it the given name.
+    given-name title addresses by given name. What stands beside
+    that word — a suffix, a nickname, a maiden name — does not make
+    the name any longer, so it does not decide this reading.
+    A title followed by exactly one name word makes that word the
+    family name, whatever suffix, nickname or maiden name stands
+    beside it, unless the title is a given-name title, which keeps
+    it the given name.
       "Mr. Johnson"               →  family="Johnson"
       "Mrs. Garcia"               →  family="Garcia"
+      "Dr. Smith née Jones"       →  family="Smith"
       "Sir John"                  →  given="John"  · boundary
     Accepted: a given-name title plus one name word leaves the
     family empty — the input names no family, and inventing one
     would be worse.
       "Sir John"                  →  family=""
-    interacts: P5 · implemented: nameparser/_pipeline/_post_rules.py
+    history: decisions.md#H1 · interacts: P2, P3, P5, M2, S1, S2, N1, N3 · implemented: nameparser/_pipeline/_post_rules.py
 
 H2. Rationale: before a name, an abbreviation is almost always a
     title — "Rev.", "Ing.", "Mag." — and no vocabulary can list
@@ -513,11 +517,17 @@ N3. Rationale: a person set down as a nickname plus one name word is
     positional reading applies.
       "'Smitty' Jones"            →  family="Jones"
       "'Smitty' John Jones"       →  given="John"  · boundary
-    Accepted: the count does not set suffixes or titles aside, so a
-    nickname plus one name word plus a suffix reads the name word
-    as given and leaves the family empty.
+    Accepted: the count does not set suffixes aside, so a nickname
+    plus one name word plus a suffix reads the name word as given
+    and leaves the family empty. A title counts against the count
+    too, but H1 then reads the title-plus-one-word name that is
+    left, so the family is named after all — unless the title is a
+    given-name title, which keeps the word in `given` and leaves no
+    family, exactly as it does anywhere else.
       "'Smitty' Jones Jr."        →  family=""
-    history: decisions.md#N3 · implemented: nameparser/_pipeline/_assign.py
+      "'Smitty' Dr. Jones"        →  family="Jones"
+      "'Smitty' Sir John"         →  given="John"
+    history: decisions.md#N3 · interacts: H1 · implemented: nameparser/_pipeline/_assign.py
 
 ## Maiden names (M)
 
@@ -587,7 +597,7 @@ M2. Rationale: a maiden marker announces that what follows it is the
     is maiden text all the same — the count it needs includes the
     very words the marker removes, so the reading is left to assign.
       "John née Jones Smith Ma"        →  maiden="Jones Smith Ma"
-    history: decisions.md#M2 · interacts: P2, P3, P5, R2, M1, S2 · implemented: nameparser/_pipeline/_group.py
+    history: decisions.md#M2 · interacts: P2, P3, P5, R2, M1, S2, H1 · implemented: nameparser/_pipeline/_group.py
 
 ## Commas & structure (C)
 

docs/design/rules.md

@@ -12,11 +12,15 @@ Release Log
 
     Nothing moved between vocabularies. The rename itself changes no
     parse at all; the parsing changes below are separate fixes, and
-    only one of them reaches the default name order -- a title no
-    longer changing how the name behind it is read, which moves four
-    names of the differential corpora. Both family-first orders change too,
-    below -- seven of those from the family-first fix, plus the names
-    the title fix moves in every order. The corpora have roughly
+    most of them reach the default name order -- each bullet names
+    the shapes and the corpus names it moves, and the gate output is
+    the source of record. The unusual part is the family-first
+    orders, which change too: seven names from the family-first fix,
+    plus the names #367's title fix moves, which it moves in every
+    order. The other title fix below, #410, is default-order only --
+    under either family-first order the word behind the title is
+    already the family, so the rule is a no-op there and moves
+    nothing (measured over all four corpora). The corpora have roughly
     doubled across this cycle (#414 added the rules doc's examples,
     #413 taught the issue harvester to read backticks), so the counts
     in the bullets below are the classified summary at the time each
@@ -69,7 +73,9 @@ Release Log
 
     - Fix a bound given-name join leaving no family name when the name also carries a maiden clause, and stop the join absorbing the marker itself. ``"abdul Berg née Jones"`` read given ``abdul Berg`` with an EMPTY family, where ``"abdul Berg"`` alone correctly reads given ``abdul``, family ``Berg``. The join reserves a name word so it always leaves a family name behind -- two name words alone do not join -- but the reserve was counted while the maiden marker and the maiden name were still part of the name, and the pass that removes them runs afterwards. Four words counted, the join fired, and when the two departed nothing was left for the family. The words a marker will take no longer count toward the reserve, so that name reads given ``abdul``, family ``Berg``, maiden ``Jones``. Excluding them is not sufficient on its own: where a suffix sits INSIDE the name the marker walk stops early, enough words survive to clear the reserve, and the join fired and took the marker -- ``"abd née Jones Jr Smith Berg"`` read given ``abd née``. The join now declines outright when the piece it would absorb is a marker, which is what the rule meant all along: it joins the bound word to a name word, and a marker is not one. A name with a word to spare is untouched -- ``"abd Allah Smith née Jones"`` still joins, giving given ``abd Allah``, family ``Smith``, maiden ``Jones``. This is pre-existing for the spellings that were already vocabulary: ``abdul``, ``abdel`` and ``abdal`` read this way at 2.0.0 and 2.1.0 alike. ``abd`` is different -- it reads CORRECTLY at 2.1.0 and acquired the defect only when this same release made it a bound given-name word, so for that spelling this fixes a regression introduced earlier in the cycle; and the particle spellings reached the reserve for the first time through the maiden-marker chain stop above. Two consequences worth knowing. The bound-given join no longer swallows a maiden marker at all, so ``"van der Berg, abdul née Jones"`` reads given ``abdul``, family ``van der Berg``, maiden ``Jones`` where it read given ``abdul née``, middle ``Jones``; the connective join is the remaining case, tracked at `#412 <https://github.com/derek73/python-nameparser/issues/412>`_. And where the bound word is ALSO suffix vocabulary, a declining join after a family comma leaves the post-nominal reading and the name has no given name: ``"Berg, abd née Jones"`` reads family ``Berg``, suffix ``abd``, maiden ``Jones``, matching how ``"Berg, abd"`` alone has always parsed. ``abd`` is the only shipped word in both sets (closes #411)
 
-    - Fix a maiden clause changing how the rest of the name is read, and a connective join keeping the marker in the surname. A maiden marker and the name it introduces are not part of the name they follow, but the grouping rules that count a name's words -- the three-word test that keeps a single-letter connective a name word, and the bound given-name reserve -- counted them, because the pass that removes them ran after those rules. ``"juan y garcia"`` reads given ``juan``, middle ``y``, family ``garcia``, but ``"juan y garcia nee jones"`` counted five words, joined the ``y``, and read given ``juan y garcia`` with NO family name at all; ``"John e Smith nee jones"``, ``"Lt.Gov. juan e garcia nee jones"`` and six more differential corpus names lose the family the same way under an appended clause. The marker pass now runs before every join, so the joins and the counts see only the name that remains, and a name of two or more name words reads as it reads without its maiden clause, plus the maiden name: ``"juan y garcia nee jones"`` is given ``juan``, middle ``y``, family ``garcia``, maiden ``jones``. (A title plus ONE name word is the pre-existing #410 shape, where the lone word sits in ``family`` alone and in ``given`` once a maiden name follows -- ``"Dr. Jane"`` against ``"Dr. Jane née Smith"`` -- and is unchanged here.) The same order closes the last of the join-swallows. The connective join used to merge the marker into a multi-word piece before the marker rule could see it, so ``"Jane van der Berg née y Jones"`` kept family ``van der Berg née y Jones`` with no maiden name; it now reads family ``van der Berg``, maiden ``y Jones`` -- the marker takes the words after it, connective included -- and ``"Jane née and Jones Smith"`` reads given ``Jane``, maiden ``and Jones Smith`` where it read middle ``née and Jones``, family ``Smith``. It also retires the particle chain's marker stop from the #399 fix above, whose condition restated the marker rule's and disagreed with it one suffix later (``"Jane van der Berg née Jr Jones"``, #417): a marker the rule declines -- nothing after it but a suffix -- is an ordinary word and rides inside the chain, which is the reading ``"Jane van der Berg née"`` already had, and the parsed fields of that name do not change. Two limits. A bound given-name word still never joins onto a marker standing as a word of its own, so ``"Berg, abdul née PhD"`` keeps given ``abdul``; a declined marker the particle chain has taken travels with the chain, so ``"Abd van der Berg née Jr Jones"`` reads given ``Abd van der Berg née`` where it read given ``Abd van der Berg``, middle ``née Jr``. And the marker rule now sees the words as written rather than as joined, so a suffix-vocabulary word inside the maiden name stops it even with a connective beside it: ``"Jane née Jr y Jones"`` read maiden ``Jr y Jones`` and now reads family ``Jr y Jones`` with no maiden name, and ``"Jane Smith née Jones Jr y Smith"`` read maiden ``Jones Jr y Smith`` and now reads maiden ``Jones``, family ``Jr y Smith``. One consequence: ``"abd née Jones Jr Smith Berg"`` reads given ``abd Jr`` -- as ``"abd Jr Smith Berg"`` already does -- where the #411 fix above read given ``abd``. Four of the 1025 differential corpus names move. The two connective-join names move at the 2.0.0 and 2.1.0 baselines alike; at 1.4.0 both were already classified as maiden-marker changes, 1.4 having no maiden field. ``"Juan y Garcia née Jones"`` and ``"Jane née Jr y Jones"``, the examples this fix adds to the rules doc and so to the rules corpus, move at all three and at 2.0.0/2.1.0 respectively. The eight corpus names that lose a family only under an APPENDED clause are not in the corpora in that form, so the gate cannot see them; the case table pins that half (closes #412, closes #417, closes #418)
+    - Fix a maiden clause changing how the rest of the name is read, and a connective join keeping the marker in the surname. A maiden marker and the name it introduces are not part of the name they follow, but the grouping rules that count a name's words -- the three-word test that keeps a single-letter connective a name word, and the bound given-name reserve -- counted them, because the pass that removes them ran after those rules. ``"juan y garcia"`` reads given ``juan``, middle ``y``, family ``garcia``, but ``"juan y garcia nee jones"`` counted five words, joined the ``y``, and read given ``juan y garcia`` with NO family name at all; ``"John e Smith nee jones"``, ``"Lt.Gov. juan e garcia nee jones"`` and six more differential corpus names lose the family the same way under an appended clause. The marker pass now runs before every join, so the joins and the counts see only the name that remains, and a name of two or more name words reads as it reads without its maiden clause, plus the maiden name: ``"juan y garcia nee jones"`` is given ``juan``, middle ``y``, family ``garcia``, maiden ``jones``. (A title plus ONE name word is the #410 shape, fixed separately below: ``"Dr. Jane née Smith"`` reads family ``Jane`` now, as ``"Dr. Jane"`` does.) The same order closes the last of the join-swallows. The connective join used to merge the marker into a multi-word piece before the marker rule could see it, so ``"Jane van der Berg née y Jones"`` kept family ``van der Berg née y Jones`` with no maiden name; it now reads family ``van der Berg``, maiden ``y Jones`` -- the marker takes the words after it, connective included -- and ``"Jane née and Jones Smith"`` reads given ``Jane``, maiden ``and Jones Smith`` where it read middle ``née and Jones``, family ``Smith``. It also retires the particle chain's marker stop from the #399 fix above, whose condition restated the marker rule's and disagreed with it one suffix later (``"Jane van der Berg née Jr Jones"``, #417): a marker the rule declines -- nothing after it but a suffix -- is an ordinary word and rides inside the chain, which is the reading ``"Jane van der Berg née"`` already had, and the parsed fields of that name do not change. Two limits. A bound given-name word still never joins onto a marker standing as a word of its own, so ``"Berg, abdul née PhD"`` keeps given ``abdul``; a declined marker the particle chain has taken travels with the chain, so ``"Abd van der Berg née Jr Jones"`` reads given ``Abd van der Berg née`` where it read given ``Abd van der Berg``, middle ``née Jr``. And the marker rule now sees the words as written rather than as joined, so a suffix-vocabulary word inside the maiden name stops it even with a connective beside it: ``"Jane née Jr y Jones"`` read maiden ``Jr y Jones`` and now reads family ``Jr y Jones`` with no maiden name, and ``"Jane Smith née Jones Jr y Smith"`` read maiden ``Jones Jr y Smith`` and now reads maiden ``Jones``, family ``Jr y Smith``. One consequence: ``"abd née Jones Jr Smith Berg"`` reads given ``abd Jr`` -- as ``"abd Jr Smith Berg"`` already does -- where the #411 fix above read given ``abd``. Four of the 1025 differential corpus names move. The two connective-join names move at the 2.0.0 and 2.1.0 baselines alike; at 1.4.0 both were already classified as maiden-marker changes, 1.4 having no maiden field. ``"Juan y Garcia née Jones"`` and ``"Jane née Jr y Jones"``, the examples this fix adds to the rules doc and so to the rules corpus, move at all three and at 2.0.0/2.1.0 respectively. The eight corpus names that lose a family only under an APPENDED clause are not in the corpora in that form, so the gate cannot see them; the case table pins that half (closes #412, closes #417, closes #418)
+
+    - Fix a title-plus-surname name losing its family name whenever anything stood beside it. ``"Dr. Smith"`` reads family ``Smith``, but ``"Dr. Smith née Jones"`` read given ``Smith`` with no family at all, and so did ``"Dr. Smith PhD"`` and ``"Dr. "Smitty" Smith"``. A title followed by a single name word names the family, and that rule declined whenever the name also carried a suffix, a nickname or a maiden name -- it counted them as further name words, which none of them is: each stands beside the name rather than in it, and none makes the name any longer. The rule now counts name words alone. ``"Dr. Smith née Jones"`` reads family ``Smith``, maiden ``Jones``; ``"Dr. Smith PhD"`` family ``Smith``, suffix ``PhD``; ``"Dr. "Smitty" Smith"`` family ``Smith``, nickname ``Smitty``; and the particle spelling this was found through, ``"Freiherr von Richthofen geb. Albrecht"``, reads family ``von Richthofen``, maiden ``Albrecht`` -- 2.2's own particle-chain stop had routed that canonical shape into the rule for the first time (#399). A given-name title is unchanged and still names no family: ``"Sir John née Jones"`` keeps given ``John`` with an empty family, exactly as ``"Sir John"`` does. One name moves where the nickname LEADS, rather than standing beside a name that already reads: ``"'Smitty' Dr. Jones"``. The lone-word nickname rule declines there because the title counts against its piece count, which leaves a title and one name word for this fix to read as the family -- given ``Jones`` through 2.1, family ``Jones`` now. Five names of the differential corpora change reading, the same five at every baseline. Four of them arrive as new diffs and are classified with this fix: ``"Dr. Smith née Jones"``, ``"Senator "Rick" Edmonds"``, ``"Xyz. (Bud) Smith"`` and ``"'Smitty' Dr. Jones"``. The fifth, ``"Freiherr von Richthofen V"``, was already classified under 2.2's particle-chain fix and stays there -- it is the suffix flavor in corpus form, and reads family ``von Richthofen`` with suffix ``V`` where it read given ``von Richthofen`` before. v1 read the two nicknames as 2.1 did; it had no maiden support at all, reading ``"Dr. Smith née Jones"`` as first ``Smith``, middle ``née``, last ``Jones``. The v1 test suite shipped the correct reading for the nickname shape as a known-failing test, which now passes (closes #410)
 
     **Deprecations**
 

docs/release_log.rst

@@ -9,7 +9,7 @@
 Lexicon.given_name_titles.
 
 Implements rules H1, P1, O1, O2 and O3 of docs/design/rules.md; each
-is cited at its code below, and P1/O1/O2's history lives in
+is cited at its code below, and H1/P1/O1/O2's history lives in
 docs/design/decisions.md.
 """
 from __future__ import annotations
@@ -174,13 +174,15 @@ def post_rules(state: ParseState) -> ParseState:
     givens = _idx(tokens, Role.GIVEN)
     middles = _idx(tokens, Role.MIDDLE)
     families = _idx(tokens, Role.FAMILY)
-    others = any(t.role in (Role.SUFFIX, Role.NICKNAME, Role.MAIDEN)
-                 for t in tokens)
 
-    # rules.md#H1: "a title followed by exactly one name word and
-    # nothing else makes that word the family name, unless the title
-    # is a given-name title" (v1 handle_firstnames)
-    if titles and givens and not middles and not families and not others:
+    # rules.md#H1: "a title followed by exactly one name word makes
+    # that word the family name, whatever suffix, nickname or maiden
+    # name stands beside it, unless the title is a given-name title,
+    # which keeps it the given name" -- counting those three as
+    # further name words is what emptied the family (#410)
+    # (known gap: the guard tests which roles are unoccupied, it does
+    # not count units -- decisions.md#H1) (v1 handle_firstnames)
+    if titles and givens and not middles and not families:
         joined = _title_key(tokens[i].text for i in titles)
         if joined not in state.lexicon.given_name_titles:
             for i in givens:

nameparser/_pipeline/_post_rules.py

@@ -153,7 +153,6 @@ def test_nickname_and_last_name(self) -> None:
         self.m(hn.last, "Edmonds", hn)
         self.m(hn.nickname, "Rick", hn)
 
-    @pytest.mark.xfail
     def test_nickname_and_last_name_with_title(self) -> None:
         hn = HumanName('Senator "Rick" Edmonds')
         self.m(hn.title, "Senator", hn)

tests/test_nicknames.py

@@ -372,7 +372,8 @@ def test_leading_period_abbreviation_case_insensitive(self) -> None:
     def test_leading_period_abbreviation_with_nickname(self) -> None:
         hn = HumanName("Xyz. (Bud) Smith")
         self.m(hn.title, "Xyz.", hn)
-        self.m(hn.first, "Smith", hn)
+        self.m(hn.first, "", hn)
+        self.m(hn.last, "Smith", hn)
         self.m(hn.nickname, "Bud", hn)
 
     def test_charge_daffaires_chains_as_title(self) -> None: