Repository navigation
Describe each word by its readings: per-word TOML vocabulary compiled to today's sets (phase 1) - #737
Merged
Merged
Conversation
nameparser/vocab/*.toml describes each word by its readings, one file per BCP 47 tag; every category is converted mechanically into und.toml and the suffix words are classified by kind and language. tools/vocab/build.py compiles them into nameparser/config/_compiled.py, which nothing imports yet. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Each nameparser.config constant is now an assignment from _compiled.py followed by its existing docstring, so the public names, their autodoc text and the deprecated-alias bridge are unchanged. The vetting comments moved into the TOML with their words, and the import-time asserts became build checks in tools/vocab/build.py. The TOML ships as package data. tools/vocab/query.py answers a decisions/ pointer (--issue, --kind, --excluded, --word). Lexicon.default() and every raw constant equal the pre-change tree's; the snapshot matches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tests/v2/test_vocab.py holds the committed _compiled.py to a fresh compile (failing with the refresh command), compiles the shipped files with warnings as errors, and gives each build check a mutation it must refuse, beside a control that the unmutated shapes compile. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
decisions/vocabulary-model records the model (Decisions 1-20) and phase 1; the README states the sanctioned exception that lets a moved word fact become a query pointer, and the Excluded: convention narrows to class-level prohibitions. generational-words, vocabulary-exclusions and salutation-titles point at tools/vocab/query.py instead of the Python blocks, W2 records the class-level half of the #307/#308 vetting note, and mechanisms.md gains COMPILED-DATA-COMMITTED. AGENTS.md's configuration-layer section, its vocabulary extension patterns and the disjointness gotcha describe the TOML and the build tool. Also found while converting: particles now state may_be_given on every reading, so neither side of the never-given default is silent, and the remaining #679 generational refusals and the Latin ml/st in sk, sl and hr moved to data with the rest. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Point salutation-titles at the #606 query and the borne notes; tag the Latin ml/st and Russian ст refusals with #661, which first refused them, so generational-words' query returns what it lists; name both refusal kinds in CURATED-VOCABULARY-ALTERNATION; give suffix-acronym-collisions' import assert its dated note; and make query.py --excluded match a refused reading by kind and a refused property only by the property's name, with a test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #737 +/- ##
==========================================
- Coverage 99.05% 99.05% -0.01%
==========================================
Files 46 47 +1
Lines 4647 4642 -5
==========================================
- Hits 4603 4598 -5
Misses 44 44 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Critical: an acronym reading with a period in its text is refused (the
parser looks acronyms up period-free, so it could never match), and a
word-level exclusion of a kind now refuses every kind compiling to the
same set, so suffix = {} or status = {} cannot put back a refused
generation reading.
Also: the build loads _lexicon.py by path, so a broken _compiled.py no
longer stops the tool that rewrites it; excluding a defaulted property
(acronym, addresses) is refused; a VocabError exits 2; the
CONSTANTS[:-1] slice is an explicit MAPPINGS filter. query.py validates
--kind, --excluded and --file, normalizes --word, and lets --kind narrow
--excluded. New mutation rows and query tests cover each. Stale and
wrong comments fixed in build.py, _invariants.py, particles.py,
capitalization.py, und.toml, AGENTS.md and the tests.
Lexicon.default() and the raw constants still equal master's; the
snapshot matches.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two regressions from widening check 6 to 'the same set': the excluded
kind itself passed once acronym moved it to the other set
(generation = { acronym = true } against a generation exclusion), and
the name kinds refused each other though they compile to nothing. A
kind now refuses itself unconditionally, other kinds only where they
feed the same set, and a name kind only itself.
A property exclusion now refuses a value other than the default rather
than the key's presence, so a particle can still state
may_be_given = false beside an exclusion of it, and writing a default
is the same reading as omitting it; the DEFAULTED refusal is gone. An
unreadable file is a VocabError (exit 2), an internal error exits 3, and
query.py reads the files inside its error handling and refuses an empty
--word. Comment corrections in build.py, _invariants.py, und.toml,
cases.py and decisions/vocabulary-model.
The new mutation rows were run against the previous commit's build.py
and each fails there. AGENTS.md records the lesson beside 'review the
fix commit too': a widened check keeps a row for every case its
narrower form refused.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- A missing or empty vocab folder is an error, not an empty vocabulary written with exit 0. Every entry but a dotfile is read, so a misnamed file (EN.TOML, en.toml.orig) or a subfolder is refused rather than skipped. - No error exits 1: _lexicon.py loads at first use inside main's error handling, all of main's work runs inside it in both tools, a module that cannot be written exits 3, and a soft finding raised under -W error exits 2. An unreadable _compiled.py reads as stale, since rebuilding replaces it. - A refused property holds over every reading of its category, and cased over every reading of the word, since both feed sets of words. - A property refusal names the refused value (acronym = false is excluded), so an exclusion on a credential no longer reads backwards. - Check 6 runs before check 4, so the recorded reason is what a contributor meets. - Wording: the CATEGORY comment and AGENTS.md agree on which kinds are generic, check 6's statement and the decisions entry say 'the value it takes unwritten', and the repeated-raise list says 'through 4 and 5'. Run against d11f89c, the 13 new or changed tests all fail: 11 on behavior, 2 on the refusal message's new wording. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A refused property now holds over its whole category, but each reading
was compared with its own unwritten value. In the suffix category
acronym is unwritten-true for a credential and unwritten-false for the
other kinds, so an exclusion of a credential's acronym called
suffix = { acronym = true } refused (the value it keeps) and let
suffix = {} past to check 4 without the reason. _overrides now takes
the exclusion and compares against a reading of the excluding kind, and
the message shows the value the refused reading takes.
Also: _shown writes a date as TOML does (2026-10-18), and a link to
nothing in the vocab folder is named as one rather than as a subfolder.
Run against aea6e79, the four new tests all fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 19, 2026
Merged
2.5.0 shipped while this was in review; the merge from master left the bullet inside the released entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73
added a commit
that referenced
this pull request
Oct 19, 2026
…#737's review (#744) * Vocabulary build: type-design follow-ups and the silent-pass refusals from #737's review Type design (no output moves; _compiled.py is byte-identical): - query.Row stores the exclusion's structure (excluded, prop) and builds `label` only for display; select() no longer parses the label back. - compile_pool/compile_files return a Compiled TypedDict, tied to the CONSTANTS roster by one import-time assert; render reads it by roster. - Reading.acronym and Reading.addresses define each default once, read by _behavior, compile_pool, _refused_by and _check_reading. - Exclusion.reason is a property over props, the object is built only once its table is valid, and Reading and Exclusion are frozen. New refusals, each with a MUTATIONS row (every row's input compiled before this change, measured by stubbing its check): - a text no token can equal: a comma or nakaguro (the tokenizer divides at them), an invisible format character other than ZWNJ/ZWJ (bidi marks are stripped, a zero-width space is kept but unseen), a phrase word with an edge period, and `cased` on a phrase (looked up per word); - a file with no [[word]] entries; - a same_as naming its own word; - a date-time where `decided` or `as_of` expects a date, and an `as_of` with no count. DIVIDERS hand-copies the pipeline's comma and name-dot separators; a test holds it to them and checks every bidi mark the tokenizer strips is refused. A .toml in a subfolder was already refused by read_dir ("no subfolders"), pinned by test_the_folder_is_read_whole. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Vocabulary build: close the gaps the #744 review found - Check 2 also refuses an emoji (stripped by default), an interpunct between two East Asian characters (divides there, rules/T3) and a control character (kept but unseen), and words its comma refusal as what it is: the main stream divides at a comma, and inside a clause rules/M3 depends on the vocabulary holding none. - The phrase-word refusal is a one-key-one-spelling rule, not a never-match one: Lexicon folds each word, so 'z. domu' would have matched as 'z domu'. Comment and message corrected. - The divider test is now driven by the tokenizer's own _ignorable over every codepoint, with EMOJI_RANGES and EAST_ASIAN_RANGES pinned to their sources, so a new separator or strip class fails it. Negative control: hiding the emoji check behind the pinned constant fails the sweep. - Exclusion validates its reason in __post_init__; Row asserts a reading refuses no property; Reading.flag() is the one definition of the may_be_* default. - as_of with a date-time gets its own message; the numbered check list says which checks run per entry and which over the pool. - New rows: control character, emoji, East Asian interpunct, a table-form exclusion reason; a query row pinning whole-name property matching; controls for a Catalan l·l and a CJK text ending in one. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Vocabulary build: hold the codepoint sweep to check 2's own refusals The sweep accepted any VocabError, so U+0020 passed only because a later check refuses a phrase title (found by the fix-commit review). It now excludes the space, a phrase's word separator, and matches check 2's messages. The _check_tokenizable and Exclusion docstrings said more than the code does; corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 19, 2026
derek73
added a commit
that referenced
this pull request
Oct 19, 2026
…s going (#748) A design overview beside nameparser-2.0-rfc.md, distilled from the working notes behind decisions/vocabulary-model, which stays the record and wins where the two differ: why readings replaced the sets; phase 1 as shipped; phase 2 (ambiguity derived from readings, the gate as a property of a reading, starting from today's six gate rules, with fils as the worked case); phase 3 (prefer and strict packs, decision 15 worked through do, a pack outranking a shape rule, and the reversal of #684's premise); phase 4 (order versus shape, the rules a pack can choose, the output model). The first-pack ranking is left to #747. Every "today" parse was measured on e9855df; corpus counts are dated. decisions/vocabulary-model: the reserved-name list gains prefer and strict (reserved in build.py since #737), and decision 11 is widened to words a file refuses, as Derek decided 2026-10-19. The vocab README links to the RFC. Closes #748. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73
added a commit
that referenced
this pull request
Oct 19, 2026
…s going (#748) A design overview beside nameparser-2.0-rfc.md, distilled from the working notes behind decisions/vocabulary-model, which stays the record and wins where the two differ. It covers why readings replaced the sets, phase 1 as shipped, phase 2 (ambiguity derived from readings, the gate for every reading kind, with fils as the worked case and the three gates that exist today), phase 3 (prefer and strict packs, the asymmetry of decision 15 worked through do, pack versus shape rule), and phase 4 (order versus shape, the rules a pack can choose, the output model). Every "today" parse was re-measured on e9855df; corpus counts are dated. It records that #684's premise is reversed by decisions 6 and 15. The first-pack ranking is left to #747. nameparser/vocab/README.md links to it, and decisions/vocabulary-model's list of reserved names gains prefer and strict, which build.py has reserved since #737. Closes #748. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
derek73
added a commit
that referenced
this pull request
Oct 19, 2026
…s going (#748) (#758) A design overview beside nameparser-2.0-rfc.md, distilled from the working notes behind decisions/vocabulary-model, which stays the record and wins where the two differ: why readings replaced the sets; phase 1 as shipped; phase 2 (ambiguity derived from readings, the gate as a property of a reading, starting from today's six gate rules, with fils as the worked case); phase 3 (prefer and strict packs, decision 15 worked through do, a pack outranking a shape rule, and the reversal of #684's premise); phase 4 (order versus shape, the rules a pack can choose, the output model). The first-pack ranking is left to #747. Every "today" parse was measured on e9855df; corpus counts are dated. decisions/vocabulary-model: the reserved-name list gains prefer and strict (reserved in build.py since #737), and decision 11 is widened to words a file refuses, as Derek decided 2026-10-19. The vocab README links to the RFC. Closes #748. Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
derek73
added a commit
that referenced
this pull request
Oct 19, 2026
…s.md, compact shipped measurements (#759) (#764) * Archive decisions/differential-ledger to tombstones and the retirement entry (#759) The ledgers #664 retired took 33,103 words across eleven arcs, every one superseded by the snapshot. Each arc is cut to a tombstone under decisions/README's rule 3 (date, issue, verdict, SUPERSEDED by 2026-10-13 #664); the retirement entry stands in full. The three passages other entries cite stay as one line each: the 'Donald mc' decline (decisions/suffix-acronym-collisions, decisions/v1-xfail-triage), the initials view (decisions/R4, the 2.3.0 release log) and the Policy-annotation Declined entry (decisions/3-0-reevaluations). The full text is pointed to at fe85bb3. mechanisms.md#RECORDED-ROSTERS said the entry keeps the ledger rosters' measurements; it now points to git through it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tidy mechanisms.md: one paragraph per label, contracts without case history (#759) Every entry's template labels (Problem shape, Contract statement, How it works, Lives in, Reach for it when) now open their own paragraph. That is load-bearing and the preamble says so: test_doc_citations reads a Contract statement to the end of its paragraph, so in the one-paragraph layout every sentence of an entry was quotable. ONE-PREDICATE-PER-QUESTION (1,856 words) and RENDER-HONORS-THE-PARSE (2,771) are cut to their contract, a short How it works and pointers. The two sentences code comments quote from outside the old contract proper ("where the reader comes AFTER the decider, ..." in _group.py, "a token the parse never saw ..." in _render.py) move into the Contract statement paragraphs, so no comment in nameparser/ changes. The instance census and the case history are in git at fe85bb3 and in the decisions entries the rosters now name; the one piece no decisions entry held, the facade's spliced-field divergence under a caller's conjunction (#528, measured 2026-09-13 and re-run on this tree), moves to decisions/R3 as a dated entry. LEDGER-RULE-SEPARATION and CROSS-RULE-OUTCOME-PINS, both already "Retired with the ledgers" stubs living in nothing, are removed; the only references left are prose names in decisions/differential-ledger. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Compact shipped measurements and superseded passages in the largest decisions files (#759) A sweep under decisions/README's rule 3 over the 22 largest files after differential-ledger, every change checked shipped in v2.5.0 or earlier (the unshipped PRs are #737, #741-#746, #755, #756). Three triggers only: a whole entry headed SUPERSEDED by, cut to a tombstone; a superseded passage inside a live entry, cut to its decision and what replaced it; a measurement paragraph of a shipped change (what moves, cost, blast radius, gate and grid results), cut to one line naming population, counts, baseline and PR. Verdicts, reasons, Accepted costs, Declined/Excluded/Open blocks, measurements of declined or unshipped changes, harvested framing lines, and anything quoted or pointed at from elsewhere stay. RECOMPUTE recipes a release-log bullet rests on stay; others fold into the one line with their PR. The two #731 entries (decisions/M2, decisions/T4), reverted by PR #733 the same day, gain the SUPERSEDED by marker rule 1 asks for and become tombstones. About 13,400 words come out across the 22 files (M2 14,882 -> 12,190, R4 13,819 -> 12,877, the rest listed in the PR). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix the design-docs review's findings on the #759 trim - decisions/T4: the #731 tombstone had cut a gap and a declined option that predate #731 (ad486f6) and still stand; both are restored verbatim under the tombstone, re-measured on this tree. - decisions/differential-ledger: the tombstones keep the tooling and dormancy arcs' harvested framing lines (rule 3: "a tombstone included"); the `_check_tree` and DEFAULT_BASELINE Declined entries, about code and process that survive the ledgers, stand as written; the initials-view tombstone says the 2.3.0 release bullet's evidence is at fe85bb3. - decisions/H5: the population line no longer calls `毛 泽东 Dr.` a rules example; the #623 line keeps the corpus-and-case-text null result and the 500-parse characterization; three bold heads keep their closing period. - decisions/S2: the #614 line no longer classifies the stress grid's 8,472 movers, which were not classified; it gives the 12-parse slice that was. - decisions/S6: the frame totals decisions/S9 says "hold exactly" are back; the verification-round head keeps "AND ITS RECIPE". - decisions/R3: the facade spliced-field divergence moved from mechanisms.md is a continuation of the 2026-09-13 #528 entry rather than a second entry of that date, so the existing pointers to "the 2026-09-13 #528 bullet" resolve to one entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Fix the second review's findings; record the compaction traps in rule 3 (#759) - decisions/differential-ledger: the initials-view tombstone gave the cluster sizes as 19 / 66 / 11 / 2, which is cluster (iii) split by vocabulary; the three clusters are 27 / 14 / 98. - decisions/T4: the restored gap paragraph was written under #731 and names "the committed grid", which the revert deleted; the lead now says so, points at PR #732, and scopes the re-measurement (the 108 and the readings hold on a rebuild; the 270 was not re-run). - decisions/H5: the #623 frames line said frames move "only" where a part holds a particle; the names without one moved too, and pay less. - decisions/README rule 3 gains the two traps the reviews found, so the next compaction meets them: a tombstone keeps what the entry recorded about the state before its change, and a one-line replacement keeps each count's own classification. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Scope T4's declined-option claim; fold the tombstone trap into rule 3's definition (#759) Third review round, on a4fed35: - decisions/T4: the lead scoped only the 270 as un-re-run; the paragraph's "closes both (0 field and 0 report differences)" is the same #731-era measurement. An approximation of the option re-run 2026-10-19 closes the fields but leaves 27 of 396 rebuilt pairs differing in their reports, so the lead now says the claim is #731's, unconfirmed since. - decisions/README rule 3: the new trap sentence changed what a tombstone keeps without amending the definition two sentences earlier, which still read as a complete list. The definition now says it, and the trap sentence keeps only the one-line-replacement half. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 1 of the per-word vocabulary redesign: the word lists in
nameparser/config/*.pyare now written per word innameparser/vocab/*.tomland compiled into a committed module. No parse, render or public name moves.What changed
nameparser/vocab/<tag>.toml: one file per BCP 47 tag (und= no language assigned yet,mul= shared on purpose). Each[[word]]has atextand readings keyed by kind, each a table. Every category is converted mechanically intound.tomlwith generic kinds (title,particle,conjunction) or their one kind (bound_given,birth_marker) and today's flags. The suffix category is fully classified: kindsgeneration,credential,honorific,professional_honorific,statusandcourtesy; languages taken from the# xxcomments;excludedrefusals with reasons. The files ship as package data.tools/vocab/build.pycompiles the TOML intonameparser/config/_compiled.py(one sorted entry per line) and runs the build checks that replace the import-time asserts. Its docstring numbers the checks.tools/vocab/query.pyanswers--issue,--kind,--excludedand--word.config/*.pykeep their public constants asNAME = _compiled.NAMEwith their autodoc docstrings and the deprecated-alias bridge. The vetting comments moved with their words. In the suffix pilot they became fields (note,issue,same_as,expands,excluded). In the mechanically converted categories they are TOML comments above the same entries, in source order, and inline comments became reading notes.tests/v2/test_vocab.py: the committed module must byte-equal a fresh compile, and the failure names the refresh command. The shipped files must pass every check with warnings as errors. Every build check has a mutation it must refuse, beside a control showing the unmutated shapes compile.decisions/vocabulary-model(Decisions 1–20 plus phase 1); the README's sanctioned exception (a moved per-word fact becomes a query pointer); the narrowedExcluded:convention; query pointers in generational-words, vocabulary-exclusions and salutation-titles; a W2 entry for the class-level half of the Spaced CJK honorifics parse as name parts (王小明 先生 → given 先生) #307/Glued CJK honorifics are absorbed into the name (山田太郎様 → family 山田太郎様) #308 note; mechanismsCOMPILED-DATA-COMMITTED; the AGENTS.md config-layer section, extension patterns and the disjointness gotcha rewritten; a CONTRIBUTING section; a release-log bullet.Two things found while converting (please look)
may_be_givenis required on every particle. The old encoding stored the NEVER-GIVEN particles, so a new particle was ambiguous unless argued otherwise (Rename the vocabulary data modules to the 2.0 terminology (prefixes.py → particles.py), with a 2.x deprecation bridge #293's safe-by-default, C-i's "under uncertainty, ambiguous"). A defaulted flag would pick one side silently, andparticle = {}reading as never-given is the unsafe side. So the build refuses a particle reading without the flag. The spec did not cover this.PARTICLESwas missing from the rendered API reference on master. A plain comment between its#:lines and the assignment split the run. That comment moved intound.toml, so the docstring now renders, with its stale "static union" sentence corrected.Beyond the spec's 22 exclusions: the rest of the #679 named refusals (hy ավագ, km ជូនៀ, fa پدر/پسر, …) also moved to data, so the decisions sentence could become one query. The Latin
ml/strefusals are recorded in all four languages the decision names (cs, sk, sl, hr). The 41 #606 titles carryissue = 606, which turns salutation-titles' recompute recipe into a query.Gate (measured 2026-10-18, Python 3.11.17)
Lexicon.default()and all 13 raw constants equal the pre-change tree's (pickled from master at 6803a26).uv run python tools/differential/snapshot.py: matches the snapshot.may_be_namefromma: 120 snapshot entries move andLexicon.default()becomes unequal;import nameparser: 48.5–48.8 ms median vs master's 49.4 ms (25 fresh processes each).ruff checkandmypyare clean, andmypy --strict tools/vocab/is clean._compiled.py.suffix = {}can no longer put back a refusedgenerationreading._lexicon.pyby path, so a broken_compiled.pycannot stop it from regenerating.casedand property exclusions hold across the word or category, and a refusal names the refused value. The 13 new or changed tests all fail against the second fix commit: 11 on behavior, 2 on the message wording.acronymexclusion misreported. It is fixed, and its four new tests fail against the earlier commit.acronymflipped got through, and the name kinds refused each other. Four of its five new test rows fail against the earlier commit; the fifth pins behavior that was already right.query.pyvalidates its arguments, normalizes--word, and lets--kindnarrow--excluded.Out of scope, and rejected by the compiler with a message naming the phase: derived ambiguity, pack semantics,
addresses = full|any, otherattachedsides.🤖 Generated with Claude Code