Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion docs/design/decisions/D1-segmenterless-activation-warning.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
- `parser_for(locales.JA)` without a segmenter warns and does not raise: the inert registration is itself a pinned property, and a warning is filterable by the caller who wants it; the Japanese install hint appears only when a Japanese script is among the dead ones, a hangul-only gap getting the generic remedies — 2026-08 #337, WARN rather than raise
- The warning re-emits from `parser_for`'s frame, so the location reported is the caller's own call — 2026-08 #337, the warning exists because
- The message offers code that type-checks, `Policy(segment_scripts=frozenset())`, pinned by a test, with the bare tuple literal in the denylist test — 2026-08-07 #339.
- `python -m nameparser --locale ja` attaches `ja_segmenter()` where namedivider is installed, and where it is not prints its own note naming the install in place of this warning; `parser_for(locales.JA)` stays explicit and keeps the warning — 2026-10-31 #984

Open: none in this file.

Expand All @@ -13,4 +14,4 @@ Open: none in this file.
- 2026-08 #337 — WARN rather than raise, deliberately: registering the JA pack without the extra must stay safe (the inert registration is itself a pinned property), and a warning is filterable by the caller who wants exactly that inertness. The Japanese install hint is conditional — it appears only when a Japanese script is among the dead scripts; a hangul-only gap gets the generic remedies.
- 2026-08 #337 — the warning exists because parser_for(locales.JA) without segmenter= used to build a parser that silently behaved like a working one minus the feature; it re-emits from the parser_for frame so the reported location is the caller's own call.
- 2026-08-07 #339 — diagnostics that hand the reader code must hand code that type-checks: the message's offered deactivation used a bare tuple literal for segment_scripts — an arg-type error under mypy in a py.typed package — and became frozenset(), pinned by a test asserting the offered spelling. The known-bad spelling is in the denylist test. The structural fact behind the recurrence: autodoc renders docstrings into the API reference, so docs/*.rst is NOT the boundary of "the docs" — a guide and the reference can teach different spellings on the same rendered page, invisibly to any .rst-only sweep.

- 2026-10-31 #984 — THE COMMAND LINE ATTACHES THE INSTALLED JAPANESE SEGMENTER; `parser_for` STAYS EXPLICIT. `python -m nameparser --locale ja` tests for namedivider with `importlib.util.find_spec`, and where it is present passes `locales.ja_segmenter()` to `parser_for`, any failure to load it (an `ImportError` from numpy, an `OSError`) propagating with its own message; only the package's absence falls back, a broad `except ImportError` having reported a broken install as a missing one. Where it is absent the CLI prints a note naming `pip install 'nameparser[ja]'` and filters this warning by its message: its other remedies (covering surnames, `segmenter=`, `Policy(segment_scripts=frozenset())`) are code a command line cannot pass, and a print shows where `PYTHONWARNINGS=ignore` would hide a warning. The CLI is a debug tool with no other way to attach one, so its reading may depend on the environment it runs in. **Declined:** doing the same in `parser_for(locales.JA)`. The same code would parse `高橋一郎` differently with and without the extra installed; a `Parser` holding `ja_segmenter()` cannot be pickled, its segmenter being a local closure, so `parser_for(locales.JA)` would stop pickling wherever the extra is installed; and the pack-and-segmenter pairing is explicit by design (`locales/ja.py`'s docstring). The inert registration this file's first entry pins stays the library's behavior.
5 changes: 4 additions & 1 deletion docs/locales.rst
Original file line number Diff line number Diff line change
Expand Up @@ -383,7 +383,10 @@ look one up dynamically by its lowercase code with

The command line accepts the same codes: ``python -m nameparser
--locale ru --json "Сидоров Иван Петрович"`` applies the pack before
parsing, equivalent to ``parser_for(locales.get("ru"))``.
parsing, equivalent to ``parser_for(locales.get("ru"))``. ``--locale
ja`` also attaches :func:`~nameparser.locales.ja_segmenter` where
``nameparser[ja]`` is installed, which ``parser_for`` never does on its
own (the ``ja`` row below).

Shipped packs
~~~~~~~~~~~~~
Expand Down
1 change: 1 addition & 0 deletions docs/release_log.rst
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ Release Log
- **Add Hindi and Bengali packs, locales.HI and locales.BN, for the romanized Indic titles.** ``parser_for(locales.HI).parse("Smt Nirmala Sitharaman")`` gives title ``Smt``, given ``Nirmala``, where 1.4.0 through 2.8.0 gave given ``Smt``, middle ``Nirmala``. ``HI`` reads ``Smt``, ``Shrimati`` and ``Shriyut`` as titles and ``BN`` ``Smt``, ``Shrimati`` and ``Adhyapak``; ``Shri`` and ``Sushri`` stay name words under both, since each opens given names (``Shri Kant``, ``Sushri Sangita``). See :doc:`locales`. :issue:`345`, :issue:`753`

- **Add --order to python -m nameparser, to read a name in another order.** ``python -m nameparser --order family-first "Kovács János Péter"`` gives family ``Kovács``, given ``János``, where the command, in 2.0.0 through 2.8.0, read only the default order and gave family ``Péter``. It takes ``given-first``, ``family-first`` or ``family-first-given-last``; a name in Han, Hangul or kana keeps its script's order. :issue:`983`
- **Divide Japanese names under python -m nameparser --locale ja when nameparser[ja] is installed.** ``python -m nameparser --locale ja "山田太郎"`` gives family ``山田``, given ``太郎``, where 2.1.0 through 2.8.0 left family ``山田太郎`` and warned. Without the extra it reads as before and prints how to install it, in place of the warning; ``parser_for(locales.JA)`` still needs ``segmenter=locales.ja_segmenter()``. :issue:`984`

**Breaking Changes**

Expand Down
10 changes: 6 additions & 4 deletions docs/usage.rst
Original file line number Diff line number Diff line change
Expand Up @@ -694,10 +694,12 @@ Exceptions are the one thing that does *not* stay inside the parse: a
segmenter is your code, so its errors propagate rather than being
absorbed as content errors.

The command line takes the pack but not the segmenter: ``python -m
nameparser --locale ja`` has no way to attach one, so it activates
nothing by itself. :doc:`customize` covers turning any of these
behaviors off.
The command line attaches the segmenter for you: ``python -m
nameparser --locale ja`` uses :func:`~nameparser.locales.ja_segmenter`
when ``nameparser[ja]`` is installed, and without it divides nothing
and prints a note naming the install. :func:`~nameparser.parser_for` never
does this on its own, so the same code parses alike wherever it runs.
:doc:`customize` covers turning any of these behaviors off.

Aggregate views
----------------
Expand Down
38 changes: 36 additions & 2 deletions nameparser/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,10 +3,14 @@
python -m nameparser "Dr. Juan Q. Xavier de la Vega III"
python -m nameparser --json "Doe, John"
python -m nameparser --order family-first "Kovács János Péter"
python -m nameparser --locale ja "山田太郎"
"""
import argparse
import dataclasses
import importlib.util
import json
import sys
import warnings

from nameparser import (
FAMILY_FIRST,
Expand All @@ -15,6 +19,7 @@
Lexicon,
Locale,
PolicyPatch,
Segmenter,
parser_for,
)

Expand All @@ -27,6 +32,16 @@
"family-first-given-last": FAMILY_FIRST_GIVEN_LAST,
}

_NO_JA_EXTRA = (
"nameparser: --locale ja divides unspaced names only with "
"namedivider-python installed: pip install 'nameparser[ja]'")


def _ja_extra_installed() -> bool:
# only the package's absence falls back: a broken install fails
# with its own error rather than a note saying to install it
return importlib.util.find_spec("namedivider") is not None


def main(argv: list[str] | None = None) -> int:
ap = argparse.ArgumentParser(
Expand All @@ -36,14 +51,17 @@ def main(argv: list[str] | None = None) -> int:
help="print the component dict as JSON")
ap.add_argument("--locale", metavar="CODE",
help="parse with a locale pack (e.g. 'ru'); see "
"nameparser.locales")
"nameparser.locales. 'ja' divides unspaced names "
"when nameparser[ja] is installed")
ap.add_argument("--order", choices=_ORDERS,
help="the order to read a name's words in "
"(default: given-first); a Han, Hangul or kana "
"name keeps its script's order, and a family "
"comma fixes the order")
args = ap.parse_args(argv)
pack = None
segmenter: Segmenter | None = None
noted = False
# `is not None`, not truthiness: --locale "" must reach get() and
# exit 2 listing the codes, not silently fall back to the default
if args.locale is not None:
Expand All @@ -52,6 +70,16 @@ def main(argv: list[str] | None = None) -> int:
pack = locales.get(args.locale)
except KeyError as exc:
ap.error(exc.args[0]) # exits 2, message to stderr
# The pack only activates division; the installed extra divides.
# Without it a note says so in the command line's terms, in
# place of the parser's warning (rules/D1), whose other remedies
# are code; a print also shows where warnings are filtered out.
if pack.code == "ja":
if _ja_extra_installed():
segmenter = locales.ja_segmenter()
else:
print(_NO_JA_EXTRA, file=sys.stderr)
noted = True
# The order is written into the pack's own patch, so it replaces a
# pack's order without a two-pack conflict warning, and the parser
# is built once, parser_for re-emitting its warnings (#337)
Expand All @@ -63,7 +91,13 @@ def main(argv: list[str] | None = None) -> int:
else:
pack = dataclasses.replace(pack, policy=dataclasses.replace(
pack.policy, name_order=order))
n = (parser_for() if pack is None else parser_for(pack)).parse(args.name)
packs = () if pack is None else (pack,)
with warnings.catch_warnings():
if noted:
warnings.filterwarnings(
"ignore", message="Policy.segment_scripts activates")
parser = parser_for(*packs, segmenter=segmenter)
n = parser.parse(args.name)
if args.json:
print(json.dumps(n.as_dict(), ensure_ascii=False))
return 0
Expand Down
145 changes: 121 additions & 24 deletions tests/v2/test_cli.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,21 @@
import importlib.util
import json
import subprocess
import sys
import warnings
from collections.abc import Callable

import pytest

import nameparser.locales
from nameparser import FAMILY_FIRST, Lexicon, Locale, PolicyPatch
from nameparser import (
FAMILY_FIRST,
Lexicon,
Locale,
PolicyPatch,
Segmentation,
Segmenter,
)
from nameparser import __main__ as cli
from nameparser import _locale

Expand Down Expand Up @@ -69,19 +79,6 @@ def test_cli_locale_empty_string_errors_not_silent_default() -> None:
assert "ru" in proc.stderr and "tr_az" in proc.stderr


def test_cli_locale_ja_activates_nothing_by_itself() -> None:
# usage.rst's claim, pinned at the layer it is about: the CLI takes
# the pack but has no way to attach a segmenter, and the ja pack
# ships no vocabulary -- so --locale ja is byte-identical to the
# default parser on the very name the pack exists to divide. A
# future --segmenter flag, or a pack that grew a surname list,
# would land here first.
packed = _run("山田太郎", "--locale", "ja")
default = _run("山田太郎")
assert packed.returncode == 0
assert packed.stdout == default.stdout


def _fields(capsys: pytest.CaptureFixture[str], *argv: str) -> dict[str, str]:
# in process, unlike _run: fast, and the warning tests can see what
# main() emits
Expand Down Expand Up @@ -140,16 +137,6 @@ def test_cli_order_leaves_a_han_name_in_its_scripts_order(
assert (fields["given"], fields["family"]) == ("泽东", "毛")


def test_cli_order_with_a_locale_warns_once_from_main() -> None:
# the parser is built once, by parser_for, which re-emits the
# segmenterless warning from its caller's frame (#337)
with pytest.warns(UserWarning, match="segment_scripts") as record:
assert cli.main(["--locale", "ja", "--order", "family-first",
"--json", "高橋一郎"]) == 0
assert len(record) == 1
assert record[0].filename == cli.__file__


def test_cli_order_unknown_value_exits_2_listing_the_choices(
capsys: pytest.CaptureFixture[str]) -> None:
with pytest.raises(SystemExit) as exc:
Expand All @@ -160,3 +147,113 @@ def test_cli_order_unknown_value_exits_2_listing_the_choices(

def test_cli_order_spellings_match_the_vocabulary_rules_tables() -> None:
assert cli._ORDERS == _locale._ORDERS


# --locale ja attaches the segmenter where namedivider is installed. The
# two subprocess runs are the real thing, each in the environment it
# needs (the extra one only in CI's ja-extra job); the in-process ones
# stand in for the install so every branch runs everywhere.

_JA_EXTRA = importlib.util.find_spec("namedivider") is not None
_JA_NOTE = "pip install 'nameparser[ja]'"


@pytest.mark.skipif(_JA_EXTRA, reason="the ja extra is installed")
def test_cli_locale_ja_without_the_extra_divides_nothing_and_says_why() -> None:
packed = _run("山田太郎", "--locale", "ja")
default = _run("山田太郎")
assert packed.returncode == 0
assert packed.stdout == default.stdout
assert _JA_NOTE in packed.stderr
assert "Warning" not in packed.stderr


@pytest.mark.skipif(not _JA_EXTRA, reason="needs the ja extra")
def test_cli_locale_ja_with_the_extra_divides() -> None:
proc = _run("山田太郎", "--locale", "ja", "--json")
assert proc.returncode == 0, proc.stderr
fields = json.loads(proc.stdout)
assert (fields["family"], fields["given"]) == ("山田", "太郎")
assert proc.stderr == ""


def test_cli_ja_extra_check_agrees_with_the_environment() -> None:
# the in-process tests below stand in for this check; the real one
# must answer what the subprocess runs' skips assume
assert cli._ja_extra_installed() is _JA_EXTRA


def _cut_after_two(text: str) -> Segmentation:
return Segmentation((2,))


def _installed(monkeypatch: pytest.MonkeyPatch,
factory: Callable[[], Segmenter]) -> None:
monkeypatch.setattr(cli, "_ja_extra_installed", lambda: True)
monkeypatch.setattr(nameparser.locales, "ja_segmenter", factory)


@pytest.mark.parametrize("code", ["ja", "JA"])
@pytest.mark.parametrize("order", [(), ("--order", "given-first"),
("--order", "family-first"),
("--order", "family-first-given-last")])
def test_cli_locale_ja_attaches_the_installed_segmenter(
capsys: pytest.CaptureFixture[str],
monkeypatch: pytest.MonkeyPatch,
code: str, order: tuple[str, ...]) -> None:
# --order rewrites the pack; the segmenter must survive it
_installed(monkeypatch, lambda: _cut_after_two)
with warnings.catch_warnings(record=True) as record:
warnings.simplefilter("always")
fields = _fields(capsys, "--locale", code, *order, "山田太郎")
assert (fields["family"], fields["given"]) == ("山田", "太郎")
assert record == []


@pytest.mark.parametrize("order", [(), ("--order", "family-first")])
def test_cli_locale_ja_without_the_extra_notes_once_and_does_not_warn(
capsys: pytest.CaptureFixture[str],
monkeypatch: pytest.MonkeyPatch, order: tuple[str, ...]) -> None:
# the note replaces the parser's warning (rules/D1), whose remedies
# are code a command line cannot pass; a print shows where warnings
# are filtered out
def unexpected() -> Segmenter:
raise AssertionError("ja_segmenter called without the extra")
monkeypatch.setattr(cli, "_ja_extra_installed", lambda: False)
monkeypatch.setattr(nameparser.locales, "ja_segmenter", unexpected)
with warnings.catch_warnings(record=True) as record:
warnings.simplefilter("always")
assert cli.main(["--locale", "ja", *order, "--json", "山田太郎"]) == 0
out, err = capsys.readouterr()
assert json.loads(out)["family"] == "山田太郎"
assert err.count(_JA_NOTE) == 1
assert record == []


@pytest.mark.parametrize("error", [
ImportError("numpy.core.multiarray failed to import"),
OSError("libomp.dylib not found"),
])
def test_cli_locale_ja_a_broken_install_fails_with_its_own_error(
monkeypatch: pytest.MonkeyPatch, error: Exception) -> None:
# namedivider is there but will not load: not the missing-extra note
def broken() -> Segmenter:
raise error
_installed(monkeypatch, broken)
with pytest.raises(type(error), match=str(error)):
cli.main(["--locale", "ja", "山田太郎"])


def test_cli_attaches_a_segmenter_only_for_the_ja_pack(
capsys: pytest.CaptureFixture[str],
monkeypatch: pytest.MonkeyPatch) -> None:
# under zh its surname list divides, and a segmenter would answer
# where the list declines
calls: list[str] = []

def factory() -> Segmenter:
calls.append("ja_segmenter")
return _cut_after_two
_installed(monkeypatch, factory)
_fields(capsys, "--locale", "zh", "王小明")
assert calls == []
Loading