diff --git a/docs/design/decisions.md b/docs/design/decisions.md index 4b7596fd..12517bce 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -520,15 +520,24 @@ Declined: ### R2 — the all-particles family_base divergence +- 2026-08-18 #404/#385 — R2's DISCRIMINATOR REVERSED. It said a family written wholly out of particle vocabulary has a base where one of those words "is itself borne as an ordinary surname", and that only the words that are never anyone's name stay particles. That is a vocabulary test, and it is the wrong one: a particle earns its name by joining forward to the word it modifies, so a particle with nothing to join is not doing a particle's work whatever it is borne as. POSITION decides, and the borne-as-surname question does not arise. + The invariant that replaces it, and the reason this is worth a rule rather than three bug fixes: a non-empty family always has a non-empty base. A particle needs a base to attach to. Measured, 43 of 751 corpus names x 3 orders violated it — "Del Toro" under FAMILY_FIRST reported family 'Del' with no base, losing the surname from the base and from the initials. + Marked rather than untagged (mechanisms.md#MARK-DONT-STRIP): `particle` is stable API and means "from the particle vocabulary wherever it lands", which stays true; UNJOINED_TAG records what was decided. That also leaves #405's ambiguity emitter a predicate to key on, which stripping would have destroyed for exactly the never-given half where the fork is most interesting. + Blast radius: 44 of 2,253 parses move and every one is VIEWS-only — the seven role fields are byte-identical everywhere, so the differential harness cannot see this change at all (it compares roles only). Tests are the whole verification, and the invariant is one of them. + - Recorded 2026-08-16, intent UNVERIFIED: the v1-era design held that a family name cannot be only particles ("Anh Do" — Do is a surname AND a particle), so last_base was guarded non-empty. The facade still guards (HumanName("Anh Do").last_base == "Do"); the v2 core does not (parse("Anh Do").family_base == "", family_particles == "Do"), and the surname vanishes from initials (parse("Anh Do").initials() == "A."). - 2026-08-16 #385 RESOLVED by the collision criterion, not on its own terms: the issue's option 3 ("guard the view only when a word is vocabulary-ambiguous") is what decisions.md#vocabulary-collisions produces when applied here. "Do" is borne as an ordinary surname, so it is ambiguous vocabulary and anchors the base; "van der" is never anyone's name, so an all-particle family there genuinely has no base. The two rows of the issue's table were never one case. This is the keystone's clearest payoff: #385 was filed as a leaf with three options and no way to choose between them, and the criterion picks one without arguing about family_base at all. -- Still open inside the resolution: whether "Do" remains in family_particles once it is also the base. Recorded here rather than left to the implementing PR to decide by accident. +- ANSWERED 2026-08-18 by the entry above, and recorded rather than left implicit: "Do" does NOT remain in family_particles once it is the base. The two views partition the family — parse("Anh Do").family_particles is "" and family_base is "Do" — which is asserted as a partition over the whole case table, not just as an example. Was: "Still open inside the resolution: whether Do remains in family_particles once it is also the base. Recorded here rather than left to the implementing PR to decide by accident." - 2026-08-16 (pre-merge coherence pass) — the resolution moves R3 too, and R3 now carries its own marker. Initials read the BASE family word, so anchoring "Do" changes parse("Anh Do").initials() from "A." to "A. D." while "Juan van der" stays "J." (no borne name, no base, and initials of a bare particle run would be nonsense). The general lesson, worth more than this instance: a deviates: marker gets written on the rule whose STATEMENT changed, but a rule can change another rule's OUTPUT without touching its statement, and nothing looks for that — the runner asserts per example line, so an unmarked downstream rule stays green precisely because its own examples avoid the affected input. When adding a marker, walk the changed rule's `interacts:` targets and ask whether any of THEIR examples move. +Declined: + +- 2026-08-18 — the GROUPING half of #404: a particle run that joins nothing does not chain, so "Jong van der" would split into middle 'van' plus family 'der'. Measured and rejected, though NOT for the reason first recorded here. The first draft said the split makes the family "no longer all-particle so the base fix stops firing" — false, and `der` and `la` are both shipped particles, so a family of either IS all-particle and the rule fires on it ("Juan Smith der" gives base 'der'). What the split actually costs is the SCOPE of the base and a stray particle relocated: grouping can decline to merge but cannot keep the words apart, because roles re-assemble them and two adjacent same-role pieces are one part at the field level. "Juan Smith van der" becomes middle 'Smith van', family 'der' — a base of 'der' rather than 'van der', and a middle name nobody wrote. Keeping the run whole in one part is what gives the base its full extent. The split reading needs the leftover distribution to know these are separate units, which is mechanisms.md#UNIT-PARTITION's problem. + ### removed-v1-surface - empty_attribute_default: removed in 2.0 (#255; deprecated in 1.4 per the bridge discipline). Origin #44 (2016): a DB-NULL convenience whose first answer — `name.title or None` — became the migration path. The in-band-signaling bug that sealed it (#254): the 2016 `.replace('None','')` scrub could not tell interpolated None from name text, so "Nonez Smith" rendered diff --git a/docs/design/mechanisms.md b/docs/design/mechanisms.md index 315f25e1..bc2309f2 100644 --- a/docs/design/mechanisms.md +++ b/docs/design/mechanisms.md @@ -31,6 +31,10 @@ Problem shape. A rule needs to know how words were JOINED (chained titles, parti Problem shape. A rule counts "one name word", but the input holds words that another rule has already joined into one name — and the joining structure it would read has been merged away. Contract statement. Three rules build multi-word units: a particle chain (P2), a conjunction join (P3), and a bound given-name pair (P5). A rule that counts name words counts those units, and takes each whole or not at all. How it works. group builds each join as a piece, but its own prefix chain then merges the joined piece into a longer one, so PIECES no longer carries the boundary — the units are rebuilt from the tags the vocabulary layer left (`particle`, `conjunction`, `vocab:bound-given`). The rebuild is RECURSIVE: what a conjunction or a bound word joins is the next UNIT, not the next word, and absorbing a single token instead strands a particle severed from the words it chains. Note the two joins arrive here for opposite reasons — the conjunction join was built and then swallowed, while the bound-given join was never built at all (P5 joins only at the first non-title piece), so restoring piece boundaries in group would fix the first and silently split the second. Lives in. nameparser/_pipeline/_post_rules.py (`_unit_end`, `_units`); rules.md P1 is the counting rule, P2/P3/P5 the joining ones. Reach for it when. A rule says "one name word" and the input can contain a join — enumerate the joining rules out of rules.md rather than the ones you remember. +## MARK-DONT-STRIP — record the decision, keep the fact + +Problem shape. A stage concludes that a vocabulary fact is not operative for one occurrence — a particle that joins nothing, a middle that will render as family. Contract statement. It MARKS the token with a namespaced marker rather than removing the vocabulary tag: the tag says what the word IS, the marker says what was decided about it, and consumers ask for the pair. The cost, which has to be paid in the docs rather than avoided: where a stable tag was documented with a RECIPE — `particle` combined with Role.FAMILY "for actual family particles" — the marker makes the recipe diverge from the view it reproduced, so the recipe's documentation has to say so. How it works. Stripping is tempting because every downstream consumer becomes correct for free, with no predicate to change; it pays for that by destroying the record, and asymmetrically. `particle` is the only tag 31 of the 33 never-given particles carry, so stripping it leaves nothing on the token, while every ambiguous one still carries `vocab:particle-ambiguous` (the two exceptions, `mc` and `vd`, carry `vocab:suffix` — the same dual membership P6's S2 precedence turns on) — the half where a later fork is most interesting is the half that loses its evidence, and ParsedName has no lexicon to re-derive from. Marking also leaves a POSITIVE predicate: "this token used to have a tag" is not testable, `UNJOINED_TAG in tags` is, which is what an ambiguity emitter needs (#405). And `particle` is documented stable API meaning "a word from the particle vocabulary wherever it lands", so removing it would be a breaking change as well as a lossy one. Lives in. nameparser/_types.py — FOLDED_TAG (O3's fold, P6's attachment) and UNJOINED_TAG (R2's standalone particle), both namespaced and internal, qualifying stable tags that stay put. Reach for it when. A rule decides a word is not doing the job its vocabulary implies. + ## STRUCTURE-GATES — comma shape as an explicit state Problem shape. A rule should fire only under one comma convention. Contract statement. segment decides the comma structure once (NO_COMMA, FAMILY_COMMA, SUFFIX_COMMA) and every later stage gates on that single decision rather than re-inspecting commas. Lives in. nameparser/_pipeline/_state.py (Structure), _pipeline/_segment.py (the one decider). Reach for it when. New code is about to count commas. diff --git a/docs/design/rules.md b/docs/design/rules.md index ed7e3835..de774bbb 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -703,32 +703,52 @@ R1. Rationale: a field is a way of reading the parse, not a stored R2. Rationale: callers need the surname with and without its particles — sorting wants "Vega", display wants "de la Vega". The family name splits into further views: the base (the family - without its leading particles) and the particles themselves. + without its leading particles) and the particles themselves. A + name part whose every word is particle vocabulary is a part where + none of them is doing a particle's work — nothing joins them to a + name — so they read as ordinary name words: they anchor the base + and leave the particles view. "Every word", not "standing alone": + a two-particle run has neither word alone and both are name words + there. Position decides that, not vocabulary; whether the word is + borne as a surname somewhere does not enter into it. "Dr. Juan Q. Xavier de la Vega III" → family_base="Vega" "Dr. Juan Q. Xavier de la Vega III" → family_particles="de la" - "Sean O'Connor" → family_base="O'Connor" · boundary - A family name written wholly out of particle vocabulary still - has a base where one of those words is itself borne as an - ordinary surname: that word anchors the base, and only the words - that are never anyone's name stay particles. - "Anh Do" → family_base="Do" deviates: #385 (today: family_base="") - "Juan van der" → family_base="" - history: decisions.md#R2 · interacts: R3 · implemented: nameparser/_types.py + "Anh Do" → family_base="Do" + "Juan van der" → family_base="van der" + "Juan van der" → family_particles="" + "Juan de la Vega" → family_base="Vega" · boundary + "Juan de la Vega" → family_particles="de la" + "Sean O'Connor" → family_base="O'Connor" + Accepted, and the invariant it exists to hold: a non-empty + family always has a non-empty base. A particle needs a base to + attach to, so a family that is all particles is a family whose + words are not acting as particles. + "Del Toro" family-first → family_base="Del" + Accepted: the test runs after every rule that moves a token + between parts, so O3's fold decides it too — a middle folded into + the family can leave the family all particles, or can give a + trailing particle the name word it was missing. + "Anh Van Do" middle_as_family → family_base="Van Do" + "Nguyen, Van Le" middle_as_family → family_particles="Le" + history: decisions.md#R2 · interacts: R3 · implemented: nameparser/_types.py, nameparser/_pipeline/_post_rules.py, nameparser/_facade.py R3. Rationale: initials abbreviate the person's name words; titles, suffixes, particles and nicknames are not name words. Initials take the first letter of each given, middle, and base family word; titles, suffixes, particles and nicknames - contribute nothing. + contribute nothing — except the particles of a part whose every + word is one, which are not acting as particles there (R2) and + initial like any other name word. A CONJUNCTION never initials, + so a base that is one contributes nothing even then. "Dr. Juan Q. Xavier de la Vega III" → initials="J. Q. X. V." - "Anh Do" → initials="A. D." deviates: #385 (today: initials="A.") + "Anh Do" → initials="A. D." + "Nguyen, Van Le" → initials="V. L. N." "Sean O'Connor" → initials="S. O." · boundary - Accepted: a family that is ALL particles contributes nothing, - so the initials are the given words alone — "van der" has no - borne name to anchor a base (R2), and initials of a bare - particle run would be nonsense. - "Juan van der" → initials="J." - history: decisions.md#R2 · interacts: R2 · implemented: nameparser/_render.py + A family that is ALL particles therefore contributes its words + rather than nothing: they are the base (R2), so they initial. + "Juan van der" → initials="J. v. d." + "Juan de y" → initials="J." + history: decisions.md#R2 · interacts: R2 · implemented: nameparser/_render.py, nameparser/_facade.py R4. Rationale: case repair is a display concern, applied only on request and never destructively. diff --git a/docs/release_log.rst b/docs/release_log.rst index 4ce25d17..96d4210a 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -32,6 +32,8 @@ Release Log - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family. Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) + - Fix a family name made only of particle words reporting no base on the 2.0 API, so the surname vanished from ``family_base`` and, on both APIs, from the initials: ``parse("Anh Do")`` gave family ``Do`` with ``family_base`` ``''`` and initials ``A.``, and under ``Policy(name_order=FAMILY_FIRST)`` ``"Del Toro"`` gave family ``Del`` the same way. The empty base was a 2.0 regression rather than a longstanding bug -- 1.4.0's own guard kept ``HumanName("Anh Do").last_base`` at ``Do``, and the facade has kept it right throughout; what 1.4.0 and 2.1 shared was the missing initial. A particle earns its name by joining forward to the word it modifies, so a particle standing alone in a name part is not doing a particle's work there and reads as an ordinary name word: it anchors the base, leaves the particles view, and contributes an initial. ``"Anh Do"`` is now base ``Do``, initials ``A. D.``; ``"Juan van der"`` is base ``van der``, initials ``J. v. d.``; ``"Nguyen, Van Le"`` initials ``V. L. N.`` where the middle name used to be dropped. Position decides this, not vocabulary -- whether the word is borne as a surname somewhere does not enter into it, which reverses the reasoning ``rules.md#R2`` carried before. The invariant it exists to hold: a non-empty family always has a non-empty base, because a particle needs a base to attach to. Where the particles DO join a name word nothing changes -- ``"Juan de la Vega"`` keeps base ``Vega``, particles ``de la`` and initials ``J. V.``. The parse fields themselves do not move: ``title``, ``given``, ``middle``, ``family``, ``suffix``, ``nickname`` and ``maiden`` are byte-identical over all 751 differential corpus names in all three name orders, and only the derived views and the initials change -- which is why the differential harness, which compares those seven roles, reports no diffs for this at any baseline. One consequence to know about: where the whole family is particles AND a tussenvoegsel was attached to it (#379), the two APIs order the initials differently -- ``parse("der, y van").initials()`` is ``y. d. v.`` against the facade's ``y. v. d.`` -- because ``initials()`` reads tokens in written order while the family FIELD renders folded words first. That affects 87 constructed inputs and no corpus name; it is a pre-existing ordering gap in ``initials()`` that this change makes visible, tracked at #408 (closes #385, closes #402) + - Fix a tussenvoegsel after a family comma being parsed as a middle name. Dutch and Belgian alphabetized listings move the particle behind the given name -- ``"Beethoven, Ludwig van"`` is how ``"Ludwig van Beethoven"`` is filed -- and the trailing particle run was read as a middle name rather than as part of the surname: ``"Beethoven, Ludwig van"`` gave middle ``van``, last ``Beethoven``, and ``"Berg, Jan van der"`` gave middle ``van der``. The run now attaches to the family the comma has already named and renders before it, so those read family ``van Beethoven`` and ``van der Berg`` with the given name unchanged. The derived views move with the parse, so ``family_particles`` is ``van`` and ``family_base`` is ``Beethoven`` where they were empty and ``Beethoven`` before. `#130 `_ asked for the split and got it in 1.3.0 as ``last_base``/``last_prefixes``; 2.0 renamed them ``family_base``/``family_particles``. What was wrong until now was the values they reported for this listing. Both halves of the particle vocabulary attach -- never-given ``de`` and may-be-given ``van`` alike -- because after a comma the family is already named and the particle has no other role to take. Two guards bound it. A name whose only given word is the particle keeps it, so ``"Nguyen, Van"`` still reads given ``Van``: the attachment needs a given word to spare. And where the word is BOTH particle and suffix vocabulary the attachment outranks the post-nominal reading, so ``"Berg, Jan vd"`` reads family ``vd Berg`` where 1.4.0 and 2.1 alike gave suffix ``vd`` -- a trailing abbreviation after a family comma is the tussenvoegsel far more often than the decoration it collides with, and the same shape sweeps in ``mc``, which 2.1 also read as a suffix. ``do`` is in ``SUFFIX_ACRONYMS_AMBIGUOUS`` and 2.1 already read a trailing one as a name word, so it attaches by the plain rule rather than by the override (closes #379, closes #380). Names without the comma are untouched: ``"Ludwig van Beethoven"`` already read family ``van Beethoven`` and is byte-identical. One of the 751 differential corpus names moves, ``"Vega, Juan de la"``, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike - Add ``abd`` to :data:`~nameparser.config.bound_given_names.BOUND_GIVEN_NAMES`, so the spellings that write the article as its own word join like the others do: ``"abd Allah Smith"`` was given ``abd``, middle ``Allah`` and is now given ``abd Allah``. ``abdul``, ``abdel`` and ``abdal`` were already there, and the Arabic-script ``عبد`` has covered the same word since 2.0, so only the Latin spelling was short. The word is also the postnominal ABD ("All But Dissertation") and stays in ``SUFFIX_ACRONYMS``: position tells the two readings apart, so ``"Jane Smith ABD"``, ``"Jane Smith, ABD"`` and ``"Jane Smith A.B.D."`` all still read the credential as a suffix. Making that work needed one fix beyond the wordlist. The join reserves enough words to leave a family name behind, and the reserve counted every piece except titles and suffixes -- so a word in BOTH vocabularies did not count as a name word even when the rule had already claimed it as one, and the join declined silently on the common three-word shape. The claimed piece now counts, which is what the reserve means: it asks whether enough OTHER words are left to spare. No shipped word other than ``abd`` is in both sets, and for any word that is not, the changed expression reduces to the old one -- so nothing else can move, by arithmetic rather than by sampling. The differential corpus agrees at all 751 names, but it holds no name carrying a dual-membership word, so that agreement tests the corpus rather than the change (#400) diff --git a/nameparser/_facade.py b/nameparser/_facade.py index dc486a5a..099e6a00 100644 --- a/nameparser/_facade.py +++ b/nameparser/_facade.py @@ -476,10 +476,14 @@ def _is_conjunction(self, text: str) -> bool: return _normalize(text) in self._lexicon.conjunctions def _split_last(self) -> tuple[list[str], list[str]]: - # v1 parser.py _split_last, verbatim: vocabulary lookup at ACCESS - # time (so assigned last names split too), with the all-particle - # guard (a family name is assumed not to consist entirely of - # particles, e.g. surname "Do" which also appears in PARTICLES) + # rules.md#R2: "a name part whose every word is particle + # vocabulary is a part where none of them is doing a + # particle's work" -- the all-particle guard + # below is this rule, and predates its statement: v1 assumed a + # family name does not consist entirely of particles, e.g. the + # surname "Do" which also appears in PARTICLES. v1 + # parser.py _split_last otherwise verbatim, vocabulary lookup + # at ACCESS time so assigned last names split too. words = " ".join(self.last_list).split() i = 0 while i < len(words) and self._is_particle(words[i]): @@ -521,20 +525,40 @@ def _process_initial(self, name_part: str, firstname: bool = False) -> str: if len(initials) > 0: return self.initials_separator.join(initials) # Return '' (never empty_attribute_default, which may be None) - # when a part has no initialable words, e.g. a middle name - # consisting only of prefixes ("de la"). Callers drop these - # parts entirely. + # when a part has no initialable words. group_initials below + # decides what that means: one such element among others is + # dropped; a group that yields nothing AND is wholly particles + # initials its words; and a group that yields nothing for any + # other reason -- a conjunction, or particles mixed with one -- + # is still dropped ("Vega, Santa de y" drops its middle). return "" def _initials_lists(self) -> tuple[list[str], list[str], list[str]]: """Initials for the first, middle and last name groups. Parts - that yield no initials (e.g. a prefix-only middle name like - "de la") are dropped rather than kept as empty strings. + that yield no initials are dropped rather than kept as empty + strings -- except a part that is wholly PARTICLES, whose words + initial as ordinary name words since #404, so the prefix-only + middle name "de la" is no longer an example of the dropping. """ def group_initials(names: list[str], firstname: bool = False) -> list[str]: - return [i for i in (self._process_initial(n, firstname) - for n in names if n) if i] + got = [i for i in (self._process_initial(n, firstname) + for n in names if n) if i] + words = [w for n in names if n for w in n.split()] + if got or not words or not all(self._is_particle(w) + for w in words): + return got + # rules.md#R3: "except the particles of a part whose every + # word is one, which are not acting as particles there" + # -- nothing survived + # the filter, so the whole group is particles. The + # facade's twin of the core's + # UNJOINED_TAG. NOT pinned against it: both case runners + # compare the seven role fields only, and Case carries no + # initials column, so the one covering test is + # tests/test_initials.py::test_initials_middle_name_all_prefixes. _split_last already applies the same guard to + # the base, which is why last_base was never empty here. + return [w[0] for w in words] return (group_initials(self.first_list, True), group_initials(self.middle_list), group_initials(self.last_list)) diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index 0da9c17b..4d551546 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -21,7 +21,7 @@ from nameparser._pipeline._assign import _name_positions from nameparser._pipeline._state import ParseState, Structure, WorkToken from nameparser._policy import PatronymicRule -from nameparser._types import FOLDED_TAG, Role +from nameparser._types import FOLDED_TAG, UNJOINED_TAG, Role # Ported verbatim from v1 (nameparser/config/regexes.py) -- layering # forbids the config import; keep in sync by hand. @@ -363,4 +363,28 @@ def post_rules(state: ParseState) -> ParseState: tokens[i] = dataclasses.replace( tokens[i], role=Role.FAMILY, tags=tokens[i].tags | {FOLDED_TAG}) + # rules.md#R2: "a name part whose every word is particle + # vocabulary is a part where none of them is doing a particle's + # work — nothing joins them to a name — so they read as ordinary + # name words" + # + # Last in the stage, because every rule above can still move a + # token between parts: + # P1's fold, P6's attachment and O3's fold all rewrite roles, and + # this reads the roles they settle on. + # + # Marked, not untagged: `particle` is stable API and says the word + # IS particle vocabulary wherever it lands, which stays true. + # + # All three roles for uniformity with the rule, not because all + # three are observable: no view filters tags on GIVEN (initials + # exempt that role outright), so restricting this loop to MIDDLE + # and FAMILY moves 0 of 4,506 parses. The GIVEN arm is marked so a + # future view reading the mark gets a consistent answer. + for role in (Role.GIVEN, Role.MIDDLE, Role.FAMILY): + part = _idx(tokens, role) + if part and all("particle" in tokens[i].tags for i in part): + for i in part: + tokens[i] = dataclasses.replace( + tokens[i], tags=tokens[i].tags | {UNJOINED_TAG}) return dataclasses.replace(state, tokens=tuple(tokens)) diff --git a/nameparser/_render.py b/nameparser/_render.py index f4846f59..487ef096 100644 --- a/nameparser/_render.py +++ b/nameparser/_render.py @@ -15,7 +15,8 @@ import re from nameparser._lexicon import Lexicon, _normalize -from nameparser._types import Ambiguity, ParsedName, Role, Token +from nameparser._types import (UNJOINED_TAG, Ambiguity, ParsedName, Role, + Token) _SPACES = re.compile(r"\s+") _SPACE_BEFORE_COMMA = re.compile(r"\s+,") @@ -31,7 +32,10 @@ #: str.format keys initials() accepts: the three name-bearing roles. _INITIALS_KEYS = (Role.GIVEN.value, Role.MIDDLE.value, Role.FAMILY.value) -#: Tags whose tokens contribute no initial outside the given group. +#: Tags whose tokens contribute no initial outside the given group -- +#: unless the token also carries UNJOINED_TAG, i.e. the whole part is +#: particles, in which case they are the part's only words and do +#: contribute (rules.md#R3, #404). #: Not STABLE_TAGS -- that also contains "initial", which must contribute. _SKIP_TAGS = frozenset({"particle", "conjunction"}) @@ -112,7 +116,8 @@ def initials(name: ParsedName, spec: str, delimiter: str, separator: str) -> str tokens = name.tokens_for(role) if role is not Role.GIVEN: tokens = tuple(t for t in tokens - if not (_SKIP_TAGS & t.tags)) + if not (_SKIP_TAGS & t.tags) + or UNJOINED_TAG in t.tags) values[key] = separator.join( t.text[0] + delimiter for t in tokens) return _format_spec(spec, values, "initials", _INITIALS_KEYS) diff --git a/nameparser/_types.py b/nameparser/_types.py index 75778d2d..9f3655b0 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -80,8 +80,11 @@ def __add__(self, other: object) -> NoReturn: # type: ignore[override] #: The four :attr:`Token.tags` values that are stable API. #: "particle" marks a word from the particle vocabulary ("de", "van") #: wherever it lands -- including a given-name "Van" -- so combine it -#: with Role.FAMILY (as family_particles does) to get actual family -#: particles; "conjunction" a joining word ("and", "y"); "initial" an +#: with Role.FAMILY to get particle-vocabulary family words -- but +#: NOT to reproduce `family_particles`, which since #404 also consults +#: UNJOINED_TAG and excludes a particle standing alone in its part +#: ("Anh Do" has a particle-tagged family word and no family +#: particles); "conjunction" a joining word ("and", "y"); "initial" an #: initial-shaped word in a script that HAS initials -- "J." or "А.", #: never "씨." (#320); #: "joined" a continuation of the previous token within one merged @@ -93,6 +96,17 @@ def __add__(self, other: object) -> NoReturn: # type: ignore[override] #: test only compares the frozenset), so edit both or neither. STABLE_TAGS = frozenset({"particle", "conjunction", "initial", "joined"}) +#: A name part whose every word is particle vocabulary is a part +#: where none of them is doing a particle's work -- nothing joins them +#: to a name -- so THREE views read them as ordinary name words: they +#: anchor `family_base`, drop out of `family_particles`, and +#: contribute initials (rules.md#R2). Capitalization does not consult +#: the mark and still lowercases them, which #407 tracks. +#: MARKED rather than untagged: `particle` is stable API and says the +#: word IS particle vocabulary wherever it lands, which stays true, and +#: keeping it leaves a later rule free to report the fork this decides. +UNJOINED_TAG = "vocab:unjoined-particle" + #: The one sanctioned view-reorder marker (namespaced = unstable API). #: Tokens cannot reorder (span order is validated), so a role fold that #: must render BEFORE the role's original tokens tags them with this; @@ -467,6 +481,32 @@ def _validated_field_strings(fields: dict[str, str]) -> dict[Role, str]: return {by_value[k]: v for k, v in fields.items()} +def _remarked(tokens: list[Token]) -> tuple[Token, ...]: + """UNJOINED_TAG recomputed over an edited token list. + + The mark says a particle stands ALONE in its part, which is a fact + about the part rather than the word, so an edit that re-roles + tokens invalidates it in both directions: replace()/revise() splice + a sub-parse's tokens into one field, and a particle marked alone + there can land beside a name word (stale mark) while an unmarked + one can end up alone (missing mark). Parser.revise strips + FOLDED_TAG for the same reason; this one is RECOMPUTED rather than + stripped, because absent is only correct for half the cases. + """ + out = list(tokens) + for role in (Role.GIVEN, Role.MIDDLE, Role.FAMILY): + part = [i for i, t in enumerate(out) if t.role is role] + alone = bool(part) and all("particle" in out[i].tags for i in part) + for i in part: + tags = out[i].tags + if alone and UNJOINED_TAG not in tags: + out[i] = dataclasses.replace(out[i], tags=tags | {UNJOINED_TAG}) + elif not alone and UNJOINED_TAG in tags: + out[i] = dataclasses.replace(out[i], + tags=tags - {UNJOINED_TAG}) + return tuple(out) + + @dataclass(frozen=True, slots=True) class ParsedName: """The immutable result of parsing one name string. Read the seven @@ -571,16 +611,24 @@ def __repr__(self) -> str: # -- string views (canonical order = Role declaration order) -------- def _text_for(self, *roles: Role, tag: str | None = None, - without_tag: str | None = None) -> str: + without_tag: str | None = None, + unless_tag: str | None = None) -> str: suffix_join = roles == (Role.SUFFIX,) parts: list[str] = [] folded: list[str] = [] for tok in self.tokens: if tok.role not in roles: continue - if tag is not None and tag not in tok.tags: + # A token carrying `unless_tag` is read as though it did + # not carry `tag`/`without_tag` at all -- so it is EXCLUDED + # by a `tag=` filter and INCLUDED by a `without_tag=` one, + # which is how an unjoined particle anchors the base and + # leaves the particles view. + waived = unless_tag is not None and unless_tag in tok.tags + if tag is not None and (tag not in tok.tags or waived): continue - if without_tag is not None and without_tag in tok.tags: + if without_tag is not None and without_tag in tok.tags \ + and not waived: continue # "joined" (stable tag) marks a continuation of the previous # token ("Ph." + "D."): attach with a space so the suffix @@ -634,11 +682,13 @@ def maiden(self) -> str: # base (the family without its leading particles) and the # particles themselves" def family_particles(self) -> str: - return self._text_for(Role.FAMILY, tag="particle") + return self._text_for(Role.FAMILY, tag="particle", + unless_tag=UNJOINED_TAG) @property def family_base(self) -> str: - return self._text_for(Role.FAMILY, without_tag="particle") + return self._text_for(Role.FAMILY, without_tag="particle", + unless_tag=UNJOINED_TAG) @property def surnames(self) -> str: @@ -718,7 +768,7 @@ def _with_field_tokens( amb for amb in self.ambiguities if all(t in new_tokens for t in amb.tokens) ) - return ParsedName(self.original, tuple(new_tokens), kept) + return ParsedName(self.original, _remarked(new_tokens), kept) # -- comparison ------------------------------------------------------- diff --git a/tests/test_initials.py b/tests/test_initials.py index 13c935bd..39e9b261 100644 --- a/tests/test_initials.py +++ b/tests/test_initials.py @@ -41,17 +41,23 @@ def test_initials_with_an_attached_tussenvoegsel(self) -> None: self.assertEqual(hn.initials(), "J. V.") def test_initials_middle_name_all_prefixes(self) -> None: - # "Nguyen, Van Le" parses with middle name "Le", which contains - # no initialable words (it is a particle). The part must be - # skipped entirely rather than emitting an empty initial: - # without the filter this reads "V. . N.". (The older version - # of this comment also named empty_attribute_default, which - # 2.0 removed in #255, so that half described a failure mode - # the test cannot reach.) + # "Nguyen, Van Le" parses with middle name "Le", every word of + # which is particle vocabulary. A particle standing alone in a + # name part is not doing a particle's work there, so it + # initials as an ordinary name word (rules.md#R2/#R3) rather + # than being dropped -- this read "V. N." until #404, losing + # the middle name entirely. hn = HumanName("Nguyen, Van Le") self.m(hn.middle, "Le", hn) - self.assertEqual(hn.initials_list(), ["V", "N"]) - self.assertEqual(hn.initials(), "V. N.") + self.assertEqual(hn.initials_list(), ["V", "L", "N"]) + self.assertEqual(hn.initials(), "V. L. N.") + + def test_initials_still_drop_a_particle_beside_a_name(self) -> None: + # The other half: where the part HAS a name word, the particle + # is doing particle work and contributes nothing. + hn = HumanName("Juan de la Vega") + self.m(hn.last, "de la Vega", hn) + self.assertEqual(hn.initials(), "J. V.") def test_initials_complex_name(self) -> None: hn = HumanName("Doe, John A. Kenneth, Jr.") diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 91500b8b..41e5dc6b 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -290,6 +290,36 @@ def __post_init__(self) -> None: "the particle, so it stays a given name rather than " "leaving the name with none. Vietnamese Van is exactly " "the case that guard protects"), + # A family made only of particle vocabulary. Position decides: + # nothing joins these words to a name, so they are not acting as + # particles and they anchor the base (rules.md#R2, #404). + Case("all_particle_family_anchors_its_own_base", "Anh Do", + {"given": "Anh", "family": "Do"}, + classification="parity", + ambiguities=("suffix-or-name",), + notes="the ROLES are parity -- 1.4.0 gives first 'Anh', last " + "'Do' too -- and the views are what #404 moved, which " + "this table cannot assert (no initials or base column; " + "rules.md#R2/#R3 carry those). Worth recording which " + "way: 1.4.0's own _split_last guard kept last_base " + "'Do', so the empty family_base was a 2.0 regression " + "and this restores it. The initials half was broken in " + "both: 'A.' at 1.4.0 and 2.1, 'A. D.' now"), + Case("all_particle_family_multi_word", "Juan van der", + {"given": "Juan", "family": "van der"}, + classification="parity", + notes="roles are parity again; R3's Accepted block used to " + "pin the views the other way, reasoning that 'van der' " + "has no borne name to anchor a base. Position trumps " + "that -- neither word joins anything here, so both are " + "name words. 1.4.0 also gave last_base 'van der'; the " + "core's '' was the 2.0 regression"), + Case("particle_beside_a_name_still_a_particle", "Juan de la Vega", + {"given": "Juan", "family": "de la Vega"}, + classification="parity", + notes="the boundary the row above needs: here the particles " + "DO join a name word, so they stay particles -- base " + "'Vega', particles 'de la', initials 'J. V.'"), Case("suffix_word_title_ambiguous_particle", "Jr. Van Johnson", {"title": "Jr.", "given": "Van", "family": "Johnson"}, classification="fix(#367)", diff --git a/tests/v2/test_cases.py b/tests/v2/test_cases.py index 731c63de..82f61153 100644 --- a/tests/v2/test_cases.py +++ b/tests/v2/test_cases.py @@ -2,10 +2,14 @@ runner (migration plan) consumes the same CASES.""" import pytest -from nameparser import Parser, Role, locales, parser_for +from nameparser import Parser, Policy, Role, locales, parser_for +from nameparser._policy import FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST from .cases import CASES, Case +_ORDER_NAMES = {FAMILY_FIRST: "family-first", + FAMILY_FIRST_GIVEN_LAST: "family-first-given-last"} + _FIELDS = tuple(r.value for r in Role) # declaration order is canonical @@ -24,3 +28,45 @@ def test_case(case: Case) -> None: kinds = sorted(a.kind.value for a in pn.ambiguities) assert kinds == sorted(case.ambiguities), \ f"{case.text!r} ({case.classification})" + + +#: The orders the invariant below is checked under. A case row carries +#: its own policy where it needs one; these are applied to every row +#: that does NOT, because the shapes this invariant is about are mostly +#: reached under a family-first order and the table has almost no rows +#: that declare one. +_INVARIANT_ORDERS = (None, FAMILY_FIRST, FAMILY_FIRST_GIVEN_LAST) + + +@pytest.mark.parametrize("order", _INVARIANT_ORDERS, + ids=lambda o: _ORDER_NAMES.get(o, "as-declared")) +@pytest.mark.parametrize("case", CASES, ids=lambda c: c.id) +def test_the_family_partitions_into_particles_and_base( + case: Case, order: tuple[Role, Role, Role] | None) -> None: + """rules.md#R2's invariant: a particle needs a base to attach to, + so a family made only of particles is a family whose words are not + acting as particles -- and therefore a non-empty family always has + a non-empty base. + + Asserted as a PARTITION, which is the stronger form: the family's + words are exactly the particles' words plus the base's words. That + catches over-marking as well as under-marking, where "the base is + non-empty" catches only the second. Word multisets rather than + strings, because the family renders in written order while the two + views render particles first ("Vega, de la" is family 'Vega de la', + particles 'de la', base 'Vega'). + """ + if case.locale is not None or (order is not None and case.policy): + pytest.skip("row carries its own policy or locale") + parser = (_parser_for_case(case) if order is None + else Parser(policy=Policy(name_order=order))) + pn = parser.parse(case.text) + if not pn.family: + return + assert pn.family_base, ( + f"{case.text!r}: family={pn.family!r} but family_base is " + f"empty (particles={pn.family_particles!r})") + assert sorted(pn.family.split()) == sorted( + (pn.family_particles + " " + pn.family_base).split()), ( + f"{case.text!r}: family={pn.family!r} is not partitioned by " + f"particles={pn.family_particles!r} + base={pn.family_base!r}") diff --git a/tests/v2/test_parser.py b/tests/v2/test_parser.py index adce383b..528e0478 100644 --- a/tests/v2/test_parser.py +++ b/tests/v2/test_parser.py @@ -669,6 +669,37 @@ def test_revise_strips_the_fold_marker() -> None: assert r.family == "Gabriel García Márquez" +def test_revise_clears_a_stale_unjoined_mark() -> None: + # UNJOINED_TAG says a particle stands alone in its PART, so an edit + # that re-roles tokens invalidates it -- the harvest splices a + # sub-parse's tokens into one field, and a particle marked alone + # there can land beside a name word. Recomputed rather than + # stripped (the fold marker above is stripped, which is only right + # for one direction). Without this, an identity revise drifted: + # base 'Toro' became 'del Toro' and initials 'T.' became 'd. T.' + p = Parser() + r = p.parse("Mr. do Jr. del Toro") + assert (r.family, r.family_base, r.family_particles) == ( + "del Toro", "Toro", "del") + again = p.revise(r, family=r.family) + assert (again.family, again.family_base, again.family_particles) == ( + "del Toro", "Toro", "del") + assert again.initials() == r.initials() + + +def test_revise_sets_a_missing_unjoined_mark() -> None: + # The other direction, and the one that made rules.md#R2's + # invariant false through this path: "Do" alone parses as a TITLE, + # so the sub-parse marks nothing, and the harvest then re-roles a + # bare particle into FAMILY. The recompute marks it there, so a + # non-empty family still has a non-empty base. + p = Parser() + revised = p.revise(p.parse("Juan de la Vega"), family="Do") + assert revised.family == "Do" + assert revised.family_base == "Do" + assert revised.family_particles == "" + + def test_revise_sub_parse_structural_behavior() -> None: # the docstring's three structural promises, pinned: delimiters # never become tokens, marker words are consumed as in parsing,