diff --git a/docs/design/decisions.md b/docs/design/decisions.md index b35ed312..4b7596fd 100644 --- a/docs/design/decisions.md +++ b/docs/design/decisions.md @@ -80,6 +80,11 @@ Open: [#360](https://github.com/derek73/python-nameparser/issues/360) which part ### P6 — the trailing orphan particle +- 2026-08-18 #379/#380 IMPLEMENTED — the rule shipped as stated, both `deviates:` markers out. Keyed on the token's VOCABULARY rather than its assigned role, which is what gives the S2 precedence its effect: assign reads a trailing `vd`/`mc`/`do` as a post-nominal, and the attachment overrides that after a comma. The words-to-spare guard is a PIECE test, not a count — every trailing piece that is wholly particles attaches, and the run must leave a GIVEN word ahead of it, so "Nguyen, Van" keeps its only given word. Not "a name word": see the defect below, where that wider reading passed on family text P1 had just produced. MIDDLE was in the test briefly and came out — 0 hits over 740,552 instrumented guard sites, and structurally unreachable, since the only rule that leaves a MIDDLE with no GIVEN ahead of it is gated on state.order, which assign never records on the family-comma path P6 runs on. + Measured: ONE of 751 corpus names moves ("Vega, Juan de la"), at all three baselines, and it needed a new ledger rule in each. The LEDGER rule's regex — not the parse rule, which has no regex and fires on the whole particle vocabulary through the tag — is a PARTIAL copy of PARTICLES, deliberately: the words that actually end a Dutch, Iberian or German listing, not all 70. The ledger guards forced that narrowing: the first draft used the neighbours' bare comma regex, which claimed 215 names and reached two protected exclusions. A member `des` had to come out as well, and for a reason worth keeping straight: it is not in PARTICLES at all, so it could copy nothing, and the text it reached was the opening of "Designated Judge David A. Ezra" rather than a person's name. `_reaches_non_vocabulary("des", PARTICLES)` prints the FRAGMENT it matched, `['Des']`, not the name containing it — which is how the retracted wording arose in the first place. + Two defects found in review, both from the rule's neighbours rather than from its own logic. P1's fold runs EARLIER in the same function and retags all of segment 1 to FAMILY, so a words-to-spare test written as "some name role remains" passed on family text P1 had just produced — and the attachment then hoisted the particle in front of a base it never preceded ("Smith, de Mesnil van" → 'van Smith de Mesnil'). 1.4 gives 'de Mesnil van Smith' for that input and 2.1 gives 'Smith de Mesnil van', so neither the defect nor the fix restores 1.4 — what the fix restores is 2.1, the pre-comma-family reading having moved back in 2.0. The test has to name GIVEN, which is what the rule says. And a post-nominal sits BEHIND the tussenvoegsel in this listing, so the backward walk stopped on it and the attachment never fired: "Berg, Jan van Jr." and "Berg, Jan van, Jr." parsed two ways on nothing but a comma. The walk now steps over a trailing piece holding no name, unless it is itself particle vocabulary — `vd` arrives suffix-roled and IS the run. + Fallout worth its own record: "Vega, Juan de la" was the input tests/test_initials.py used to build an ALL-PARTICLE middle name, and this rule takes that shape away from THAT input. It does not remove the shape — "Vega, Santa de la", "Berg, Van der" and "Nguyen, Van Le" all still produce one in the default order, because the trailing run swallows the given word too and the attachment then declines. The test is repointed at the last of those rather than deleted. Whether such a middle should exist at all is #402 — for a never-given particle it is #365's defect, and for an ambiguous one the parser reports it as a middle name and then declines to initial it, which is a miscount either way. + - 2026-08-16 (order-precedence keystone; #379, #380, #365) — a particle ending the name has nothing to link forward to, and no particle is a name by itself, so it attaches to the family name standing beside it and renders BEFORE it. The distinction from a chain is what makes this a rule rather than an exception: a chained group has a head word and can be positioned; an orphan has no head, so position has nothing to work with. - Scope: the COMMA form only, deliberately. "Jong, Anke de" is unambiguous — the comma has already named the family. Without the comma the written shape is not settled: "Jong Anke de" may be a misformatted listing (arguably a missing comma under a declared family-first order) and "Jong de" may be a given name beside a particle. Those keep their positional reading and are not tracked as deviations. - The words-to-spare guard is load-bearing, not incidental. #379's own subject is "van", which is in the AMBIGUOUS half — so a rule keyed to never-given particles alone would not fix the issue it was filed for, while a rule with no guard breaks Vietnamese diff --git a/docs/design/rules.md b/docs/design/rules.md index 10732638..ed7e3835 100644 --- a/docs/design/rules.md +++ b/docs/design/rules.md @@ -245,24 +245,54 @@ P6. Rationale: a particle ending the name has nothing to link forward to, so it is not doing a particle's work there. A never-given particle in that position cannot be a name at all and must belong to the family written beside it; an ambiguous - particle could genuinely be the name (Vietnamese "Van"), which - is what the words-to-spare test below is for, not an - afterthought to it. Dutch and Flemish names are listed exactly + particle could genuinely be the name (Vietnamese "Van"), and + after a comma there is no signal that separates the two + readings. Dutch and Flemish names are listed exactly this way ("Beethoven, Ludwig van"), the tussenvoegsel trailing the given name but belonging to the surname. Where a family comma has already named the family, a particle ending the name attaches to that family name and is written before it — provided at least one given word remains, so that a name whose only given word is the particle keeps it (the - words-to-spare test S2 applies to ambiguous suffixes). Where + words-to-spare test S2 applies to ambiguous suffixes). A + post-nominal is written BEHIND the particle in this listing, so + it does not end the name for this purpose: the run is found by + looking past trailing words that hold no name — unless such a + word is itself particle vocabulary, which makes it part of the + run rather than something to look past. Where the word is BOTH a particle and suffix vocabulary, this attachment outranks the suffix reading (S2): a trailing abbreviation after a family comma is the tussenvoegsel far more often than the decoration it collides with. - "Jong, Anke de" → family="de Jong" deviates: #379 (today: family="Jong") - "Beethoven, Ludwig van" → family="van Beethoven" deviates: #379 (today: family="Beethoven") - "Berg, Jan vd" → family="vd Berg" deviates: #380 (today: family="Berg") + "Jong, Anke de" → family="de Jong" + "Beethoven, Ludwig van" → family="van Beethoven" + "Berg, Jan vd" → family="vd Berg" + "Berg, Jan van der" → family="van der Berg" + "Vega, Juan de la" → family_particles="de la" + "Beethoven, Ludwig van" → family_base="Beethoven" + "Beethoven, Ludwig van" → family_particles="van" "Nguyen, Van" → given="Van" · boundary + Accepted: an ambiguous particle attaches on the same terms as a + never-given one, so a Vietnamese name written in this listing + loses its given name — but only in the UNACCENTED + transliteration. Vân carries a diacritic and is not particle + vocabulary, so the correctly spelled name never reaches this + rule. It is the ASCII spelling that collides, and there the two + traditions write the same string. + "Nguyen, Thi Van" → family="Van Nguyen" + "Nguyễn, Thị Vân" → family="Nguyễn" + Accepted: no ambiguity is reported for that collision, which A1 + would call for. The fork is decided here, and assign's emitter is + scoped to the no-comma shapes on the reasoning that a comma has + fixed the family — true of the family, not of the particle behind + it. Tracked at #405. + Accepted: the colliding spelling has a format that reads + correctly, and it is ONE order, not both: FAMILY_FIRST still + sends the given name to the middle, and only + FAMILY_FIRST_GIVEN_LAST recovers it. + "Nguyen Thi Van" family-first-given-last → family="Nguyen" + "Nguyen Thi Van" family-first-given-last → given="Van" + "Nguyen Thi Van" family-first → middle="Van" Accepted: without a family comma the name's written shape is not settled — "Jong Anke de" may be a misformatted listing, and a bare "Jong de" may be a given name beside a particle — so the @@ -272,9 +302,11 @@ P6. Rationale: a particle ending the name has nothing to link Accepted: the precedence over S2 is stated for the shape, so it sweeps in every word that is both particle and suffix vocabulary — today vd, do and mc. Only vd's reading was - weighed; the other two inherit it, which is the shape's cost - and is tracked with the other contested memberships. - history: decisions.md#P6 · interacts: C1, P1, S2 · tracked: #379, #380 + weighed; mc inherits it, which is the shape's cost and is + tracked with the other contested memberships. `do` sits in the + AMBIGUOUS acronym half and was already read as a name word + there, so the precedence decides nothing for it. + history: decisions.md#P6 · interacts: C1, P1, S2 · implemented: nameparser/_pipeline/_post_rules.py ## Suffixes: generational & credentials (S) @@ -660,8 +692,9 @@ R1. Rationale: a field is a way of reading the parse, not a stored string. Every field is a view computed from the parsed words at read time, joining its words in written order — except folded family - words (O3), which render before the rest of the family wherever - they stood in the string. + words — O3's fold and, since #379, P6's attached tussenvoegsel — + which render before the rest of the family wherever they stood in + the string. "Dr. Juan Q. Xavier de la Vega III" → family="de la Vega" "Hassan, Mohamad Ahmad Ali" middle_as_family → family="Ahmad Ali Hassan" "Hassan, Mohamad Ahmad Ali" → family="Hassan" · boundary diff --git a/docs/release_log.rst b/docs/release_log.rst index 805ef280..4ce25d17 100644 --- a/docs/release_log.rst +++ b/docs/release_log.rst @@ -32,6 +32,8 @@ Release Log - Fix a name opening with a particle that is *never* a given name being split at the particle under a family-first name order -- ``Policy(name_order=FAMILY_FIRST)`` and ``Policy(name_order=FAMILY_FIRST_GIVEN_LAST)`` alike, and identically: ``"de Mesnil"`` read as family ``de``, given ``Mesnil``, and ``"de la Vega"`` as family ``de``, given ``la Vega``. Each is now the whole surname, as it has always been in the default order. The rule enforcing it asked for the particle by the ``GIVEN`` role, which under a family-first order belongs to the token *after* the particle, so the test read the wrong word and declined. It now also asks by position -- the piece that opens the name -- so both shapes of the same rule are caught: where such a particle stands alone as a piece, either opening the name or in the given position, the name is left with no given name at all, the given and the middles folding into the family. Standing *alone* is the whole of it, and the rule claims nothing wider: ``"Juan de la Vega"`` under ``FAMILY_FIRST`` still reports given ``de la Vega``, because there the particle chained onto the words after it rather than standing alone, and a bare ``"de"`` with nothing to fold into is still reported as the given name. The decision behind the fix: a word that can never be a given name leaves ``name_order`` nothing to decide, so declaring family-first is not a reason to make ``de`` a surname on its own. A leading particle that *may* be a given name is genuinely order-dependent and is untouched -- ``"van Gogh"`` still reads as family ``van``, given ``Gogh`` under both family-first orders. This is also what gives ``Lexicon.particles_ambiguous`` an effect outside the default order: taking a word out of it now changes the parsed fields under a family-first order, where before it moved only the ambiguity report. Seven of the 751 differential corpus names move, the same seven under each family-first order; default-order output is byte-identical over all 751, at the 1.4.0, 2.0.0 and 2.1.0 differential baselines alike (closes #359) + - Fix a tussenvoegsel after a family comma being parsed as a middle name. Dutch and Belgian alphabetized listings move the particle behind the given name -- ``"Beethoven, Ludwig van"`` is how ``"Ludwig van Beethoven"`` is filed -- and the trailing particle run was read as a middle name rather than as part of the surname: ``"Beethoven, Ludwig van"`` gave middle ``van``, last ``Beethoven``, and ``"Berg, Jan van der"`` gave middle ``van der``. The run now attaches to the family the comma has already named and renders before it, so those read family ``van Beethoven`` and ``van der Berg`` with the given name unchanged. The derived views move with the parse, so ``family_particles`` is ``van`` and ``family_base`` is ``Beethoven`` where they were empty and ``Beethoven`` before. `#130 `_ asked for the split and got it in 1.3.0 as ``last_base``/``last_prefixes``; 2.0 renamed them ``family_base``/``family_particles``. What was wrong until now was the values they reported for this listing. Both halves of the particle vocabulary attach -- never-given ``de`` and may-be-given ``van`` alike -- because after a comma the family is already named and the particle has no other role to take. Two guards bound it. A name whose only given word is the particle keeps it, so ``"Nguyen, Van"`` still reads given ``Van``: the attachment needs a given word to spare. And where the word is BOTH particle and suffix vocabulary the attachment outranks the post-nominal reading, so ``"Berg, Jan vd"`` reads family ``vd Berg`` where 1.4.0 and 2.1 alike gave suffix ``vd`` -- a trailing abbreviation after a family comma is the tussenvoegsel far more often than the decoration it collides with, and the same shape sweeps in ``mc``, which 2.1 also read as a suffix. ``do`` is in ``SUFFIX_ACRONYMS_AMBIGUOUS`` and 2.1 already read a trailing one as a name word, so it attaches by the plain rule rather than by the override (closes #379, closes #380). Names without the comma are untouched: ``"Ludwig van Beethoven"`` already read family ``van Beethoven`` and is byte-identical. One of the 751 differential corpus names moves, ``"Vega, Juan de la"``, at the 1.4.0, 2.0.0 and 2.1.0 baselines alike + - Add ``abd`` to :data:`~nameparser.config.bound_given_names.BOUND_GIVEN_NAMES`, so the spellings that write the article as its own word join like the others do: ``"abd Allah Smith"`` was given ``abd``, middle ``Allah`` and is now given ``abd Allah``. ``abdul``, ``abdel`` and ``abdal`` were already there, and the Arabic-script ``عبد`` has covered the same word since 2.0, so only the Latin spelling was short. The word is also the postnominal ABD ("All But Dissertation") and stays in ``SUFFIX_ACRONYMS``: position tells the two readings apart, so ``"Jane Smith ABD"``, ``"Jane Smith, ABD"`` and ``"Jane Smith A.B.D."`` all still read the credential as a suffix. Making that work needed one fix beyond the wordlist. The join reserves enough words to leave a family name behind, and the reserve counted every piece except titles and suffixes -- so a word in BOTH vocabularies did not count as a name word even when the rule had already claimed it as one, and the join declined silently on the common three-word shape. The claimed piece now counts, which is what the reserve means: it asks whether enough OTHER words are left to spare. No shipped word other than ``abd`` is in both sets, and for any word that is not, the changed expression reduces to the old one -- so nothing else can move, by arithmetic rather than by sampling. The differential corpus agrees at all 751 names, but it holds no name carrying a dual-membership word, so that agreement tests the corpus rather than the change (#400) - Change how far a leading never-given particle takes the surname when a family-first ``name_order`` is declared. ``Policy(name_order=FAMILY_FIRST)`` read ``"de Mesnil Juan"`` as family ``de Mesnil Juan`` -- the whole name -- and now reads family ``de Mesnil``, given ``Juan``. Declaring a family-first order asserts that what follows the family is not more surname, and where the surname run ends is exactly that question, so the declaration settles it. The default order is unchanged, deliberately: with no order declared nothing marks where the surname ends, and a particle followed by several words really can be all surname -- ``von Bergen Wessels`` is one such name. Nothing in ``"de Mesnil Juan"`` distinguishes it from that reading except a declared order or a comma. A caller who means family ``de la Vega`` plus given ``Juan`` in the default order writes the comma, which already parses that way. The run takes one name WORD rather than one token: a conjunction-joined run and a bound given-name pair each count once, so the stop cannot land inside one -- ``"de la Vega y Santos Juan"`` reads family ``de la Vega y Santos``, and ``"ibn Awf abdul Rahman"`` reads given ``abdul Rahman``. Where two or more words are left over the two family-first orders differ from each other for the first time: ``"de la Cruz Juan Carlos"`` reads given ``Juan``, middle ``Carlos`` under ``FAMILY_FIRST`` and middle ``Juan``, given ``Carlos`` under ``FAMILY_FIRST_GIVEN_LAST``. An ambiguous leading particle is untouched in every order -- ``"van Gogh Jan Pieter"`` still reads family ``van`` under both family-first orders -- and so is a family comma, where the comma has already fixed the surname (``"Smith, de Mesnil"`` keeps family ``Smith de Mesnil``). One of the 751 differential corpus names moves, ``"de Mesnil Garcia"`` to family ``de Mesnil``, given ``Garcia``, under each family-first order; default-order output is byte-identical over all 751. This reverses the answer #364 was closed on, and the reasoning is recorded at ``docs/design/decisions.md#P1`` (closes #395) diff --git a/nameparser/_pipeline/_post_rules.py b/nameparser/_pipeline/_post_rules.py index f294df32..0da9c17b 100644 --- a/nameparser/_pipeline/_post_rules.py +++ b/nameparser/_pipeline/_post_rules.py @@ -277,6 +277,82 @@ def post_rules(state: ParseState) -> ParseState: _retag(tokens, m2, Role.MIDDLE) _retag(tokens, f, Role.MIDDLE) _retag(tokens, g, Role.FAMILY) + # rules.md#P6: "a particle ending the name attaches to that family + # name and is written before it" -- where a family comma has + # already named the family, and provided at least one given word + # remains (history: decisions.md#P6). The Dutch alphabetized + # listing: + # "Beethoven, Ludwig van" is how "Ludwig van Beethoven" is filed. + # + # Keyed on the token's VOCABULARY, not its assigned role, which is + # what gives the attachment its stated precedence over S2. `vd`, + # `mc` and `do` are the three words in both vocabularies; assign + # reads a trailing `vd` or `mc` as a post-nominal, so those two + # need the override. `do` is in the AMBIGUOUS acronym half, which + # already leaves it a name word, so it attaches by the plain rule. + # After a family comma the tussenvoegsel is the commoner reading. + # + # The words-to-spare guard is a piece test, not a count: every + # trailing piece that is wholly particles attaches, and the run + # must leave a GIVEN word ahead of it, so "Nguyen, Van" keeps its + # only given word rather than being left with none. Only the + # DEGENERATE Vietnamese listing is protected by that -- "Nguyen, + # Thi Van" has a given word to spare, so `Van` attaches and the + # given name is lost. rules.md#P6 records why that is accepted. + # + # mechanisms.md#FOLDED_TAG does the rest: tokens never move, so + # the family view reads the tag and renders these before the base. + if state.structure is Structure.FAMILY_COMMA and len(state.pieces) > 1: + seg = state.pieces[1] + # A post-nominal sits BEHIND the tussenvoegsel in this listing + # ("Berg, Jan van Jr."), so the run is found by walking past a + # trailing piece that holds no name -- but only one that is not + # itself particle vocabulary, since `vd` arrives suffix-roled + # and IS the run. Without this the same name parsed two ways on + # whether a comma preceded the credential. + end = len(seg) + while (end + and not any(tokens[i].role in _NAME_ROLES + for i in seg[end - 1]) + and not all("particle" in tokens[i].tags + for i in seg[end - 1])): + end -= 1 + k = end + while k and all("particle" in tokens[i].tags for i in seg[k - 1]): + k -= 1 + # GIVEN alone, which is what P6 says ("provided at least one + # given word remains"). Not `_NAME_ROLES`: P1's fold runs + # earlier in this function and retags all of segment 1 to + # FAMILY, so a test for "some name word remains" passes on + # family text P1 just produced, and the rule then hoists the + # particle in front of a base it never preceded ("Smith, de + # Mesnil van" -> 'van Smith de Mesnil'). MIDDLE was in this + # test until review found no input where it decides anything + # -- 0 hits over 740,552 instrumented guard sites. The reason + # is structural: the only rule that can leave a MIDDLE with no + # GIVEN ahead of it in segment 1 is P1's family-first + # redistribution, which is gated on `state.order`, and assign + # never records an order on the FAMILY_COMMA path (the comma + # has already fixed the family). P6 runs only on that path, + # so the branch cannot be reached from here. + if k and any(tokens[i].role is Role.GIVEN + for piece in seg[:k] for i in piece): + # A range, though only ever one piece today: grouping's + # prefix chain makes a non-leading particle absorb what + # follows, so a trailing run splits into several pieces + # only where nothing ahead of it holds a given role -- + # the run opening the segment ("Berg, de van"), or only + # titles ahead of it ("Berg, Sir de la", 8% of them). The + # guard then declines either way. Measured over 95,180 + # generated multi-piece runs: `end - k` is never above 1 + # where the guard passes. Written as a range because the + # guard, not this loop, is what bounds it. + for piece in seg[k:end]: + for i in piece: + tokens[i] = dataclasses.replace( + tokens[i], role=Role.FAMILY, + tags=tokens[i].tags | {FOLDED_TAG}) + # rules.md#O3: "every middle word joins the family name and is # rendered before it" (v1 handle_middle_name_as_last). v1 # PREPENDED middle_list to last_list; mechanisms.md#FOLDED_TAG: diff --git a/nameparser/_types.py b/nameparser/_types.py index 7661613a..75778d2d 100644 --- a/nameparser/_types.py +++ b/nameparser/_types.py @@ -610,8 +610,8 @@ def middle(self) -> str: @property # rules.md#R1: "every field is a view computed from the parsed # words at read time, joining its words in written order — except - # folded family words (O3), which render before the rest of the - # family" + # folded family words" (O3's fold and P6's tussenvoegsel, which + # render before the rest of the family) def family(self) -> str: return self._text_for(Role.FAMILY) diff --git a/tests/test_initials.py b/tests/test_initials.py index c6ab90a5..13c935bd 100644 --- a/tests/test_initials.py +++ b/tests/test_initials.py @@ -28,16 +28,31 @@ def test_initials_all_empty_renders_empty_string(self) -> None: hn = HumanName("", constants=Constants()) self.assertEqual(hn.initials(), "") - def test_initials_middle_name_all_prefixes(self) -> None: - # "Vega, Juan de la" parses with middle name "de la", which contains - # no initialable words (both are prefixes). The part must be skipped - # entirely — not emit an empty initial ("J. . V.") and not crash when - # empty_attribute_default is None. + def test_initials_with_an_attached_tussenvoegsel(self) -> None: + # "Vega, Juan de la" used to park "de la" in the middle name; #379 + # attaches it to the family instead, so the initials come from + # 'Juan' and the family BASE. The all-prefix MIDDLE this test + # was written for now comes from "Nguyen, Van Le" below -- + # whether such a middle should exist at all is #402. hn = HumanName("Vega, Juan de la") - self.m(hn.middle, "de la", hn) + self.m(hn.middle, "", hn) + self.m(hn.last, "de la Vega", hn) self.assertEqual(hn.initials_list(), ["J", "V"]) self.assertEqual(hn.initials(), "J. V.") + def test_initials_middle_name_all_prefixes(self) -> None: + # "Nguyen, Van Le" parses with middle name "Le", which contains + # no initialable words (it is a particle). The part must be + # skipped entirely rather than emitting an empty initial: + # without the filter this reads "V. . N.". (The older version + # of this comment also named empty_attribute_default, which + # 2.0 removed in #255, so that half described a failure mode + # the test cannot reach.) + hn = HumanName("Nguyen, Van Le") + self.m(hn.middle, "Le", hn) + self.assertEqual(hn.initials_list(), ["V", "N"]) + self.assertEqual(hn.initials(), "V. N.") + def test_initials_complex_name(self) -> None: hn = HumanName("Doe, John A. Kenneth, Jr.") self.m(hn.initials(), "J. A. K. D.", hn) diff --git a/tests/v2/cases.py b/tests/v2/cases.py index 2dd749b5..91500b8b 100644 --- a/tests/v2/cases.py +++ b/tests/v2/cases.py @@ -213,6 +213,83 @@ def __post_init__(self) -> None: "the whole suite passed with name_order discarded from " "it; on this branch the same mutation fails three " "tests, this row among them"), + # The Dutch alphabetized listing: "Beethoven, Ludwig van" is how + # "Ludwig van Beethoven" is filed, the tussenvoegsel moved behind + # the given name but belonging to the surname (#379). + Case("tussenvoegsel_after_family_comma", "Beethoven, Ludwig van", + {"given": "Ludwig", "family": "van Beethoven"}, + classification="fix(#379)", + notes="1.4.0 gives middle 'van', last 'Beethoven'. The " + "particle attaches to the family the comma already " + "named and renders before it, so the derived views " + "move with it -- family_particles 'van', family_base " + "'Beethoven', which is what #130 asked for"), + Case("tussenvoegsel_multiword", "Berg, Jan van der", + {"given": "Jan", "family": "van der Berg"}, + classification="fix(#379)", + notes="the whole run attaches, not just its last word"), + Case("tussenvoegsel_outranks_the_suffix_reading", "Berg, Jan vd", + {"given": "Jan", "family": "vd Berg"}, + classification="fix(#380)", + notes="'vd' is particle AND suffix vocabulary, and assign " + "read the trailing one as a post-nominal (1.4.0 and " + "2.1 alike gave suffix 'vd'). After a family comma the " + "tussenvoegsel abbreviation is far more often the " + "reading meant; P6 states that precedence over S2"), + Case("tussenvoegsel_behind_a_post_nominal", "Berg, Jan van Jr.", + {"given": "Jan", "family": "van Berg", "suffix": "Jr."}, + classification="fix(#379)", + notes="the credential sits BEHIND the tussenvoegsel in this " + "listing, so the run is found by walking past it. " + "Without that walk the same name parsed two ways on " + "whether a comma preceded the credential -- " + "'Berg, Jan van, Jr.' attached and this one did not"), + Case("tussenvoegsel_declines_with_no_given_word_left", + "Smith, de Mesnil van", + {"family": "Smith de Mesnil van"}, + classification="fix(comma-precomma-family)", + notes="P1's fold has already made all of segment 1 the " + "family, so no GIVEN word remains and the " + "attachment declines. Testing for any NAME role here " + "instead would pass on family text P1 just produced, " + "and hoist 'van' in front of a base it never preceded " + "('van Smith de Mesnil'). NOT parity: 1.4.0 leaves no " + "given name either, but renders last 'de Mesnil van " + "Smith' -- it treats only the leading 'de' as a " + "last-prefix and leaves 'van' inside the base. The " + "structure agrees, the string does not"), + Case("tussenvoegsel_behind_a_comma_post_nominal", + "Berg, Jan van, Jr.", + {"given": "Jan", "family": "van Berg", "suffix": "Jr."}, + classification="fix(#379)", + notes="the credential in its own comma segment, which is the " + "spelling the no-comma row is defined against -- both " + "must read the same, and gating the rule on a two" + "-segment name silently reverts this one"), + Case("tussenvoegsel_behind_a_title", "Berg, Dr. Jan van", + {"title": "Dr.", "given": "Jan", "family": "van Berg"}, + classification="fix(#379)", + notes="the words-to-spare test asks whether ANY word ahead " + "of the run holds a given role, not whether all of " + "them do: the title does not, and the rule must still " + "fire. decisions.md#P6 calls that guard load-bearing"), + Case("tussenvoegsel_takes_the_vietnamese_reading", "Nguyen, Thi Van", + {"given": "Thi", "family": "Van Nguyen"}, + classification="fix(#379)", + notes="the accepted cost, pinned so it cannot move without " + "someone deciding to move it: Nguyen Thi Van is " + "family-middle-given, so the given name Van is lost " + "here. The listing is identical to the Dutch one and " + "nothing separates them. The comma-LESS family-first " + "spelling reads it correctly, which is what makes the " + "loss acceptable -- see rules.md#P6"), + Case("tussenvoegsel_needs_a_given_word_to_spare", "Nguyen, Van", + {"given": "Van", "family": "Nguyen"}, + classification="parity", + notes="the words-to-spare boundary: the only given word IS " + "the particle, so it stays a given name rather than " + "leaving the name with none. Vietnamese Van is exactly " + "the case that guard protects"), Case("suffix_word_title_ambiguous_particle", "Jr. Van Johnson", {"title": "Jr.", "given": "Van", "family": "Johnson"}, classification="fix(#367)", diff --git a/tests/v2/test_ledger_guards.py b/tests/v2/test_ledger_guards.py index dc8d7d2e..e610e803 100644 --- a/tests/v2/test_ledger_guards.py +++ b/tests/v2/test_ledger_guards.py @@ -45,6 +45,7 @@ class declares, which members an alternation offers. Those are exact # written to forbid exactly that. from nameparser._lexicon import _normalize from nameparser.config.maiden_markers import MAIDEN_MARKERS +from nameparser.config.particles import PARTICLES from nameparser.config.suffixes import ( GLUED_HONORIFICS, SUFFIX_ACRONYMS_AMBIGUOUS, SUFFIX_WORDS) @@ -679,6 +680,16 @@ class _LatinCopy(NamedTuple): "ambiguous-surname-acronym": _LatinCopy( vocabulary=SUFFIX_ACRONYMS_AMBIGUOUS, covers=frozenset({"do", "ma"})), + # The tussenvoegsel rule copies PARTICLES, and copies it PARTLY on + # purpose: it needs the words that actually end a Dutch, Iberian or + # German listing, not all 70. A wider copy would claim comma names + # whose trailing word is a particle nobody files a name with, which + # is the reach this roster exists to keep honest. + "fix(#379)": _LatinCopy( + vocabulary=PARTICLES, + covers=frozenset({"de", "del", "den", "der", "di", "do", "dos", + "du", "la", "le", "los", "mc", "van", "vd", + "von", "zu"})), } #: Alternations that copy no vocabulary, so discovery must not demand a @@ -1016,6 +1027,8 @@ def _claim(rule: dict) -> _Claim: _Claim(4, ('family', 'maiden', 'middle'), "b31dc2e2bbc4"), "fix(cjk-maiden-marker) maiden marker consumed, compounding with the CJK order flip": _Claim(3, ('family', 'given', 'maiden', 'middle'), "cf5c9d671c14"), + "fix(#379) a tussenvoegsel after a family comma attaches to the family": + _Claim(2, ('family', 'middle'), "f3a43bebfb91"), "fix(comma-family) lone post-comma piece routes to suffix/title, not first": _Claim(215, ('given', 'suffix', 'title'), "f16a0e79cba3"), "fix(comma-precomma-family) pre-comma run reads as family, not given": @@ -1050,6 +1063,8 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), }, "expected_since_2.0.0.toml": { + "fix(#379) a tussenvoegsel after a family comma attaches to the family": + _Claim(2, ('family', 'middle'), "f3a43bebfb91"), "fix(#271/#272/#298) native-script CJK: family-first order, hangul segmentation, the kana license and the dots": _Claim(97, ('_ambiguities', 'family', 'given', 'middle'), "66e71d60a075"), "fix(#308/#312/#319/#320) glued CJK honorific peeled off the name into suffix": @@ -1066,6 +1081,8 @@ def _claim(rule: dict) -> _Claim: _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), }, "expected_since_2.1.0.toml": { + "fix(#379) a tussenvoegsel after a family comma attaches to the family": + _Claim(2, ('family', 'middle'), "f3a43bebfb91"), "fix(#367) a title no longer displaces a leading particle out of the leading position": _Claim(1, ('family', 'given', 'middle'), "dce0ae6df4be"), }, diff --git a/tools/differential/expected_since_1.4.0.toml b/tools/differential/expected_since_1.4.0.toml index dde79d00..ad8976f3 100644 --- a/tools/differential/expected_since_1.4.0.toml +++ b/tools/differential/expected_since_1.4.0.toml @@ -179,6 +179,47 @@ issue = "fix(comma-precomma-family) pre-comma run reads as family, not given" name_regex = "^[\\u0000-\\u024f]*,[\\u0000-\\u024f]*$" fields = ["given", "family"] +[[change]] +issue = "fix(#379) a tussenvoegsel after a family comma attaches to the family" +# 'Vega, Juan de la': the Dutch alphabetized listing moves the +# tussenvoegsel behind the given name -- "Beethoven, Ludwig van" is how +# "Ludwig van Beethoven" is filed -- and 1.4 read the trailing particle +# run as a middle name. 2.2 attaches it to the family already named by +# the comma and renders it before the base, so `middle` empties and +# `family` grows. rules.md#P6 states it; the derived views move with it +# (family_particles 'de la', family_base 'Vega'). +# +# Fields are {middle, family}: the given name does not move. `suffix` +# is deliberately NOT here, though the #380 half of the change does +# move a token out of it ('Berg, Jan vd' was suffix 'vd' at every +# baseline and is now family 'vd Berg'). No corpus name exercises that +# shape, and adding `suffix` to claim it costs three shapes this rule +# has no business explaining -- {family,suffix} taken from +# fix(suffix-routing), and {middle,suffix} and {middle,family,suffix} +# taken from UNEXPLAINED, the last two being diffs in which `family` +# does not move at all. A rule that pre-claims shapes it has never seen +# is the #372 failure mode. If a vd-shaped name ever enters the corpus +# it should arrive unexplained and be read once. +# +# Latin-anchored over the WHOLE name, not just the trailing word: with +# a leading .* this absorbed non-Latin comma names whose diff has +# another cause -- a Cyrillic 'Smith, de la' moves {family} because the +# particle's RENDERING ORDER inside the family changed, which no rule +# in this ledger names, and it classified here. (It now falls to the +# fields-only catch-all, so the absorption moved rather than vanished; +# worth a narrower rule if such a name ever appears.) The class is the +# neighbours' and compare.py's own: U+0000-U+024F, which stops at +# Latin Extended-B, so a diacritic-bearing 'Nguyen, Van de la' in NFD +# or Latin Extended Additional is outside it. +# +# What the rule still over-claims is vocabulary, not script: any Latin +# comma name ending in an alternation member is claimed whatever the +# naming tradition, so a future {middle,family} regression on +# 'Nguyen, Van' -- one of the two corpus names it reaches, and the one +# that does NOT move -- would classify as intended. +name_regex = "(?i)^[\\u0000-\\u024f]*,[\\u0000-\\u024f]*\\s(de|del|den|der|di|do|dos|du|la|le|los|mc|van|vd|von|zu)(\\s+(de|del|den|der|di|do|dos|du|la|le|los|mc|van|vd|von|zu))*\\s*$" +fields = ["middle", "family"] + [[change]] issue = "fix(suffix-routing) two-token name with unambiguous trailing suffix stays suffix" # 'Johnson PhD' / 'Smith Jr.' / 'John V' / 'QC MP': v1 routed a lone diff --git a/tools/differential/expected_since_2.0.0.toml b/tools/differential/expected_since_2.0.0.toml index d8cb0fe7..2863a8d8 100644 --- a/tools/differential/expected_since_2.0.0.toml +++ b/tools/differential/expected_since_2.0.0.toml @@ -283,3 +283,44 @@ issue = "fix(#367) a title no longer displaces a leading particle out of the lea # classification. name_regex = "(?i)^mr\\.\\s+van\\b" fields = ["given", "middle", "family"] + +[[change]] +issue = "fix(#379) a tussenvoegsel after a family comma attaches to the family" +# 'Vega, Juan de la': the Dutch alphabetized listing moves the +# tussenvoegsel behind the given name -- "Beethoven, Ludwig van" is how +# "Ludwig van Beethoven" is filed -- and 1.4 read the trailing particle +# run as a middle name. 2.2 attaches it to the family already named by +# the comma and renders it before the base, so `middle` empties and +# `family` grows. rules.md#P6 states it; the derived views move with it +# (family_particles 'de la', family_base 'Vega'). +# +# Fields are {middle, family}: the given name does not move. `suffix` +# is deliberately NOT here, though the #380 half of the change does +# move a token out of it ('Berg, Jan vd' was suffix 'vd' at every +# baseline and is now family 'vd Berg'). No corpus name exercises that +# shape, and adding `suffix` to claim it costs three shapes this rule +# has no business explaining -- {family,suffix} taken from +# fix(suffix-routing), and {middle,suffix} and {middle,family,suffix} +# taken from UNEXPLAINED, the last two being diffs in which `family` +# does not move at all. A rule that pre-claims shapes it has never seen +# is the #372 failure mode. If a vd-shaped name ever enters the corpus +# it should arrive unexplained and be read once. +# +# Latin-anchored over the WHOLE name, not just the trailing word: with +# a leading .* this absorbed non-Latin comma names whose diff has +# another cause -- a Cyrillic 'Smith, de la' moves {family} because the +# particle's RENDERING ORDER inside the family changed, which no rule +# in this ledger names, and it classified here. (It now falls to the +# fields-only catch-all, so the absorption moved rather than vanished; +# worth a narrower rule if such a name ever appears.) The class is the +# neighbours' and compare.py's own: U+0000-U+024F, which stops at +# Latin Extended-B, so a diacritic-bearing 'Nguyen, Van de la' in NFD +# or Latin Extended Additional is outside it. +# +# What the rule still over-claims is vocabulary, not script: any Latin +# comma name ending in an alternation member is claimed whatever the +# naming tradition, so a future {middle,family} regression on +# 'Nguyen, Van' -- one of the two corpus names it reaches, and the one +# that does NOT move -- would classify as intended. +name_regex = "(?i)^[\\u0000-\\u024f]*,[\\u0000-\\u024f]*\\s(de|del|den|der|di|do|dos|du|la|le|los|mc|van|vd|von|zu)(\\s+(de|del|den|der|di|do|dos|du|la|le|los|mc|van|vd|von|zu))*\\s*$" +fields = ["middle", "family"] diff --git a/tools/differential/expected_since_2.1.0.toml b/tools/differential/expected_since_2.1.0.toml index 796b1062..875c5f1d 100644 --- a/tools/differential/expected_since_2.1.0.toml +++ b/tools/differential/expected_since_2.1.0.toml @@ -94,3 +94,44 @@ issue = "fix(#367) a title no longer displaces a leading particle out of the lea # classification. name_regex = "(?i)^mr\\.\\s+van\\b" fields = ["given", "middle", "family"] + +[[change]] +issue = "fix(#379) a tussenvoegsel after a family comma attaches to the family" +# 'Vega, Juan de la': the Dutch alphabetized listing moves the +# tussenvoegsel behind the given name -- "Beethoven, Ludwig van" is how +# "Ludwig van Beethoven" is filed -- and 1.4 read the trailing particle +# run as a middle name. 2.2 attaches it to the family already named by +# the comma and renders it before the base, so `middle` empties and +# `family` grows. rules.md#P6 states it; the derived views move with it +# (family_particles 'de la', family_base 'Vega'). +# +# Fields are {middle, family}: the given name does not move. `suffix` +# is deliberately NOT here, though the #380 half of the change does +# move a token out of it ('Berg, Jan vd' was suffix 'vd' at every +# baseline and is now family 'vd Berg'). No corpus name exercises that +# shape, and adding `suffix` to claim it costs three shapes this rule +# has no business explaining -- {family,suffix} taken from +# fix(suffix-routing), and {middle,suffix} and {middle,family,suffix} +# taken from UNEXPLAINED, the last two being diffs in which `family` +# does not move at all. A rule that pre-claims shapes it has never seen +# is the #372 failure mode. If a vd-shaped name ever enters the corpus +# it should arrive unexplained and be read once. +# +# Latin-anchored over the WHOLE name, not just the trailing word: with +# a leading .* this absorbed non-Latin comma names whose diff has +# another cause -- a Cyrillic 'Smith, de la' moves {family} because the +# particle's RENDERING ORDER inside the family changed, which no rule +# in this ledger names, and it classified here. (It now falls to the +# fields-only catch-all, so the absorption moved rather than vanished; +# worth a narrower rule if such a name ever appears.) The class is the +# neighbours' and compare.py's own: U+0000-U+024F, which stops at +# Latin Extended-B, so a diacritic-bearing 'Nguyen, Van de la' in NFD +# or Latin Extended Additional is outside it. +# +# What the rule still over-claims is vocabulary, not script: any Latin +# comma name ending in an alternation member is claimed whatever the +# naming tradition, so a future {middle,family} regression on +# 'Nguyen, Van' -- one of the two corpus names it reaches, and the one +# that does NOT move -- would classify as intended. +name_regex = "(?i)^[\\u0000-\\u024f]*,[\\u0000-\\u024f]*\\s(de|del|den|der|di|do|dos|du|la|le|los|mc|van|vd|von|zu)(\\s+(de|del|den|der|di|do|dos|du|la|le|los|mc|van|vd|von|zu))*\\s*$" +fields = ["middle", "family"]