# DeTars-PII-v1 — evaluation details (release 3.1) All results are self-run, descriptive and single-seed; no statistical significance is claimed. Every source file and its SHA256 is listed in `evaluation-sources.json`. Metrics: - **Leak %**: in-scope gold characters left unmasked ÷ gold characters. Lower is better. - **Over-mask / 1k**: masked non-gold characters ÷ input characters × 1000. Lower is better. - **Strict F1 %**: exact span boundaries and shared category after label projection. Higher is better. Leak and over-mask are character coverage measures and ignore category correctness. ## 1. How the bundle routes text Routing is decided once per original document, before protected ranges are blanked or the text is chunked: - **Han route**: Han characters present and no Hiragana, Katakana or Hangul. 15 labels; base pass plus an address LoRA pass; per-label width caps; a door-number address filter; threshold 0.5. This route is byte-identical between 3.0 and 3.1. - **Multilingual route**: every other document. 18 labels (the 15 above plus LOCATION, DATE and URL); balanced word splitter; one multilingual LoRA pass; threshold 0.65 in 3.1 (0.25 in 3.0). This is a script rule, not language identification: mixed Han + Latin text can use the Han route, and Japanese written only in Han without kana can too. Both routes share one ONNX encoder; the adapters are switched by gate inputs. `LOCATION` maps to the existing ADDRESS category; generic `DATE` and `URL` map to IDENTIFIER (not birthday or email). In 3.1 the multilingual adapter is rank 16: the 3.0 rank-8 adapter and the round-3 rank-8 adapter (trained on the 3.0 merged model) concatenated exactly. On the 920-document EU development set at threshold 0.65, ONNX spans equal locked CPU PyTorch spans on 920/920 documents (largest common-span score difference 0.0087). ## 2. Production regression: 3.0 vs 3.1 The full production chain (the DeTars runtime-6 detector service → `previewPrivacyText(depth=model)` → final mask selection, including rules and overlap selection) on 3,090 documents per version, both run serially in one session on the same machine (Apple M2) and source. Character-weighted micro averages. The sets: - **Fresh900**: 100 documents each in en, de, fr, es, it, nl, pt, ja, ko from the Ai4Privacy OpenPII validation split, frozen before 3.0 was trained. - **Confirmation set**: 750 documents across 30 languages from Ai4Privacy datasets. - **newlang6**: 240 DeTars-synthetic documents in de, it, pt, nl, ko, ru, including negatives. - **Independent hard negatives**: 40 documents (4 per language) with no gold PII. - **pii-bench-zh**: 1,000 Chinese documents from `wan9yu/pii-bench-zh`. - **v4 Chinese / v4 English**: an earlier internal regression set, not blind (136 Chinese, 24 English documents). Fresh900 and the confirmation set have been scored before, so they are regression and confirmation material, not new blind tests. | Set | N | 3.0 leak % | 3.1 leak % | 3.0 over / 1k | 3.1 over / 1k | negative-doc hits 3.0 → 3.1 | |---|---:|---:|---:|---:|---:|---:| | Fresh900 (9 languages) | 900 | 2.30 | 2.08 | 16.57 | 12.92 | 1/11 → 3/11 | | Confirmation set (30 languages, reused) | 750 | 2.50 | 2.51 | 18.81 | 18.40 | 0/4 → 0/4 | | newlang6 (synthetic) | 240 | 3.60 | 3.39 | 21.58 | 17.08 | 29/48 → 24/48 | | Independent hard negatives | 40 | — | — | 19.25 | 13.53 | 4/40 → 1/40 | | pii-bench-zh | 1000 | 2.06 | 2.06 | 14.29 | 14.29 | — | | v4 Chinese | 136 | 2.49 | 2.49 | 11.71 | 11.71 | 4/19 → 4/19 | | v4 English | 24 | 6.13 | 4.24 | 5.37 | 3.22 | — | | Korean, all sets | 169 | 9.73 | 9.87 | 56.27 | 38.78 | 10/13 → 6/13 | | Japanese, all sets | 129 | 6.93 | 6.28 | 20.52 | 23.93 | 1/6 → 2/6 | | All | 3090 | 2.55 | 2.42 | 17.44 | 15.29 | 38/122 → 32/122 | ### Fresh900 by language (100 documents each) | Set | N | 3.0 leak % | 3.1 leak % | 3.0 over / 1k | 3.1 over / 1k | negative-doc hits 3.0 → 3.1 | |---|---:|---:|---:|---:|---:|---:| | en | 100 | 1.47 | 1.11 | 25.67 | 17.90 | — | | de | 100 | 0.76 | 0.28 | 8.04 | 5.75 | 0/2 → 0/2 | | fr | 100 | 1.10 | 1.36 | 12.81 | 11.64 | — | | es | 100 | 1.84 | 1.42 | 20.74 | 18.46 | 0/1 → 0/1 | | it | 100 | 1.19 | 1.12 | 13.20 | 9.84 | — | | nl | 100 | 1.76 | 2.01 | 10.48 | 8.03 | 0/2 → 0/2 | | pt | 100 | 1.65 | 0.72 | 6.60 | 7.37 | 0/3 → 0/3 | | ja | 100 | 7.35 | 6.59 | 19.01 | 22.69 | 0/2 → 2/2 | | ko | 100 | 5.98 | 6.61 | 49.26 | 25.52 | 1/1 → 1/1 | ### newlang6 by language (40 synthetic documents each) | Set | N | 3.0 leak % | 3.1 leak % | 3.0 over / 1k | 3.1 over / 1k | negative-doc hits 3.0 → 3.1 | |---|---:|---:|---:|---:|---:|---:| | de | 40 | 0.00 | 0.00 | 5.77 | 3.94 | 5/8 → 4/8 | | it | 40 | 4.44 | 3.59 | 8.83 | 5.74 | 4/8 → 3/8 | | pt | 40 | 0.19 | 0.24 | 6.07 | 4.14 | 5/8 → 4/8 | | nl | 40 | 0.16 | 0.61 | 25.71 | 22.17 | 5/8 → 6/8 | | ko | 40 | 19.97 | 19.93 | 82.04 | 68.61 | 6/8 → 4/8 | | ru | 40 | 3.10 | 2.24 | 11.93 | 6.98 | 4/8 → 3/8 | ### Independent hard negatives by language (4 documents each, no gold PII) | Set | N | 3.0 leak % | 3.1 leak % | 3.0 over / 1k | 3.1 over / 1k | negative-doc hits 3.0 → 3.1 | |---|---:|---:|---:|---:|---:|---:| | en | 4 | — | — | 0.00 | 0.00 | 0/4 → 0/4 | | de | 4 | — | — | 0.00 | 0.00 | 0/4 → 0/4 | | fr | 4 | — | — | 0.00 | 0.00 | 0/4 → 0/4 | | es | 4 | — | — | 0.00 | 0.00 | 0/4 → 0/4 | | it | 4 | — | — | 0.00 | 0.00 | 0/4 → 0/4 | | nl | 4 | — | — | 0.00 | 0.00 | 0/4 → 0/4 | | pt | 4 | — | — | 0.00 | 0.00 | 0/4 → 0/4 | | ja | 4 | — | — | 45.87 | 0.00 | 1/4 → 0/4 | | ko | 4 | — | — | 238.81 | 194.03 | 3/4 → 1/4 | | ru | 4 | — | — | 0.00 | 0.00 | 0/4 → 0/4 | ### Memory and speed Apple M2, 4 threads (production default), model load excluded. Peak is lifetime peak physical footprint of the one resident sidecar process. | Run | Documents | Preview seconds | Characters / s | Peak bytes | |---|---:|---:|---:|---:| | 3.0 full set | 3,090 | 787.6 | 798.7 | 804,997,088 | | 3.1 full set | 3,090 | 733.1 | 858.1 | 806,717,336 | | 3.1 live smoke (3) + two-route 896-subword stress (4) | 7 | — | — | 863,045,360 | ## 3. European blind set (23 languages, 2,300 documents) Model level (detector output, not the full production chain). The set was sampled from the Ai4Privacy OpenPII validation split and frozen before round-3 training began; 3.1's threshold was locked on a separate EU development split, and the blind set was scored once. Peer operating points were locked on the benchmark development split before any test scoring: Fastino `fastino/gliner2-privacy-filter-PII-multi` @0.25, NVIDIA `nvidia/gliner-PII` @0.25, OpenAI Privacy Filter `openai/privacy-filter` at privacy_bias −2 (an emission shift, not a threshold). NVIDIA and OPF were scored only after the 3.0 / 3.1 / Fastino blind scores had been locked. Gold follows policy-v3 (section 5). old7 = en, de, fr, es, it, nl, pt; new16 = the other 16 languages. Each cell: **leak % / over-mask per 1k / strict F1 %**. | Language | 3.0 @0.25 | 3.1 @0.65 | Fastino @0.25 | NVIDIA @0.25 | OPF bias −2 | |---|---|---|---|---|---| | en | 1.45 / 22.21 / 81.5 | 0.42 / 20.63 / 90.4 | 5.79 / 11.46 / 66.9 | 8.16 / 9.54 / 57.0 | 17.01 / 5.95 / 66.7 | | de | 1.40 / 8.58 / 84.5 | 0.66 / 8.64 / 92.4 | 3.99 / 3.07 / 70.6 | 8.86 / 5.22 / 57.2 | 13.60 / 2.95 / 73.2 | | fr | 1.84 / 14.21 / 83.8 | 1.29 / 13.55 / 90.1 | 5.13 / 6.82 / 71.0 | 9.95 / 6.48 / 58.8 | 8.27 / 4.36 / 73.8 | | es | 2.12 / 10.01 / 87.2 | 0.87 / 6.12 / 93.5 | 8.58 / 5.31 / 70.7 | 9.77 / 8.47 / 55.7 | 11.11 / 4.53 / 76.5 | | it | 1.01 / 11.52 / 85.9 | 1.53 / 7.88 / 91.5 | 4.85 / 4.45 / 71.1 | 7.88 / 9.92 / 55.0 | 9.66 / 2.81 / 74.1 | | nl | 1.83 / 8.07 / 83.6 | 1.50 / 8.90 / 91.9 | 4.56 / 5.17 / 67.6 | 7.66 / 4.28 / 59.6 | 14.35 / 3.73 / 73.2 | | pt | 1.97 / 12.51 / 85.6 | 1.39 / 8.54 / 93.8 | 6.17 / 2.44 / 69.8 | 7.85 / 7.03 / 57.6 | 12.17 / 7.14 / 74.1 | | bg | 2.52 / 7.98 / 84.6 | 2.95 / 5.43 / 91.1 | 8.39 / 7.14 / 66.1 | 13.50 / 10.67 / 53.1 | 14.84 / 15.76 / 66.4 | | pl | 3.49 / 9.45 / 79.9 | 1.56 / 6.50 / 92.0 | 7.65 / 6.04 / 64.7 | 10.03 / 6.99 / 54.3 | 26.03 / 6.71 / 68.7 | | cs | 3.41 / 7.29 / 83.7 | 3.59 / 4.61 / 90.9 | 7.12 / 4.35 / 68.7 | 12.17 / 5.78 / 57.5 | 27.47 / 7.48 / 66.9 | | sv | 2.06 / 10.29 / 81.8 | 1.36 / 10.36 / 90.7 | 5.43 / 5.93 / 65.2 | 10.80 / 3.68 / 56.6 | 23.33 / 3.92 / 64.3 | | ro | 2.32 / 4.64 / 87.9 | 1.76 / 3.60 / 93.9 | 6.62 / 4.45 / 69.8 | 9.84 / 6.38 / 58.6 | 28.90 / 4.75 / 69.8 | | lt | 1.57 / 13.09 / 83.9 | 1.75 / 13.78 / 90.2 | 5.41 / 6.17 / 70.9 | 10.59 / 42.66 / 49.2 | 40.00 / 4.47 / 64.3 | | et | 2.59 / 13.28 / 81.3 | 1.96 / 12.96 / 87.9 | 6.99 / 6.82 / 62.9 | 11.51 / 26.02 / 52.9 | 22.11 / 12.99 / 66.4 | | hr | 2.92 / 6.71 / 83.5 | 3.12 / 4.94 / 92.3 | 6.30 / 3.78 / 67.2 | 9.68 / 8.36 / 54.7 | 19.81 / 8.76 / 70.0 | | fi | 0.60 / 10.20 / 84.2 | 0.91 / 7.59 / 92.4 | 3.29 / 7.32 / 66.9 | 8.74 / 8.42 / 56.1 | 33.60 / 10.89 / 62.0 | | lv | 1.74 / 11.47 / 81.7 | 1.33 / 12.08 / 89.6 | 5.10 / 6.81 / 66.2 | 13.20 / 36.87 / 48.2 | 46.64 / 3.19 / 57.8 | | hu | 1.69 / 17.60 / 83.2 | 1.28 / 17.10 / 88.7 | 5.68 / 6.66 / 67.4 | 10.02 / 22.00 / 52.3 | 25.99 / 12.01 / 63.4 | | sk | 2.19 / 13.81 / 83.1 | 2.79 / 10.00 / 92.4 | 4.20 / 3.15 / 68.8 | 9.49 / 12.62 / 54.7 | 22.73 / 9.87 / 66.3 | | sl | 3.31 / 7.20 / 84.4 | 2.58 / 10.59 / 90.2 | 6.63 / 6.72 / 67.6 | 7.77 / 8.86 / 57.6 | 28.93 / 5.92 / 66.2 | | da | 1.63 / 10.57 / 83.2 | 0.65 / 7.63 / 93.8 | 6.55 / 4.45 / 66.3 | 9.24 / 6.60 / 57.2 | 24.31 / 4.21 / 68.5 | | el | 2.26 / 7.87 / 86.4 | 1.65 / 6.85 / 92.0 | 5.75 / 2.85 / 71.1 | 10.47 / 8.98 / 55.8 | 43.40 / 4.36 / 59.4 | | sr | 2.77 / 14.26 / 84.4 | 1.95 / 9.91 / 91.7 | 7.13 / 4.94 / 71.2 | 10.81 / 10.90 / 59.0 | 24.28 / 9.41 / 65.3 | | **old7 macro** | 1.66 / 12.44 / 84.6 | 1.10 / 10.61 / 91.9 | 5.58 / 5.53 / 69.7 | 8.59 / 7.28 / 57.3 | 12.31 / 4.49 / 73.1 | | **old7 micro** | 1.69 / 12.43 / 84.6 | 1.09 / 10.54 / 92.0 | 5.71 / 5.53 / 69.6 | 8.63 / 7.28 / 57.3 | 12.31 / 4.54 / 73.2 | | **new16 macro** | 2.32 / 10.36 / 83.6 | 1.95 / 9.00 / 91.2 | 6.14 / 5.47 / 67.6 | 10.49 / 14.11 / 54.9 | 28.27 / 7.79 / 65.4 | | **new16 micro** | 2.31 / 10.33 / 83.6 | 1.94 / 9.03 / 91.2 | 6.15 / 5.49 / 67.6 | 10.46 / 14.21 / 54.7 | 28.25 / 7.73 / 65.5 | | **eu23 macro** | 2.12 / 10.99 / 83.9 | 1.69 / 9.49 / 91.5 | 5.97 / 5.49 / 68.2 | 9.91 / 12.03 / 55.6 | 23.41 / 6.79 / 67.7 | | **eu23 micro** | 2.11 / 11.00 / 83.9 | 1.67 / 9.51 / 91.5 | 6.01 / 5.50 / 68.2 | 9.88 / 12.01 / 55.5 | 23.19 / 6.72 / 68.1 | Against Fastino, NVIDIA and OPF, 3.1 has the lowest leak in 23/23 languages and the highest strict F1 in 23/23 (also above 3.0 in strict F1 everywhere). 3.0 itself leaks slightly less than 3.1 in 7 languages (it, bg, cs, lt, hr, fi, sk). On the aggregates, 3.1's over-mask is above Fastino's and OPF's everywhere, and above NVIDIA's on old7 (10.61 vs 7.28 / 1k). ### Matched over-mask against Fastino 3.1's leak, interpolated along its own seven-point threshold curve at Fastino's over-mask (Fastino @0.25), per language. No extrapolation: languages where Fastino's over-mask is outside 3.1's curve are omitted (15 languages: en, de, fr, it, pt, sv, lt, et, lv, hu, sk, sl, da, el, sr). | Language | Fastino @0.25 over / 1k | Fastino leak % | 3.1 leak % at that over-mask | |---|---:|---:|---:| | es | 5.31 | 8.58 | **1.33** | | nl | 5.17 | 4.56 | **1.62** | | bg | 7.14 | 8.39 | **1.99** | | pl | 6.04 | 7.65 | **3.59** | | cs | 4.35 | 7.12 | **4.04** | | ro | 4.45 | 6.62 | **1.23** | | hr | 3.78 | 6.30 | **3.82** | | fi | 7.32 | 3.29 | **2.15** | 3.1 leaks less in all 8 languages where the curves overlap and more in none. On the aggregates, 3.1's curve does not reach Fastino's over-mask level. ## 4. Chinese, Japanese and Korean against peers Model-level benchmark rows on one set per language. Japanese and Korean are the Fresh900 ja / ko subsets (100 documents each), measured with the 3.0 multilingual route; 3.1 changes that route (see section 2 for the 3.0 → 3.1 change). The Chinese row was measured with release 2 (runtime 5); the Han route's outputs have not changed since (release 2, 3.0 and 3.1 produced identical spans on all 1,162 Han-route documents in the production chain). Peer models: `fastino/gliner2-privacy-filter-PII-multi`, `nvidia/gliner-PII`, `urchade/gliner_multi_pii-v1`, `openai/privacy-filter`. On other, earlier Japanese sets with the release-2 model, DeTars was not always first (for example Ai4Privacy Japanese strict F1: OPF 51.75 vs DeTars 50.70). | | DeTars-PII | Fastino | NVIDIA | Urchade | OpenAI PF | |---|---:|---:|---:|---:|---:| | Chinese, `wan9yu/pii-bench-zh` (300-document subset), strict F1 % | **92.13** | 46.07 | 18.55 | 28.19 | 56.21 | | Japanese (Fresh900 ja), leak % / over-mask / strict F1 | **7.36 / 25.54 / 69.49** | 34.71 / 94.91 / 28.72 | 32.99 / 100.47 / 24.07 | 33.93 / 131.89 / 28.50 | 12.69 / 39.63 / 58.99 | | Korean (Fresh900 ko), leak % at Fastino's over-mask (35.05 / 1k) | **8.16** | 9.15 | 13.13 | 45.95 | 14.71 | Fastino in these rows is @0.25; NVIDIA and Urchade @0.25; OPF privacy_bias −2 (Korean matched value interpolated along each curve). At its own operating point the 3.0 Korean route had higher over-mask (53.25 / 1k) than Fastino (35.05), NVIDIA (36.90) and OPF (29.27), though lower than Urchade (69.30); the matched comparison removes that difference. ## 5. Gold policy v3: address connectors Round-3 training and the EU development and blind gold use policy-v3 (`address-connectors-v3`) for addresses. Source address components (STREET, BUILDINGNUM, CITY, ZIPCODE) are merged into one ADDRESS span only when every gap between consecutive components is a connector: - the gap is at most 20 code points and contains no line break, `!`, `?`, `;` or CJK sentence punctuation; - it consists only of separators (space, tab, `,`, `,`, `、`, `·`, `-`, `–`, `—`, `(`, `)`) and whole connector atoms, matched case-insensitively and never as the prefix of a longer word: `#`, `a`, `br.`, `broj`, `ciudad`, `cod poştal`, `cod poștal`, `code postal`, `codigo postal`, `codul poştal`, `codul poștal`, `cp`, `cu`, `código postal`, `della via`, `di via`, `din`, `en`, `g.`, `grad`, `hausnummer`, `i`, `iela`, `ielas`, `ielu`, `im`, `in`, `kod pocztowy`, `meste`, `n.`, `n.°`, `n.º`, `no.`, `nr`, `nr.`, `number`, `numer`, `numero`, `numur`, `n°`, `nº`, `número`, `oraşul`, `orașul`, `plz`, `postal code`, `postiindeks`, `postnummer`, `poštanski broj`, `poštanski kod`, `psč`, `s`, `szám`, `u`, `utca`, `utcán`, `v`, `w`, `zip`, `zip code`, `à`, `č.`, `číslo`, `αριθ.`, `αριθμός`, `в`, `номер`, `пощенски код`, `с`, `№`; - periods are accepted only inside those abbreviations, never as standalone sentence punctuation; - no other annotated entity lies between the components, and no label repeats within one group; - the group is anchored: it contains a STREET or BUILDINGNUM, or it is a postal CITY + ZIPCODE pair. Otherwise each component stays a separate span, and a lone component maps to LOCATION. Outer boundaries never extend beyond the source components and no source entity is dropped. ## 6. Release 3.1: criteria and exceptions **Production regression criteria.** Written before either production run, then left unchanged: | ID | Criterion (written before the runs) | 3.0 → 3.1 | Result | |---|---|---|---| | P1 | both versions complete 3,090/3,090 documents, no error or degraded row | 3,090 / 3,090 each | pass | | C1 | all 1,162 Han-route documents byte-identical to 3.0 (tolerance 0) | 1,162 documents, 0 differences | pass | | C2 | Fresh900 leak ≤ 3.0 + 0.30 pp | 2.30 → 2.08 | pass | | C3 | 750-document confirmation leak ≤ 3.0 + 0.30 pp | 2.50 → 2.51 | pass | | C4 | newlang6 leak ≤ 3.0 + 0.30 pp | 3.60 → 3.39 | pass | | C5 | independent hard-negative triggered documents ≤ 3.0 | 4/40 → 1/40 | pass | | C6 | Korean leak ≤ 3.0 + 1.00 pp | 9.73 → 9.87 | pass | | C7 | Japanese leak ≤ 3.0 + 1.00 pp | 6.93 → 6.28 | pass | | C8 | Korean over-mask ≤ 3.0 | 56.27 → 38.78 | pass | | C9 | peak memory ≤ 1.10 × 3.0 and ≤ 1.5 GB; 896-subword stress ≤ 1.5 GB | 804,997,088 → 806,717,336 bytes; stress 863,045,360 | pass | | C10 | throughput ≥ 0.85 × 3.0 | 798.7 → 858.1 chars/s | pass | | C11 | live sidecar de/ko/zh mask + vault restore and stress cases pass, empty stderr | 7/7 cases, stderr 0 bytes | pass | **Training-time promotion rule.** Round 3 had a promotion rule of 11 items fixed before training. The 3.1 weights failed 4 of them (EU development over-mask at Fastino's level, ja / ko / en development regressions, organization recall, and the combined EU blind check), and the release was approved by the maintainer as an exception after the production regression above. The items not met: - **Over-mask relative to Fastino.** The rule asked for EU over-mask at most 1.10 × Fastino and at least 25% below 3.0. On the development split the 25% reduction was met (27.0%) but the Fastino limit was not (8.25 vs ≤ 6.46 / 1k). On the blind set both parts failed: eu23 macro over-mask is 9.49 / 1k, 1.73 × Fastino's 5.49 / 1k, and 13.7% below 3.0 (10.99 / 1k). 3.1's threshold curve does not reach Fastino's over-mask level on the aggregates. - **ja / ko / en development regressions.** On the existing development sets of 8 documents per language, fixed in the rule beforehand: Japanese leaked characters 116 → 129 and Korean 17 → 19 (over-masked characters fell in both, and strict F1 rose 51.66 → 59.57 and 45.95 → 50.00), English strict F1 64.29 → 58.49. - **Organization recall.** On 22 reviewed real-organization fragments (a silver set, not complete company gold), any-category masking recall fell from 1.0000 to 0.9832: 468 of 476 characters, one fragment missed. Masking of generic organization phrases fell 1.00 → 0.64 and of named non-organizations 1.00 → 0.72 (lower is better for these two). **Observed after promotion.** In the production regression, Japanese over-mask rose from 20.52 to 23.93 / 1k across all sets, and Fresh900 Japanese negative documents triggered 2/2 (3.0: 0/2), while Japanese leak fell 6.93 → 6.28%. The production criteria bound Japanese leak, not Japanese over-mask, so this did not fail them. ## 7. Limitations - Test material is synthetic or public. Ai4Privacy OpenPII validation documents come from the same source family as part of the training data. Base-model pretraining contamination cannot be excluded. - Single training seed; no confidence intervals. - Korean over-mask remains high in absolute terms (newlang Korean 68.61 / 1k in 3.1). - English full-entity boundaries remain imperfect (v4 English rows above). - Russian evidence is synthetic regression only. - Only macOS was measured; Windows and Linux hardware were not part of this evaluation.