Introducing Temperate PII

Results

We measured Temperate PII on 2,041 texts in English, Dutch, German, French, Spanish and Italian. They hold 2,720 values that must be masked, and 4,307 that must stay as they are, such as company names, product codes and order numbers. Temperate PII missed 13 values and masked 8 that it should have left alone.

We ran the PII service of Azure AI Language over the same texts and scored it with the same scorer [1]. It missed 402 values and masked 145 by mistake.

Every type of personal data

On email addresses, IBANs and street addresses, the two are close. The gap is in the numbers that differ per country. Azure found 48% of the VAT numbers, 62% of the passport numbers and 88% of the driving licences. Temperate PII found all VAT and passport numbers, and 98.8% of the driving licences. Azure has no category for API keys and tokens, so it found none of the 211 in the benchmark.

Azure finds more person names: 99.5% against our 98.0%. It is the one type where our recall is lower.

How it works

Most personal data has a format. An IBAN carries a checksum, a Dutch BSN passes the eleven test, and a card number passes the Luhn check. Rules find these values, with a pack per country for Belgium, France, Germany, Italy, the Netherlands, Spain and the United Kingdom. Names and street addresses have no format, so the model finds those.

The model is a student of six layers, distilled from a larger teacher and built on Multilingual MiniLM. It has 107 million parameters and runs as an int8 ONNX model. We trained it on open data: the PII masking set of ai4privacy and the multilingual set of nym [2][3].

A model that finds names also finds words that only look like names. A precision step after the model drops weak spans, such as a surname on its own with no title, greeting or earlier full name. "Vraag het aan Bakker van financiën" stays as it is: masking every word that could be a surname would also mask company and place names.

Every language

We trained the model on six languages, and its recall stays above 99% in each of them. The service accepts all 24 official languages of the EU. For a language the model was not trained on, the answer says so, and lists the types it could not check.

It is also fast enough to sit in front of every request. For a text of 2,000 characters, the median answer took 23 ms and the 95th percentile 31 ms, one request at a time on an Apple M1 Max with four threads, while other work ran on the same machine.

Where it misses

The benchmark is text we built, so we also wrote 26 business texts by hand: emails, invoices, contracts and meeting notes, in six languages. They hold 83 names and addresses. Temperate PII finds 76 of the 83, and masks one of the 112 company, product and place names in them by mistake. Three of its misses, as they are:

• "Katrin Weber", a full name in a German email.
• "Mark", a first name on its own in an English support mail.
• "79098 Freiburg im Breisgau", a postcode with its town in a German invoice.

We found one more while we wrote this page. In "Bel Sanne de Vries op 06 12345678", the model finds the phone number but not the name. With a greeting before it, it finds the name.

How we measure

1. 1The benchmark. 2,041 texts. 963 are passages of openly licensed prose, most of them from Wikipedia, in which we replaced the real names and put in made-up identifiers that pass their national checks. We generated 975 more and wrote 103 by hand.
2. 2Kept apart from training. A check compares every benchmark text with the training data: by its source, by its exact text, and by runs of eight words in a row. It found 30 shared runs of eight words, stock phrases, and none at ten words. No benchmark text is in the training data.
3. 3One scorer for both. The same code scores Temperate PII and Azure. We mapped 39 of Azure's categories onto our 13 types. Azure also finds organizations and dates, which we do not score, and those spans did not count against it.
4. 4Speed. 1,000 requests of 2,000 characters each, one at a time, after 100 requests to warm up.
5. 5Checking it. The benchmark, the scorer and Azure's raw answers are in the repository, so you can run every number on this page again [4].

What these numbers do not say

1. 1We wrote the cases, and that helps us. In our own measurements, scores on text like this run up to about 15 points higher than scores on real documents.
2. 2Each figure comes from one run. Azure's comes from a run on 12 September 2026, and Azure can change its model under the same name.
3. 3The model was trained on six languages. In the other 18 languages of the EU, the rules still work, but names and addresses are not checked.
4. 4The speed was measured on one laptop. A server with other cores gives other numbers.

Try it

The weights are on Hugging Face as temperate-works/pii, and the code, the benchmark and the training scripts are on GitHub [4][5]. Both are under Apache-2.0. It runs from source with Rust, or in a container with two commands.

docker build -t temperate-pii .
docker run -p 8090:8090 temperate-pii