Last week, I wrote about what over-redaction does to your tool’s architecture: blanking out too much strips away the context the model needs and destroys your blueprint. Your data is valuable because it’s yours, and that specificity is the first thing that redaction removes.
I recently spoke to a founder who had closed his business and sold the company’s data as part of this closing process. His team de-identified it by hand, and the buyer rejected it three times. Each time, the buyer said either the personal information hadn’t all been found or it hadn’t been de-identified well enough. He also told me the most valuable data was from before 2020, because it was more human and not yet saturated with AI.
There’s an irony there – proprietary data is worth so much because people wrote it, and it’s being sold to train the models that make new data less human.
But the part that made me think was about the detection: even careful human detection still wasn’t good enough for the buyer.
It made me look at the protection process from the start. Redaction has a huge effect on how well data performs, but detection comes first, and it sets the ceiling. Nothing can be protected unless it’s identified first.
We’re drawn to high numbers, like which vendor beats which or who tops the latest benchmark. The gap below 100% gets less attention. When Hou and colleagues compared four ways of detecting private details before text reached a larger model, the best one caught 65.7% of the names, places and organisations in it. Roughly one in three went to the model untouched.
Whatever slips through is just leaked, obviously. A company can face regulatory consequences for that, but the bigger failure is to the people whose data is now exposed.
So why doesn’t everyone have perfect detection? Most teams start with Presidio, a spaCy NER model or a prompt to a small LLM. The best performer in the Hou study was an off-the-shelf BERT NER model, the same kind of tool many of us already run. Each of these catches the obvious cases, and each misses in its own way.
Part of the problem is definition. “Personal” and “identifiable” aren’t straightforward terms and pinning them down means first sorting identifiers into direct and indirect. Direct identifiers point to one person on their own: a name, a home address, a National Insurance number. Indirect identifiers, also called quasi-identifiers or contextual identifiers, only point to someone in combination. A date of birth, a job title and a town might be harmless one at a time, but identifying together.
Take a line like “I spoke to the only paralegal in our Leeds office, who joined in March.” Since there’s no name in it, a detector looking for names would return nothing. To anyone at that firm, it describes exactly one person.
Those indirect details are exactly what models are good at piecing together. Patsakis and Lykousas took descriptions of 20 UK public figures that had already been run through an anonymisation tool, with names, places, jobs and dates stripped out. GPT-3.5 still named the right person about 73 times in 100. Human readers, in an earlier study on the same texts, managed about 26. The researchers then removed every word the model itself identified as giving the person away. It still identified 59 out of 100 of the texts it had named before.
The subjects were celebrities, which makes guessing easier, but anyone holding background data on customers is in a similar position. Catching names isn’t enough if what’s left still describes someone.
Part of good detection is training. Getting a model to catch every form of an entity means a lot of fine-tuning and building domain-specific ontologies, and that work has to be repeated for every entity type. For sensitive data, the engine also can’t send requests out to an external service, which rules out third-party LLMs for this step.
Then there’s the opposite failure: if you push sensitivity too far false positives might climb. Over-detection does the same damage as over-redaction. The data turns sterile, and a person has to go back through it by hand.
Detecting well means defining identifiers for your own domain, training on semantic ontologies you outline and define, keeping the whole process in-house, and holding false positives down while you do it. None of this is quick. It can also be unglamorous.
