You ask a company to delete your data. In a normal system that is a database operation: find the rows that are yours, remove them, done. The mental model of “delete” that privacy law is built on — the GDPR’s right to erasure, most obviously — assumes exactly this: that your data sits somewhere as a discrete record you can locate and destroy.
A trained AI model breaks that assumption. Answer first: your data isn’t stored in the model as a record at all. It is dissolved into the model’s parameters — billions of numbers, each nudged a little during training by every example it saw, yours included. There is no row labelled with your name to delete. Removing your influence means changing the numbers, and doing that cleanly is a genuine research problem, not a setting with a toggle.
Where your data actually goes when a model “learns” it
Training a large model is a process of adjustment. The model makes a prediction, it’s wrong, and an optimiser tweaks its parameters a fraction to make that particular kind of error slightly less likely next time. Repeat across trillions of tokens and those fractional tweaks accumulate into a system that has, in a distributed and lossy way, absorbed patterns from its training data.
The key word is distributed. A single document doesn’t live in one identifiable place in the weights; its contribution is smeared across many parameters that also encode a great many other things. Two consequences follow, and they are the whole reason this is hard. First, you cannot point at the part of the model that is “you.” Second, deleting the original document from the training set does nothing to the model that already trained on it — the lesson has been learned and the textbook has been closed. The data is gone; the influence remains.
Erasing your data from the training set is like removing a single lump of sugar from a cake that has already been baked. The lump is gone from the recipe. The sweetness is still in the cake.
The clean fix that nobody can afford to do often
There is a correct, boring answer to all of this: retrain the model from scratch on the dataset with your data left out. This is sometimes called exact unlearning, and it is the gold standard because the resulting model provably never saw your data. If cost were no object, every deletion request would be honoured this way.
Cost is very much an object. Training a frontier-scale model is a multi-week run on enormous clusters, with an energy and compute bill that industry estimates put anywhere from the high hundreds of thousands into the millions of dollars for a single training run. Doing that again for one person’s erasure request — and then again for the next request, and the next — is not something any lab will do routinely. So exact unlearning, the one method with a real guarantee, is precisely the one that doesn’t scale to the volume of requests a popular product generates.
“Approximate unlearning”: cheaper, and no promises
The workaround is a fast-moving research field called machine unlearning, which tries to make a model act as though it never saw specific data without paying the full retraining cost. The techniques are ingenious and worth knowing by name, because they define what “we’ll remove your data” can realistically mean:
- Gradient ascent — roughly, train away from the data you want forgotten, nudging the weights in the opposite direction to how they were nudged when learning it. Cheap, but a heavy hand can degrade the model’s general ability.
- Influence functions — estimate how much a given data point shaped the model and subtract that estimated influence. Elegant, but it’s an approximation of an approximation on systems this large.
- Sharded retraining — split the training data into shards up front so that forgetting a point only requires retraining the shard it lived in, not the whole model. Practical, but it constrains how you train and still isn’t free.
All of these buy efficiency by giving up guarantees. Approximate unlearning is typically judged not by a proof but by empirical tests — can we still extract the data, does the model still complete the forgotten passage — and there is no settled definition of what counts as “successfully forgotten” in a probabilistic system. Benchmarks exist: a 2023 Machine Unlearning Challenge organised with Google framed the problem around three goals in tension — forgetting quality, keeping the model useful, and doing it efficiently — and treated a method as “efficient” only if it cost a small fraction of full retraining. But framing the problem well is not the same as solving it.
Even “forgotten” data can come back
The uncomfortable finding from recent research is that erasure can be less final than it looks. Work published in 2025 demonstrated that data could in some cases be extracted after supposed exact unlearning, because traces survive in places the procedure didn’t fully account for. If even the gold-standard method can leave recoverable residue under the right probing, the approximate methods — the ones that actually get used at scale — should be read as reducing risk rather than guaranteeing absence. There is progress on cost, too: 2025 methods have shown unlearning at roughly half the price of retraining while preserving performance. But “cheaper and pretty good” is a different promise from “gone.”
Why this should change how you read a privacy promise
None of this is a reason to shrug and assume erasure is fake. It is a reason to read the specific wording. When a service says it will “delete your data,” the honest, verifiable part is usually about the training set and the databases — your records stop being stored and stop being fed into future runs. That is real and it matters. The much harder, much vaguer part is what happens to the model that already learned from you, and that is where “and we’ll retrain” does a lot of quiet work.
This is not abstract. It is exactly the promise sitting under, for instance, Atlassian’s pledge to remove opted-out data and retrain its models, and under every consumer AI product’s data-deletion page. It also sharpens why memorisation is such a problem in the first place: if models can leak the data they were trained on, then incomplete forgetting isn’t just a compliance footnote, it’s a live exposure. And it is one more front in the unresolved fight over who owns the words that trained your AI — ownership is moot if removal is impossible.
The regulation is trying to catch up. The right to erasure was written for a world of databases, and applying it to distributed model weights is an open legal-technical question that unlearning research and the wider push to regulate AI are both circling. Until it’s resolved, the pro-consumer posture is the sceptical one: treat “we’ll delete it from the model” as a claim with a hard engineering problem behind it, ask which deletion is being promised — the dataset, or the model — and don’t assume the two are the same thing. The delete key works on your files. It does not yet work on a model’s memory, and pretending otherwise is the part worth not forgetting.
Originally published at theaidownside.com — evidence-first reporting on the costs and trade-offs behind AI products.
