We spent a few months building a food scanner: point a phone at a plate, get calories and macros back. The naive version took an afternoon. Making it trustworthy took considerably longer, and the reason turned out to be one specific blind spot in vision models that no amount of prompting fixes.
This is what we measured, what broke, and how we ended up structuring it.
The actual problem isn't recognition
The reason people quit food tracking isn't motivation — it's friction. Searching a database for "grilled chicken thigh, skin removed" three times a day gets old within about two weeks. So the goal was never "identify food in an image" (that part is close to solved). The goal was seconds per meal, end to end, without the result being nonsense.
So: photo in, dish identified, portion estimated, numbers out. Simple pipeline. Here's where it went wrong.
Vision models are good at density and bad at mass
This is the finding that shaped everything else.
Ask a modern multimodal model what's on the plate and how many calories it holds, and you get a single number that quietly bundles two very different estimates:
- What the food is, expressed as nutritional density — kcal per 100g, protein per 100g, and so on.
- How much of it there is — the mass in grams.
We tested these separately against weighed reference meals, and the split was stark. On density, the models were excellent. Chicken breast came back at roughly 165 kcal / 31g protein per 100g — that matches USDA reference data essentially exactly. Peanuts landed within a few percent of the published figure. That's within noise for a nutrition app.
On mass, they were consistently, directionally wrong. A chicken portion we'd weighed at 162g came back around 180g. Across the models we tested, overestimates ranged from roughly +10% to well over +100% on identical images. Every model overestimated. None of them underestimated.
That asymmetry is the tell. This isn't random error you can average away — it's a systematic bias, and it makes sense once you think about what these models actually learned. There's no depth information in a single photo, no reference object, no scale. The model has seen millions of pictures of food captioned with descriptions, not with gram values from a kitchen scale. It's pattern-matching to "a plate of chicken looks like this," and a plate of chicken is usually about 180g in the training distribution.
You cannot prompt your way out of this. We tried. "Be conservative with portion sizes," anchoring examples, explicit references to plate diameter — the estimates didn't converge, they just got noisier. One prompt variant produced a 680g portion for a normal dinner. Adding instructions gave the model more room to be creatively wrong.
So we split the problem
Once we stopped treating "calories in this photo" as one question, the architecture became obvious:
- The model answers what it's good at: identify the dish, return per-100g composition.
- The database answers what it knows better: verified macros from national food-composition sources, keyed to the identified food.
- The user answers what neither can know: the actual portion, via a one-tap gram adjustment, pre-filled with the model's estimate.
That last step is the part teams tend to skip because it feels like admitting defeat. It isn't. The user is standing in front of the food. They know it's a big portion. Confirming or nudging a number takes under a second, and it converts a systematically biased guess into something the user actually trusts — which matters more than the raw accuracy figure, because a number people distrust gets abandoned regardless of how good it is.
Score every macro separately
We ran reference meals with known ground truth (weighed, with composition calculated from reference data) across several models and compared average error.
The single most useful thing that came out of it wasn't the ranking — it was discovering that a blended accuracy number hides the failure that actually matters.
One model we evaluated got calories nearly right — within about 2% — while overshooting protein by nearly 60%. If we'd scored it on "calories accuracy" alone, it would have looked like the winner. It wasn't. Protein error is what users notice and complain about, because protein targets are the thing people actively chase. Ours surfaced as a support ticket reading, roughly, "two eggs is not 26g of protein" — and the user was right.
So: score each macro as its own metric. Calories, protein, fat, carbs. The one that's silently broken is the one that generates support load.
Stability deserves its own test too. Some models returned identical values across repeated runs on the same image; others swung by 20% (one gave three noticeably different calorie figures for the same chicken across three calls). For a tracking app, a number that changes when you rescan the same plate is worse than a number that's slightly off but consistent — users read the variance as the app being broken.
Two counterintuitive things about image input
Bigger images can be free. We assumed downscaling would cut token costs, so we tested several resolutions on the same photos. The token count came back essentially identical across all of them, because the provider tiles images into a fixed budget. But at the smallest resolution, accuracy dropped noticeably. We were about to make the app worse to save money we were never actually spending. Measure before you optimise.
JPEG quality above roughly 80 bought nothing. Higher quality produced several times the bytes and no accuracy gain. On one model it actually shifted estimates upward.
Also worth knowing: reasoning/"thinking" modes roughly tripled latency on this task for no measurable accuracy benefit, and support for disabling them is inconsistent — some model variants reject the config outright with a 400. If you're switching between models, that setting needs to be conditional rather than global. We lost a little time to that one.
For what it's worth, we settled on Gemini 3 Flash. I'd resist reading that as a recommendation, though — model rankings in this space go stale in months, and the right answer depends entirely on your own reference set. The transferable part isn't which model we picked; it's that we had a weighed reference set to pick it with.
The database is the unglamorous half
The scanner gets the attention, but a fast scan on top of bad data is just fast nonsense.
Most food databases in this category are user-submitted, which is why the same food shows up with five different calorie counts and why "wrong food data" is one of the most common complaints across app store reviews in the category (we read a lot of them — the pattern is remarkably consistent). We built ours from national food-composition sources instead: millions of foods with verified macros, deduplicated and normalised, which is a genuinely tedious data-engineering problem and absolutely not the fun part of the project.
But it's what makes the model's output checkable. When the model says "chicken breast," we don't take its per-100g numbers on faith — we resolve the identified food against a known-good record. The model does recognition. The database does nutrition. Neither is asked to do the other's job.
What I'd tell someone starting this
- Decompose the estimate before you evaluate it. Density and mass fail differently; a single accuracy number hides which one is broken.
- Score every output dimension separately. The macro that's silently wrong is the one that generates support tickets.
- Test stability, not just accuracy. Run the same input repeatedly. Variance is a product bug even when the mean is fine.
- Put the human where the model is structurally blind. Not as a fallback for failure — as a designed step. There's no depth data in a flat photo; asking the model to hallucinate it harder isn't engineering.
- Measure your "optimisations." Ours would have degraded quality to save zero tokens.
The finished thing is BodyPal if you want to see where this landed — photo in, adjust the grams, done. But the architecture generalises well beyond food: any time you're asking a vision model for a physical quantity it has no way to observe, you're better off splitting the question and letting the model answer only the half it can actually see.
Happy to go deeper on the evaluation methodology in the comments — especially curious how others are handling the "model is confidently wrong about a physical quantity" problem in their own domains.
