The face that forgot how to count
Mosaic is the UI running on a robot's face and kiosk display — GLTF facial animation, tool results, conversational state, all streamed over NATS and rendered live. Most of the time it's seamless. But every so often, after a dropped Wi-Fi connection reconnected, something would look... off. A response would appear twice. A tool result that had already been shown would echo back into the UI a second time, like the robot was repeating itself for no reason.
Nothing was wrong in the steady state. The bug only ever showed up right after the network had a hiccup and recovered — which meant it was almost invisible in normal testing, and only ever bit in exactly the moment reliability mattered most: right when something had already gone a little wrong and the system was trying to recover.
Event-sourced UI, and a guarantee I hadn't questioned
Mosaic's state isn't just rendered directly — it's built from a stream of events coming over NATS: a tool call result here, a UI fragment there, a state update from the conversation pipeline. The store applies each event as it arrives and that's what ends up on screen.
That model works great — until you actually think about what "arrives" means over a message bus. NATS gives you at-least-once delivery. That's a deliberate, sensible guarantee: after a dropped connection, NATS will redeliver messages you might have missed, rather than silently lose them. Good default. But it means the same event can legitimately arrive twice.
My store didn't know that. It applied every incoming event as if "arrived" meant "new." Which is true almost all the time — until a reconnect happens, NATS does exactly what it's designed to do and redelivers a message you'd already processed, and the store dutifully applies it again.
Finding the gap between two different guarantees
This is the kind of bug that's obvious once you say it out loud and completely invisible while you're staring at the UI code, because the UI code isn't wrong — Svelte's rendering the state correctly. The state itself is wrong.
The real insight was separating two things that had been quietly conflated:
- A stream of updates — what NATS gives you: a sequence of events, possibly including redeliveries.
- The current true state — what the UI actually needs to render correctly.
I'd been treating the first as if it were automatically the second. It isn't. A replay stream tells you things happened; it doesn't promise you're hearing about each one exactly once, and it doesn't promise the order you get them in after a reconnect matches what you'd expect if nothing had ever dropped.
The fix: make "again" a no-op, and don't trust the stream blindly
Two changes, and they mattered in different ways:
1. Made event application idempotent. Every event now has an identity, and the store dedupes on that identity before it's allowed to mutate anything. A redelivered event, or an echoed tool result, becomes a no-op instead of a double-apply.
before: event arrives -> apply to store (always)
after: event arrives -> seen before? -> skip
-> not seen? -> apply + remember id
2. Added an authoritative resync path. After a reconnect specifically, the kiosk doesn't just trust whatever the replay stream sends first — it pulls the true current state directly. The event stream is great for staying in sync once you're already there; it's the wrong tool for re-establishing sync after you've been offline. Those are different problems, and I'd been using one mechanism for both.
The result
State stays correct through drops and redelivery, full stop. The kiosk can lose its NATS connection, reconnect, and the display doesn't quietly lie about what state it's in.
That matters more for a robot face than it might for a typical dashboard — a duplicated toast notification is mildly annoying; a robot's face flickering into a stale or doubled expression right in front of someone reads as broken, immediately, to anyone watching. UI bugs on embodied systems don't get to hide in a support ticket. They happen live, in front of a person.
Takeaways
- At-least-once delivery pushes idempotency onto you, the consumer. If your message bus can redeliver — and most production-grade ones can — "I received this event" and "this event is new" are two different claims. Conflate them and you will eventually double-apply something.
- A stream of updates isn't the same as a source of current truth. Replaying an event log is great for staying in sync; it's the wrong primitive for re-establishing sync after a gap. Sometimes the right move after a reconnect is to ask "what's actually true right now?" instead of "what did I miss?"
- Test the unhappy path on purpose. The steady-state path will always work in a demo. The reconnect/redelivery path is the one that's actually worth writing a deliberate test for, because it's the one nobody exercises by accident.
- Bugs on visible, embodied systems have zero grace period. A backend duplicate-processing bug might sit quietly in a log for a week. The same bug on a robot's face is a bug someone's watching happen. Prioritize accordingly.
Over to you
If you're building anything event-sourced on top of a message bus with at-least-once semantics — NATS, Kafka, SQS, whatever — I'd bet you have a version of this bug somewhere, even if it hasn't bitten you yet. Have you hit redelivery duplication before, and how did you handle dedup — event IDs, content hashing, something else?
I write about the systems-level bugs that show up when real-time UI meets distributed messaging — if that's your kind of problem too, follow along, and let me know what you're building.
