216 passing tests. A feature that was completely broken. Here is the gap between those two facts, and what I changed afterwards.
The feature
I am building an encrypted messenger. Messages are end to end encrypted, and the server relaying them cannot read anything. That part worked.
What I added was offline delivery. If you message someone whose app is closed, the server should hold the message, hand it over when they come back, then delete it. Nothing kept longer than it needs to be.
I wrote it. I wrote tests for it: unit tests for the storage layer, integration tests against a real Postgres, end to end tests over real WebSocket connections. Every one passed.
Then I ran it against the deployed build, closed one browser, sent two messages, and reopened.
Nothing arrived.
Bug 1: a race my tests could not have
The server hands over held messages the instant the connection opens. The client, meanwhile, loads its decryption keys from browser storage, which is asynchronous.
So the messages arrived before there was anything to decrypt them with, and were dropped.
My tests never saw it because in tests the key loading was effectively instant. The window between "connected" and "ready to decrypt" existed only on a real machine doing real I/O.
// Before: connect, then restore. The gap is where messages die.
socket = connect(room)
state = await loadSavedState()
// After: restore, then connect. No gap.
state = await loadSavedState()
socket = connect(room)
The lesson: if your test setup completes instantly and production does not, you are not testing the same system. Anywhere your code says await between "we are live" and "we are ready", something can arrive in between.
Bug 2: acknowledging the wrong event
This one was worse, because it destroyed data.
The server deletes a held message once the client confirms it. My client confirmed on arrival.
Arrival is not delivery. The message had arrived at the socket, but the app had not decrypted it, had not stored it, had not shown it to anyone. The server deleted it anyway. The message was gone from both sides.
The fix was to confirm only after the message had actually been handled, and to leave anything unhandled with the server so it comes again next time.
// Before
onMessage(m) { show(m); confirm(m.id) } // confirm fires even if show() threw
// After
onMessage(m) { if (handled(m)) confirm(m.id) } // unhandled stays on the server
There is one deliberate exception: a message this device can never read, because it belongs to a conversation whose keys are gone, is still confirmed. Asking for it again would not help.
The lesson: "received" and "handled" are different events. Only one of them is safe to delete on. If you are building any at-least-once delivery, be precise about which one you are acting on.
Bug 3: a cleanup path that never ran
Once messages were flowing again, I looked in the database and found a copy of every message I had sent, including ones delivered instantly while both people were online.
The logic was: store every message, delete it when the recipient confirms. But confirmation only happens for messages delivered from storage. A message delivered live goes straight to the other person, so nothing ever confirms it, so nothing ever deletes it.
An active conversation was quietly accumulating a server side copy of itself, and only a weekly sweep cleared it.
The fix was to store only when nobody is there to receive it.
const othersPresent = room.size - 1 > 0
broadcast(room, message)
if (!othersPresent) hold(message) // only when it cannot be delivered now
The lesson: a delete that only runs on one code path is not a delete. When you write "we clean this up later", check that every path reaches the cleanup, not just the one you had in mind.
This one also mattered beyond storage. A server holding copies of an entire conversation is a very different privacy claim from one holding a message for a few seconds.
Bug 4: state that outlived its owner
Removing a chat deleted the messages but left the encryption keys behind. Re-add the same person and the app cheerfully tried to resume a conversation the other side had thrown away. The two ends no longer agreed on anything, and nothing could be decrypted.
The lesson: when you delete a thing, delete everything derived from it. Orphaned state does not sit there harmlessly; it gets picked up later by code that assumes it is still valid.
What I actually changed
Not "write more tests". I had plenty, and they were all green while the feature did not work at all.
What these bugs had in common is that every one of them lived in the gap between components: between the socket opening and the keys loading, between the server's idea of delivered and the client's, between one delivery path and another. Each component behaved correctly in isolation. The system did not.
So the rule I now follow is simple: anything that touches real storage, a real network or a real other machine gets exercised against the deployed build before I call it done. Not the dev server. Not a harness. The thing I actually shipped, with a real second device, doing the thing a user would do.
That one test run found four bugs, two of which lost user data. It took about ten minutes.
The honest footnote
I found these because I went looking. The feature had shipped in the sense that it was written, reviewed, tested and deployed. If I had trusted the green checkmarks, it would have reached testers as a messenger that silently ate messages.
Tests tell you the parts work. They are much worse at telling you the whole thing does.
This is from a privacy-focused messenger I am building on Midnight, where identity is proved on-chain with zero knowledge proofs and messages never touch the chain. A longer technical write-up, including the on-chain contract and real transactions you can inspect, is here on the Midnight forum.
