The Hardest System I Ever Built Was for Patients Who Could Not Afford for It to Fail

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

There is a version of this story where I walk you through the technical challenges in a clean, detached voice. Structured. Professional. Composed.

This is not that version.

Seven months. A hundred pages of documentation written alongside the code, not after it. A live system serving rural clinics in Delta State, Nigeria. And three bugs so specific and so cruel that each one felt designed to find the exact gap between what I thought I knew and what I actually knew.

The Delta Health Information and Appointment Booking System was my final year project at the University of Port Harcourt. That sentence makes it sound smaller than it was. This was a real system for real clinics, serving patients who travel for hours to reach a healthcare facility only to find the clinic is full, the doctor is not in, or nobody told them their appointment was cancelled. These are not hypothetical users. These are people whose access to healthcare depends on whether the software works.

That is a different kind of pressure than getting a good grade. And it did not start at deployment. Before a single line of code was written, I had to pitch the entire concept to my university supervisor, the Head of Department, the Faculty Dean, and an external examiner. The system had to be defensible not just technically but practically. It had to demonstrate that it solved a real problem that real people in Delta State were actually facing. I was presenting to people with decades of experience who would ask questions I had not thought of yet. That kind of scrutiny either sharpens you or breaks you. I chose to let it sharpen me.

The stack was React.js, Node.js, PHP with Laravel, MySQL, AWS, and Docker. Communication channels included SMS gateways, WhatsApp Business API, and USSD because not every patient in rural Delta State has a smartphone. The system had to run on 2G connections, on entry-level Android phones with 2GB RAM, in areas where electricity is inconsistent and internet is a bonus, not a guarantee.

Three bugs tested everything I thought I understood about building for real environments.

Bug 1: The Clinic That Appeared in the Ocean

I was proud of the location feature. Genuinely proud.

When a patient booked an appointment, the system pulled the GPS coordinates stored in the database and surfaced nearby clinics in order of proximity. For patients in rural areas who might not even know the name of the nearest clinic, this mattered. They were not booking at one specific facility. They were asking which healthcare facility they could actually get to.

The clinics table stored latitude and longitude as DECIMAL(10,7) fields. The booking module queried them, the frontend rendered them, and in development everything worked perfectly. Then we moved into field testing with real clinic data that real administrators had entered manually from printed forms.

That is when a clinic in Delta State appeared off the coast of Ghana.

Not metaphorically. On the map. In the ocean.

I sat back and just stared at it for a minute.

The database was not wrong. MySQL had accepted every value without complaint because the values were technically valid decimals. The problem was simpler and more human than that. Clinic profiles were being set up by administrators entering coordinates manually from printed reference sheets, and transcription errors crept in during that process. Some records had latitude and longitude completely swapped. Others had digits missing or duplicated from misreading handwritten numbers. Six degrees of longitude and five degrees of latitude are both perfectly valid decimal numbers. MySQL stored them without question. They just pointed to the ocean.

The fix had two parts. I added a server-side validation layer in the Node.js API that checked every incoming coordinate pair against Nigeria's geographic bounding box before writing to the database. If a coordinate fell outside that box, the record was flagged for review and not saved. Then I wrote a one-time audit script to go through every existing record, pull the outliers, and mark them for manual correction.

// Validate coordinates fall within Nigeria's bounding box
function isValidNigerianCoordinate(lat, lng) {
  const NIGERIA_BOUNDS = {
    minLat: 4.0,
    maxLat: 14.0,
    minLng: 2.7,
    maxLng: 15.0
  };
  return (
    lat >= NIGERIA_BOUNDS.minLat &&
    lat <= NIGERIA_BOUNDS.maxLat &&
    lng >= NIGERIA_BOUNDS.minLng &&
    lng <= NIGERIA_BOUNDS.maxLng
  );
}

if (!isValidNigerianCoordinate(lat, lng)) {
  return res.status(400).json({ 
    error: 'Coordinates fall outside expected geographic boundary. Please verify and resubmit.' 
  });
}

Six clinics had swapped coordinates. Three had digits that had been misread during manual entry. All nine were placing facilities somewhere they had no business being.

The lesson: Validate coordinates against geography, not just against data types. A number can be perfectly valid and completely wrong at the same time.

Bug 2: The Notifications That Went Nowhere

This one hurt differently.

The communication layer was the part of this system I had worried about most during planning. Patients needed appointment confirmations and reminders through whatever channel their phone supported. SMS for basic phones. WhatsApp for smartphones. USSD as a fallback. The notifications table tracked every outgoing message with a type, content, status, and timestamp.

During integration testing, everything looked correct. API calls returned 200 status codes. The notifications table showed a status of sent for every message dispatched. I moved into field testing feeling reasonably confident.

Then a clinic administrator in the pilot told me that none of the patients had received their reminders.

None.

I pulled up the notifications table. Every record showed sent. I checked the API logs. Every call had returned 200. From every technical indicator I had, the messages were going out.

They were not going out.

It took three days to understand what was actually happening. The SMS gateway had a two-stage delivery model I had not fully read in the documentation. When our server sent a batch of notifications, the gateway accepted them and returned 200. That response confirmed the gateway had received the messages. It did not confirm delivery to the recipient. Actual delivery was asynchronous. The gateway would attempt delivery and then, if we had registered a webhook URL, call back to update the final delivery status. We had not registered a webhook. Messages were entering the gateway's queue, some delivering and some failing, and we had no visibility into any of it because we were reading sent as if it meant delivered.

It did not mean delivered. It meant accepted by the gateway. Those are different things.

// What we had before (wrong)
const sendNotification = async (userId, message) => {
  const response = await smsGateway.send({ to: phone, body: message });
  if (response.status === 200) {
    await db.notifications.update({ status: 'sent' }); 
    // WRONG: this is gateway receipt, not delivery
  }
};

// What we built after (correct)
const sendNotification = async (userId, message) => {
  const response = await smsGateway.send({ 
    to: phone, 
    body: message,
    webhookUrl: `${process.env.BASE_URL}/webhooks/delivery-receipt`
  });
  await db.notifications.update({ status: 'queued' }); // Honest status
};

// Webhook handler for actual delivery confirmation
app.post('/webhooks/delivery-receipt', async (req, res) => {
  const { messageId, status } = req.body;
  await db.notifications.update({ 
    messageId, 
    status: status === 'delivered' ? 'delivered' : 'failed' 
  });
  if (status === 'failed') {
    await retryQueue.add({ messageId }); // Queue for retry
  }
  res.sendStatus(200);
});

After these changes, confirmed delivery in the pilot went from what I can only describe as catastrophically low to 94%. The remaining 6% were phones that were switched off or numbers that were unreachable, which is a real baseline in any rural environment.

The lesson: A 200 from a messaging API is an acknowledgement, not a guarantee. Read the documentation past the quickstart. Implement webhooks. Never trust a status you did not actively confirm.

Bug 3: The Bug That Only Existed in the Real World

This one was the most frustrating because I could not reproduce it in development.

The system was built to be bandwidth-tolerant. The health education hub had offline caching. The UI was optimized for slow connections. I thought I had accounted for the network.

What I had not accounted for was what happens to a database transaction when a patient starts a booking, the connection drops mid-transaction, connectivity returns, and the patient hits submit again because nothing happened the first time.

During the field pilot we started seeing duplicate appointments. Same patient. Same clinic. Same time slot. Two confirmed bookings in the database. I could not reproduce this in development because our development environment had stable internet. The bug only existed where the internet was not stable, which was the exact environment this system was built for.

The sequence was this: patient initiates booking, client generates the request, connection drops before the request reaches the server, client shows a loading state, connection returns, patient re-submits, both requests eventually reach the server, both are accepted and written because nothing in the database logic detected they were the same request.

The fix was idempotency keys.

// Client side: generate a unique key per booking attempt
const initiateBooking = async (bookingData) => {
  const idempotencyKey = `booking_${userId}_${clinicId}_${slotId}_${Date.now()}`;

  // Store key locally before sending
  localStorage.setItem('pendingBookingKey', idempotencyKey);

  const response = await fetch('/api/appointments', {
    method: 'POST',
    headers: { 
      'Content-Type': 'application/json',
      'Idempotency-Key': idempotencyKey
    },
    body: JSON.stringify(bookingData)
  });

  if (response.ok) {
    localStorage.removeItem('pendingBookingKey');
  }
};

// Server side: check key before processing
app.post('/api/appointments', async (req, res) => {
  const idempotencyKey = req.headers['idempotency-key'];

  const existing = await db.appointments.findByKey(idempotencyKey);
  if (existing) {
    return res.status(200).json(existing); // Return original response
  }

  const appointment = await db.appointments.create({
    ...req.body,
    idempotencyKey
  });

  res.status(201).json(appointment);
});

I also added a client-side request state machine so the UI accurately reflected whether a booking was pending, confirmed, or failed. And I added local caching so a booking initiated during a dropped connection would retry automatically when connectivity returned.

The lesson: Test your system in the conditions it will actually run in. A stable development environment hides entire categories of bugs that only exist where infrastructure is unreliable. Idempotency is not a nice-to-have. It is the difference between one booking and two.

What Seven Months Felt Like

I want to be clear about something. These bugs were not exotic. They were not especially clever to find. What made them hard was that each one required me to step outside the assumptions I had built into my mental model and confront the reality of how real people in a real environment were actually using the system.

Administrators made transcription errors entering coordinates from printed sheets because the data entry process had no validation to catch them. Patients re-submitted forms because nothing visually confirmed their action had worked. A gateway returning 200 seemed like enough confirmation because in most environments it would be.

None of my assumptions was unreasonable. They just were not built for this specific context.

I was also documenting everything in parallel because I was building a system that had to outlast my involvement. The team maintaining this after my final year ended needed to understand not just what the code did but why every decision was made. Writing down the reasoning behind every fix, not just the fix itself, is something I carry into every project now.

The system is live at deltahealth.usestudybuddy.org. Patients in Delta State use it. Clinics that had no digital infrastructure now have a real-time view of their appointment flow and a way to reach patients before they make a wasted journey.

I am proud of what it does. I am prouder of how many times it almost didn't work and we fixed it anyway.

I am Tobore, a full-stack developer and Best Author on DEV for React writing. I built the Delta State Health Information and Appointment Booking System and I write about tools that actually change how developers work. Find more at dev.to/toboreeee.