I Built a Mesh Network That Works When the Internet Doesn't

What happens when the power grid fails? When cellular towers collapse? When the internet simply disappears?

You get silence. Isolation. No way to call for help.

I built OEMN (Offline Emergency Mesh Network) to answer this: What if your laptop could become part of an automatic emergency communication system, no infrastructure required?

THE PROBLEM: CONNECTIVITY DOESN'T SCALE

During the Turkey-Syria earthquakes in February 2023, communication networks collapsed. The Indian Ocean tsunami. Wildfire zones. Disaster areas where people are trapped but can't reach anyone.

Why? Because infrastructure is fragile.

Cell networks are centralized. One point of failure cascades. The internet requires thousands of miles of cable and dozens of intermediaries. As long as even one link breaks, you're isolated.

But what if you didn't need infrastructure at all?

What if laptops, phones, and devices could talk to each other directly, forming their own network on-the-fly? Even if cellular fails. Even if the internet is gone.

That's what OEMN does.

WHY IT'S ACTUALLY HARD

Building a mesh network looks simple on a whiteboard: nodes talk to neighbors, messages hop across multiple nodes, done.

The reality is three layers of hard problems:

PROBLEM 1: HOW DO NODES FIND EACH OTHER?

Without DNS, DHCP, or a central server, nodes are just devices on a network. They don't know who exists, where they are, or how to reach them. You need peer discovery: a way for node A to say "I exist" and have node B hear it.

PROBLEM 2: HOW DO YOU ROUTE MESSAGES EFFICIENTLY?

In a mesh of 50 nodes, there could be hundreds of paths from A to Z. Which one is fastest? Which one still works after node failures? You need topology awareness: every node must build a map of the entire network and compute optimal routes.

This is the classic shortest-path problem. But it needs to recompute in real-time as nodes join and leave.

PROBLEM 3: HOW DO YOU GUARANTEE DELIVERY?

UDP (the transport layer) doesn't guarantee messages arrive. They can be lost, duplicated, reordered, or corrupted. You need reliability: ACKs, retries, deduplication. But you can't do what TCP does (overhead is too high). You need something in between.

Add encryption. Add replay protection (so attackers can't resend old messages). Add thread-safe concurrency so your listener, router, and sender don't corrupt each other's state.

Now you have a mesh network.

THE ARCHITECTURE: FOUR THREADS, ONE SHARED QUEUE

OEMN's elegance comes from its simplicity.

FOUR THREADS, each with a single responsibility:

Listener Thread
receives packets from UDP, validates encryption, updates topology

Router Thread
runs Dijkstra, builds shortest-path routing table

Send Queue (mutex-protected, condition variable signaled)

Sender Thread
transmits packets, handles ACKs and retries

The listener doesn't transmit. The sender doesn't route. The router doesn't listen. Each thread owns its data. They communicate through one shared send queue.

This is THREAD SAFETY BY DESIGN, not by accident.

Here's what happens when a packet arrives:

1. Listener validates AES-256-GCM encryption and checks replay protection
2. Listener checks if this is a topology update (TOPO message)
3. If topology changed, listener signals router to recompute
4. Router runs Dijkstra algorithm, updates routing table
5. If packet is for us, listener enqueues it to send an ACK
6. Sender dequeues, transmits ACK, waits for confirmation
7. If no ACK within timeout, sender retries with exponential backoff

The entire system is coordinated by a single atomic shutdown flag. When you hit Ctrl+C, all threads gracefully drain their queues and exit.

DIJKSTRA SHORTEST-PATH ROUTING

Here's where topology matters.

Every node runs Dijkstra's algorithm to compute the shortest path to every other node. A naive implementation is O(V²). That's too slow for real-time rerouting.

OEMN uses a BINARY MIN-HEAP, reducing complexity to O((V + E) log V). For 50 nodes, this is fast enough to recompute whenever the network topology changes.

Here's the actual implementation from router.c:

static void run_dijkstra_heap(const oemn_router_t *r, int src, uint32_t *dist_out, int *parent_out)
{
    int n = r->n_verts;
    for (int i = 0; i < n; ++i) {
        dist_out[i] = OEMN_INF_DIST;
        parent_out[i] = -1;
    }

    heap_ent_t *heap = calloc(OEMN_HEAP_CAP, sizeof(*heap));
    int hs = 0;

    dist_out[src] = 0;
    heap_push(heap, &hs, (heap_ent_t){0u, src});

    while (hs > 0) {
        heap_ent_t x;
        if (heap_pop(heap, &hs, &x) != 0)
            break;
        int u = x.v;
        if (x.d != dist_out[u])
            continue;
        for (int k = 0; k < r->deg[u]; ++k) {
            int v = r->nbr[u][k];
            uint32_t ew = r->w[u][v];
            if (ew == OEMN_INF_DIST)
                ew = OEMN_EDGE_DEFAULT_COST;
            uint32_t nd = dist_out[u] + ew;
            if (nd < dist_out[v]) {
                dist_out[v] = nd;
                parent_out[v] = u;
                heap_push(heap, &hs, (heap_ent_t){nd, v});
            }
        }
    }
    free(heap);
}

The routing table maps: destination → next hop → distance in hops

When you send a message to node 5, it automatically routes through the neighbor closest to node 5. If that neighbor fails, the next topology update (every 2-4 seconds) recomputes and reroutes through an alternate path.

This is AUTOMATIC FAILOVER. No manual intervention. No central coordinator.

ENCRYPTION & REPLAY PROTECTION: DEFENDING AGAINST ATTACKS

I use AES-256-GCM (via OpenSSL) or ChaCha20-Poly1305 (via libsodium) for authenticated encryption.

Every message is encrypted with a per-source key, and every message has a sequence number embedded in its ID.

But encryption alone isn't enough. I had to prevent replay attacks:

Attacker sees: "Node A sends $100 to Node B"
Attacker captures the encrypted packet
Attacker replays the same encrypted packet
Node B doesn't know it's a duplicate
Attacker just transferred $100 twice

This is trivial to execute on UDP because there's no session state.

OEMN solves this with a PER-SOURCE SLIDING WINDOW:

For each source, track the last 64 received sequence numbers
If we see sequence N where N <= (current_seq - 64), reject as duplicate
If we see N where (current_seq - 64) < N < current_seq, reject as duplicate
Only accept N > current_seq

Here's the actual code from replay.c:

int oemn_replay_check_and_update(oemn_replay_t *r, uint32_t src_id, const uint8_t msg_id[16])
{
    uint64_t seq = msg_seq_from_id(msg_id);

    pthread_mutex_lock(&r->mu);
    int idx = -1;
    for (int i = 0; i < OEMN_MAX_PEERS; ++i) {
        if (r->slots[i].used && r->slots[i].src_id == src_id) {
            idx = i;
            break;
        }
    }

    oemn_replay_slot_t *s = &r->slots[idx];

    if (seq > s->max_seq) {
        uint64_t gap = seq - s->max_seq;
        if (gap >= 64)
            s->window = 1ULL;
        else
            s->window = (s->window << gap) | 1ULL;
        s->max_seq = seq;
        pthread_mutex_unlock(&r->mu);
        return 0;  // Accept: fresh sequence
    }

    uint64_t delta = s->max_seq - seq;
    if (delta >= 64) {
        pthread_mutex_unlock(&r->mu);
        return 1;  // Reject: too old
    }

    uint64_t mask = 1ULL << delta;
    if (s->window & mask) {
        pthread_mutex_unlock(&r->mu);
        return 1;  // Reject: duplicate
    }

    s->window |= mask;
    pthread_mutex_unlock(&r->mu);
    return 0;  // Accept: fresh
}

This costs 64 bits of memory per source. At 50 nodes, that's 400 bytes total. Negligible.

The attacker can't replay old messages. They can only forge new ones (which requires the encryption key, which they don't have).

BUFFER POOLING: FROM MALLOC THRASHING TO SLAB ALLOCATION

One thing I discovered: malloc/free on every message is a performance killer.

OEMN uses pre-allocated SLAB BUFFERS:

From fwd_pool.h:

#define OEMN_FWD_POOL_SLOTS 48
#define OEMN_FWD_BUFSZ (OEMN_HDR_SIZE + OEMN_MAX_PAYLOAD)

typedef struct {
    pthread_mutex_t mu;
    uint8_t slab[OEMN_FWD_POOL_SLOTS][OEMN_FWD_BUFSZ];
    int free_stack[OEMN_FWD_POOL_SLOTS];
    int nfree;
} oemn_fwd_pool_t;

Instead of malloc-ing on every message, OEMN allocates one giant slab upfront. Each thread acquires a buffer from the pool, uses it, then returns it. No heap fragmentation. No allocation overhead.

This is a simple optimization, but it matters more than you'd expect. Systems that malloc constantly spend more CPU managing memory than doing useful work.

LESSONS LEARNED

1. THREAD SAFETY IS HARD BUT WORTH GETTING RIGHT

I spent weeks hunting data races using ThreadSanitizer. The payoff: zero race conditions under stress testing. This is how production systems should be built. Most open-source projects skip this. Don't.

1. TOPOLOGY CHANGES ARE YOUR PERFORMANCE BOTTLENECK

Dijkstra is O((V + E) log V). For 50 nodes, it's still fast. But if you rerun it on every message (amateur mistake), you're doing wasteful computation.

OEMN only recomputes on topology changes. This is the right optimization target.

1. UDP ISN'T UNRELIABLE, IT'S JUST HONEST

TCP hides failures through retries and buffering. UDP shows them immediately.

In a disaster scenario, seeing failures is better than silently dropping 10% of messages under congestion (which TCP does). You need visibility into what's happening.

1. BUFFER POOLING WINS OVER MALLOC/FREE

Pre-allocated slabs eliminate allocation overhead. The threading model means each buffer acquisition is an O(1) stack pop. No freelist searching. No fragmentation.

1. WRITING THINGS IN C MAKES YOU UNDERSTAND HARDWARE

Python and Rust abstract away memory. C forces you to think about:

• Cache line locality
• Malloc overhead
• Mutex contention
• Thread synchronization primitives

By the end, you understand networking differently.

WHAT'S NEXT

The current roadmap has three priorities:

1. LATENCY-WEIGHTED ROUTING: Currently, OEMN finds the shortest path (fewest hops). But 10 hops of low-latency LAN is better than 3 hops of high-latency satellite. Adding EWMA-based RTT measurement per edge is next.

2. DELTA TOPOLOGY UPDATES: Right now, nodes send full topology advertisements. With 100+ nodes, that's bandwidth-heavy. Delta updates (send only changes) could cut bandwidth by 10x.

3. FUZZING: I want to throw 10,000 random packet corruptions at the protocol and watch it survive. libFuzzer + AddressSanitizer.

WHY THIS MATTERS

Most software engineers will never think about resilience. You build on Kubernetes, AWS, and 99.99% uptime guarantees.

But when infrastructure fails, when the easy assumptions break, mesh networks become critical infrastructure.

Understanding how to build one teaches you:

• How to program without central authority (coordination, consensus)
• How to handle adversarial networks (encryption, replay protection)
• How to design concurrent systems that don't have race conditions
• How to measure and optimize for real-world constraints (packet loss, node failures)

OEMN isn't just a project. It's a masterclass in systems engineering.

But more importantly: it works. No dependencies. No cloud. Just peers talking to peers.

TRY IT

OEMN is on GitHub: github.com/bharqav/oemn

To run the demo locally:

make
./oemn --id 1 --port 7777
./oemn --id 2 --port 7778

Then connect them (in the CLI):

# on node 1: peer add 2 127.0.0.1 7778
# on node 2: peer add 1 127.0.0.1 7777
# on node 1: send 2 hello

Watch the message travel. Watch the ACK come back. Watch the routing table update in real-time.

The code is production-grade:

• 48-slot buffer pool
• Thread-safe by design
• Dijkstra shortest-path routing
• AES-256-GCM encryption
• Per-source replay protection
• Graceful shutdown

Read the docs in the repo. They're detailed. Run the benchmarks yourself. See what performance you get on your hardware.

If you're curious about how systems really work, build your own. Build a mesh network. Build an autograd engine. Build a shell.

Don't just use other people's abstractions. Understand the layer beneath.

That's where the real learning happens.

OEMN is open-source (MIT License). Contributions welcome: github.com/bharqav/oemn