🏠 HomeLab Chronicles: Episode 3 - Ghost in the Mesh

Hey all 👋

Last episode I said the network was going to need luck instead of the laptop. I'd like to formally apologize to the network. It didn't need luck. It needed an exorcism.

The task itself was supposed to be the single most boring item on the entire mission list: Step 2, remote SSH access. Generate a key, copy it over, done, never touch a keyboard directly on this machine again. Ten minutes, tops. I've done it a hundred times on a hundred boxes.

This time it took an evening, a diagram, and a newfound distrust of my own WiFi.

🔑 The Part That Was Supposed to Be Boring

Standard stuff, nothing to see here:

ssh-keygen -t ed25519 -C "msi@homelab"
ssh-copy-id msi@192.168.68.110
ssh msi@192.168.68.110

Key on the client, key copied to the server, connect, no password, go live my life. That's the whole plan. It fell apart before I even got through it.

🚫 Connection Refused, Apparently

$ ssh msi@192.168.68.110
ssh: connect to host 192.168.68.110 port 22: Connection refused

Refused. Not timed out — refused, like the server had looked at me personally and said no. Which felt extra unfair, because when I actually checked the server, it had done nothing wrong:

$ systemctl status ssh
● ssh.service - OpenBSD Secure Shell server
     Active: active (running)

$ sudo ss -tlnp | grep :22
LISTEN  0  128  0.0.0.0:22  0.0.0.0:*  users:(("sshd",...))

sshd up, listening on every interface, no firewall even configured yet to blame. By every metric the server had access to about itself, it was innocent. And yet.

📡 Clue One: A Ping With Main Character Energy

Before spiraling into SSH configs, I did the normal thing and pinged it.

$ ping 192.168.68.110
64 bytes from 192.168.68.110: icmp_seq=1 ttl=64 time=312 ms
64 bytes from 192.168.68.110: icmp_seq=2 ttl=64 time=298 ms

It answered! Great, host's up, must be an SSH-level problem — except 300ms on a LAN isn't a number, it's a cry for help. Same-subnet pings on wired gear should be sub-millisecond. 300ms is what you get pinging a different continent, not a laptop three feet away on the same switch. I almost let that slide. I am glad I didn't.

🕵️ Clue Two: The ARP Table Doesn't Lie (Unlike Ping, Apparently)

Here's the part nobody warns you about: a ping reply only proves something answered at that IP. It doesn't prove it was the thing you meant. On a LAN, IP-to-hardware resolution happens over ARP, and your machine caches whatever answer it got first. Wrong cache, wrong destination — for every protocol, ping included.

$ ip neigh show 192.168.68.110
192.168.68.110 dev eth0 lladdr aa:bb:cc:11:22:33 REACHABLE

Then, straight from the server's own mouth:

$ ip link show enp0s3
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP
    link/ether aa:bb:cc:44:55:66 brd ff:ff:ff:ff:ff:ff

aa:bb:cc:11:22:33 versus aa:bb:cc:44:55:66. Two different pieces of silicon, one IP address, and my laptop had confidently bet on the wrong one.

👻 So Who's the Impostor?

Somewhere on my network, something else was also answering to .110. And once I actually thought about my network topology instead of just interrogating my server, I had a solid suspect.

Here's the setup: fiber comes into a modem, the modem feeds a TP-Link mesh router, and that mesh router has an extender hanging off it to cover the dead zone at the far end of the apartment.

Internet → Modem → TP-Link Mesh (main node)
                         │
                         ├── extender ── tp-link device, same IP as my server
                         │
                         └── my server (when the mesh feels like it)

Mesh systems are supposed to make all of this invisible — one big happy network, roam anywhere, nobody notices the seams. In practice, an extender bolted onto a mesh is still, underneath, its own little hop with its own opinions about who's connected where. My working theory: something on the extender's side ended up holding .110 at the same time my server did, and depending on which hop answered the ARP broadcast first, my laptop's cache latched onto the extender's guest instead of my actual box.

• Ping worked because whatever was squatting on .110 answered ICMP just fine — a real, alive device, just not mine.
• SSH got refused, not timed out, because a refusal means the SYN reached a live host that slammed the door on port 22 — nothing was listening there. A timeout would've meant something dropping packets silently. A refusal meant I'd reached a machine. Just not the machine.
• The 300ms latency fits the theory too — an extra wireless hop through an extender is exactly the kind of detour that turns sub-millisecond LAN pings into "did this packet go on vacation" pings.

Nothing here was actually broken. It was all working exactly as designed. The design just never accounted for me.

🛠️ The Fix: Stop Negotiating, Just Move

I could've gone spelunking through the mesh app's client list hunting for the impostor. I did not have that kind of patience left in me. I gave the server a new, boring, unclaimed IP instead.

sudo nano /etc/netplan/99-static.yaml

network:
  version: 2
  renderer: networkd
  ethernets:
    enp0s3:
      dhcp4: false
      dhcp6: false
      addresses:
        - 192.168.68.230/24
      routes:
        - to: default
          via: 192.168.68.1
      nameservers:
        addresses:
          - 1.1.1.1
          - 8.8.8.8

sudo netplan try

netplan try deserves a fan club. It applies the change immediately and auto-reverts after 120 seconds unless you confirm — extremely relevant when you're one YAML indent away from locking yourself out over the very connection you're using to make the change. Confirmed the box was still reachable, then made it permanent:

sudo netplan apply

✅ Receipts

$ ping 192.168.68.210
64 bytes from 192.168.68.210: icmp_seq=1 ttl=64 time=0.91 ms

$ ip neigh show 192.168.68.210
192.168.68.210 dev eth0 lladdr aa:bb:cc:44:55:66 REACHABLE

Sub-millisecond, and the MAC finally matches the server's actual NIC. Proof, at last, that I was talking to my own hardware.

🔑 Back to the Boring Part, Finally

ssh-copy-id msi@192.168.68.210
ssh msi@192.168.68.210

Straight in. No password. The ten-minute task, delivered roughly one evening and one network diagram late:

sudo nano /etc/ssh/sshd_config
# PasswordAuthentication no

sudo systemctl restart ssh

🧠 What This Taught Me

• Ping proves a device answered. Not which device. Don't let it clear a network problem it never actually investigated.
• "Refused" and "timed out" are different diagnoses wearing the same "it didn't work" costume — refused means a live host said no, timeout means something's silently eating your packets. Learn to tell them apart fast.
• Weird LAN latency is a symptom, not background noise. Sub-millisecond is the bar; three hundred milliseconds is a red flag wearing a ping reply as a disguise.
• ip neigh plus ip link settles an identity crisis in about ten seconds.
• Mesh WiFi plus a bolted-on extender is a surprisingly good recipe for exactly this kind of ghost — worth remembering before blaming the server for anything again.
• Reserve the address at the router level (MAC-based DHCP reservation), not just on the box. A static IP the router doesn't know about is one collision away from becoming somebody else's problem — namely mine, again.

📋 Quick Reference (For Skimmers)

🚀 What's Next

Given how well "one server, one IP" just went, the obvious next move is clearly to add more servers and more IPs. So: the Dell Inspiron isn't staying benched after all — it's getting reflashed with the same minimal Ubuntu Server + LXD/LXC combo that worked the first time, except this time with a job description it can actually survive, alongside an old Raspberry Pi I dug out of a drawer. Three nodes, one fledgling cluster.

And since apparently I haven't suffered enough at the hands of networking this episode, the plan is to bring in Cilium — eBPF-based pod networking, or as I now think of it, round two against the concept of "IP address" as a stable idea — to actually handle traffic across all of it instead of hoping defaults figure it out on their own.

Given this episode's track record, I would like the record to reflect I have concerns.

💬 Final Thoughts

Ghost: identified, evicted. SSH: finally, actually working. Trust in my own home WiFi: still recovering.

Stay tuned. Popcorn 🍿 mandatory this time, not optional.