I Had 5 Servers Sending Metrics But Couldn't Tell Which One Was Which

What happened

I had one server. It sent metrics (CPU, errors, etc.) to my monitoring dashboard. Fine.

Then I added more servers same service, running on 4-5 machines now.

All the metrics from all the machines showed up in the same dashboard, but with no way to tell which machine sent which metric. If CPU spiked, I couldn't tell if it was server 1 or server 4. If errors went up, same problem.

This is useless when something breaks. "Something is wrong" isn't helpful I need "server 3 is wrong."

Why it happened

Each server was supposed to tag its own metrics with its own name like a label saying "this metric came from server-3."

I had set up a config value meant to do exactly that. But when I checked, that name was never actually reaching the code that sends metrics. The setting existed in a file, but it wasn't making it into the running program the way I assumed it would.

So every server, silently, was sending metrics with no name attached or the same default name. No error. No warning. It just quietly didn't work.

How I fixed it

Instead of hoping a config file setting would "just work" and reach the right place automatically, I made it explicit:

• When a server starts up, its name gets set directly, as a hard requirement.
• That name gets passed by hand, step by step, into the exact piece of code that sends metrics.
• Nothing is assumed to "just be available" it's handed over directly, on purpose, every time.

I checked it worked by looking at the actual metrics in the dashboard afterward and confirming each server's name showed up correctly.

The real lesson

If your app depends on a setting "just working" through some automatic config-loading magic double check it's actually arriving where you think it is. Don't assume. Go look.

And before you scale from 1 server to many: make sure you can tell them apart in your monitoring before you actually need to, during a real problem, at 2am, with no way to know which machine to even look at.