Hunting the Ghost in the Machine: Why Our Linux Servers Kept Spiking to 100% CPU (And Why the Monitoring Script Was Lying)

A deep dive into CPU delta calculations, isolated Docker metric samplers, and disabling rogue systemd timers.

The Telegram notification arrived at 03:14 AM.

ALERT: [prod-app-02] CPU utilization is at 100.0% (Threshold: >90.0% for 1m)

I rolled out of bed, grabbed my laptop, opened a terminal, and authenticated into the instance over SSH. I expected to find a runaway process or a denial of service attack consuming every compute cycle.

I ran htop.

Nothing was happening. The 15-minute load average sat at 0.12. Idle CPU hovered around 98.4%. The memory graph was flat. Aside from the standard background noise of systemd-journald and our container runtime, the machine was basically asleep.

I closed my laptop and went back to sleep. Two hours later, my phone buzzed again with the exact same alert on the exact same server.

When you log into a server that your monitoring stack claims is on fire and find an operating system that is barely breathing, you have a ghost in your system. The problem was not that our workload was overwhelming the hardware. The problem was that our monitoring daemon was calculating CPU usage incorrectly, our Docker healthcheck was creating its own CPU spikes, and a rogue systemd timer was waking up in the background and locking the disk.

Here is how we tracked down every single one of those issues, fixed the underlying mathematics, and silenced the false alarms for good.

The Illusion of Point-in-Time CPU Usage

Most developers and sysadmins think of CPU usage like a speedometer in a car. You look at the dial right now, and it tells you that you are driving at 65 miles per hour.

A CPU does not work that way.

A modern CPU core is a digital state machine governed by a clock frequency. At any given nanosecond, a processor core is either executing an instruction or it is halted waiting for an interrupt. It is either 100% busy or 0% busy. There is no physical state where a CPU transistor is operating at "42% capacity."

When an operating system reports that CPU usage is at 42%, it is reporting a statistical calculation averaged over a specific window of time:

CPU Usage % = (Time spent executing tasks / Total elapsed time) * 100

Without an elapsed time window (a delta), CPU percentage is a mathematically meaningless number.

Yet countless custom monitoring scripts, bot daemons, and internal status checks make the mistake of attempting to query CPU usage instantaneously. When a script asks the operating system for current CPU load without measuring a time difference between two discrete samples, it ends up reading whatever transient spike or kernel accounting tick happened to coincide with that exact millisecond.

If that millisecond happens to catch a context switch, a JSON parsing loop, or an encrypted TLS handshake, the calculation returns 100%.

Your monitoring daemon sees 100%, panics, and fires an alert webhook to your phone.

How Linux Actually Tracks CPU Cycles

To understand why custom monitoring scripts report garbage numbers, you have to look directly at the pseudo-filesystem where the Linux kernel exposes CPU metrics: /proc/stat.

Inspect /proc/stat on any Linux box:

head -n 1 /proc/stat

The kernel outputs a single line containing raw numbers:

cpu  2450123 12450 678910 89234120 145020 452 12050 0 0 0

These values are not percentages, and they are not seconds. They are cumulative CPU clock ticks, known inside the Linux kernel as jiffies, counted since the system booted. On most Linux distributions (including Ubuntu and Debian on x86_64), the kernel clock runs at CONFIG_HZ=250 or USER_HZ=100, meaning each tick represents 10 milliseconds or 4 milliseconds of processor time.

The ten numeric fields represent where the processor cores spent their ticks:

• user: Normal processes executing in user space (applications, web servers, scripts).
• nice: Low-priority user processes whose scheduling priority was reduced via nice.
• system: Kernel space execution (system calls, driver operations, context switching).
• idle: No runnable tasks present; the processor was halted executing an idle loop.
• iowait: CPU was idle while at least one I/O request was in flight to a block device.
• irq: Servicing hardware interrupts.
• softirq: Servicing software interrupts (network packet handling, timer queues).
• steal: In virtualized cloud environments, time spent waiting for the hypervisor to allocate a physical CPU core to your virtual machine.
• guest: Running a guest operating system inside a virtual machine managed by KVM.
• guest_nice: Running a niced guest operating system.

Because these counters continuously increment from the moment the kernel boots, reading /proc/stat a single time tells you only one thing: the total CPU time spent since the server was turned on.

If you read /proc/stat once, sum the fields, and calculate:

Total = user + nice + system + idle + iowait + irq + softirq + steal
Busy = Total - idle
Usage % = (Busy / Total) * 100

You have just calculated the average CPU usage of your server over its entire lifetime since boot, not its current usage.

The Common Script Trap: top -b -n 1

When engineers realize they cannot easily parse /proc/stat with a single command, many take a dangerous shortcut in their Bash or Node.js monitoring daemons:

# DO NOT USE THIS FOR INSTANTANEOUS METRICS
CPU_USAGE=$(top -b -n 1 | grep "Cpu(s)" | awk '{print $2 + $4}')

This is one of the most common anti-patterns in Linux system administration.

If you check the man page for top, you will find a critical warning: the first iteration of top (-n 1) always displays values averaged since the last system boot.

It does not measure the current system load. To get a real measurement from top, you must run at least two iterations with a delay between them:

# Correct measurement using top
top -b -n 2 -d 1 | grep --line-buffered "Cpu(s)" | tail -n 1 | awk '{print 100 - $8 "%"}'

The second iteration computes the delta across the one-second delay (-d 1), producing a meaningful number. But running a full top process every five seconds inside a monitoring cron job is heavy, allocates process table slots, and wastes memory.

We needed a clean, native solution inside our Node.js monitoring daemon.

Fixing the Metric Sampler in Node.js

Our infrastructure monitoring service ran a lightweight Node.js daemon that reported health stats to our Telegram alerting pipeline.

When we opened the daemon codebase, we found the bug immediately. Whenever the reporting cron ticked, it called a function that looked roughly like this:

// THE BROKEN IMPLEMENTATION: Single-point snapshot
const os = require('os');

function getCpuUsageInstant() {
  const cpus = os.cpus();
  let totalIdle = 0;
  let totalTick = 0;

  for (const cpu of cpus) {
    for (const type in cpu.times) {
      totalTick += cpu.times[type];
    }
    totalIdle += cpu.times.idle;
  }

  // BUG: This compares cumulative ticks to cumulative ticks since boot!
  // Or worse, when run with a tiny micro-sleep, floating-point jitter creates 100% spikes.
  return (1 - totalIdle / totalTick) * 100;
}

Because os.cpus() in Node.js wraps the uv_cpu_info function from libuv (which reads /proc/stat directly), querying it without comparing it to an earlier sample produced wild, unreliable results.

Worse, when the previous engineer tried to fix this by adding a tiny 50-millisecond delay between two os.cpus() calls, the measurement window was so microscopic that any minor kernel context switch happening during those 50ms swung the math from 0% to 100%.

We rewrote the metric collector to run an isolated, continuous background sampling loop with a fixed 2000ms delta window.

Here is the exact production implementation in metrics.js:

// THE ROBUST DELTA SAMPLER: metrics.js
const os = require('os');

let lastCpuSnapshot = null;
let currentCpuPercent = 0.0;

function takeCpuSnapshot() {
  const cpus = os.cpus();
  let idle = 0;
  let total = 0;

  for (let i = 0; i < cpus.length; i++) {
    const times = cpus[i].times;
    idle += times.idle;
    total += (times.user + times.nice + times.sys + times.idle + times.irq);
  }

  return { idle, total };
}

// Background sampler running every 2000ms
function startCpuSampler(intervalMs = 2000) {
  lastCpuSnapshot = takeCpuSnapshot();

  setInterval(() => {
    const current = takeCpuSnapshot();

    const idleDelta = current.idle - lastCpuSnapshot.idle;
    const totalDelta = current.total - lastCpuSnapshot.total;

    if (totalDelta > 0) {
      const busyPercent = ((totalDelta - idleDelta) / totalDelta) * 100;
      // Clamp between 0.0 and 100.0 to prevent floating precision anomalies
      currentCpuPercent = Math.max(0.0, Math.min(100.0, parseFloat(busyPercent.toFixed(2))));
    }

    lastCpuSnapshot = current;
  }, intervalMs);
}

function getSystemMetrics() {
  return {
    cpuPercent: currentCpuPercent,
    freeMemMB: Math.round(os.freemem() / (1024 * 1024)),
    totalMemMB: Math.round(os.totalmem() / (1024 * 1024)),
    uptimeHours: (os.uptime() / 3600).toFixed(1)
  };
}

module.exports = {
  startCpuSampler,
  getSystemMetrics
};

By decoupling the measurement loop from the alert check, the daemon stopped taking ad-hoc, microscopic snapshots. The alert loop simply read the pre-computed, rolling 2-second moving average.

This eliminated the phantom mathematical spikes. But our investigation was only half finished.

Even with the new delta sampler running, we noticed that every two minutes, our CPU usage genuinely jumped from 2% to 85% for a brief 3-second pulse.

The monitoring math was now accurate. The remaining spikes were real.

The Docker Healthcheck Trap: Killing Next.js With SSR

We inspected running processes on the server using pidstat to catch whatever process was consuming CPU cycles:

pidstat -u 1 10

pidstat samples process activity every 1 second for 10 iterations. During the next micro-spike, the output lit up:

11:42:15 AM   UID      PID    %usr %system  %guest   %wait    %CPU   CPU  Command
11:42:16 AM  1001    24810   82.00   14.00    0.00    1.00   96.00     1  node /app/server.js

Our primary application container was a Next.js production build running under Node.js. Why would a web service with virtually zero traffic spike to 96% CPU like clockwork?

We opened docker-compose.yml and found the culprit under the healthcheck directive:

# THE WRONG WAY: Hammering the root SSR route
services:
  web:
    image: our-registry/web-frontend:v1.4.2
    restart: unless-stopped
    ports:
      - "127.0.0.1:3000:3000"
    healthcheck:
      test: ["CMD", "wget", "-q", "-O", "-", "http://127.0.0.1:3000/"]
      interval: 15s
      timeout: 5s
      retries: 3

Notice what wget was requesting: http://127.0.0.1:3000/.

In Next.js, the root path (/) is not a static HTML file sitting on an Nginx webroot. It is a dynamic React server component. Every time Docker executed that healthcheck probe:

1. wget established a TCP handshake with Node.js on port 3000.
2. Next.js received the HTTP GET request for /.
3. The server runtime triggered a complete Server-Side Rendering (SSR) cycle.
4. Next.js rendered the React component tree into HTML strings.
5. It evaluated layout wrappers, serialized hydration payloads, and encoded headers.
6. It streamed the complete 85-kilobyte homepage response back to wget.
7. wget piped the payload to stdout (discarded) and exited.

On a modest cloud VPS with 1 or 2 virtual CPU cores, executing a full React SSR pipeline takes 200 to 400 milliseconds of concentrated CPU time.

Because Docker was repeating this healthcheck every 15 seconds, and because our monitoring script was checking utilization around the same time, the monitoring daemon routinely caught the server in the middle of rendering its own homepage for an internal health probe.

The server was not under load from real users. The server was spending all its energy proving to Docker that it was alive.

The Fix: Dedicated Zero-Overhead Healthcheck Route

Never point container healthchecks at server-rendered web pages, authenticated routes, or complex application handlers.

A healthcheck should do exactly one thing: verify that the HTTP process event loop is responsive and that socket connections are accepted.

In Next.js (using the App Router), we created a minimal API route at app/api/health/route.ts:

// app/api/health/route.ts
export const dynamic = 'force-dynamic';

export async function GET() {
  return new Response(JSON.stringify({ status: 'ok', uptime: process.uptime() }), {
    status: 200,
    headers: {
      'Content-Type': 'application/json',
      'Cache-Control': 'no-store, no-cache, must-revalidate',
    },
  });
}

If you are using the Pages Router, create pages/api/health.ts:

// pages/api/health.ts
import type { NextApiRequest, NextApiResponse } from 'next';

export default function handler(req: NextApiRequest, res: NextApiResponse) {
  res.status(200).json({ status: 'ok', uptime: process.uptime() });
}

Then we updated docker-compose.yml to target the lightweight API route:

# THE RIGHT WAY: Ultra-lightweight JSON endpoint
services:
  web:
    image: our-registry/web-frontend:v1.4.3
    restart: unless-stopped
    ports:
      - "127.0.0.1:3000:3000"
    healthcheck:
      test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:3000/api/health | grep -q 'ok' || exit 1"]
      interval: 30s
      timeout: 3s
      start_period: 10s
      retries: 3

We also increased the interval from 15 seconds to 30 seconds. A container does not need to be interrogated four times every minute when a 30-second window is more than fast enough for load balancers.

The impact was immediate:

• Before (SSR /): 320ms execution time, 94% single-core CPU utilization during request, 85 KB response payload.
• After (/api/health): 1.2ms execution time, 0.01% CPU consumption, 42 bytes response payload.

The container was now healthy, quiet, and cool.

Killing Rogue Systemd Timers: The fwupd Nightmare

With the monitoring daemon patched and the Docker healthchecks optimized, false alerts dropped by 95%.

Yet once every day or two, an alert still slipped through. This time, however, pidstat showed that Node.js was completely blameless. The culprit was an operating system utility: fwupd-refresh.

What is fwupd?

It is the Linux Vendor Firmware Service (LVFS) daemon. Its job is to detect motherboard hardware, query online repositories for updated firmware (such as BIOS updates, Thunderbolt dock firmware, or SSD controller patches), and notify the system administrator.

On a physical Dell PowerEdge rack server or a ThinkPad laptop, fwupd is a useful utility.

On a virtualized cloud VPS running on AWS, DigitalOcean, Hetzner, or Linode, it is completely useless. A guest virtual machine running inside KVM or Xen does not have access to physical motherboard hardware. It cannot flash BIOS firmware, and it never will.

Yet on standard Ubuntu 22.04 and Debian cloud images, fwupd and its associated systemd timer are installed and enabled by default.

How fwupd-refresh.timer Causes Spikes

Let us see what timers are scheduled to run on your Linux system:

systemctl list-timers --all

Look for fwupd-refresh.timer. When this timer triggers, systemd launches fwupd-refresh.service.

On a resource-constrained 1 GB or 2 GB RAM cloud instance, fwupd-refresh does the following:

1. It awakens and initializes internal hardware bus inspection libraries.
2. It downloads remote firmware metadata archives (often 20MB to 50MB of compressed XML).
3. It decompresses these large cabinet archives in memory using libxml2.
4. It parses tens of thousands of XML nodes into an in-memory database to cross-reference against device GUIDs that do not exist on virtual machines.
5. It writes temporary cache databases to /var/lib/fwupd/.

On a virtual machine with single-thread CPU constraints and slow shared storage, this metadata decompression pins the CPU at 100% and saturates disk I/O for 15 to 45 seconds.

The Remediation: Disable and Mask Unnecessary Timers

If your server is a virtual cloud instance, disable and mask fwupd immediately:

# Stop running services
sudo systemctl stop fwupd-refresh.timer fwupd-refresh.service fwupd.service

# Disable them from starting on boot
sudo systemctl disable fwupd-refresh.timer fwupd-refresh.service fwupd.service

# Mask the services so other packages cannot trigger them
sudo systemctl mask fwupd-refresh.service fwupd.service

Masking points the systemd unit symlink to /dev/null. Even if another package (like an apt hook) tries to invoke fwupd, systemd refuses to launch it.

Other Background Services Worth Auditing

While inspecting systemd timers, audit these other common cloud culprits:

# Check all active timers
systemctl list-timers --no-pager

• apt-daily.timer and apt-daily-upgrade.timer: Automates package index updates. On managed production nodes where updates are controlled via Ansible or CI/CD pipelines, unannounced background apt runs lock /var/lib/dpkg/lock-frontend and spike CPU. If you manage updates manually, disable these timers.
• man-db.timer: Rebuilds the manual page index cache. On a headless container host, you do not need an automated job burning CPU cycles indexing man pages you will never read.
• mlocate.timer or plocate.timer: Periodically indexes the entire filesystem for locate. On a server running Docker with hundreds of thousands of overlayfs files, this job causes severe disk thrashing.

To safely disable man-db indexing:

sudo systemctl stop man-db.timer
sudo systemctl disable man-db.timer
sudo systemctl mask man-db.service

Live Patching Inside Docker: Volume Mount Workarounds

During this debugging session, our monitoring daemon was running as a containerized service alongside our web apps. We needed to test changes to metrics.js and verify our delta calculations against live system ticks without tearing down our entire production stack or triggering false downtime.

In development environments, developers often use volume bind mounts. In this specific incident, having a volume bind mount configured in our docker-compose.yml allowed us to hot-patch the monitoring daemon live:

services:
  system-monitor:
    image: node:20-alpine
    working_dir: /app
    volumes:
      - ./monitor-src:/app/src:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
    environment:
      - NODE_ENV=production
      - HOST_PROC=/host/proc
    command: ["node", "src/index.js"]
    restart: always

Because ./monitor-src was mounted directly into /app/src, we were able to update the delta calculation logic directly on disk.

A simple docker kill -s SIGHUP <container_id> or a process manager restart inside the container reloaded the Node.js script in 80 milliseconds without dropping the underlying network namespace or interrupting surrounding containers.

Once verified over a 24-hour baseline, we baked the final code into a versioned container image and removed the host bind mount for production immutability.

The Linux Performance Diagnostic Toolbox

When a server claims to be at 100% CPU, do not restart it immediately. Restarting destroys the volatile process state and clears kernel counters, leaving you blind to the root cause.

Run through this diagnostic sequence in order:

1. Check System Wide Queue Pressure (vmstat)

vmstat 1 5

The output gives you immediate ground truth:

procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 1  0      0 812404  45120 412890    0    0     0    12  320  450  2  1 97  0  0

Focus on four specific columns:

• r (runnable): Number of processes waiting for CPU time. If r exceeds your core count (e.g. r=8 on a 2-core machine), you have true CPU starvation.
• b (blocked): Processes sleeping in uninterruptible disk sleep. If b is high, your "CPU spike" is actually disk I/O lockup.
• id: Percentage of CPU time spent idle.
• wa: Percentage of CPU time spent waiting on disk or network storage.

2. Identify the Culprit Process (pidstat)

# Track active CPU consumers every 1 second
pidstat -u 1 5

Unlike top, which sorts dynamically and rearranges lines while you are trying to read, pidstat outputs sequential timestamped lines. It shows you the exact command, PID, and whether the time was spent in user space (%usr) or kernel space (%system).

3. Check for Interrupted Hardware Events (mpstat)

If you have sysstat installed, inspect core distribution:

mpstat -P ALL 1 3

This reveals whether an interrupt storm (%irq or %soft) is pinning a single core while the remaining cores sit completely idle. A broken network driver or misconfigured packet queue often pins CPU 0 at 100% softirq while total system usage looks deceptively moderate.

An Interesting Fact About Linux CPU Accounting

Linux does not track CPU percentages anywhere in kernel memory. The kernel's scheduler tracks only elapsed time in nanoseconds per task via runtime structures (se.sum_exec_runtime inside the Completely Fair Scheduler / EEVDF).

When /proc/stat is read, the kernel translates those internal nanosecond clocks into legacy architecture ticks defined by USER_HZ (standardized to 100 ticks per second on modern POSIX systems).

If a 32-bit Linux system runs continuously without a reboot for approximately 497 days, a 32-bit jiffies counter wraps around from 4,294,967,295 back to 0. In the late 1990s and early 2000s, this integer overflow routinely caused unpatched monitoring daemons and uptime utilities to crash or report that systems were operating at negative 10,000% CPU utilization.

Summary Checklist: Eliminating False 100% CPU Spikes

Before you trust an alert claiming your Linux server is pegged at maximum capacity, verify these four fundamentals:

1. Audit Metric Math: Ensure monitoring scripts compare two discrete snapshots across a defined delta window ($\ge 1000\text{ms}$). Never calculate instantaneous usage from single-point readings.
2. Beware top -b -n 1: The first iteration of top reports lifetime averages since boot, not current usage. Always discard the first iteration or use /proc/stat deltas.
3. Inspect Docker Healthchecks: Never probe dynamic, server-rendered web routes like Next.js root (/). Build a dedicated, zero-dependency /api/health endpoint that returns static JSON in under 2ms.
4. Kill Rogue Cloud Timers: Mask fwupd-refresh.timer on cloud virtual machines where motherboard firmware cannot be flashed. Audit secondary timers like man-db and unmanaged apt-daily services.
5. Trust the Run Queue Over Percentages: Use vmstat 1 and inspect the r (runnable) queue. If the run queue is below your core count, your system is not suffering from computational exhaustion regardless of what a dashboard says.

Have you ever spent hours chasing a phantom 100% CPU alert on your servers only to discover the monitoring tool was measuring the wrong metric? What was the rogue script or misconfigured healthcheck in your environment?

If this breakdown saved you hours of debugging or gave you something practical to use in production, consider buying me a coffee. Your support directly fuels independent, zero-fluff Linux and DevOps technical guides.

About the Author

Asep Sayyad is a Linux and DevOps engineer passionate about Linux administration, automation, cloud technologies, containers, and open-source software. He enjoys solving real-world infrastructure challenges and sharing practical knowledge through in-depth technical articles, tutorials, and hands-on guides.

His goal is to help aspiring and experienced engineers build stronger Linux and DevOps skills with content focused on real production scenarios rather than theory alone.

Connect with Me

• Portfolio: asepsayyad007.in
• Blog: asepsayyad007.in/blogs
• GitHub: github.com/asepsayyad007
• LinkedIn: linkedin.com/in/asepsayyad
• Medium: asepsayyad007.medium.com
• Support: buymeacoffee.com/asepsayyad007

Enjoyed this article?

If this guide saved you hours of debugging or gave you something practical for production, consider:

• Buying me a coffee: Your support directly fuels independent, zero-fluff Linux and DevOps engineering breakdowns.
• Starring my open-source projects on GitHub.
• Sharing this article with fellow Linux and DevOps engineers.

You can also follow me for more practical content on Linux, DevOps, Cloud, Containers, Automation, and Open Source. Thanks for reading, and enjoy your learning!

© 2026 Asep Sayyad