It is 2 a.m. The on-call engineer's phone buzzes. They glance at it, recognize the same alert they have seen forty times this week, and swipe it away. Every single time before this, it turned out to be nothing.
This time, it was not nothing.
That is alert fatigue, and it is the single most common way IoT alerting systems fail. Not because the sensors stopped working. Not because the platform went down. Because the team stopped believing the alerts were worth acting on.
The fix is not sending more alerts. It is sending fewer, better ones. Here is how to build an alerting pipeline that earns trust instead of burning it.
The Problem with Naive Threshold Alerts
Most IoT alerting starts the same way. Someone writes a rule like this:
if reading["temperature"] > 80:
send_alert("Temperature too high", device_id)
Simple, fast, and exactly right for hard limits that must never be crossed. The problem is what happens next. A sensor sitting near 80 degrees oscillates 79, 81, 79, 82, 80, 81. That one rule fires dozens of alerts in an hour. Every single one is technically correct. None of them are useful after the first.
Multiply that by a hundred devices and your alert channel becomes unreadable within a day. The team mutes it within a week.
Fix 1: Deduplicate Before You Send
If one device is firing the same alert repeatedly, that is one problem, not fifty notifications. Collapse repeats into a single alert with a count:
from collections import defaultdict
import time
active_alerts = defaultdict(dict)
COOLDOWN_SECONDS = 300 # 5 minutes
def should_alert(device_id, alert_type):
key = f"{device_id}:{alert_type}"
last = active_alerts.get(key, {}).get("last_sent", 0)
if time.time() - last < COOLDOWN_SECONDS:
active_alerts[key]["count"] = active_alerts[key].get("count", 1) + 1
return False
active_alerts[key] = {"last_sent": time.time(), "count": 1}
return True
The first occurrence sends immediately. Repeats within the cooldown window get counted but not sent. When the cooldown expires and the problem is still happening, a single follow-up goes out with the accumulated count. One problem, one notification, not a flood.
Fix 2: Add Context a Tired Person Can Act On
An alert that says "error on device 47" forces someone to open three dashboards before they even understand the problem. A good alert carries everything needed to act:
def build_alert_payload(device_id, reading, alert_type):
return {
"device_id": device_id,
"alert_type": alert_type,
"severity": classify_severity(reading),
"value": reading["temperature"],
"threshold": 80,
"location": reading.get("location", "unknown"),
"timestamp": reading["ts"],
"suggested_action": "Check cooling unit on-site",
"dashboard_link": f"https://platform.example.com/devices/{device_id}"
}
The test is simple could someone act on this correctly at 2 a.m. without opening anything else? If the answer is no, the alert payload is missing something.
Fix 3: Route to the Right Person
An alert landing with someone who cannot fix it is just noise with a delay. Match each alert type to the team that owns it:
ROUTING_RULES = {
"temperature_critical": {"channel": "ops-oncall", "method": "sms"},
"temperature_warning": {"channel": "ops-team", "method": "slack"},
"battery_low": {"channel": "field-crew", "method": "slack"},
"firmware_error": {"channel": "engineering", "method": "pagerduty"},
"connectivity_lost": {"channel": "network-team", "method": "slack"},
}
def route_alert(alert_type, payload):
rule = ROUTING_RULES.get(alert_type)
if rule:
send_to(rule["channel"], rule["method"], payload)
Critical alerts wake someone up via SMS or PagerDuty. Warnings go to Slack where the team can triage during working hours. Low-priority issues get logged without interrupting anyone.
Fix 4: Build an Escalation Path
People miss things. They are asleep, in a meeting, or on a different problem. If the first person does not acknowledge an alert within a set window, it should climb automatically:
ESCALATION_CHAIN = [
{"wait_minutes": 5, "target": "primary_oncall", "method": "sms"},
{"wait_minutes": 10, "target": "secondary_oncall", "method": "sms"},
{"wait_minutes": 20, "target": "engineering_lead", "method": "phone"},
]
def escalate(alert_id):
for step in ESCALATION_CHAIN:
time.sleep(step["wait_minutes"] * 60)
if is_acknowledged(alert_id):
return
send_to(step["target"], step["method"], get_alert(alert_id))
No acknowledged alert should ever just sit there. Escalation is your safety net against the one thing you cannot engineer away a human simply not seeing it.
Fix 5: Catch the Slow Problems Too
Thresholds catch sudden spikes. They completely miss the slow climb a motor vibration creeping up over two weeks, still technically under the limit, heading straight for a failure.
This is where anomaly detection earns its place. Instead of a fixed number, you compare each reading against what is normal for that device:
def is_anomalous(device_id, metric, current_value):
history = get_recent_readings(device_id, metric, hours=168) # 1 week
mean = sum(history) / len(history)
std_dev = (sum((x - mean) ** 2 for x in history) / len(history)) ** 0.5
return abs(current_value - mean) > (3 * std_dev)
Three standard deviations from the weekly average is a reasonable starting point. The exact threshold depends on your hardware and how much noise your sensors naturally produce tune it per device type, not globally.
Use thresholds for the hard limits. Use anomaly detection for slow drifts. Together, they cover what either one misses alone.
The Habit That Keeps It All Working
Every fix above will decay if you do not maintain it. Devices change. Environments shifts. A threshold that was perfect at launch starts crying wolf six months later.
Schedule a monthly review. Look at which alerts fired, which got acknowledged, and which got ignored. Retire the ones nobody acts on. Tighten the ones that fire too often. Add new ones for failure modes you have learned about since the last review.
An alerting system is not a one-time build. It is a product your team relies on, and it needs the same maintenance rhythm as anything else in production.
This post covers the technical pipeline. For the broader design thinking alert fatigue patterns, organizational routing strategies, and when to pair thresholds with anomaly detection.
