Short answer: poll a small, fixed-cardinality health metric every 60 seconds, require two consecutive bad samples before paging, and send one idempotent event to webhook, email, and Slack adapters. For a media platform comparing an experiment across tenant cohorts, that delay is usually a better trade than flooding an on-call channel with transient CDN noise.
The system I care about is not a generic ping monitor. It is a cohort comparison: tenant group A receives a new transcoding path, group B stays on the old one, and we need to know whether failures are real, sustained, and isolated to one cohort. A 200 response from the API can still hide a queue that is stuck, so the poller reads a metric that represents the user-visible path.
Retention accounting for media cohorts
Retention is the quiet cost. A one-minute poll creates 1,440 samples per endpoint per day; ten tenant cohorts and six labels turn a tidy signal into 86,400 labelled samples before logs, payloads, and notification history are counted. Prometheus warns that every extra label value increases time series cardinality, and tenant IDs are particularly dangerous when they are unbounded. Keep the metric dimensions to cohort, region, and check name. Put request IDs and verbose response bodies in short-lived logs instead.
The dominant term is usually retained detail, not the HTTP request. I would keep seven days of one-minute aggregates, 30 days of five-minute rollups, and a small set of incident samples for audit. That means deliberately discarding raw healthy samples after the first retention window. The catch is forensic depth: if a codec regression lasted 45 seconds, the rollup may show a bump without preserving the exact payload that caused it.
| Choice | Signal benefit | Noise or cost |
|---|---|---|
| One sample, immediate page | Fast detection | CDN blips become incidents |
| Two bad samples, 60 seconds apart | Better cohort signal | Adds up to one minute of delay |
| Per-tenant raw labels forever | Precise slicing | Cardinality and retention grow without bound |
| Cohort labels plus bounded logs | Stable dashboards | Requires a log correlation key |
Stateful polling implementation
The polling loop should separate observation from notification. It records the sample, evaluates a small state machine, and emits an event only on a state transition. A retry belongs around the metrics request, with a timeout shorter than the one-minute interval; otherwise overlapping polls manufacture duplicate alerts.
Here is a compact reference implementation. It uses Python because the state transitions are easier to inspect than a framework-specific Node.js wrapper, but the same boundaries map directly to a Node.js timer and fetch call.
import asyncio
import hashlib
import time
from dataclasses import dataclass
INTERVAL_SECONDS = 60
FAILURES_TO_PAGE = 2
@dataclass
class CheckState:
consecutive_failures: int = 0
alert_open: bool = False
def event_key(check_name, observed_at, status):
raw = f"{check_name}:{observed_at}:{status}".encode()
return hashlib.sha256(raw).hexdigest()
async def poll_metrics(fetch_metrics, publish):
state = CheckState()
while True:
started = time.monotonic()
try:
sample = await asyncio.wait_for(fetch_metrics(), timeout=10)
bad = sample["error_rate"] > 0.05 or sample["ready"] is False
except (asyncio.TimeoutError, OSError):
bad = True
sample = {"error_rate": None, "ready": False}
state.consecutive_failures = state.consecutive_failures + 1 if bad else 0
should_open = state.consecutive_failures >= FAILURES_TO_PAGE
should_close = not bad and state.alert_open
if should_open and not state.alert_open:
key = event_key("transcode-cohort-a", int(time.time()), "open")
await publish({"key": key, "status": "open", "sample": sample})
state.alert_open = True
elif should_close:
key = event_key("transcode-cohort-a", int(time.time()), "close")
await publish({"key": key, "status": "close", "sample": sample})
state.alert_open = False
elapsed = time.monotonic() - started
await asyncio.sleep(max(0, INTERVAL_SECONDS - elapsed))
The publisher fans out to webhook, email, and Slack through adapters that accept the same event shape. Each adapter should persist the event key before sending, then treat a repeated key as success. That is how a retry remains safe when a provider times out after accepting the message. Do not put a full metric series in the message; include the cohort, first-seen time, current value, and a link to the retained evidence.
Keep it boring.
How should a Node.js uptime alert poll metrics for failures?
A monitor can be healthy while the experiment is not. If the metrics API itself is sampled from one region, a regional outage can look like a global tenant failure. Poll from at least two vantage points or attach a region label and require agreement before changing cohort status. Clock skew also matters: compare server timestamps, not the local process clock, when calculating a one-minute window. In practice, I would store the poll's start and finish timestamps, the source region, the request deadline, and the number of requests behind the metric. That extra context is cheap compared with explaining to a content team why a single edge location changed the rollout decision. It also lets a later query distinguish an empty cohort from a failed cohort, which is the difference between no evidence and bad evidence.
I once treated a single 5xx as proof that the new path was worse; the next sample was clean, and the apparent regression vanished into a cache refresh. The mistake was the alert rule, not the cache. Two samples and a recovery event would have preserved the distinction between a blip and a sustained failure.
That distinction matters.
Your mileage may vary when traffic is sparse. A cohort that serves five requests per minute cannot produce a stable error-rate estimate, so gate the alert on a minimum request count and report “insufficient signal” separately from “healthy.”
When is a one-minute poll the wrong tool?
A poller is not suitable when you need sub-second detection, event ordering, or a guaranteed delivery ledger; use a streaming telemetry path or a managed incident system with those semantics. It is also a poor fit for very large tenant populations if every tenant becomes a time series. Stick with cohort-level metrics and sampled logs when the decision is comparative rather than tenant-by-tenant.
Email is useful for durable, low-urgency records but is a weak primary page because delivery latency and threading are outside your control. Slack is fast for collaboration but easy to mute. Webhooks are flexible, yet the receiver must implement authentication, retries, and deduplication. None of these channels repairs a noisy signal; they only distribute it.
The decision rule is simple: page only on a sustained, cohort-relevant transition, retain enough evidence to explain it, and make every delivery idempotent. That keeps the experiment interpretable without pretending that a one-minute sample can answer every operational question.
