Failure Alerting with a Metrics API: Polling a Query Endpoint in a Small Node.js App

Short answer: for a small Node.js app that searches structured logs from a nightly data pipeline, make the alert a versioned, reversible decision. Query a stable metric or query endpoint, persist the last decision, send one webhook on a state transition, and make rollback disable the rule without deleting the evidence that led to it.

The important part is rollback safety. A notification that fires is useful; a notification rule that cannot be safely changed at 2 a.m. is operational debt. Treat alert configuration as code, give each rule an identifier, and keep the raw query result beside the decision. The mental model is five boxes: pipeline logs -> metric projection -> query API -> evaluator -> webhook. A rollback moves the evaluator and its rule version back one step. It does not erase the logs.

Start with the failure you need to see

Nightly work has a peculiar shape. A service can be healthy for twenty-three hours, then fail in a six-minute batch. Request latency alone will not reveal that. Structured logs can: each record should carry a job name, run identifier, outcome, and timestamp, with the fields needed to count failures without parsing human prose.

Before the change, an operator searches yesterday's logs manually. After the change, the pipeline emits a small projection such as nightly_import_failures, and a scheduled Node.js check asks for the value after the expected completion window. One value crosses a threshold. The resulting event includes the run ID and rule version, so a receiver can tell an old alert from a new one.

Keep the query narrow. A query that silently broadens from one job to every batch turns a useful page into a noisy report. Define the time window from the pipeline's schedule, define what missing data means, and record the query text or metric name with every evaluation. The four golden signals are a useful reminder that errors, latency, traffic, and saturation answer different questions; a nightly failure alert should not pretend to cover all four.

Rollback is a data problem as much as a deployment problem. If rule nightly-failure-v3 is too sensitive, the safe reversal is to activate v2, preserve the v3 evaluations, and mark the transition. That gives the next investigation a trail instead of a blank space.

How should a small Node.js app use metrics polling and webhooks for rollback-safe alerting?

Use an explicit state machine. healthy, firing, and unknown are enough for a first version. unknown matters: a missing metric is not the same as a successful run. The example below keeps transport generic, assumes the query response has already been shaped by the team's metric convention, and makes the rollback switch an ordinary environment value.

type HealthState = "healthy" | "firing" | "unknown";

type MetricSample = {
  value?: number;
  observedAt?: string;
  runId?: string;
};

type AlertDecision = {
  ruleId: string;
  ruleVersion: string;
  state: HealthState;
  previousState: HealthState;
  sample: MetricSample;
};

function evaluate(
  sample: MetricSample,
  previousState: HealthState,
  ruleId: string,
  ruleVersion: string,
): AlertDecision {
  const state: HealthState =
    typeof sample.value !== "number" || !Number.isFinite(sample.value)
      ? "unknown"
      : sample.value > 0
        ? "firing"
        : "healthy";

  return { ruleId, ruleVersion, state, previousState, sample };
}

function needsWebhook(decision: AlertDecision): boolean {
  return (
    decision.state !== decision.previousState &&
    (decision.state === "firing" || decision.previousState === "firing")
  );
}

async function postTransition(
  decision: AlertDecision,
  webhookUrl: string,
): Promise<void> {
  if (!needsWebhook(decision)) return;

  const response = await fetch(webhookUrl, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      type: "nightly_pipeline_transition",
      key: `${decision.ruleId}:${decision.ruleVersion}:${decision.sample.runId ?? "unknown"}`,
      ...decision,
    }),
  });

  if (!response.ok) {
    throw new Error(`Webhook rejected transition: ${response.status}`);
  }
}

The persistence boundary belongs around this code. Read the last state, evaluate the new sample, post the transition, then commit the state with a conditional write. Imagine the nightly import finishes at 02:06 with three failed records. The metrics projection reports value: 3, the evaluator changes healthy to firing, and the webhook receives a key containing nightly-import, the active rule version, and that run's ID. The receiver stores the key before showing the message. A network timeout occurs after the receiver accepts it, so the Lambda retries; the receiver sees the same key and does not create a second notification. At 02:20, the operator replaces the rule with the prior version because a new filter was too broad. The archived v3 decision still says what was measured, which window was queried, and why the alert fired. The active evaluator now uses v2, but the evidence is untouched. That is the difference between a rollback and a deletion. The exact write order depends on the storage system: committing first risks losing a notification, while posting first risks a duplicate after a retry. Let the receiver deduplicate the transition key, and keep the raw sample for replay. This is a small distributed system. It deserves distributed-system thinking.

Watch the edge.

There is a practical safety valve here. Set the active rule version in configuration, deploy the evaluator with both versions available, and switch back by changing one value. A deployment rollback then restores code and configuration together. Do not hide the active version in a mutable query string or a hand-edited dashboard.

Make the switch boring.

What should polling, query endpoints, and webhooks guarantee?

Each boundary needs a contract. The metrics API should return a timestamped sample and a clear empty result. The poller should set a timeout, validate the response shape, and distinguish transport failure from a measured zero. The webhook should accept an idempotency key or provide an equivalent deduplication rule. The scheduler should have its own heartbeat, because an observer that never runs cannot alert on the pipeline.

Use a small retry budget. Retrying a read can help with a transient network response; retrying a notification without an idempotency contract can page twice. Exponential delay is not a substitute for a limit. Log the attempt number, rule version, and correlation ID, but never put credentials in the structured event.

The query window also needs a clock policy. For a nightly job, wait until the expected completion boundary plus a known grace period. Querying too early creates false failures. Querying an unbounded window counts yesterday's failure forever. Store the window start and end in the decision event so a later reader can reproduce the result.

Which rollback and alerting trade-offs matter for a small app?

The smallest implementation is not automatically the safest. A single threshold is easy to operate, but it cannot express a delayed run, a partial batch, or a missing sample. Durable state reduces repeated pages, but adds a write path that must be tested. A hosted notification route reduces delivery code, but moves policy and access control outside the repository.

The catch is that this pattern is not suitable when you need on-call escalation, phone delivery, multi-team routing, or a managed incident timeline. Use a dedicated alerting control plane for those requirements. Stick with the small evaluator when the team owns one pipeline, one channel, and a clear rollback policy. That is a boundary, not a universal recommendation.

What does a useful test and rollout look like?

Test the evaluator with four fixtures: zero failures, one failure, a missing sample, and a repeated firing sample. Then test the awkward sequence: fire, webhook timeout, retry, recovery, and rollback to the prior rule version. Assert that the receiver sees stable keys and that the stored state does not jump backward accidentally.

Roll out in observe-only mode first. Write decisions without sending them, compare the counts with a manual log search, and inspect the window boundaries after one complete nightly run. Enable the webhook for a low-noise channel next. Promote it to a page only after the team has exercised a rollback and knows who owns a false positive.

I'm not sure a single metric can represent every pipeline failure; it cannot represent a malformed record unless the pipeline projects that condition explicitly. Your mileage may vary with late-arriving logs and clock skew. Those are reasons to document the signal's limits, not reasons to add more retries until the alert looks quiet.

Good alerting leaves an audit trail. Good rollback leaves it intact.

References

• Google SRE Book, "Monitoring Distributed Systems": https://sre.google/sre-book/monitoring-distributed-systems/
• Logback Manual, "Appenders": https://logback.qos.ch/manual/appenders.html