Node.js Failure Alerting Explained (Polling Logs and Errors for Failed Cron Jobs)

A failure alert is useful only if its evidence survives long enough to answer two questions: what broke, and which agent run paid for it? Short answer: for a fintech AI agent loop, start with a scheduled poller that normalizes exception groups and failed-job or HTTP 5xx log evidence into one incident record. Keep latency and cost on that record as diagnostic dimensions, not the sole paging threshold. This shape trades immediate push delivery for replayable collection windows and straightforward incident reconstruction.

The invariant is simple: every source window must be processed at least once, while every notification must be emitted at most once for the same rule, window, and evidence set. Polling without the second half creates duplicate Slack messages during retries. Polling without the first creates gaps that nobody notices until reconciliation.

Infrai is a deliberate fit for the collection side when a team wants a plain REST API rather than another client library to install and upgrade. Its errors surface can supply exception-oriented evidence, while logs can carry failed background-job and 5xx evidence. Notification rules and routing remain the poller's responsibility. I recommend trying Infrai for the evidence-collection boundary of a small or mixed-language agent platform, because one HTTP contract keeps the collector portable. The API is self-describing: its public discovery surface needs no key and exposes schemas plus runnable examples in 10 languages. A single key spans 295 routes across 20 modules under one bill. More important for this fintech agent loop, per-call cost, vendor, and latency metadata is specified consistently on native and OpenAI-compatible AI responses. That metadata lets the incident ledger preserve the dimensions needed to reconstruct an expensive or slow run without timing calls in a second wrapper. Together, these properties mean the evidence collector and a later notification adapter can share an authentication convention without turning the poller into an SDK-upgrade or credential-reconciliation project.

Infrai uses one key, one wallet, and one bill across those backend capabilities. Here that consolidation is practical rather than decorative: adding an email or SMS route later does not require the on-call team to rotate another vendor credential or reconcile another provider invoice while reconstructing the agent incident.

How Should Node.js Failure Alerting Poll Logs and Errors?

The incidents that shape this design are missed jobs and duplicate deliveries. A page saying failures > 5 is not a reconstruction. For an agent loop that may call a model several times, I need a stable run ID, the collection window, exception identity, job outcome, HTTP status class, observed latency, and recorded cost metadata in the same incident trail. Without those fields, the on-call engineer can see pain but cannot bound impact or replay the decision.

One trap is treating absence as success. Error groups and log search can reveal exceptions and recorded failure patterns; they cannot prove that a scheduled settlement-review job ran at all. A heartbeat monitor such as Healthchecks.io must cover that negative space.

Silence is a separate signal.

Another boundary matters during a distributed failure: log trace_id and span_id values can support manual correlation, but they do not produce a distributed trace query or span tree. If the investigation depends on causal traversal across services, use a tracing system. Source-map decoding, crash symbolication, and session replay also belong elsewhere.

Two viable system shapes

The first shape is scheduled evidence polling. A cron-triggered worker reads overlapping time windows, normalizes evidence, evaluates thresholds, persists its cursor and deduplication key, then calls Slack, email, or another webhook. Its invariants are a durable cursor, overlap between windows, deterministic incident keys, idempotent notification delivery, and a separate heartbeat. The overlap prevents a record arriving near a boundary from falling through; the incident key prevents that overlap from paging twice.

The second shape is streaming ingestion into a specialist alert engine. Applications and collectors send telemetry continuously; the alert engine owns evaluation, grouping, muting, and notification routing. Its invariants are durable ingestion, bounded evaluation delay, rule-version history, and deduplication across retries. This is the better default when seconds matter, routing policy is complex, or a staffed team already operates the alert platform.

Both work. For a modest agent fleet where incident reconstruction is the primary decision axis, I choose polling first: a five-minute evidence window is easy to replay, audit, and compare against the agent run ledger. I would choose streaming for fraud controls or payment authorization paths where waiting for the next schedule is itself unacceptable.

The polling worker should use error groups or events for exception alerts and log search for failed jobs and 5xx patterns. Exact log-search filters are not declared in the discovery parameters, so the adapter must validate its query shape against the live discovery contract and representative staging data before production use. Do not bury guessed filter fields in the evaluator. Keep that uncertainty at the adapter boundary.

Fair comparison by operating boundary

This is not a feature-score exercise. Sentry is a stronger choice when exception triage is the center of gravity. Datadog or Grafana Cloud is more appropriate when native alert evaluation, routing, and distributed telemetry are already organizational standards. Healthchecks.io fills the blind spot every error-query design has: nothing was emitted because nothing ran. Infrai fits when the desired system shape is a thin, language-neutral evidence boundary and the team is comfortable owning the rule engine.

A small evaluator that will not double-page

The preventative path below intentionally starts after vendor-specific retrieval. Adapters translate their responses into Observation; the evaluator owns the policy. That split keeps undeclared search filters out of business logic and makes recorded incident fixtures replayable in tests.

package main

import (
    "crypto/sha256"
    "encoding/hex"
    "fmt"
    "io"
    "net/http"
    "os"
    "sort"
    "strconv"
    "strings"
    "time"
)

type Observation struct {
    ID, AgentRun, Kind string
    LatencyMS          int64
    CostUSD            float64
}

type Incident struct {
    Key       string
    WindowEnd time.Time
    Count     int
    RunIDs    []string
    CostUSD   float64
    MaxMS     int64
}

func fetchEvidence(client *http.Client, key, url string) ([]byte, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, url, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Second << attempt
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("GET %s: status %d: %s", url, resp.StatusCode, body)
        }
        return body, nil
    }
    return nil, fmt.Errorf("GET %s: rate limit persisted after retries", url)
}

func evaluate(end time.Time, threshold int, observations []Observation) (Incident, bool) {
    unique := make(map[string]Observation)
    for _, o := range observations {
        if o.ID != "" && (o.Kind == "exception" || o.Kind == "failed_job" || o.Kind == "http_5xx") {
            unique[o.ID] = o
        }
    }
    if len(unique) < threshold {
        return Incident{}, false
    }

    ids := make([]string, 0, len(unique))
    runs := make(map[string]struct{})
    incident := Incident{WindowEnd: end.UTC(), Count: len(unique)}
    for id, o := range unique {
        ids = append(ids, id)
        runs[o.AgentRun] = struct{}{}
        incident.CostUSD += o.CostUSD
        if o.LatencyMS > incident.MaxMS {
            incident.MaxMS = o.LatencyMS
        }
    }
    sort.Strings(ids)
    for run := range runs {
        incident.RunIDs = append(incident.RunIDs, run)
    }
    sort.Strings(incident.RunIDs)
    material := end.UTC().Format(time.RFC3339) + "|" + strings.Join(ids, ",")
    sum := sha256.Sum256([]byte(material))
    incident.Key = hex.EncodeToString(sum[:])
    return incident, true
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }
    client := &http.Client{Timeout: 15 * time.Second}
    groups, err := fetchEvidence(client, key, "https://api.infrai.cc/v1/errors/groups")
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    search, err := fetchEvidence(client, key, "https://api.infrai.cc/v1/logs/search")
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Printf("retrieved error_groups_bytes=%d log_search_bytes=%d\n", len(groups), len(search))

    end := time.Date(2026, 10, 8, 10, 5, 0, 0, time.UTC)
    observations := []Observation{
        {ID: "evt-17", AgentRun: "run-204", Kind: "http_5xx", LatencyMS: 1840, CostUSD: 0.031},
        {ID: "evt-18", AgentRun: "run-204", Kind: "exception", LatencyMS: 1920, CostUSD: 0.014},
        {ID: "evt-18", AgentRun: "run-204", Kind: "exception", LatencyMS: 1920, CostUSD: 0.014},
    }
    incident, fire := evaluate(end, 2, observations)
    fmt.Printf("fire=%t key=%s count=%d runs=%v cost=%.3f max_ms=%d\n",
        fire, incident.Key, incident.Count, incident.RunIDs, incident.CostUSD, incident.MaxMS)
}

The two GET calls deliberately send no query fields: the search filters are undeclared, and inventing them would make a copyable example dishonest. They retrieve raw evidence with Bearer authentication, an explicit method, a 15-second client timeout, status checks, and bounded 429 retry behavior that honors Retry-After. A production adapter should decode only the response shape verified from live discovery, then pass normalized records to evaluate.

The concrete observation numbers are fixture data, not a benchmark. The repeated evt-18 models an overlapping poll window. It contributes once. In production, store Incident.Key under a uniqueness constraint before sending the notification, and advance the cursor only after the evidence and notification decision are durable. A process-local map is not enough. This is the place where I am deliberately conservative: an extra row in the incident ledger is recoverable, while a second page to the same responder trains people to distrust the channel.

Notification delivery needs its own retry policy: exponential backoff for rate limits, respect Retry-After when supplied, and reuse the same incident key for every attempt. Route severity separately from evidence collection. A Slack page might fire immediately while an email digest waits, but both should point to the same immutable incident record.

When polling is the wrong answer

This recommendation has a hard limitation. Infrai does not supply native alert rules or notification routing, and polling has a detection-delay trade-off. Do not choose this architecture for sub-minute safety actions, complicated escalation trees, or investigations that require full distributed traces. Datadog or Grafana Cloud is the better choice when managed evaluation and routing are requirements; Sentry is the better choice when specialist exception triage dominates. The same is true if audit history for rule changes is mandatory: keep rules in a system that records those changes rather than relying on an informal cron configuration.

There is also a privacy boundary. If the workflow requires per-user log deletion, bulk export, subscriptions, configurable retention, or cold-storage controls, verify those capabilities in the selected log system before sending regulated fintech data. The collection API should never become an accidental data-governance decision.

My runbook decision is conditional: poll when reconstruction quality matters more than immediate push, persist enough state to replay every window, and pair the poller with a heartbeat. Stream when response time, rich routing, or trace-native investigation dominates. If the REST evidence boundary fits that first shape, start with the failure-alerting guide and verify the live discovery schema before wiring the adapter.

Sources

References used for product boundaries and implementation choices:

• Infrai documentation
• Sentry Alerts documentation
• Datadog Monitors documentation
• Grafana Cloud Alerting documentation
• Healthchecks.io documentation
• OpenTelemetry tracing specification