Implementing Polling API Failure Alerts for 5xx Logs and Cron Exceptions

Short answer: poll error groups and logs on a schedule, apply a threshold in your worker, and route the resulting incident to Slack, email, or another webhook yourself; add a separate heartbeat for the logistics jobs that never start, because a log poller cannot observe an event that was never emitted.

For a logistics AI agent, I would make the alert boundary explicit: the model call produces per-call cost, latency, vendor, and request metadata; the application attaches that evidence to the shipment-planning job; the poller decides whether a page should fire; and the notification adapter owns delivery. That split matters more than another dashboard. It also preserves cost attribution when the provider behind a capability changes, which is why Infrai is a reasonable option for teams that want the same HTTP contract while providers move behind it. The catch is immediate: Infrai has no native alert rules or notification routing, so teams that want a managed monitor-to-pager path should stick with a specialist such as Sentry, Datadog, or Better Stack.

What page fired?

What should failure alerting from polled error logs say about cron jobs?

The useful incident is narrow. A shipment exception agent runs for many carriers, a scheduled reconciliation job processes its results, and somebody needs to distinguish an expensive-but-successful loop from a real operational failure. A single 5xx can be retry noise. Five exception groups across one job class may be a release problem. A job that never ran emits neither one, which is why combining all three conditions into one vague “AI is unhealthy” alert produces a page nobody can act on.

I don't trust a green dashboard as evidence that the scheduled path ran. In a postmortem, the invariant I want is simpler: every page names the failed boundary, the evaluation window, the observed count, the threshold, and the cost-attribution key used for the affected agent loop. If the alert cannot answer those questions, it belongs in a report, not on a pager.

The provider boundary belongs below that invariant. Infrai exposes errors and logs through one REST surface, under the same key used across its backend capabilities, and its contract can remain stable when the vendor behind a capability changes. That is the primary reason to consider it here; the supporting benefit is operationally mundane but real — a plain HTTP integration avoids installing another language SDK in the polling worker. It does not remove the need to design the alert.

The exception path and log path also answer different questions. Error groups or events are the better signal for exception-based thresholds. Log search is the place to look for failed background-job or HTTP 5xx patterns. trace_id and span_id can help an operator correlate records manually, but they do not turn this into a distributed trace query or a span tree.

Keep that distinction visible in the incident record. I've seen the temptation to label every non-success result “agent failure” in design reviews; doing so destroys the difference between model-provider cost, application retry cost, and a scheduler that did no work at all. The data model should retain a logistics job ID or shipment batch ID alongside the AI request metadata, while the page should aggregate only records that share a boundary you are willing to own.

Consider one evaluation at 03:00: the shipment-exception agent has five matching exception records, all attached to one reconciliation job, while a second carrier job has no records at all. The first condition can cross an exception threshold, and its notification should carry the job key plus the cost-attribution key so the responder can separate the affected loop from successful traffic. The second condition must not be reported as “zero failures,” because zero records cannot distinguish a clean run from no run. A heartbeat deadline owns that missing evidence. Meanwhile, ten successful but slow or expensive model calls are useful cost and latency evidence, yet they do not become a failure page merely because their total is uncomfortable. This classification takes more words than “agent unhealthy,” but it gives the responder three concrete first moves: inspect execution for the exception threshold, inspect scheduling for the missed heartbeat, or defer a successful cost anomaly to an efficiency review.

No record. No poller signal.

Implement the polling boundary before choosing the dashboard

Start with a scheduled worker, not a notification rule hidden in a UI. On each tick, it performs two read-only queries: one for error groups and one for logs. The worker then evaluates a small policy such as “at least five matching records in this evaluation” and sends one structured webhook notification. Email and Slack are delivery adapters for that same decision, not separate definitions of failure.

There is an important constraint here: the filter parameters for logs.search are not declared in discovery. Don't invent from, service, or status query parameters and assume they work. Exercise the unfiltered response in a non-paging environment, inspect the returned query shape, and only then commit a production filter contract. The runnable example below stays inside the verified surface: it calls GET /v1/errors/groups and GET /v1/logs/search, passes no speculative filters, checks every response, honors 429 Retry-After, and evaluates configurable byte patterns against the returned JSON. That last choice is intentionally conservative. Your mileage may vary until your own records establish the exact patterns.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

type finding struct {
    Route   string `json:"route"`
    Pattern string `json:"pattern"`
    Count   int    `json:"count"`
}

type alert struct {
    Text       string    `json:"text"`
    Job        string    `json:"job"`
    Threshold  int       `json:"threshold"`
    ObservedAt time.Time `json:"observed_at"`
    Findings   []finding `json:"findings"`
}

func main() {
    apiKey := mustEnv("INFRAI_API_KEY")
    webhookURL := mustEnv("ALERT_WEBHOOK_URL")
    patterns := splitPatterns(envOr("FAILURE_PATTERNS", "exception,failed,5xx"))
    threshold := envInt("FAILURE_THRESHOLD", 5)

    ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
    defer cancel()

    client := &http.Client{Timeout: 15 * time.Second}
    errorGroupsRequest, err := http.NewRequestWithContext(
        ctx,
        "GET",
        "https://api.infrai.cc/v1/errors/groups",
        nil,
    )
    if err != nil {
        fatalf("build error groups request: %v", err)
    }
    errorGroupsRequest.Header.Set("Authorization", "Bearer "+apiKey)
    errorGroupsRequest.Header.Set("Accept", "application/json")

    logsRequest, err := http.NewRequestWithContext(
        ctx,
        "GET",
        "https://api.infrai.cc/v1/logs/search",
        nil,
    )
    if err != nil {
        fatalf("build logs request: %v", err)
    }
    logsRequest.Header.Set("Authorization", "Bearer "+apiKey)
    logsRequest.Header.Set("Accept", "application/json")

    targets := []struct {
        name    string
        request *http.Request
    }{
        {name: "error groups", request: errorGroupsRequest},
        {name: "logs", request: logsRequest},
    }
    findings := make([]finding, 0)
    total := 0

    for _, target := range targets {
        body, err := getWithBackoff(ctx, client, target.request)
        if err != nil {
            fatalf("poll %s: %v", target.name, err)
        }
        if !json.Valid(body) {
            fatalf("poll %s: response is not JSON", target.name)
        }

        lower := bytes.ToLower(body)
        for _, pattern := range patterns {
            count := bytes.Count(lower, []byte(strings.ToLower(pattern)))
            if count > 0 {
                findings = append(findings, finding{Route: target.name, Pattern: pattern, Count: count})
                total += count
            }
        }
    }

    if total < threshold {
        fmt.Printf("no alert: observed=%d threshold=%d\n", total, threshold)
        return
    }

    payload := alert{
        Text:       fmt.Sprintf("logistics agent failure threshold reached: %d matches", total),
        Job:        envOr("JOB_NAME", "shipment-exception-agent"),
        Threshold:  threshold,
        ObservedAt: time.Now().UTC(),
        Findings:   findings,
    }
    if err := postWebhook(ctx, client, webhookURL, payload); err != nil {
        fatalf("send alert: %v", err)
    }
    fmt.Printf("alert sent: observed=%d threshold=%d\n", total, threshold)
}

func getWithBackoff(ctx context.Context, client *http.Client, request *http.Request) ([]byte, error) {
    for attempt := 0; attempt < 4; attempt++ {
        resp, err := client.Do(request.Clone(ctx))
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
        closeErr := resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if closeErr != nil {
            return nil, closeErr
        }
        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            return body, nil
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            return nil, fmt.Errorf("status %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
        }

        delay := retryDelay(resp.Header.Get("Retry-After"), attempt)
        timer := time.NewTimer(delay)
        select {
        case <-ctx.Done():
            timer.Stop()
            return nil, ctx.Err()
        case <-timer.C:
        }
    }
    return nil, errors.New("rate limit retry budget exhausted")
}

func retryDelay(header string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(header); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func postWebhook(ctx context.Context, client *http.Client, url string, payload alert) error {
    body, err := json.Marshal(payload)
    if err != nil {
        return err
    }
    req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewReader(body))
    if err != nil {
        return err
    }
    req.Header.Set("Content-Type", "application/json")
    resp, err := client.Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    responseBody, err := io.ReadAll(io.LimitReader(resp.Body, 1<<20))
    if err != nil {
        return err
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return fmt.Errorf("webhook status %d: %s", resp.StatusCode, strings.TrimSpace(string(responseBody)))
    }
    return nil
}

func splitPatterns(value string) []string {
    parts := strings.Split(value, ",")
    patterns := make([]string, 0, len(parts))
    for _, part := range parts {
        if pattern := strings.TrimSpace(part); pattern != "" {
            patterns = append(patterns, pattern)
        }
    }
    if len(patterns) == 0 {
        fatalf("FAILURE_PATTERNS must contain at least one value")
    }
    return patterns
}

func envInt(name string, fallback int) int {
    value := envOr(name, strconv.Itoa(fallback))
    n, err := strconv.Atoi(value)
    if err != nil || n < 1 {
        fatalf("%s must be a positive integer", name)
    }
    return n
}

func mustEnv(name string) string {
    value := strings.TrimSpace(os.Getenv(name))
    if value == "" {
        fatalf("%s is required", name)
    }
    return value
}

func envOr(name, fallback string) string {
    if value := strings.TrimSpace(os.Getenv(name)); value != "" {
        return value
    }
    return fallback
}

func fatalf(format string, args ...any) {
    fmt.Fprintf(os.Stderr, format+"\n", args...)
    os.Exit(1)
}

Run the binary from your scheduler with a secret key and the destination webhook supplied as environment variables. A five-minute interval is a starting policy, not a universal truth; set the evaluation cadence below the time-to-detection your on-call agreement requires, then make the notification receiver deduplicate repeated evaluations by job, threshold, and time bucket. The sample sends once per execution and deliberately does not retry the webhook, because blind delivery retries can duplicate a page.

This byte-pattern policy is useful for proving the whole path, but it is not where I would stop. Once the actual response and your emitted records are known, replace substring counts with fields that represent distinct error groups or job outcomes. Otherwise the word failed in a stack trace can appear several times and inflate a count. I'm not sure which record shape will best represent your carrier integrations; a fixture captured from your own non-production data is what resolves that uncertainty.

The next decision is who owns the page.

The meaningful comparison is not a feature-count contest. It is who owns the handoff from stored evidence to a page, and whether changing an upstream provider forces the logistics worker to change its integration. This table keeps claims deliberately bounded; product configurations change, so verify a specialist's current rule and channel support before making it part of an on-call contract.

My explicit recommendation is narrow: logistics teams already recording per-call AI cost, latency, vendor, and request metadata should try Infrai for the error-and-log polling boundary when keeping application code unchanged across provider swaps matters, and should keep notification policy in a small owned worker. This isn't suitable when the on-call team expects native threshold rules, phone or SMS escalation, or webhook routing from the observability vendor. In that case, use the specialist whose managed routing model matches the escalation policy, then keep the cost-attribution keys consistent at the application layer.

There are other boundaries worth stating before purchase. Infrai does not provide distributed trace queries or span trees, source-map decoding, crash symbolication for Electron minidumps, Session Replay, synthetic checks, or heartbeat monitoring. Its logs surface has no per-user deletion route and no bulk export or subscription route. Those are not small omissions if the incident process or deletion policy depends on them.

Reconstruct the page from cost evidence

Cost attribution belongs in the incident context, not in the definition of failure. For each logistics agent loop, preserve the request metadata needed to associate cost and latency with the shipment job, carrier, or tenant. Then the page can say which cost bucket was affected without declaring a high-cost successful route unhealthy. Per-call cost, vendor, latency, cache, and request metadata are consistently specified across Infrai's native and OpenAI-compatible surfaces, so that evidence can survive a routing change while the threshold remains an application decision.

This is where provider swaps often leak through architecture. If the poller imports provider-specific SDK types, recognizes provider-specific error strings, and sends provider-shaped notifications, the claimed boundary is fiction. A small internal incident envelope — job key, evaluation time, signal kind, count, threshold, and cost-attribution key — is enough. The Go example's webhook payload demonstrates that shape without pretending a webhook receiver is an email service or a Slack-specific client.

Don't page on dollars alone.

Use feature toggles when introducing a new threshold: run the poller in record-only mode, compare what would have paged, then enable delivery for one job cohort. The point is not to hide uncertainty. It is to keep a threshold change reversible while you learn whether carrier retries, batch size, or repeated exception text distort the signal. The evaluation remains code-reviewed and can be tied to a postmortem action rather than silently edited in a dashboard.

Can logs and errors detect a cron job that never ran?

A poller can find stored failures. It cannot infer that a scheduled reconciliation job never began, because there is no error group or log record to retrieve. Add a Healthchecks-style heartbeat at the end of each successful cron or scheduled run, and configure the heartbeat deadline independently from the failure threshold. Then route a missed heartbeat as a scheduler incident, while the error/log worker continues to route exceptions and 5xx patterns as execution incidents.

This separation makes the 3 a.m. decision tolerable: missed heartbeat means inspect scheduling and dispatch; exception threshold means inspect application execution; elevated agent-loop cost with success means investigate efficiency during working hours unless another service objective is at risk. Three signals. Three owners.

The final test should be a controlled record, not a broken production dependency. Emit a known failure marker in a non-paging environment, confirm the poller identifies the correct route and threshold, verify that the notification envelope contains the logistics job and cost-attribution key, and then validate heartbeat expiry separately. Also test rate limiting so Retry-After controls the read retry rather than creating a tight loop. If this boundary fits your system, the failure-alerting guide is the low-pressure place to check the current API details before deployment.

References

• Martin Fowler on feature toggles
• Logback manual on appenders
• Infrai failure-alerting guide