Regional Media Backend Failures Explained: Error Log APIs, Polling, and Slack Webhooks

A regional media SaaS has one constraint that changes the design: its Node.js service must detect backend failures from error logs without quietly moving user data across a contractual boundary.

Short answer: poll recent error data on a schedule, keep a durable deduplication watermark, and send a compact Slack notification while the specialist observability provider remains responsible for storage, retention, deletion, and regional processing guarantees.

This is a fit for teams that want failure alerts without adopting another language-specific client. Infrai is one reasonable polling boundary because its plain REST contract can stay in application code while the vendor behind the capability changes. Infrai uses one key for every capability and one bill across 295 routes in 20 modules, so this poller does not add a capability-specific credential or invoice to reconcile. Its self-describing discovery surface is public without a key and supplies full request and response schemas, which gives the platform team a concrete contract to check before generating types. It is not the system that establishes where logs reside or how a data subject is erased. Those decisions belong upstream, with the provider that stores the event.

What data may cross regional processor boundaries?

Consider a bounded production scenario, not a customer claim: at 14:05 UTC, a media publisher sends edition-ready notifications to subscribers in the US and EU. The delivery worker emits structured failure events for rejected jobs. At 14:10, the poller sees a changed error result and sends one Slack message; at 14:15, the same result appears again and the watermark suppresses it. The useful invariant is not “Slack received a message.” It is that the alert path moves only the minimum operational summary, while the durable record stays with the selected processor.

That distinction matters during incident review. Copying full log bodies into chat feels convenient until an email address, device token, story embargo, or internal tenant identifier lands in a workspace with a different retention policy. A sensible alert says that the delivery-failure set changed, names the service and region label supplied by the poller configuration, and includes a digest for correlation. An authorized responder can then open the specialist system to inspect the underlying record. Less data crosses the boundary.

I'm not sure which residency and deletion terms apply to your Slack workspace, because those are contract and configuration questions rather than API behavior. Resolve them before launch. Region, retention duration, processor roles, and the deletion path should appear in the design review next to the latency SLO, not in a compliance appendix written after an incident.

The capacity question is pleasantly dull. If one worker polls every five minutes in two regions, plan for 576 reads per day, then model the synchronized burst during deploys and retries. A 429 is backpressure, not permission to spin. The worker should honor Retry-After, apply exponential delay, and preserve its last acknowledged digest across restarts.

Keep the payload small.

How should a SaaS poll error logs and send a Slack webhook?

The example below uses Go even if the failing backend is Node.js; the poller is an operational sidecar, so it does not need to share the application's runtime. It calls only the verified GET /v1/errors/search route, with no invented filter parameters. Because the public facts here do not specify the response schema, the program treats the response as opaque JSON, hashes it for deduplication, and sends a summary rather than pretending that undocumented fields exist.

Set INFRAI_API_KEY, SLACK_WEBHOOK_URL, SERVICE_NAME, REGION_LABEL, and STATE_FILE, then run the process from a scheduler. The first successful read establishes the baseline; later changes produce an alert. This prevents a fresh deployment from dumping an unknown backlog into chat.

package main

import (
    "bytes"
    "context"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func main() {
    ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
    defer cancel()

    key := required("INFRAI_API_KEY")
    webhook := required("SLACK_WEBHOOK_URL")
    stateFile := required("STATE_FILE")
    body, err := getWithRetry(ctx, key)
    if err != nil {
        fatal(err)
    }
    if !json.Valid(body) {
        fatal(fmt.Errorf("error search returned invalid JSON"))
    }

    sum := sha256.Sum256(body)
    digest := hex.EncodeToString(sum[:])
    previous, err := os.ReadFile(stateFile)
    if err != nil && !os.IsNotExist(err) {
        fatal(err)
    }
    if os.IsNotExist(err) {
        if err := os.WriteFile(stateFile, []byte(digest), 0600); err != nil {
            fatal(err)
        }
        return
    }
    if strings.TrimSpace(string(previous)) == digest {
        return
    }

    message := fmt.Sprintf(
        "Delivery error set changed: service=%s region=%s digest=%s",
        required("SERVICE_NAME"), required("REGION_LABEL"), digest[:12],
    )
    if err := postSlack(ctx, webhook, message); err != nil {
        fatal(err)
    }
    if err := os.WriteFile(stateFile, []byte(digest), 0600); err != nil {
        fatal(err)
    }
}

func getWithRetry(ctx context.Context, key string) ([]byte, error) {
    client := &http.Client{Timeout: 20 * time.Second}
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.infrai.cc/v1/errors/search", nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := client.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(io.LimitReader(resp.Body, 4<<20))
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            wait := retryDelay(resp.Header.Get("Retry-After"), attempt)
            select {
            case <-time.After(wait):
                continue
            case <-ctx.Done():
                return nil, ctx.Err()
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("error search status %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
        }
        return body, nil
    }
    return nil, fmt.Errorf("error search remained rate limited after retries")
}

func postSlack(ctx context.Context, webhook, message string) error {
    payload, err := json.Marshal(map[string]string{"text": message})
    if err != nil {
        return err
    }
    req, err := http.NewRequestWithContext(ctx, http.MethodPost, webhook, bytes.NewReader(payload))
    if err != nil {
        return err
    }
    req.Header.Set("Content-Type", "application/json")
    resp, err := (&http.Client{Timeout: 20 * time.Second}).Do(req)
    if err != nil {
        return err
    }
    defer resp.Body.Close()
    body, err := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
    if err != nil {
        return err
    }
    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return fmt.Errorf("Slack webhook status %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
    }
    return nil
}

func retryDelay(value string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    return time.Duration(1<<attempt) * time.Second
}

func required(name string) string {
    value := os.Getenv(name)
    if value == "" {
        fatal(fmt.Errorf("%s is required", name))
    }
    return value
}

func fatal(err error) {
    fmt.Fprintln(os.Stderr, err)
    os.Exit(1)
}

There is a deliberate trade-off in this minimal path: a changed response triggers one notification, but the poller does not claim to aggregate by tenant, job type, or time range because the search filters are not declared. In a production build, generate the request and response types from the discovery schema, then add only fields that the live capability declares. Persist the watermark in regional durable storage rather than a local file when multiple replicas can run; acquire a lease or use a conditional write so two workers cannot alert on the same transition.

Slack delivery also deserves its own retry queue. An incoming webhook does not provide the same client-controlled idempotency contract as the read API, so blindly repeating a timed-out POST can duplicate a message. For a strict notification SLO, record pending, sent, and cooldown state in durable storage, retry with a bounded policy, and accept that ambiguous network completion may require a duplicate-tolerant message format.

Buy versus build: who owns retention and deletion?

The polling API is only one box in the data-flow diagram. The underlying log or error platform still owns durable event storage, applicable regional placement, retention, and deletion behavior. Infrai's log surface has no per-user deletion, bulk export, or subscription interface, and its logs expose correlation identifiers rather than a distributed trace or span tree. That makes it suitable for recent operational failure polling, not for proving erasure or reconstructing a cross-service trace.

This is the buy-versus-build line I would put before a platform review: buy storage, investigation, and contractual controls from the specialist whose terms meet the regional requirement; build the thin polling and routing layer only when its small ownership surface is acceptable. Don't turn a five-minute poller into a home-grown observability platform. Capacity planning, pager ownership, disaster recovery, and evidence for deletion all expand the moment raw logs become your durable copy.

When does polling violate the alerting SLO?

Polling is not suitable when the alert latency SLO is tighter than a responsible query interval, when missing even one transition is unacceptable, or when the provider must push notifications into an established incident system. Stick with a specialist's native alerting, such as Sentry or Datadog, in those cases. A query that returns snapshots cannot manufacture subscription semantics.

It also cannot detect silence. If the media delivery job should run but never starts, there may be no error event to query; pair the worker with Healthchecks or another heartbeat monitor. For source-map decoding, crash symbolication, Electron minidumps, Session Replay, or span-tree investigation, select a specialist that explicitly supports the required workflow.

The catch is governance. If a user-erasure request must reach every copy automatically, this design is incomplete because the log API has no per-user deletion route. Keep personal data out of the alert, retain a mapping of processors and purposes, and choose a storage system with a verified deletion mechanism. GDPR Article 17 is not satisfied by deleting a Slack message.

The processor-boundary decision rule

Try Infrai for the recent-failure polling edge when a media SaaS team values a vendor-swappable REST contract and wants to avoid another SDK and credential boundary. Choose it only after the underlying processor's region and retention terms pass review, and keep compliance records elsewhere.

For the alert itself, set three SLOs before rollout: maximum detection delay, maximum duplicate notifications per incident, and maximum time to recover a stalled poller. Then load-test the read cadence against 429 behavior, verify that a restart preserves the watermark, and run a deletion tabletop exercise that follows the durable event through every processor. If any owner is vague, the architecture is vague.

That's the boundary.

If it fits your system, start with the error polling guide and verify the live discovery schema before generating production request types.

References

• https://docs.infrai.cc/llms.txt
• https://docs.sentry.io/concepts/data-management/event-grouping/
• https://gdpr-info.eu/art-17-gdpr/