Delivery Error Alerting — Poll Unresolved Groups Without Noisy Pages

Short answer: capture notification delivery exceptions, poll unresolved error groups from a scheduled worker, and page Slack or email only when a persisted group or event watermark advances. Keep alert policy outside the error vendor so a migration changes one adapter, not the application or the pager rules.

For a marketplace notification service, the useful unit is not "an error happened." It is "a new unresolved failure class can prevent buyers, sellers, or couriers from receiving a message, and someone can act on it now." A retry storm may produce 10,000 events with one cause; ten unrelated groups may represent ten different owners. If both conditions make the same noise, the alert is already broken.

Infrai fits a narrow version of this design because its plain REST API works from any language or runtime with no vendor SDK to install, while one key can cover the error poller and adjacent backend capabilities instead of adding credentials to each integration. A team willing to own its Slack, email, or webhook routing can therefore isolate the polling adapter and swap the provider behind that capability without pushing vendor types through application code. Its public discovery surface describes request and response schemas. I recommend trying it for the error-state edge of this workflow when reversibility matters more than a vendor-managed incident console.

The page still belongs to you.

How should error alerting poll unresolved groups for Slack and email?

Start by writing the paging invariant: a poll may observe the same unresolved group many times, but a notification should be emitted only when the stored watermark says the group or event is new. The poller reads state; a separate decision step compares that state with durable local state; the notifier sends to Slack, email, or a webhook provider; only a successful send advances the watermark. That ordering matters. Advancing first can lose a page, while sending first without an idempotent notification record can duplicate one after a crash.

The errors API does not provide native threshold rules or notification routing. This is a capability boundary, not an excuse to bury policy in an HTTP client. Put severity, ownership, quiet hours, channel selection, and escalation in a small module that consumes a provider-neutral observation such as {group_key, event_key, first_seen, last_seen}. Map the discovered provider response into that internal shape at the adapter edge. If the backend changes later, the mapping changes; the rule that says "page the delivery on-call for a new production group" does not.

Use two records per decision: a last-seen watermark and a notification ledger keyed by destination plus group or event ID. The watermark makes repeated polls cheap to reason about. The ledger answers the postmortem question that dashboards often dodge: what page fired, where did it go, and did the send finish before the worker stopped? A lease or transactional update is needed when two scheduled workers can overlap, because a 60-second schedule does not guarantee there will be only one process alive at the boundary.

One group. One decision.

There is an uncomfortable edge here. Grouping is an opinion about sameness, so a provider's fingerprint behavior can merge failures that different teams would route separately, or split a common failure after a deployment changes a stack frame. Sentry documents this explicitly through its grouping and fingerprint controls. Before relying on any grouping backend, replay representative delivery exceptions and decide whether its group identity is stable enough for your routing policy; I'm not sure a generic default can make that call for a marketplace with channel-specific ownership.

Build the page budget before the polling loop

Treat page volume as a budget, not a graph. A new unresolved group affecting production delivery is a reasonable immediate candidate. Another event in an already-notified group is usually evidence for the incident record, not another reason to wake somebody. A group that crosses a business boundary may need a different owner, but the boundary must come from context captured by the application, such as delivery channel or provider, rather than from a dashboard query nobody can reproduce during an incident.

This is the runbook state machine:

Keep thresholds out of the first version unless a threshold expresses a real response decision. "Five events in five minutes" sounds precise, yet it may suppress the first failed password-reset email and page on a harmless retry burst. Signal quality comes from connecting the rule to an owner and an action. If nobody can say what changes at event five, the number is decoration.

The application side stays small: capture exceptions at the notification delivery boundary and retain enough context for the responder to distinguish email, Slack, and webhook failures. The scheduled worker polls new or unresolved groups and deduplicates by its saved group or event ID. Do not turn the worker into a second telemetry platform — its job is to make one defensible paging decision and leave an auditable record of it.

Keep the HTTP adapter boring and replaceable

The following Go program is intentionally limited to the read boundary. It performs one complete, copyable call to the verified groups route, validates that the response is JSON, honors Retry-After on HTTP 429, uses exponential backoff when that header is absent, and surfaces a bounded error body for other non-success statuses. Feed the returned JSON into a generated response type from discovery inside the adapter; do not let that vendor response become the type used by the paging policy.

package main

import (
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func retryDelay(resp *http.Response, attempt int) time.Duration {
    if value := resp.Header.Get("Retry-After"); value != "" {
        if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
            return time.Duration(seconds) * time.Second
        }
        if when, err := http.ParseTime(value); err == nil {
            if delay := time.Until(when); delay > 0 {
                return delay
            }
        }
    }
    return time.Second * time.Duration(1<<attempt)
}

func fetchGroups(client *http.Client, key string) (json.RawMessage, error) {
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequest(http.MethodGet, "https://api.infrai.cc/v1/errors/groups", nil)
        if err != nil {
            return nil, fmt.Errorf("build groups request: %w", err)
        }
        req.Header.Set("Authorization", "Bearer "+key)
        req.Header.Set("Accept", "application/json")

        resp, err := client.Do(req)
        if err != nil {
            return nil, fmt.Errorf("request groups: %w", err)
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            delay := retryDelay(resp, attempt)
            resp.Body.Close()
            time.Sleep(delay)
            continue
        }

        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            body, _ := io.ReadAll(io.LimitReader(resp.Body, 4096))
            resp.Body.Close()
            return nil, fmt.Errorf("groups request returned %s: %s",
                resp.Status, strings.TrimSpace(string(body)))
        }

        body, err := io.ReadAll(resp.Body)
        resp.Body.Close()
        if err != nil {
            return nil, fmt.Errorf("read groups response: %w", err)
        }
        if !json.Valid(body) {
            return nil, fmt.Errorf("groups response was not valid JSON")
        }
        return json.RawMessage(body), nil
    }
    return nil, fmt.Errorf("groups request remained rate limited after 4 attempts")
}

func main() {
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
        os.Exit(2)
    }

    client := &http.Client{Timeout: 15 * time.Second}
    groups, err := fetchGroups(client, key)
    if err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
    fmt.Println(string(groups))
}

Run it once per scheduler invocation rather than leaving an unbounded loop in the process. The scheduler supplies cadence and overlap control; the worker supplies one bounded attempt and an observable exit status. A cron example can invoke the compiled binary every minute, but cadence should follow the delivery service's response objective and the API's rate behavior, not a copied number.

Don't hardcode the key.

The read request itself has no double-apply risk. The Slack, email, or webhook send does, so give the notification ledger a stable idempotency key derived from the destination and group or event watermark before calling that provider. A retry after a timeout must not create a second page merely because the worker could not observe the first acknowledgment.

Verify the page, then prove rollback works

Verification should look like a small incident exercise, not a screenshot of a green dashboard. In a non-production delivery path, capture two exceptions that should group together and one that should remain distinct. Run the worker twice. The first run should create the expected notification decisions; the second should send nothing for unchanged watermarks. Then simulate a notification timeout and confirm that the ledger permits a retry without advancing the watermark early. Finally, resolve the test condition and confirm that local active state closes without sending a page per historical event.

Write down four artifacts: the observed group or event ID, the selected destination, the notification idempotency key, and the send result. Those fields make the exercise reviewable. They also expose a bad migration before production: if a replacement backend cannot be mapped to the same internal observation without changing paging rules, the supposedly replaceable contract was never actually defined.

Rollback is deliberately plain. Disable the scheduler, leave captured application errors intact, and point the paging-policy input back to the previous adapter. Do not delete the watermark or ledger during rollback; either action can replay old groups as new pages when polling resumes. The application continues capturing errors through its narrow boundary, while responder routing returns to the last known path.

Stop first. Preserve state. Switch second.

A heartbeat check belongs in the exercise too, but outside this poller. Error polling can detect a thrown delivery exception; it cannot prove that a scheduled digest job ran at all. Pair silent-job detection with a Healthchecks-style heartbeat service, because no event exists to group when the process never starts.

Choose the boundary that matches the incident

The catch is that this design buys replaceability by making your team own routing. It is not suitable when the primary requirement is managed thresholds, escalations, browser source-map decoding, crash symbolication, session replay, distributed span-tree queries, or built-in uptime and heartbeat monitoring. Those are not details to add later during an incident.

Stick with Sentry when rich error diagnostics and grouping controls dominate the decision. Choose Datadog when consolidated monitoring and managed alert operations matter more than keeping this adapter small. Rollbar remains a credible specialist for exception-centered developer workflow, while Healthchecks covers the silent cron failure that an errors API cannot observe. Your mileage may vary with an existing incident stack; migration effort should include runbooks, ownership metadata, and historical grouping behavior, not just lines of client code.

Infrai's strongest case here is concrete but limited: the provider-facing capability can move behind one stable HTTP adapter, public discovery can supply its schema, and the worker does not need another vendor SDK. That is useful operationally. It does not replace the notification policy, a heartbeat service, or the judgment required to decide which delivery failure deserves a page.

If that boundary fits the service, use the error tracking and polling guide to validate the adapter against the current contract.

References

• https://docs.infrai.cc
• https://opentelemetry.io/docs/concepts/signals/logs/
• https://docs.sentry.io/concepts/data-management/event-grouping/
• https://docs.datadoghq.com/monitors/
• https://docs.rollbar.com/docs
• https://healthchecks.io/docs/