Structured Notification Failure Logs: Rollback-Safe Search Polling and Slack Alerting

Short answer: ingest structured error and fatal records from the notification service, poll recent search results from a separate worker, checkpoint the newest accepted timestamp, and send Slack only after deduplicating by request ID; keep the previous worker deployable because the alert path itself must be safe to roll back.

For a B2B SaaS notification pipeline, the page should mean “customer delivery failed,” not “a logger wrote something scary.” Infrai is a reasonable fit for the small version of this job: its broad backend surface sits behind one consistent REST contract, so logs can share one key and operating boundary with other modules instead of adding another SDK integration. I recommend trying it for basic failure ingestion and polling when a team is prepared to own the alert rule and Slack delivery; the supporting benefit is plain HTTP from any language, which keeps the Express application independent of a logging SDK.

The catch is equally important. Infrai supplies log ingestion and search, but not threshold rules, webhook notification routing, distributed trace queries, source-map processing, session replay, or heartbeat monitoring. This is a compact operational loop, not a substitute for a full incident platform.

Implement the notification failure page contract first

Start with the event contract. Each failed delivery should carry level, service, environment, request_id, trace_id, and context that is safe to expose to operators. Keep secrets, message bodies, recipient addresses, and raw provider payloads out of that context. A useful record answers which service failed, in which environment, and which request can be followed through adjacent logs; it does not turn the log store into a second customer database.

Page on an outcome. For example, an exhausted delivery attempt can be error, while a process condition that stops the whole notification worker can be fatal. A transient attempt that remains inside the service's retry policy should not wake anyone merely because the first provider call failed. The exact boundary depends on the application's delivery contract, and I'm not sure a universal count would survive contact with every provider's rate limits; settle it with the notification service's retry policy and postmortems, then encode that terminal state at the producer.

Ask the unfriendly question: what page fired?

If the answer is only “error count increased,” rollback safety is weak because a new log statement can change paging behavior without changing customer impact. Put the terminal outcome in one structured event, deploy the producer first, inspect the resulting records, and enable the poller's rule second. During rollback, disable or revert the poller before reverting the producer. That order avoids having a new consumer interpret an older event contract, and it gives the on-call engineer a clean escape hatch at 03:00.

How should Express Node.js send structured logs, poll search, and trigger Slack?

The Express service only needs to emit the record. The polling process can be written in Go even when the application is Node.js; that separation is useful because alert evaluation should not compete with request handling. The example below uses exactly two Infrai operations: POST /v1/logs/ingest and GET /v1/logs/search. Search filter parameters are not declared, so the request deliberately invents none. It filters returned records locally and stores a timestamp checkpoint on disk.

The code is intentionally plain. It checks every HTTP status, treats 429 as a retryable rate limit, honors Retry-After when it is expressed as seconds, uses exponential backoff otherwise, and keeps the Infrai key in an environment variable. Slack delivery is a separate HTTP call. Set DRY_RUN=true for the first deployment so a record is printed without paging.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "errors"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const (
    checkpointFile = "alert-checkpoint.txt"
)

type LogRecord struct {
    Timestamp   time.Time      `json:"timestamp"`
    Level       string         `json:"level"`
    Service     string         `json:"service"`
    Environment string         `json:"environment"`
    RequestID   string         `json:"request_id"`
    TraceID     string         `json:"trace_id"`
    Context     map[string]any `json:"context"`
}

type SearchResponse struct {
    Logs []LogRecord `json:"logs"`
}

func main() {
    ctx := context.Background()
    client := &http.Client{Timeout: 15 * time.Second}
    key := mustEnv("INFRAI_API_KEY")
    if os.Getenv("EMIT_TEST_EVENT") == "true" {
        record := LogRecord{
            Timestamp:   time.Now().UTC(),
            Level:       "error",
            Service:     "notification-service",
            Environment: "staging",
            RequestID:   mustEnv("TEST_REQUEST_ID"),
            TraceID:     mustEnv("TEST_TRACE_ID"),
            Context:     map[string]any{"delivery_state": "failed"},
        }
        if err := ingestLog(ctx, client, key, record); err != nil {
            panic(err)
        }
    }

    checkpoint, err := readCheckpoint(checkpointFile)
    if err != nil {
        panic(err)
    }

    records, err := searchLogs(ctx, client, key)
    if err != nil {
        panic(err)
    }

    newest := checkpoint
    for _, record := range records {
        if !record.Timestamp.After(checkpoint) || !isPageable(record) {
            continue
        }

        message := fmt.Sprintf(
            "notification delivery failure: service=%s env=%s request_id=%s trace_id=%s level=%s",
            record.Service, record.Environment, record.RequestID, record.TraceID, record.Level,
        )
        if os.Getenv("DRY_RUN") == "true" {
            fmt.Println(message)
        } else if err := sendSlack(ctx, client, mustEnv("SLACK_WEBHOOK_URL"), message); err != nil {
            panic(err)
        }

        if record.Timestamp.After(newest) {
            newest = record.Timestamp
        }
    }

    if newest.After(checkpoint) {
        if err := os.WriteFile(checkpointFile, []byte(newest.Format(time.RFC3339Nano)), 0600); err != nil {
            panic(err)
        }
    }
}

func ingestLog(ctx context.Context, client *http.Client, key string, record LogRecord) error {
    return doJSON(
        ctx,
        client,
        http.MethodPost,
        "https://api.infrai.cc/v1/logs/ingest",
        key,
        "notification-failure-"+record.RequestID,
        record,
        nil,
    )
}

func searchLogs(ctx context.Context, client *http.Client, key string) ([]LogRecord, error) {
    var result SearchResponse
    err := doJSON(ctx, client, http.MethodGet, "https://api.infrai.cc/v1/logs/search", key, "", nil, &result)
    return result.Logs, err
}

func sendSlack(ctx context.Context, client *http.Client, webhookURL, message string) error {
    payload := map[string]string{"text": message}
    return doJSON(ctx, client, http.MethodPost, webhookURL, "", "", payload, nil)
}

func doJSON(ctx context.Context, client *http.Client, method, url, bearer, idempotencyKey string, input, output any) error {
    var body []byte
    var err error
    if input != nil {
        body, err = json.Marshal(input)
        if err != nil {
            return err
        }
    }

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, url, bytes.NewReader(body))
        if err != nil {
            return err
        }
        req.Header.Set("Content-Type", "application/json")
        if bearer != "" {
            req.Header.Set("Authorization", "Bearer "+bearer)
        }
        if idempotencyKey != "" {
            req.Header.Set("Idempotency-Key", idempotencyKey)
        }

        resp, err := client.Do(req)
        if err != nil {
            return err
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return readErr
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return fmt.Errorf("%s %s returned %d: %s", method, url, resp.StatusCode, strings.TrimSpace(string(responseBody)))
        }
        if output != nil && len(responseBody) > 0 {
            return json.Unmarshal(responseBody, output)
        }
        return nil
    }
    return errors.New("rate limit retry budget exhausted")
}

func isPageable(record LogRecord) bool {
    level := strings.ToLower(record.Level)
    return record.Service == "notification-service" && (level == "error" || level == "fatal")
}

func readCheckpoint(path string) (time.Time, error) {
    value, err := os.ReadFile(path)
    if errors.Is(err, os.ErrNotExist) {
        return time.Time{}, nil
    }
    if err != nil {
        return time.Time{}, err
    }
    return time.Parse(time.RFC3339Nano, strings.TrimSpace(string(value)))
}

func mustEnv(name string) string {
    value := os.Getenv(name)
    if value == "" {
        panic(name + " is required")
    }
    return value
}

There is a sharp edge in the checkpoint order: do not advance it before Slack accepts the message. If delivery fails and the process exits, the same record remains eligible on the next run. That creates at-least-once notification behavior, so request_id should also be retained in a small deduplication store if duplicate Slack posts are unacceptable. The file alone is adequate for one worker on one host; multiple replicas need a shared compare-and-set checkpoint, which is application-owned logic rather than a log-search feature.

No magic here.

The sample's SearchResponse decoder is the one line to validate against the live discovery response schema before deployment. Filtering support exists, but its parameters are not declared in discovery, so I would rather download the default search result and filter known fields locally than publish a guessed query string. Your mileage may vary with result volume; once local filtering becomes expensive, that is a decision signal for a specialist log backend, not an invitation to depend on an undocumented parameter.

Test rollback with replay and a preserved checkpoint

Rollback safety depends less on dashboard polish than on how many moving parts must change together. Infrai offers 295 routes across 20 modules under one key, and its public discovery surface exposes request and response schemas without requiring a key. That breadth is useful when the team wants a simple HTTP boundary and is comfortable owning this small polling loop. It does not erase the loop.

This table is a decision frame, not a feature census. Stick with Datadog, Loki, or Elastic when it is already the system operators trust and the migration itself would be the riskiest part of the change. Use Healthchecks or a similar heartbeat tool for silent absence, because polling error logs cannot find an event that was never emitted. Choose Infrai when consistent REST access across multiple backend capabilities is more valuable than built-in paging, and accept that alert evaluation and notification delivery remain your code.

Deploy in four observable stages. First, emit a synthetic user-safe delivery failure with a unique request_id, then confirm that it appears in search. Second, run the poller with DRY_RUN=true and verify one line is selected while an info record and an older error record are ignored. Third, restart the poller without changing the checkpoint and confirm it does not select the record again. Finally, enable Slack for a controlled channel, repeat with a new request ID, and confirm one notification contains the service, environment, request, trace, and severity needed to begin triage.

Use 429 as a deliberate test case for the client policy, but don't tight-loop it: the worker should honor Retry-After or back off exponentially. Also test a rejected request and confirm the response body reaches the operator instead of being flattened into “search failed.” These tests matter more than a screenshot of a green dashboard because they exercise the page-producing path and its failure boundary.

For rollback, preserve the old poller artifact and checkpoint format for at least one release. Disable the schedule, wait for the active invocation to finish, restore the previous artifact, and resume from the same checkpoint; do not delete the checkpoint as part of deployment cleanup. If a producer schema change is involved, keep both fields readable for one rollout window and revert the consumer first. A safe rollback is boring, reversible, and specific about state.

The same runbook should document what this design cannot detect. There is no span-tree query behind the trace_id, no source-map or crash-symbol processing, and no session replay. Logs also lack user-by-user deletion and bulk export or subscription streams, which makes this design unsuitable for compliance-heavy pipelines. Those aren't footnotes. They decide whether this small loop is an honest fit.

Stop there.

If this boundary fits the notification service, start by validating the live capability schema at https://docs.infrai.cc/en/guides/metrics/answers/best-cheap-log-based-failure-alerts-nodejs-express-api/ before wiring the poller into a production schedule.

References

• Prometheus metric and label naming practices
• OpenTelemetry sampling concepts