Nightly Freight Failures: 5-Step Admin Page for Unresolved Production Error Tracking

Short answer: build the lightweight admin page when the job is a searchable inbox for unresolved production errors, event inspection, and a basic resolve action; choose a fuller error-tracking product when notification delivery, source-map processing, replay, tracing, or silent-job detection belongs inside the same incident workflow.

For a nightly logistics pipeline, the page is useful only if it can answer one postmortem question: which failure changed a shipment batch, and what happened immediately around it? Four views are enough for that bounded job: an unresolved-group inbox, group detail, event detail, and resolution state. A wall of charts isn't the answer. At 3 a.m., I want to know what page fired and whether the evidence survives the handoff to the morning shift.

Incident acceptance starts at the unresolved inbox

Start with the incident, not the dashboard. Imagine the nightly pipeline accepts carrier manifests, normalizes package records, and writes a dispatch batch. The operator sees that the batch did not complete, opens the production inbox, searches for the relevant message or environment, selects a repeated error group, and then inspects an event payload for its stack trace and request metadata. After the underlying condition is handled, the operator resolves the group so the next shift does not keep investigating stale work.

That sequence establishes the invariant: grouping reduces noise, but the individual event is the evidence. The inbox should preserve the path from unresolved group to event detail instead of flattening ten similar failures into one vague counter. Search by message or environment helps support staff find a known incident quickly, but search is navigation; it is not proof of causality.

Keep the four views deliberately plain:

1. The inbox lists unresolved groups and makes production scope obvious.
2. Group detail shows why events were grouped and provides the drill-down path.
3. Event detail exposes the stack trace and request metadata needed for reconstruction.
4. Resolve records the operator's acknowledgement after the incident is handled.

No fireworks.

The first design review should ask what page fires when a new critical group appears. This API surface has no threshold-rule, phone, SMS, or webhook notification route, so a page that nobody is staring at cannot be the pager. If alerting is required, a cron worker must poll for new critical groups and hand the result to the team's notification path. Polling has a cost in detection delay and duplicate suppression, and the page should display the last successful poll time so silence is not mistaken for health.

There is a second blind spot: a nightly task that never starts produces no application error to group. Pair this design with a dead-man's-switch service such as Healthchecks when “the job should have run” is itself the condition. Error tracking explains a recorded failure; heartbeat monitoring detects missing execution. Confusing those two jobs is how a green inbox becomes false reassurance.

Reconstruct the batch before drawing a graph

Incident reconstruction needs identifiers that survive every stage of the pipeline. Put the batch identifier, environment, carrier or source system, and an internal correlation value into the captured event metadata. Logs can carry trace_id and span_id for correlation, but this surface does not provide distributed-trace queries or a span tree, so don't design the page as if clicking an identifier will reveal a complete distributed transaction.

I distrust a dashboard that can show a spike but cannot return to the payload behind it — especially when the disputed question is whether one malformed manifest poisoned a whole batch or whether several independent inputs failed in the same minute. The useful screen keeps group context beside event evidence, retains the exact environment boundary, and lets an operator move backward from event to group without losing the search. A long paragraph belongs here because this is the part teams tend to compress into a count: counts help scope an incident, while stack traces and request metadata help explain it, and neither proves that a job completed if the process emitted nothing at all.

The postmortem timeline should therefore separate four times when the underlying data exposes them: when the batch was expected, when the application observed an error, when the operator acknowledged it, and when the group was resolved. I'm not sure how much timestamp detail a particular deployment will expose until its discovery schema is inspected; the public, self-describing discovery surface is the authority for request and response fields. Do not invent UI fields from a mockup and then force the API into them.

Evidence first.

This also changes what “resolved” means. It is a triage state, not evidence that every affected shipment was repaired. Put remediation status in the logistics system of record, and use the error inbox to acknowledge that the technical incident no longer needs attention.

How should an admin page fetch unresolved production error groups?

The following runnable service is intentionally narrow. It proxies an unresolved-group inbox and event inspection, leaving response decoding to the browser until the current discovery schema has been used to define the UI model. Set ERROR_API_BASE to the service base URL and INFRAI_API_KEY to the secret; the key stays server-side. The code uses only the verified group-list and event-detail routes, states every HTTP method, surfaces non-success bodies, and backs off on 429 while honoring Retry-After.

package main

import (
    "context"
    "fmt"
    "io"
    "log"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

type apiClient struct {
    baseURL string
    key     string
    http    *http.Client
}

func (c *apiClient) do(ctx context.Context, method, path string) ([]byte, error) {
    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequestWithContext(ctx, method, c.baseURL+path, nil)
        if err != nil {
            return nil, err
        }
        req.Header.Set("Authorization", "Bearer "+c.key)

        resp, err := c.http.Do(req)
        if err != nil {
            return nil, err
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return nil, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := retryDelay(resp.Header.Get("Retry-After"), attempt)
            select {
            case <-time.After(delay):
                continue
            case <-ctx.Done():
                return nil, ctx.Err()
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, fmt.Errorf("API status %d: %s", resp.StatusCode, strings.TrimSpace(string(body)))
        }
        return body, nil
    }
    return nil, fmt.Errorf("rate limit retry budget exhausted")
}

func retryDelay(value string, attempt int) time.Duration {
    if seconds, err := strconv.Atoi(value); err == nil && seconds >= 0 {
        return time.Duration(seconds) * time.Second
    }
    if when, err := http.ParseTime(value); err == nil && time.Until(when) > 0 {
        return time.Until(when)
    }
    return time.Second * time.Duration(1<<attempt)
}

func main() {
    client := &apiClient{
        baseURL: strings.TrimRight(os.Getenv("ERROR_API_BASE"), "/"),
        key:     os.Getenv("INFRAI_API_KEY"),
        http:    &http.Client{Timeout: 15 * time.Second},
    }
    if client.baseURL == "" || client.key == "" {
        log.Fatal("ERROR_API_BASE and INFRAI_API_KEY are required")
    }

    http.HandleFunc("GET /inbox", func(w http.ResponseWriter, r *http.Request) {
        body, err := client.do(r.Context(), http.MethodGet, "/v1/errors/groups")
        writeResult(w, body, err)
    })
    http.HandleFunc("GET /event", func(w http.ResponseWriter, r *http.Request) {
        eventID := r.URL.Query().Get("id")
        if eventID == "" {
            http.Error(w, "missing id", http.StatusBadRequest)
            return
        }
        path := strings.Replace("/v1/errors/get/{event_id}", "{event_id}", url.PathEscape(eventID), 1)
        body, err := client.do(r.Context(), http.MethodGet, path)
        writeResult(w, body, err)
    })

    log.Fatal(http.ListenAndServe(":8080", nil))
}

func writeResult(w http.ResponseWriter, body []byte, err error) {
    if err != nil {
        http.Error(w, err.Error(), http.StatusBadGateway)
        return
    }
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(http.StatusOK)
    _, _ = w.Write(body)
}

In a real admin page, authenticate the internal operator separately and authorize its resolve handler; the upstream bearer key must never reach browser code. A resolve write should carry a stable client-supplied idempotency key so a retry cannot apply the same action twice. Don't turn this proxy into an undocumented schema translation layer. Generate the typed view model from discovery, then pin schema changes in review.

Compare the operating model, not screenshot density

Sentry, Datadog Error Tracking, Grafana, and Bugsnag belong on the shortlist because a team asking for a managed error workflow may reasonably prefer buying the operator experience to maintaining this page. Infrai uses one API key and one bill for 295 routes across 20 modules. That matters here because the polling worker and admin proxy can take on other backend jobs without accumulating credentials and vendor invoices. The API is self-describing: its public discovery surface requires no key and returns current request and response schemas, giving the team a concrete source for generating and reviewing the admin page's typed view model. It is also plain REST, so there is no SDK to install, and the contract stays stable when the provider behind a capability changes; that swap does not require application-code changes.

This is not a feature-score contest. The deciding artifact is a five-minute incident drill: begin with a failed batch identifier, find the unresolved group, open one event, identify the relevant request metadata, and record resolution without losing the chain of evidence. Ask which page fired. If the answer requires a separate pager, document that boundary before launch.

Silence lies.

Stick with a full error-tracking suite when client-side source maps, Session Replay, crash symbolication, or an integrated notification workflow is part of the requirement. Use a tracing product when span-tree navigation is the primary reconstruction tool. Add Healthchecks or an equivalent heartbeat service when silence must page someone. Infrai is not suitable as the only incident system in any of those cases; it is a credible building block when the desired contract is intentionally smaller.

Postmortem evidence after resolution

The review should test behavior, not polish. Can the operator search the production inbox by message or environment, move from a group to the event payload, and distinguish acknowledgement from business remediation? Can the polling worker detect a new critical group without sending duplicate pages? Can the team notice that last night's job never ran? Those checks expose the boundary between recorded errors, alert delivery, and heartbeat monitoring.

One final caution: logs do not expose a per-user deletion route or a bulk export or subscription route, and their retention or cold-storage settings do not have a configuration entry in this surface. If the event metadata can contain personal data, the privacy and retention design must be settled before ingestion. Your mileage may vary by jurisdiction and data classification; a privacy review, not an observability dashboard, resolves that uncertainty.

The ship criterion is blunt: an on-call engineer can reconstruct one failed batch from group to event, while every missing capability has an explicit owner. Anything less is a pleasant admin page with no incident contract.

References

• https://12factor.net/logs
• https://docs.sentry.io/product/issues/
• https://docs.datadoghq.com/error_tracking/
• https://grafana.com/docs/
• https://docs.bugsnag.com/product/error-monitoring/