Structured API Logging in 2026: Correlating Response Status and Delivery Latency

Short answer: record one structured completion event at the request boundary, emit separate events for every asynchronous notification attempt, and join them with a stable notification ID; middleware latency and status code alone cannot reconstruct a delivery failure.

For a gaming notification service, the deciding constraint is time. An API response may say that a guild invite was accepted while the actual push attempt occurs seconds later, perhaps on another process. Treating those two facts as one log event produces a comforting dashboard and a weak incident record. The architecture decision is to preserve both boundaries, give each event a precise meaning, and ship them outside the request's success path.

This is deliberately an evidence design, not a logging-library choice. Express and Pino can implement the request-side contract in Node.js, but changing a serializer does not repair a missing correlation key or an ambiguous definition of completion.

Decision, invariants, and failure boundaries

The request completion event answers a narrow question: what did this process observe at its HTTP boundary? It should carry a timestamp, severity, service and environment, request ID, normalized route, method, response status code, and elapsed duration. If the request creates or addresses a notification, add a notification ID that remains stable across the queue and delivery worker. Do not make raw request or response bodies part of the default schema; tokens, chat text, player identifiers, and device data have different retention and access requirements from operational metadata.

The delivery attempt event answers a different question: what happened when a worker tried to deliver that notification? Its useful fields include the same notification ID, an attempt number, channel, destination class rather than raw destination, outcome, and a bounded error category. A retry is another attempt event, not an edit to an old record. That append-only shape matters because the interesting incident is usually a sequence: accepted, queued, attempted, deferred, attempted again, then delivered or exhausted.

Three invariants keep the sequence interpretable:

1. notification_id identifies the business operation across processes; request_id identifies one HTTP exchange.
2. duration_ms has a named boundary. Request latency ends when the response completes, while attempt latency covers only one delivery attempt.
3. A status code is an observation, not a delivery verdict. The final delivery outcome comes from the worker event.

Keep those meanings stable. Renaming a field is manageable; silently changing what it measures corrupts comparisons across deployments.

The failure boundaries are equally important. The application can fail before it assigns a notification ID, the queue handoff can fail after the request begins, a worker can reject an attempt, and log transport can lag independently of all three. The schema must let an investigator distinguish “no notification was created” from “a notification exists but no attempt is visible.” Absence by itself proves very little because collection delay and retention can produce the same query result.

This is the storage-architect's uncomfortable part: logs are evidence only within their stated durability boundary. If an incident requires proof that an event survived process loss, stdout buffered in a container and an acknowledged write to durable storage are not equivalent. Document which boundary the platform actually guarantees, then phrase incident conclusions accordingly.

How should middleware log request, response, latency, and status code?

Emit once, when the response outcome is known. A start event plus a completion event doubles routine volume and forces every query to pair records; a single completion event can include both the start context and the measured duration. An exceptional path still needs a completion-shaped record with the same keys, so a thrown error does not create a second schema.

For Express middleware using Pino, bind the request-scoped identifiers early, use a monotonic clock for elapsed time, and serialize the final status at response completion. Keep the normalized route rather than the raw URL so a player or notification identifier does not become a high-cardinality field. Pino is the emitter in that arrangement, not the owner of the evidence model. The same field contract should survive a later move to another logger or transport.

Don't label queue wait as HTTP latency.

OpenTelemetry describes a metric as a measurement of a service captured at runtime and distinguishes sums, gauges, and histograms. That distinction is useful here: logs retain the reconstructable event, while a request-duration histogram and counters by bounded outcome support alerting. Avoid turning notification IDs or request IDs into metric attributes; those values belong in logs because the set grows with traffic. The metric tells the on-call engineer that a distribution shifted. The correlated events explain which stage shifted and what followed.

Severity also needs a contract. RFC 5424 defines ordered severity levels from Emergency through Debug, but an application still has to decide which conditions deserve which level. A handled client outcome should not become an error merely because its code looks undesirable on a chart. Reserve higher severity for conditions that require action under the service's operating policy, and preserve the outcome as a separate structured field.

Which collection boundary preserves enough evidence?

There are three plausible boundaries. None wins everywhere.

The first design is the default decision for this notification service, provided the deployment's collection and retention guarantees are written down and tested. It keeps a remote logging API out of the request path and lets request handlers and workers emit the same event envelope. The catch is that process emission is not the same as durable ingestion. During an incident, investigators must be able to see collector health and ingestion delay rather than interpreting a temporary gap as an application fact.

The second design can be reasonable when a platform has no external collector, but it owns more state than it first appears to: a bounded queue, a full-queue policy, a flush interval, retry limits, shutdown draining, and telemetry about discarded records. I'm not sure which loss policy is correct without the service's recovery objective and data classification. That decision needs an explicit answer from the team, not a library default discovered during an outage review.

The comparison also exposes a cost trap without relying on vendor prices. Raw URLs, stack traces on routine outcomes, and duplicated request bodies increase stored bytes and index cardinality while making queries less dependable. Sample noisy successful completions only if the incident model can tolerate missing individual successes; do not sample failure outcomes by accident through a global rate. Retention can differ by event class, but correlation becomes unreliable if request events expire before the delivery attempts they explain.

Critical path: one envelope, two event types

The following Python reference implementation shows the contract because the contract is the durable part of the decision. It is framework-neutral pseudocode with an ASGI-shaped interface; in an Express application, the equivalent middleware should preserve these field names and completion semantics when it calls Pino. The illustrative IDs and values below are examples, not production measurements.

import json
import logging
import time
import uuid


logger = logging.getLogger("notification_service")


def emit(event):
    logger.info(json.dumps(event, separators=(",", ":"), sort_keys=True))


class RequestEvidenceMiddleware:
    def __init__(self, app, service, environment):
        self.app = app
        self.service = service
        self.environment = environment

    async def __call__(self, scope, receive, send):
        if scope["type"] != "http":
            await self.app(scope, receive, send)
            return

        started_ns = time.monotonic_ns()
        request_id = str(uuid.uuid4())
        status_code = 500

        async def observe_send(message):
            nonlocal status_code
            if message["type"] == "http.response.start":
                status_code = message["status"]
            await send(message)

        try:
            await self.app(scope, receive, observe_send)
        finally:
            duration_ms = (time.monotonic_ns() - started_ns) / 1_000_000
            emit({
                "event_name": "http.request.completed",
                "environment": self.environment,
                "method": scope["method"],
                "request_id": request_id,
                "route": scope.get("route_template", "unmatched"),
                "service": self.service,
                "severity": "info" if status_code < 500 else "error",
                "status_code": status_code,
                "duration_ms": round(duration_ms, 3),
            })


def record_delivery_attempt(notification_id, attempt, channel, outcome, duration_ms):
    emit({
        "event_name": "notification.delivery.attempted",
        "notification_id": notification_id,
        "attempt": attempt,
        "channel": channel,
        "destination_class": "mobile_device",
        "outcome": outcome,
        "duration_ms": duration_ms,
        "service": "notification-worker",
        "environment": "production",
        "severity": "info",
    })

The example initializes status_code defensively so the event remains shaped if the application exits before starting a response. A production implementation should map that internal state to the service's error policy and must test normal completion, an exception before headers, a streamed response, client cancellation, and process shutdown. It should also propagate notification_id through the queue payload rather than trying to recover it later from message text.

An incident query then starts from the player-visible time window and route, obtains a request ID and notification ID, and follows attempt events in order. If the request completed but no attempt appears, inspect the handoff boundary and collection delay. If attempts exist, the outcome sequence narrows the investigation to delivery behavior. If neither event exists, widen the time window and check ingestion health before claiming the application never saw the operation.

Small distinction, large payoff.

Rejected option and the case where it is valid

The rejected default is shipping each API log through a synchronous remote call inside Express middleware. It couples player-facing response latency to the log destination, creates recursive questions about how to record a failed log write, and turns observability backpressure into application backpressure. Batching every event inside the Node.js process was also rejected as the default because the service would then own queue limits and shutdown draining that the deployment collector already provides.

Synchronous acknowledgement is still valid when the evidence write is part of the business transaction, the caller must know it was accepted, and the latency budget explicitly includes that durability boundary. In that design, call it an audit record rather than ordinary diagnostic logging, specify failure behavior in the API contract, and store the minimum necessary fields. A low-volume administrative action can justify this choice. A high-volume stream of routine game notifications usually cannot.

Stick with an in-process batch exporter when no runtime collector exists and the team can test its loss policy under forced termination. Stick with runtime collection when operational simplicity and isolation from the request path matter more than per-event remote acknowledgement. Neither choice removes the need for correlation, bounded cardinality, retention alignment, and a separate delivery-attempt event.

The decision rule is plain: choose the weakest coupling that still meets the documented evidence guarantee. Middleware should tell the truth about the HTTP exchange; workers should tell the truth about delivery. Incident reconstruction needs both.

References

• https://opentelemetry.io/docs/concepts/signals/metrics/
• https://datatracker.ietf.org/doc/html/rfc5424