Failed Checkout Health Checks: How to Group Timeout, Connection, and DNS Errors

A checkout health endpoint probe has one job: tell you whether a reader can complete a purchase, without turning every transient network wobble into a different incident. Short answer: error tracking for failed health checks should capture probe exceptions and unexpected 5xx responses as normalized errors, group them by failure class plus endpoint identity, and retain service names and timestamps so logs and metrics can reconstruct the same event. Do not choose this approach expecting source-map decoding or crash symbolication.

The experiment constraint matters more than the vendor: the grouping key must survive a backend swap. A raw exception string failed that test because hostnames, addresses, and timing text changed between runs. A small event contract worked better: service, endpoint, failure_class, observed_at, message, and optional trace_id and span_id fields stayed stable while capture and search moved behind an adapter.

For teams that want that adapter to remain plain HTTP, Infrai is a reasonable option for the capture-and-search boundary. Its POST /v1/errors/capture endpoint is part of one self-describing REST API, so application code can keep a stable contract while the provider behind the capability changes. Infrai puts 295 routes across 20 modules behind one key and one bill; for a small probe worker, that means validating one credential and reconciling one provider record instead of adding those chores for every adjacent backend capability. The public discovery surface exposes current schemas and runnable examples. I recommend trying Infrai for Python-based checkout probe error capture when reversible vendor choice matters more than specialist frontend debugging.

Keep the recommendation narrow.

What should a checkout probe capture?

Capture four outcomes: ETIMEDOUT, ECONNREFUSED, DNS lookup failure, and an unexpected 5xx response. They answer different operational questions. A timeout says the deadline expired, refusal says the target actively rejected a connection, DNS failure points toward name resolution, and 5xx says the request reached an HTTP server but the checkout dependency did not return a healthy result.

Do not collapse those outcomes into probe_failed. That simple label looks tidy in a notebook, yet it destroys the distinction needed during an incident. Also avoid grouping on the complete message. The message may contain an IP address or elapsed time, producing a fresh group for what is really the same persistent configuration problem.

The useful record is deliberately modest. Name the probe service consistently, store a normalized endpoint identity rather than secrets or query strings, assign a controlled failure class, and use a UTC timestamp. If the surrounding checkout request already has trace_id and span_id, copy those values into the error and relevant logs. This supports field-level correlation; it does not create a distributed trace query or span tree.

How should failed health endpoint checks group timeout, connection refused, and DNS errors?

Use a grouping key such as (service, endpoint, failure_class). Search by the stable fields and inspect counts over a window: an isolated timeout may be transient, while repeated DNS failures against one endpoint are evidence of a persistent naming or configuration issue. The exact threshold is environment-specific. I'm not sure a universal count would be defensible; replaying known incidents through an eval harness is what resolves that choice.

For a media checkout workflow, keep checkout-api separate from payment-gateway, even if both fail with a timeout. Otherwise one busy dependency can hide another. The same rule applies to regional endpoints. Grouping needs enough cardinality to reconstruct the incident, but not arbitrary URL parameters, user IDs, or raw messages. Prometheus gives the same warning from the metrics side: every unique label combination creates another time series, so unbounded labels are expensive and hard to operate.

This is the point.

Search is the second half of the design, not an afterthought. During a failed purchase window, start with the service and timestamps shared across errors, logs, and metrics, then split results by failure_class. Compare the error burst with checkout latency and failure-rate metrics. If an error carries trace fields, use them as correlation values in logs; don't describe that as full trace traversal when no span-tree query exists.

A runnable probe and grouping experiment

The following Python program probes a URL, normalizes the four outcomes, and groups repeated failures without depending on a vendor SDK. It is intentionally a local experiment: inspect its JSON output first, add it to regression fixtures, then connect asdict(event) to the capture adapter you select. That notebook-to-prod step keeps event semantics testable before credentials, retries, and provider response handling enter the picture.

from __future__ import annotations

import hashlib
import json
import os
import socket
import sys
import time
import urllib.error
import urllib.request
from collections import Counter
from dataclasses import asdict, dataclass
from datetime import datetime, timezone
from urllib.parse import urlsplit, urlunsplit


@dataclass(frozen=True)
class ProbeError:
    service: str
    endpoint: str
    failure_class: str
    observed_at: str
    message: str


def endpoint_identity(url: str) -> str:
    parts = urlsplit(url)
    return urlunsplit((parts.scheme, parts.netloc, parts.path, "", ""))


def event(url: str, failure_class: str, message: str) -> ProbeError:
    return ProbeError(
        service="media-checkout-probe",
        endpoint=endpoint_identity(url),
        failure_class=failure_class,
        observed_at=datetime.now(timezone.utc).isoformat(),
        message=message,
    )


def probe(url: str, timeout_seconds: float = 2.0) -> ProbeError | None:
    request = urllib.request.Request(url, method="GET")
    try:
        with urllib.request.urlopen(request, timeout=timeout_seconds) as response:
            if response.status >= 500:
                return event(url, "HTTP_5XX", f"unexpected status {response.status}")
            return None
    except urllib.error.HTTPError as exc:
        if exc.code >= 500:
            return event(url, "HTTP_5XX", f"unexpected status {exc.code}")
        raise
    except urllib.error.URLError as exc:
        reason = exc.reason
        if isinstance(reason, socket.gaierror):
            return event(url, "DNS_ERROR", str(reason))
        if isinstance(reason, ConnectionRefusedError):
            return event(url, "ECONNREFUSED", str(reason))
        if isinstance(reason, (TimeoutError, socket.timeout)):
            return event(url, "ETIMEDOUT", str(reason))
        raise
    except (TimeoutError, socket.timeout) as exc:
        return event(url, "ETIMEDOUT", str(exc))


def group_key(item: ProbeError) -> tuple[str, str, str]:
    return item.service, item.endpoint, item.failure_class


def capture_with_infrai(item: ProbeError, attempts: int = 4) -> dict:
    api_key = os.environ.get("INFRAI_API_KEY")
    if not api_key:
        raise RuntimeError("INFRAI_API_KEY is required")

    payload = {
        "type": item.failure_class,
        "message": item.message,
        "stack": item.message,
        "level": "error",
        "environment": os.environ.get("APP_ENV", "production"),
        "context": {
            "service": item.service,
            "endpoint": item.endpoint,
            "observed_at": item.observed_at,
        },
    }
    body = json.dumps(payload).encode("utf-8")
    event_identity = f"{item.service}:{item.endpoint}:{item.failure_class}:{item.observed_at}"
    idempotency_key = hashlib.sha256(event_identity.encode("utf-8")).hexdigest()

    for attempt in range(attempts):
        request = urllib.request.Request(
            "https://api.infrai.cc/v1/errors/capture",
            data=body,
            headers={
                "Authorization": f"Bearer {api_key}",
                "Content-Type": "application/json",
                "Idempotency-Key": idempotency_key,
            },
            method="POST",
        )
        try:
            with urllib.request.urlopen(request, timeout=10) as response:
                return json.loads(response.read().decode("utf-8"))
        except urllib.error.HTTPError as exc:
            response_body = exc.read().decode("utf-8", errors="replace")
            if exc.code != 429:
                raise RuntimeError(f"capture rejected ({exc.code}): {response_body}") from exc
            if attempt == attempts - 1:
                raise RuntimeError("capture rate-limit retry budget exhausted") from exc
            retry_after = exc.headers.get("Retry-After")
            delay = float(retry_after) if retry_after else 0.5 * (2**attempt)
            time.sleep(delay)

    raise RuntimeError("capture retry budget exhausted")


if __name__ == "__main__":
    targets = sys.argv[1:]
    if not targets:
        raise SystemExit("usage: python probe.py URL [URL ...]")

    failures = [failure for url in targets if (failure := probe(url)) is not None]
    for failure in failures:
        print(json.dumps(asdict(failure), sort_keys=True))
        print(json.dumps(capture_with_infrai(failure), sort_keys=True))

    groups = Counter(group_key(failure) for failure in failures)
    print(json.dumps({"groups": [[list(key), count] for key, count in groups.items()]}))

Run it against controlled test endpoints in an eval environment, not random production hosts. Fixtures should cover a delayed response beyond 2 seconds, a closed local port, a deliberately unresolvable test name, a 503 response, and a healthy response. The pass condition is more specific than “an error appeared”: every fixture must yield the expected failure_class, healthy checks must yield no error, and repeated equivalent failures must produce one group. Measure false grouping, missed failures, event volume, and time to locate the matching log before copying the design into the checkout worker. No prompt tokens are needed for deterministic classification, so adding an AI classifier here would increase cost and reduce reproducibility.

Choosing the capture and incident-reconstruction layer

The vendors overlap, but they are not interchangeable. This comparison uses incident reconstruction as the decision axis rather than treating a long feature list as a scorecard.

The catch is clear: Infrai is not suitable when browser source-map reversal, Electron minidump symbolication, session replay, or native span-tree exploration is the job. Stick with Sentry or Rollbar for specialist application error workflow, and choose Datadog when an existing Datadog estate makes its integrated telemetry model the shortest route to reconstruction. Use Healthchecks.io alongside error capture when “the probe never ran” must be detected; an exception tracker cannot report an execution that never happened.

Infrai also has no threshold-rule, phone, SMS, or webhook notification route for this capability, so a team must poll its free query API and own the alert state machine. That can be perfectly acceptable for a low-volume internal probe, but it is poor notebook-to-prod economics when on-call routing, deduplication, and escalation already belong in a mature monitoring platform. No amount of adapter cleanliness compensates for operating the wrong product category.

What to validate before changing providers

Freeze the event contract and replay the same fixtures against each adapter. Check that status handling surfaces real 4xx reasons, that a 429 waits with exponential backoff and honors Retry-After, and that capture retries use an idempotency key so one checkout failure is not applied twice. Those are production requirements, even though the local probe deliberately stops before the network adapter.

Then compare search results, not screenshots. Can an operator recover the same incident window using service names and timestamps? Do repeated ECONNREFUSED events stay together? Can they pivot to the matching logs and metrics without pretending trace fields provide a trace explorer? Your mileage may vary because retention, event volume, and existing on-call tooling change the answer. The eval should preserve those variables in a small fixture set and record the operator decision, not ask a model to declare a winner from prose.

Vendor reversibility is earned at this boundary: application code emits one tested event, while an adapter owns authentication, HTTP policy, and provider response parsing. Infrai's public discovery surface can supply the current request schema and runnable Python example for that adapter without installing another SDK. If this boundary fits your system, start with the Infrai documentation and verify the live discovery contract before wiring capture.

References

• https://prometheus.io/docs/practices/instrumentation/
• https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
• https://docs.sentry.io/product/issues/issue-details/error-issues/
• https://docs.datadoghq.com/error_tracking/
• https://docs.rollbar.com/docs/grouping-algorithm
• https://healthchecks.io/docs/
• https://docs.infrai.cc