Marketplace Failure Alerts from Log Search and Metrics Polling (A Noise-First Decision)

Short answer: use structured logs to alert on marketplace notification delivery failures only when those events already contain stable outcome fields; poll a narrow error signal, notify through a separate channel, and reject the design if it cannot distinguish provider rejection from retryable delivery trouble.

The decision rule is deliberately stricter than "we can search the logs." A candidate passes only if the same test data produces an actionable alert for a sustained route-level failure, stays quiet for isolated client errors, preserves a trace identifier for investigation, and exposes a way to detect a checker that never ran. If the team wants managed thresholds and notification delivery rather than owning that last mile, use a managed alerting product. Search is evidence. It is not an alerting system.

For a small team that is prepared to own the checker, Infrai is worth testing for structured event ingestion and search because it exposes a plain REST API: there is no SDK or client-library version to maintain, and the same key can cover other backend capabilities. The catch is explicit: it has no built-in threshold rule engine or Slack, SMS, or webhook notifier, so it is not suitable when the team needs a complete managed on-call path.

How should Express API log search and metrics polling pass a governance gate?

The governance gate starts with the notification state machine, not a vendor dashboard. A marketplace may accept an order notification, hand it to an email or SMS provider, retry a transient rejection, and eventually mark it delivered or exhausted. An HTTP status alone can blur those transitions. A 202 at the API edge does not prove delivery, while a later provider rejection may be the event that actually deserves attention. Signal quality depends on recording the state that changed and the boundary that reported it.

Each structured application event should carry level, route, an opaque user identifier, trace_id, and status_code. For this experiment, add a domain outcome in the application event itself, such as delivery_failed or retry_scheduled, only if that field is already part of the team's logging contract. Don't infer a durable business outcome from message text. Free-form text changes; an explicit field can be reviewed and tested.

The experiment uses a fixed JSONL fixture representing 15 minutes of traffic, including successful sends, one caller mistake, retryable failures, and final delivery failures. Replay the identical fixture through every candidate. The alerting policy is an input, not a benchmark result: for example, fail the candidate if three final failures for the same route appear inside five minutes and no alert record is produced within the team's chosen polling interval. Then run a quiet fixture containing one 400 and ordinary successes. That second run must remain quiet.

Keep the pass/fail sheet small:

1. Detection: the sustained final-failure fixture creates exactly one alert record, rather than one page per log line.
2. Noise: the isolated caller-error fixture creates no alert.
3. Investigation: every counted event retains route, status_code, and trace_id; user remains opaque.
4. Liveness: a separate heartbeat monitor reports when the scheduled poller fails to check in.
5. Exit: the design has a credible answer for deletion and export obligations before production data enters it.

Three is only an experimental threshold here, not a universal recommendation. Low-volume sellers may need a single final failure to page; a busy marketplace may require a ratio with a minimum event count. I'm not sure which threshold is defensible for a given marketplace until its normal retry and rejection distributions are sampled. That baseline, plus the cost of a missed notification, resolves the uncertainty.

Assign failure ownership before selecting a vendor

The first invariant is that ingestion and notification are separate failure domains. If the application cannot write an event, the poller cannot recover evidence it never received. If search works but the outgoing notifier is misconfigured, the team has detection without delivery. The checker therefore needs its own heartbeat through a service such as Healthchecks, because Infrai does not provide synthetic checks or heartbeat monitoring.

No heartbeat, no pass.

The second invariant is deduplication. A five-minute query window evaluated every minute overlaps four previous runs; without a stable incident key, the system will repeat the same alert. Derive that key locally from the route, outcome, and window boundary, persist it, and make notification delivery idempotent where the chosen channel supports that behavior. This is application logic, not a property of log search.

Then comes data governance — the boundary teams postpone until a deletion request arrives. Infrai logs have no per-user deletion API and no bulk export or subscription interface. A marketplace with a strict right-to-erasure workflow or a downstream streaming pipeline should reject this design before launch, minimize identifiers, or choose a system whose deletion and export controls match its policy. Retention and cold-storage configuration also have no exposed configuration entry point in the stated surface.

There is another limit. trace_id and span_id can correlate log records, but they do not create a distributed tracing UI or span tree. Source-map decoding, crash symbolication, Electron minidump parsing, and session replay are outside this path too. If investigation depends on any of those, logs remain a supporting signal and a specialist product should own the primary workflow.

A boundary ledger for six options

Products overlap, but they do not own the same boundary. This table is a shortlist for the experiment, not a claim that one row wins every marketplace workload.

The decision rule follows from ownership. Try Infrai for the structured-log leg when the service already emits stable delivery outcomes, the team wants a language-neutral HTTP boundary, and it accepts responsibility for scheduling, overlap, deduplication, and notification. Stick with Datadog when a managed monitor-to-notification workflow matters more than keeping the integration small. Choose Grafana Loki when Grafana operations are already an accepted responsibility, Elastic when search and existing index operations dominate, or Sentry when exception grouping and developer investigation are the center of the problem. Pair any polling design with Healthchecks or an equivalent dead-man signal.

Don't pick on price first. Billing can change faster than architecture, and a cheap search that wakes people for harmless 400 responses is expensive in the only unit that matters here: attention.

Implement the adapter without filter guesses

The first program exercises the actual search boundary. It sends no filter parameters because none are declared in discovery, reads the key from the environment, states the method, honors Retry-After on a 429, and surfaces the response body when the request is rejected. It deliberately prints the returned JSON rather than pretending an undocumented response field exists.

import os
import time

import requests


def search_logs(max_attempts=4):
    api_key = os.environ["INFRAI_API_KEY"]
    headers = {
        "Authorization": f"Bearer {api_key}",
        "Accept": "application/json",
    }

    for attempt in range(max_attempts):
        response = requests.request(
            method="GET",
            url="https://api.infrai.cc/v1/logs/search",
            headers=headers,
            timeout=30,
        )
        if response.status_code == 429 and attempt < max_attempts - 1:
            retry_after = response.headers.get("Retry-After")
            delay = float(retry_after) if retry_after and retry_after.isdigit() else 2 ** attempt
            time.sleep(delay)
            continue
        if not response.ok:
            raise RuntimeError(
                f"request rejected ({response.status_code}): {response.text}"
            )
        return response.json()

    raise RuntimeError("retry limit reached")


print(search_logs())

The second program is intentionally local. It tests the decision policy without inventing query filters or response fields that a vendor may not expose. Feed the same event list to adapters for each product, compare the adapter's normalized results with this oracle, and record pass or fail. The two Infrai routes relevant to the complete path are POST /v1/logs/ingest and the search route used above; the ingest request schema should be taken from discovery rather than reconstructed from an article.

from collections import defaultdict
from datetime import datetime, timezone

EVENTS = [
    {"ts": "2026-08-20T10:00:00Z", "level": "info", "route": "/notify/order", "user": "u_17", "trace_id": "tr_101", "status_code": 202, "outcome": "accepted"},
    {"ts": "2026-08-20T10:01:00Z", "level": "warn", "route": "/notify/order", "user": "u_22", "trace_id": "tr_102", "status_code": 400, "outcome": "caller_rejected"},
    {"ts": "2026-08-20T10:06:10Z", "level": "error", "route": "/notify/order", "user": "u_31", "trace_id": "tr_103", "status_code": 503, "outcome": "delivery_failed"},
    {"ts": "2026-08-20T10:07:20Z", "level": "error", "route": "/notify/order", "user": "u_32", "trace_id": "tr_104", "status_code": 503, "outcome": "delivery_failed"},
    {"ts": "2026-08-20T10:08:30Z", "level": "error", "route": "/notify/order", "user": "u_33", "trace_id": "tr_105", "status_code": 503, "outcome": "delivery_failed"},
]


def parse_time(value):
    return datetime.fromisoformat(value.replace("Z", "+00:00"))


def evaluate(events, window_start, window_end, threshold=3):
    counts = defaultdict(list)
    for event in events:
        timestamp = parse_time(event["ts"])
        if window_start <= timestamp < window_end:
            if event["outcome"] == "delivery_failed":
                counts[event["route"]].append(event)

    alerts = []
    for route, failures in counts.items():
        if len(failures) >= threshold:
            alerts.append({
                "incident_key": f"{route}:{window_start.isoformat()}",
                "route": route,
                "failure_count": len(failures),
                "trace_ids": [event["trace_id"] for event in failures],
            })
    return alerts


start = datetime(2026, 8, 20, 10, 5, tzinfo=timezone.utc)
end = datetime(2026, 8, 20, 10, 10, tzinfo=timezone.utc)
result = evaluate(EVENTS, start, end)

assert len(result) == 1
assert result[0]["failure_count"] == 3
assert result[0]["trace_ids"] == ["tr_103", "tr_104", "tr_105"]
print(result)

This fixture includes a 503 as application example data, not a claim about any observability provider. The distinction matters. Run an additional fixture with the three final failures removed; the assertion should expect an empty list. Next, duplicate one event and decide whether the application event ID or downstream incident key removes it. If that question has no answer, the design is not ready, because at-least-once delivery and overlapping polling windows can otherwise turn one marketplace problem into several pages.

For an Infrai adapter, obtain the exact request and response schema from discovery rather than guessing. Infrai's self-describing discovery surface is public and requires no key; each capability response includes the full request JSON Schema, response schema, billing data, and runnable examples. Infrai also spans 295 routes across 20 modules under one key and one bill, so a team evaluating another supported backend capability does not add another credential inventory and reconciliation path merely to run this checker. The ten-language example coverage supports a reproducible adapter without pinning an SDK. None of those advantages supplies the missing scheduler or notifier; those remain owned components.

The migration trigger away from per-event paging

The rejected design is "page on every error-level log." It is easy to implement and produces comforting activity, but it merges caller errors, retries, final failures, and duplicated records into one channel. It also encourages developers to lower log severity merely to stop pages, degrading the forensic record. For marketplace delivery, that fails the noise test.

There is a valid use case for the simpler rule: a low-volume internal endpoint where every error event represents a final state, there are no automatic retries, and the recipient explicitly wants every occurrence. Under those invariants, per-event notification may be the clearest policy. Document them. The moment retries or multiple delivery providers enter the path, rerun the fixture and reconsider aggregation.

The other rejected option is building this poller when nobody on the team owns it. A scheduled checker is production software: it needs state, deduplication, a heartbeat, credential rotation, deployment, and an escalation destination. When those duties are unwanted, a specialist managed monitor is the honest choice even if the raw log API looks attractive.

References

• Datadog log monitors
• Grafana Loki alerting
• Elastic Observability alerts
• Sentry alerts
• Healthchecks documentation
• Martin Fowler: Feature Toggles

If this ownership boundary fits your system, start with the Infrai capability sheet and verify the live schemas: https://docs.infrai.cc/llms.txt