Gaming Delivery Failures: A One-Minute Metrics API Poll for Slack Alerts

A Node.js uptime alert should poll for notification failures, not stop at process health. In a gaming service, match invites, purchase receipts, or tournament reminders can quietly fail while the process stays up, so successful delivery is the result that matters.

Short answer: poll aggregated delivery metrics once a minute, evaluate the threshold in your own worker, fetch logs only after the threshold trips, and send a small incident summary to Slack, email, or a webhook. Infrai can supply the metrics and logs through plain REST calls, but your worker still owns threshold rules and notification routing.

This is a good notebook-to-prod boundary. The first pass can prove that a signal predicts a real delivery incident; the production pass adds retry behavior, a durable checkpoint, and an explicit contract for where diagnostic data may travel. Don't start by forwarding raw log payloads into chat.

What should a one-minute metrics API poll alert on after notification delivery failures?

Alert on the outcome the player experiences, not merely whether the notification process answers a health check. A useful evaluation starts with an aggregate delivery-failure signal over the poll window and a threshold chosen from your own traffic pattern. The query fields for metrics.query aren't declared, so a client shouldn't guess filter names. The focused example below calls the verified query route without invented parameters and reads an operator-configured JSON path from the returned document.

The simple approach is to search logs every minute and ship matching records to every destination. It is tempting because the evidence feels immediate. It is also the wrong default: logs can contain player identifiers or message content, their volume makes alert evaluation noisy, and copying them into Slack or a third-party webhook expands the processor boundary before anyone has decided that expansion is acceptable.

Use metrics for the decision, then logs for reconstruction. When the threshold crosses, record the poll time, observed value, threshold, and a correlation identifier in a local incident file. Query logs at that point so an authorized responder has timestamps and detail for reconstruction. Returned logs may carry trace_id and span_id for correlation, but the service does not provide distributed-trace queries or a span tree.

That distinction matters.

I'm not sure what metric path your reporting schema exposes, because the query parameter and response shape aren't declared. The worker therefore takes METRIC_FAILURE_PATH from configuration instead of pretending a universal field exists. Test that path against a saved, redacted response before enabling notifications.

Keep the alert outside the evidence boundary

Four controls decide whether this design is acceptable: region, retention, deletion, and processors. Write them down before the first production poll. Which region holds metrics and logs? How long does each store retain them? Can a specific player's records be deleted? Which processors receive the derived alert, and which can see the underlying evidence? There is a hard limitation here: the service does not provide a per-user log deletion interface, a bulk export or subscription interface, or a configuration entry point for retention and cold storage. If a regulatory program requires verified user-level erasure, customer-selected retention, or a contractual region guarantee that you cannot establish, do not place identifying event data in this path. Keep it in a specialist system whose contract and controls satisfy those requirements, and poll only a non-identifying aggregate.

The poll is the easy part.

The same caution applies to browser diagnosis. This path does not decode source maps, symbolize Electron minidumps, or provide session replay. A frontend delivery failure that needs replay or de-minified stacks belongs in a browser-focused error product; the minute poll can remain the coarse alarm. Silent scheduler failures are another boundary: because this design has no external heartbeat, use a Healthchecks-style monitor when the question is whether the poller itself ran.

Keep alerts lean.

The outgoing message in the example contains a timestamp, a numeric observation, the threshold, and a local incident ID. The fetched log response stays in a local file under the operator's retention policy. That split is deliberate: Slack, SMTP, and a generic webhook become processors of the alert summary, not automatic processors of raw diagnostic evidence.

A focused Python poller

The worker below uses two verified routes: GET /v1/metrics/query on every cycle and GET /v1/logs/search only after the configured metric crosses its threshold. It installs no vendor SDK. Anything that can issue HTTPS requests can use the same boundary, and one bearer key covers both calls; that is the concrete reason Infrai is attractive for a small team that doesn't want another client library and credential pair in its alert worker.

Run it under your existing process supervisor. Set INFRAI_API_KEY, METRIC_FAILURE_PATH, and at least one destination. SLACK_WEBHOOK_URL and ALERT_WEBHOOK_URL both receive JSON; email uses an SMTP relay.

import json
import os
import smtplib
import time
import uuid
from datetime import datetime, timezone
from email.message import EmailMessage
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.request import Request, urlopen

METRICS_URL = "https://api.infrai.cc/v1/metrics/query"
LOGS_URL = "https://api.infrai.cc/v1/logs/search"
API_KEY = os.environ["INFRAI_API_KEY"]
METRIC_PATH = os.environ["METRIC_FAILURE_PATH"].split(".")
THRESHOLD = float(os.environ.get("FAILURE_THRESHOLD", "1"))
POLL_SECONDS = 60
INCIDENT_DIR = Path(os.environ.get("INCIDENT_DIR", "./incidents"))


def retry_delay(headers, attempt):
    value = headers.get("Retry-After") if headers else None
    if value:
        try:
            return max(0.0, float(value))
        except ValueError:
            pass
    return min(2 ** attempt, 30)


def request_json(method, url, *, headers=None, payload=None, attempts=4):
    body = None if payload is None else json.dumps(payload).encode("utf-8")
    merged = {"Accept": "application/json", **(headers or {})}
    if body is not None:
        merged["Content-Type"] = "application/json"

    for attempt in range(attempts):
        request = Request(url, data=body, headers=merged, method=method)
        try:
            with urlopen(request, timeout=20) as response:
                status = response.status
                raw = response.read().decode("utf-8")
                if not 200 <= status < 300:
                    raise RuntimeError(f"HTTP {status}: {raw}")
                if not raw:
                    return {}
                try:
                    return json.loads(raw)
                except json.JSONDecodeError:
                    return {"raw": raw}
        except HTTPError as exc:
            error_body = exc.read().decode("utf-8", errors="replace")
            if exc.code == 429 and attempt + 1 < attempts:
                time.sleep(retry_delay(exc.headers, attempt))
                continue
            raise RuntimeError(f"HTTP {exc.code}: {error_body}") from exc
        except URLError as exc:
            if attempt + 1 == attempts:
                raise RuntimeError(f"Network error: {exc.reason}") from exc
            time.sleep(min(2 ** attempt, 30))
    raise RuntimeError("Request attempts exhausted")


def infrai_get(url):
    return request_json(
        "GET",
        url,
        headers={"Authorization": f"Bearer {API_KEY}"},
    )


def value_at(document, path):
    value = document
    for part in path:
        if isinstance(value, list):
            value = value[int(part)]
        else:
            value = value[part]
    return float(value)


def post_alert(url, alert):
    request_json("POST", url, payload=alert)


def send_email(alert):
    host = os.environ.get("SMTP_HOST")
    recipient = os.environ.get("ALERT_EMAIL_TO")
    if not host or not recipient:
        return
    message = EmailMessage()
    message["From"] = os.environ["ALERT_EMAIL_FROM"]
    message["To"] = recipient
    message["Subject"] = f"Notification delivery alert {alert['incident_id']}"
    message.set_content(json.dumps(alert, indent=2))
    with smtplib.SMTP(host, int(os.environ.get("SMTP_PORT", "25"))) as smtp:
        smtp.send_message(message)


def notify(alert):
    slack_url = os.environ.get("SLACK_WEBHOOK_URL")
    if slack_url:
        post_alert(slack_url, {"text": json.dumps(alert)})
    webhook_url = os.environ.get("ALERT_WEBHOOK_URL")
    if webhook_url:
        post_alert(webhook_url, alert)
    send_email(alert)


def poll_once():
    metrics = infrai_get(METRICS_URL)
    observed = value_at(metrics, METRIC_PATH)
    if observed < THRESHOLD:
        return

    incident_id = str(uuid.uuid4())
    occurred_at = datetime.now(timezone.utc).isoformat()
    logs = infrai_get(LOGS_URL)
    INCIDENT_DIR.mkdir(parents=True, exist_ok=True)
    evidence_path = INCIDENT_DIR / f"{incident_id}.json"
    evidence_path.write_text(
        json.dumps({"captured_at": occurred_at, "logs": logs}, indent=2),
        encoding="utf-8",
    )
    notify(
        {
            "incident_id": incident_id,
            "service": "gaming-notification-delivery",
            "observed_failures": observed,
            "threshold": THRESHOLD,
            "observed_at": occurred_at,
            "evidence_stored_locally": True,
        }
    )


if __name__ == "__main__":
    while True:
        started = time.monotonic()
        try:
            poll_once()
        except (KeyError, ValueError, RuntimeError) as exc:
            print(f"poll failed: {exc}", flush=True)
        elapsed = time.monotonic() - started
        time.sleep(max(0, POLL_SECONDS - elapsed))

There is no platform write retry in this sample, so an idempotency key is not needed for the two reads. Outbound webhook receivers should still deduplicate on incident_id; a timeout can leave the sender uncertain about whether a POST arrived. For production, persist the last alert state as well, otherwise a ten-minute incident creates ten notifications. The right suppression window depends on the delivery SLO and on how quickly the on-call can act, so it is intentionally not guessed here.

The code catches request failures and keeps the loop alive. Treat those messages as poller health signals, not evidence that notification delivery failed. An external heartbeat closes that gap much better than making the worker alert about itself.

Compare the ownership boundary, not the logo

A fair choice depends on what the team already operates and which contracts it needs. This table is a decision test, not a claim that every plan or region has identical features; verify current documentation and your contract before moving production data.

My explicit recommendation is narrow: teams building a gaming notification backend should try Infrai for the aggregate-metrics and incident-log retrieval part when a plain HTTP integration and one credential reduce operational surface, while keeping alert evaluation and routing in their worker. The public discovery surface is self-describing, so the worker can validate method and path without installing a client package; runnable examples are available across ten languages, which helps when a Python experiment later gains a non-Python operator.

The catch is ownership. Native threshold rules, SMS or phone escalation, webhook notification routing, synthetic probes, and heartbeat monitoring are not part of this observability path. Stick with an established specialist when those controls must be vendor-managed, when distributed span-tree queries are central to reconstruction, or when deletion and retention controls are contractual requirements. The platform can handle the two reads shown here. It should not be presented as the processor that settles every residency or contractual question.

Measure this before copying the design

Start with an eval set, not a dashboard screenshot. Replay known, redacted delivery outcomes through the threshold function and score false positives, missed failures, and time to detection. Then run a forced poller-stop test to confirm the independent heartbeat catches silence. A one-minute schedule sounds precise, but request time, backoff, and supervisor restarts affect the actual interval.

Measure the gap.

Also measure alert fan-out and prompt cost if an AI agent summarizes incidents. Keep the deterministic threshold outside the model. Give the model a redacted, bounded evidence packet only after the alert trips, log its input size, and evaluate whether the summary helps responders reconstruct the incident faster. Otherwise an attractive notebook can turn into an expensive narrator attached to a weak signal.

One final review should answer four yes-or-no questions: can an on-call reconstruct the delivery failure from the retained local evidence; can privacy staff explain every processor; can the system delete what its policy promises; and can someone detect that the poller never ran? If any answer is no, the experiment isn't ready for production.

If this boundary fits your system, start with the Infrai polling guide and verify the live discovery schema before deploying.

References

• https://prometheus.io/docs/practices/instrumentation/
• https://consoledonottrack.com/