How to Implement 3 Custom Metrics-Based Failure Alerting Rules — For a SaaS API

Metrics-based failure alerting for a SaaS API should measure the customer-support agent loop, not merely HTTP responses. TL;DR: use three signals at the loop boundary: terminal failure ratio, end-to-end latency, and cost per resolved conversation. Page only on sustained, user-impacting failure or missing work; route latency and cost drift to investigation unless they cross a separately agreed service objective. A dashboard explains. A page demands action.

I have been paged for missed scheduled jobs and duplicate deliveries. The invariant those incidents exposed was uncomfortable: a successful poll is not evidence that useful work happened. For an AI support loop, the unit of work is a conversation turn or case, not an HTTP request. Give that unit a stable ID, record one terminal outcome, and make retries idempotent before tuning any threshold.

This article builds the smallest useful path in Go: an outcome record, a bounded collector, and alert decisions with explicit denominators. The examples use only the standard library and can be run locally.

Which metrics should drive SaaS API failure alerting?

Start with the failure that has an operator action. A provider timeout may be retryable. A malformed tool result may end the loop. A queue consumer that stops completing turns is different again: it creates no failure events, so an error-rate alert stays green.

Silence matters.

That gives us three failure modes to cover:

1. Terminal errors among completed attempts.
2. Completion latency above the service objective.
3. Missing completions from a scheduler or queue.

Cost belongs on the same operational view because runaway loops can remain technically successful. It usually should not page by itself. Token use and monetary cost can change because the request mix changed, so an urgent threshold without workload context produces noise. Compare cost per resolved conversation within a stable route, model class, and customer tier; send sustained drift to a ticket or daytime channel.

The event below carries bounded dimensions. Do not put conversation IDs, prompts, email addresses, error text, or arbitrary tool names into metric labels. OWASP recommends excluding or masking sensitive data such as access tokens, passwords, and personal data from logs; the same restraint is useful for telemetry payloads. High-cardinality identifiers remain in protected logs or traces for diagnosis, while metrics retain only dimensions with a controlled set of values.

package main

import (
    "encoding/json"
    "fmt"
    "time"
)

type Outcome struct {
    Route       string  `json:"route"`
    Status      string  `json:"status"` // resolved, escalated, failed
    LatencyMS   int64   `json:"latency_ms"`
    CostUSD     float64 `json:"cost_usd"`
    Attempt     int     `json:"attempt"`
    CompletedAt string  `json:"completed_at"`
}

func main() {
    event := Outcome{
        Route: "billing", Status: "resolved", LatencyMS: 1840,
        CostUSD: 0.012, Attempt: 2,
        CompletedAt: time.Now().UTC().Format(time.RFC3339),
    }
    b, err := json.Marshal(event)
    if err != nil {
        panic(err)
    }
    fmt.Println(string(b))
}

The exact cost calculation depends on the model and accounting contract, so keep it outside the alert evaluator. Feed the evaluator a normalized cost that the billing pipeline can reproduce. If it cannot be reproduced, it is not yet an alerting signal.

Step 1: Count terminal outcomes once

Retries are where clean-looking metrics lie. If every attempt increments the denominator, a burst of retries can make success volume look healthy. If both an original worker and its retry report completion, one support turn can be counted twice.

Use a stable work ID as the idempotency key and accept only the first terminal outcome. The in-memory store here makes the rule executable; production storage needs an atomic insert-if-absent operation and a retention period longer than the maximum retry window.

package main

import (
    "fmt"
    "sync"
)

type Counter struct {
    mu     sync.Mutex
    seen   map[string]struct{}
    totals map[string]int
}

func (c *Counter) Record(workID, status string) bool {
    c.mu.Lock()
    defer c.mu.Unlock()
    if _, exists := c.seen[workID]; exists {
        return false
    }
    c.seen[workID] = struct{}{}
    c.totals[status]++
    return true
}

func main() {
    c := &Counter{seen: map[string]struct{}{}, totals: map[string]int{}}
    fmt.Println(c.Record("turn-1042", "failed"))
    fmt.Println(c.Record("turn-1042", "failed")) // retry: ignored
    fmt.Println(c.totals)
}

Count resolved, escalated, and failed separately. An escalation is not necessarily a system failure; for some support policies it is the correct result. Put only outcomes that violate the service contract in the failure numerator, and write that classification beside the runbook.

No traffic means no denominator. Do not turn 0 / 0 into zero percent failures. Mark the ratio unavailable, then let the missing-completion check decide whether silence is expected.

Step 2: Evaluate a window, not a single poll

A polling process should evaluate complete time windows and remember which window it last processed. Poll overlap is harmless when evaluation is idempotent. Poll gaps are visible when the evaluator records its own last-success timestamp.

The following decision function requires both a minimum sample and consecutive bad windows. Those two gates suppress pages caused by one failed conversation or a brief burst, while preserving a clear reason in the alert. The sample values are examples, not universal objectives; derive yours from support volume, error budget, and the response time an operator can influence.

package main

import "fmt"

type Window struct {
    Total        int
    Failed       int
    P95LatencyMS int64
}

type Policy struct {
    MinTotal           int
    MaxFailureRatio    float64
    MaxP95LatencyMS    int64
    RequiredBadWindows int
}

func bad(w Window, p Policy) (bool, string) {
    if w.Total < p.MinTotal {
        return false, "insufficient sample"
    }
    ratio := float64(w.Failed) / float64(w.Total)
    if ratio >= p.MaxFailureRatio {
        return true, fmt.Sprintf("failure ratio %.3f", ratio)
    }
    if w.P95LatencyMS >= p.MaxP95LatencyMS {
        return true, fmt.Sprintf("p95 latency %dms", w.P95LatencyMS)
    }
    return false, "healthy"
}

func main() {
    policy := Policy{50, 0.05, 5000, 2}
    windows := []Window{{100, 7, 2200}, {120, 8, 2400}}
    consecutive := 0
    for _, w := range windows {
        isBad, reason := bad(w, policy)
        if isBad {
            consecutive++
        } else {
            consecutive = 0
        }
        fmt.Printf("bad=%v consecutive=%d reason=%s page=%v\n",
            isBad, consecutive, reason, consecutive >= policy.RequiredBadWindows)
    }
}

There is a deliberate trade-off: two windows increase detection time. I accept that delay for a moderate-volume support route because a single-window page is too sensitive to a small denominator. A high-volume route with a short error budget may warrant a faster multi-window burn-rate policy instead. A low-volume route may need synthetic work or per-event failure notification because ratio statistics remain weak.

Latency needs a distribution, not an average. Store histogram buckets or compute quantiles from a bounded window; do not average already calculated percentiles. Separate queue wait from agent execution when the remediation differs. Otherwise the dashboard can show that the loop is slow but cannot tell the on-call engineer whether to inspect scheduling capacity or downstream calls.

Step 3: Make silence an explicit signal

The worst scheduled-job failure can produce no error at all. The scheduler did not start, the worker stopped consuming, or the metrics exporter stalled. A heartbeat from the same broken worker is weak evidence, so derive freshness from the durable completion stream and monitor the evaluator independently.

Short code is enough.

package main

import (
    "fmt"
    "time"
)

func completionMissing(now, last time.Time, expectedEvery, grace time.Duration) bool {
    return now.Sub(last) > expectedEvery+grace
}

func main() {
    now := time.Date(2026, 10, 7, 12, 0, 0, 0, time.UTC)
    last := now.Add(-16 * time.Minute)
    fmt.Println(completionMissing(now, last, 10*time.Minute, 5*time.Minute))
}

Set expectedEvery from the schedule or observed arrival contract, not from wishful thinking. The grace period must cover normal queue delay and clock skew without swallowing the response window. During declared quiet periods, suppress the expectation rather than muting the alert after it fires.

A missing-completion page should include the route, last successful completion time, expected cadence, current queue age, and a runbook link. It should not include a customer transcript. The first runbook checks are mechanical: confirm whether input arrived, compare enqueue and completion counts, inspect oldest-work age, and verify that the evaluator itself completed its last poll.

Dashboard, page, or ticket?

Use delivery severity to encode urgency, not uncertainty. The table is the policy I would review with support operations before deployment.

Keep the dashboard compact: throughput and terminal outcomes first, latency distribution second, cost per resolved case third, then evaluator freshness. Every chart should use the same time range and expose its denominator. Overlay deployments and policy changes because they shorten diagnosis, but do not make a deployment marker proof of causation.

Before enabling paging, replay known windows through the evaluator, inject a missing-completion interval, and run the policy in shadow mode. Review pages with two questions: did a customer-impacting condition exist, and could the recipient act immediately? If either answer is repeatedly no, change the signal or its route. Do not merely raise the threshold until the noise disappears.

This design does not apply unchanged to a one-off batch, a route with fewer than a handful of daily cases, or safety-critical decisions where a single bad outcome requires immediate review. Those need event-level handling and different escalation policy. For a recurring support-agent loop, though, the three outcomes establish a useful contract: count terminal results once, evaluate complete windows, and treat unexpected silence as failure.

Sources

• https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html