Silent Breakage Detection: Poll Queries, Cron Jobs, and SaaS API Error Rates

Short answer: use an error-rate counter for busy API traffic, an externally observed last-success timestamp for cron jobs, and a small replay test that proves each alert fires and recovers before anyone is paged.

The deciding constraint isn't the dashboard. It's whether the signal can distinguish a failed outcome from a request that merely completed, and whether it can notice work that never started. A beautiful chart built from shallow signals will confidently report the wrong thing.

For a notebook-to-prod workflow, I treat this as an eval problem. Define the expected outcome, create a few controlled failures, and score the alert path against those cases. The collector, query language, and notification system are replaceable parts; the failure contract is the durable part.

Start with silence.

How should custom metrics poll a SaaS API for failed cron jobs?

A poll should verify the narrowest stable business outcome available in the API contract. Transport success only says that an exchange occurred. If a scheduled export is meant to create an artifact, for example, the poller should check the documented completion state or read back the artifact rather than equating any response with success.

Use three bounded signals. First, count API attempts by a small outcome set such as ok, rejected, rate_limited, and timeout. Second, record whether the latest scheduled run met its completion predicate. Third, retain the Unix timestamp of the latest verified success. The counter supports an error-rate query; the result flag exposes an explicit failed job; the timestamp exposes a skipped or stalled schedule that emitted no failure event.

That third signal does the unusual work. A process can't report that it never ran.

The poller therefore has to run outside the job it observes. It also needs a deadline and a completion predicate derived from the API contract. Keep tenant IDs, request IDs, prompts, exception bodies, and access tokens out of metric labels. Those values create high-cardinality dimensions, and some can expose sensitive data. OWASP's logging guidance recommends excluding or masking secrets and sensitive personal data; telemetry deserves the same care.

Here is a focused Python example. It evaluates already-collected samples, so it doesn't assume an undocumented endpoint or metric backend. The numbers are test inputs, not universal production thresholds.

from dataclasses import dataclass


@dataclass(frozen=True)
class Window:
    attempts: int
    failures: int
    last_run_ok: bool
    seconds_since_verified_success: int


def evaluate(window: Window, *, min_attempts: int, max_error_rate: float,
             max_success_age_seconds: int) -> set[str]:
    alerts: set[str] = set()

    if window.attempts >= min_attempts:
        error_rate = window.failures / window.attempts
        if error_rate > max_error_rate:
            alerts.add("api_error_rate")

    if not window.last_run_ok:
        alerts.add("cron_last_run_failed")

    if window.seconds_since_verified_success > max_success_age_seconds:
        alerts.add("cron_success_stale")

    return alerts

Notice what isn't encoded: customer identity, raw error text, or a route assembled from actual IDs. Put diagnostic detail in access-controlled structured logs, joined by an identifier that isn't used as a metric dimension. Metrics answer whether a pattern is happening; logs help inspect a particular event; traces help follow work across component boundaries. Asking one signal type to do all three jobs makes both cost and debugging worse.

Build the query from failure semantics

An API error ratio is failed attempts / all attempts over a chosen window. It needs a traffic floor, because one rejected request out of one attempt is mathematically 100% and may still be a poor reason to wake someone. The floor, window, and threshold must come from the service objective and normal traffic distribution. I'm not sure a five-minute window is meaningful for a low-volume integration; your mileage may vary, and a replay of representative traffic is what resolves that uncertainty.

Cron work needs separate rules for explicit failure and absence. The latest-run flag answers, "Did the last observed run fail?" The age query answers, "Has too much time passed since a verified success?" Do not reset the success timestamp on failure. Its increasing age is the evidence the rule needs.

The alert state also needs a recovery condition. If the trigger is a ratio over a rolling window, decide whether recovery uses the same boundary or a lower one to avoid repeated state changes near the threshold. If the trigger is stale success, a newly verified completion can clear it. A notification without a tested resolution path leaves an operator guessing whether the service recovered or the alert pipeline stopped talking.

For AI-backed API routes, I place token use and retry count next to the core failure signals, but neither proves answer quality. A request can return successfully while an eval rejects the answer; retries can raise prompt cost while obscuring an unstable dependency. Keep task-quality evals alongside operational metrics, and make side-effecting tool calls idempotent. This is where prompt-cost awareness earns its keep: retry volume is an input to diagnosis, not permission to turn every prompt field into a label.

The dashboard comes after the rules. One compact view is enough to start: request volume, error ratio, latency, last-run status, verified-success age, retry volume, and current alert state. Every panel should answer a question that appears in a runbook. Otherwise it's decoration.

No magic panel.

Which operating model can pass the same three failure replays?

The useful comparison isn't a feature matrix. Run the same fixtures through any viable stack: a self-operated collector and rule evaluator, a hosted metrics system, or the observability platform the company already uses. Score the result on detection, notification context, recovery, access control, maintenance, and the time needed to reach useful evidence.

My minimum eval has three cases. The first contains 20 attempts with four failures, which exercises the ratio calculation without claiming that 20% is the right threshold. The second marks the latest run as failed while the prior verified-success timestamp remains intact. The third advances the clock beyond the allowed schedule gap without producing a new run record. I also include healthy and low-traffic controls so an evaluator can't pass by alerting on everything.

def test_alert_contract() -> None:
    policy = {
        "min_attempts": 10,
        "max_error_rate": 0.10,
        "max_success_age_seconds": 900,
    }

    cases = {
        "healthy": (
            Window(20, 0, True, 120),
            set(),
        ),
        "low_traffic_control": (
            Window(1, 1, True, 120),
            set(),
        ),
        "ratio_breach": (
            Window(20, 4, True, 120),
            {"api_error_rate"},
        ),
        "explicit_failure": (
            Window(20, 0, False, 600),
            {"cron_last_run_failed"},
        ),
        "missing_run": (
            Window(20, 0, True, 901),
            {"cron_success_stale"},
        ),
    }

    for name, (sample, expected) in cases.items():
        actual = evaluate(sample, **policy)
        assert actual == expected, f"{name}: {actual} != {expected}"

This is intentionally small. Run it without paging, then send notifications to a non-paging destination and verify the service name, environment, observed condition, evaluation window, and runbook location. Do not include raw payloads or secrets. Finally, exercise recovery with a healthy fixture. A moving graph is not acceptance evidence; a known failure reaching the intended destination with actionable context is.

The operating choice follows ownership.

The catch is that self-operation isn't simple when those responsibilities are unassigned. Sticking with an existing company platform can still be the sound choice when changing tools would add another alert path for operators to learn.

An external heartbeat can detect a missing run from outside the scheduled process. It is not suitable as the only check when a job can signal on schedule without completing its business side effect. Likewise, polling is a poor primary mechanism when an API contract provides authenticated event delivery with the required completion evidence and timeliness. Even in an event-driven design, an age signal remains useful for detecting a silent event path.

What should a failure alert dashboard prove?

It should prove four transitions: healthy to pending, pending to firing, firing to recovered, and stale input to an explicit data-quality state. Test notification delivery independently from rule evaluation — the watcher needs evidence too — while keeping the design proportional to the service.

Measure true positives, false positives, missed test cases, detection delay, time to the first useful operator action, and telemetry volume. For scheduled work, also measure normal run duration and the acceptable completion delay before setting the age boundary. For request traffic, inspect the distribution of volume and failures across several candidate windows. These measurements are more defensible than copying somebody else's threshold.

Metrics-based alerts have firm limits. They are not suitable for reconstructing individual requests, producing audit evidence, or explaining causality across a distributed workflow. Use protected structured logs and traces for those tasks. Also, a failure counter cannot prove output quality: a RAG answer or agent action needs its own checked criterion, just as it did in the notebook.

Before adopting the design, stage one ratio breach, one explicit failed run, and one missing run. Then confirm that healthy and low-traffic controls stay quiet. If the system detects those cases, delivers useful context, recovers predictably, and has a named owner, the simple architecture is doing enough.

References

• OWASP Logging Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html