A cheap metrics dashboard plus email failure alerts sounds sufficient for a media pipeline, but the operational constraint changes the design: the alert must arrive soon enough to stop bad output from advancing, yet a delayed poll must never turn yesterday's failure into a reason to roll back today's deployment.
Short answer: record a monotonic failure counter at the Node.js job boundary, poll its query on a fixed schedule, persist the last observed value and alert state, and send email only after correlating a new failure with the batch and deployment that produced it. The dashboard is a view of that same data, not the component that decides whether to alert or roll back.
This architecture is inexpensive to operate because it has few moving parts, but cost is not the decision rule. Rollback safety is. A cheap metrics dashboard that cannot distinguish a new failure from an old counter value is an expensive source of false decisions.
How should a startup SaaS poll custom metric queries and send email failure alerts?
The concrete workload is a nightly media pipeline that validates a manifest, transcodes assets, writes derived objects, and publishes an index used by search. Its structured logs remain the place to investigate individual items. Metrics answer the narrower operational question: did this run cross a failure boundary, and did that happen after the deployment under consideration? Trying to encode every log field as a metric label would confuse those jobs and, as the Prometheus instrumentation guidance warns, create labels with unbounded cardinality.
The first invariant is that a failed run increments a counter once at a defined ownership boundary. Retries need their own accounting; they must not quietly erase the original failure. The second is that the poller stores a checkpoint durably, including the counter value it observed and the alert state associated with it. The third is that rollback remains a guarded action. An email may recommend investigation, but the rollback decision also needs the batch identifier, deployment identifier, last known good checkpoint, and evidence that the failing execution used the candidate deployment.
Never roll back from a gauge alone.
A gauge can return to zero between polls, so it can hide a short failure. A cumulative counter preserves the transition, while the poller's saved checkpoint turns that transition into delta = current - previous. Counter resets are a named failure mode, not an exotic corner: if current < previous, treat the sample as a reset, establish a new baseline, and do not manufacture a negative incident count.
These are the failure boundaries I would put in the record:
- Producer boundary: the job emits a success or failure exactly where the run outcome becomes final.
- Query boundary: missing, stale, or malformed samples produce an internal poller error, not a healthy result.
- Notification boundary: email delivery and alert detection have separate state, so a mail retry cannot rediscover the same metric transition.
- Rollback boundary: only failures tied to the candidate deployment can contribute to a rollback recommendation.
The catch is storage. A process-local checkpoint disappears on restart and can resend old alerts. Use a small durable record with compare-and-swap semantics, or serialize the poller so only one instance owns the transition. If neither is possible, this design is not suitable for automatic rollback advice; keep the alert informational until checkpoint ownership is reliable.
Poll a counter, not the dashboard page and not the structured logs. The Node.js producer should expose one stable custom metric for terminal run outcomes, with bounded labels such as pipeline, stage, and status. Keep batch_id, object keys, customer identifiers, and raw error messages in structured logs or an event record where they can be searched without multiplying time series. A report can join the time window and deployment metadata later.
For an illustrative schedule, suppose the nightly batch has a 30-minute rollback hold and the poll interval is five minutes. Requiring two consecutive observations of a positive delta can filter a single incomplete scrape while still leaving time for review. Those numbers are policy inputs, not universal defaults. I'm not sure two polls fit a pipeline whose retry cycle lasts 20 minutes; replaying several real run timelines against the state machine is what resolves that uncertainty.
There is a subtle trap here — the query and email loops should not be one stateless scheduled script. If mail submission succeeds and the process exits before saving state, the next run may send the same alert again. Save a pending notification with an idempotency key derived from stable event metadata, attempt delivery, and then mark it sent. If the mail system does not accept idempotency keys, the local outbox still prevents most duplicate sends, although no design can claim exactly-once delivery without cooperation across the boundary.
The dashboard query should show at least the run rate, failure delta, last successful run age, and alert state. It should not infer recovery merely because no new failures appeared. Recovery means a later terminal success for the affected pipeline and a checkpoint newer than the failed batch. That's stricter. It is also auditable.
State comes first.
Four designs at the rollback boundary
The useful comparison is not a vendor leaderboard. It is where state lives, how a missed observation behaves, and whether the design can explain a rollback recommendation after the operator has slept.
| Option | Failure memory | Rollback safety | Operational limit | Valid use case |
|---|---|---|---|---|
| Direct email inside the pipeline | Usually tied to one execution | Weak unless deployment metadata and deduplication are built in | Mail latency or retry logic extends the job's critical path | Small batch jobs where every failure requires a human and duplicates are acceptable |
| Poll a monotonic metric with a durable checkpoint | Counter plus explicit alert state | Strong when batch and deployment metadata are correlated outside high-cardinality labels | Poll delay is intentional; checkpoint ownership must be enforced | Nightly pipelines with a review window before promotion |
| Evaluate structured logs on a schedule | Full event detail | Potentially strong, but only with a stable schema and bounded query window | Late log arrival and shifting windows can repeat or omit matches | Investigations that need per-object context more than a compact health signal |
| Run an external synthetic check | Independent of producer instrumentation | Weak for rollback attribution unless it records deployment context | It observes output behavior, not every internal stage | Detecting publication or search regressions after the pipeline finishes |
The metric-and-checkpoint option wins for this scenario because it separates detection from investigation while preserving a small, reviewable state transition. It loses when the batch has no stable terminal boundary, when failures are meaningful only per media object, or when operators need sub-poll-interval reaction. In the first two cases, stick with structured event evaluation. In the last, use a push-based event path with durable delivery.
The checkpoint transition in Python
The producer can be Node.js; the alert evaluator below is Python because the contract is plain JSON over HTTP and SMTP, not an SDK-specific integration. METRIC_QUERY_URL is a configured internal endpoint, so the example does not assume a vendor route. Its response contract is deliberately narrow: a nonnegative cumulative value plus stable batch and deployment metadata for the most recent terminal failure.
The code uses a local JSON state file to make the transition readable. For production, place the same state behind a durable single-writer or compare-and-swap store. Don't run two copies against one file.
import json
import os
import smtplib
import ssl
import urllib.request
from dataclasses import asdict, dataclass
from email.message import EmailMessage
from pathlib import Path
@dataclass
class AlertState:
value: int = 0
consecutive_failure_polls: int = 0
sent_key: str = ""
def load_state(path: Path) -> AlertState:
if not path.exists():
return AlertState()
return AlertState(**json.loads(path.read_text(encoding="utf-8")))
def save_state(path: Path, state: AlertState) -> None:
temporary = path.with_suffix(".tmp")
temporary.write_text(json.dumps(asdict(state)), encoding="utf-8")
temporary.replace(path)
def query_failure_counter() -> dict:
request = urllib.request.Request(
os.environ["METRIC_QUERY_URL"],
data=json.dumps({"metric": "media_pipeline_runs_total", "status": "failed"}).encode(),
headers={"Content-Type": "application/json"},
method="POST",
)
with urllib.request.urlopen(request, timeout=10) as response:
sample = json.load(response)
if not isinstance(sample.get("value"), int) or sample["value"] < 0:
raise ValueError("metric value must be a nonnegative integer")
return sample
def send_email(sample: dict, delta: int) -> None:
message = EmailMessage()
message["From"] = os.environ["ALERT_FROM"]
message["To"] = os.environ["ALERT_TO"]
message["Subject"] = f"Nightly media pipeline: {delta} new failed run(s)"
message.set_content(
"\n".join(
[
f"Batch: {sample['batch_id']}",
f"Deployment: {sample['deployment_id']}",
f"Failure counter delta: {delta}",
"Action: inspect structured logs before deciding on rollback.",
]
)
)
context = ssl.create_default_context()
with smtplib.SMTP_SSL(os.environ["SMTP_HOST"], 465, context=context) as smtp:
smtp.login(os.environ["SMTP_USER"], os.environ["SMTP_PASSWORD"])
smtp.send_message(message)
def main() -> None:
state_path = Path(os.environ.get("ALERT_STATE_PATH", "alert-state.json"))
state = load_state(state_path)
sample = query_failure_counter()
current = sample["value"]
if current < state.value:
save_state(state_path, AlertState(value=current))
return
delta = current - state.value
consecutive = state.consecutive_failure_polls + 1 if delta > 0 else 0
event_key = f"{sample['batch_id']}:{sample['deployment_id']}:{current}"
next_state = AlertState(current, consecutive, state.sent_key)
if consecutive >= 2 and event_key != state.sent_key:
send_email(sample, delta)
next_state.sent_key = event_key
save_state(state_path, next_state)
if __name__ == "__main__":
main()
This sample is intentionally conservative. A query timeout raises an error and leaves the previous checkpoint intact; it does not reinterpret absence as zero. A counter reset establishes a baseline without emailing. A repeated value clears the consecutive-poll count. The state is saved only after mail submission, which favors possible duplicate notification over silently losing an alert; an outbox is the next step when that trade-off is unacceptable.
One more limit matters: SMTP submission confirms that the server accepted the message, not that a human read it. For high-consequence rollback gates, pair email with an owned queue or incident workflow and test the escalation path. Email alone is a notification channel, not an acknowledgement protocol.
Why direct job email is rejected, and when it isn't
I would reject direct email from the nightly job for this system. It couples notification retries to media processing, spreads recipient configuration into the producer, and makes deduplication depend on every retry path doing the same thing. Still, it has a valid use case: a small internal job with no automated rollback, one operator, and failures rare enough that a duplicate message is harmless. Architecture decisions need that boundary; otherwise “rejected” is just branding.
Before deployment, test the evaluator with a table of sequences rather than a single happy-path run: 10, 10, 11, 11 should alert once under the two-poll policy; 11, 2 should record a reset without a failure email; a malformed response should preserve state; and two schedulers racing for the same checkpoint should be rejected by the storage layer. These are constructed test vectors, not observed production measurements.
Also test a rollback mismatch. If batch batch-0042 failed under deployment release-b but the currently staged candidate is release-c, the email must not recommend rolling back release-c. Search the structured logs using the batch identifier, confirm the deployment association, and only then apply the runbook. Fast alerts are useful. Correct attribution is better.
Scheduled automation, including a repository workflow runner, can host the poller, but ownership remains part of the design: protect credentials, prevent overlapping executions, retain evaluator errors, and ensure a missed schedule becomes visible. The official GitHub Actions documentation is one primary reference for that execution model. If those controls become harder to reason about than a continuously running evaluator, the scheduler is no longer the cheap option in engineering time.
The final decision rule is compact: choose polled custom metrics when the job has a stable terminal counter, the response window exceeds the polling interval, and durable alert state is available. Choose structured-event evaluation when attribution requires per-object fields. Choose a pushed event when waiting for the next poll violates the rollback hold. Keep the dashboard read-only in all three designs.
