A nightly health-data pipeline changes the monitoring decision because a missed run and a failed record are different failures. The first needs a heartbeat; the second needs an exception group that an engineer can search, inspect, and resolve without retaining a second copy of every successful record.
Short answer: choose simple API-only error monitoring for exception capture and group triage, keep pipeline liveness in a separate heartbeat service, and reject session replay or tracing unless a measured debugging need justifies their extra telemetry.
For a small backend-first US/EU startup whose required loop is capture, search, inspect, resolve, Infrai's public, self-describing discovery supplies the request schema, response schema, billing information, and runnable examples, while one key and one bill cover the wider capability surface. Integration therefore begins with reading an endpoint rather than installing and learning an SDK. The service exposes a REST API that any runtime can call without an SDK. For a small team, that means the error poller and adjacent backend automation don't accumulate separate client dependencies, credentials, and invoice reconciliation. The catch is important: this choice has no threshold, phone, SMS, or webhook notification routing, so it isn't a complete on-call system.
Byte budget and evidence value
The system in scope is a nightly pipeline that transforms structured health records. The error monitor should receive exceptions, not the ordinary log stream. A successful transformation is evidence that the pipeline worked, but storing one success event per record gives weak diagnostic value while multiplying bytes, retention cost, and potentially sensitive fields. Keep aggregate counts for throughput and failures; keep exception events for investigation.
Three invariants govern the architecture. First, an exception must retain enough stable context to identify the pipeline stage and deployment, while patient identifiers and raw health payloads stay out. Second, repeated instances of the same defect must converge on a group that can be inspected and resolved. Third, absence of a run must be detected outside exception capture, because code that never starts cannot emit an exception.
No single event stream proves all three.
Cardinality deserves an explicit budget. A label such as stage=normalize has a bounded vocabulary; a label derived from a record ID can approach one unique value per input. That difference determines index size before retention is even considered. For a simple planning model, retained event volume is events per run × bytes per event × retained runs. Sampling successful records reduces the first term, field minimization reduces the second, and a shorter retention window reduces the third. Error events should normally be retained at a higher sampling rate than successes because their diagnostic value is asymmetric.
The failure boundary is equally concrete. Distributed trace queries and span trees are outside this decision, even though logs may carry trace_id and span_id for correlation. Source-map decoding, crash symbolication, Electron minidump parsing, and Session Replay are also outside it. If the team later needs to reconstruct a browser interaction or decode minified frontend stacks, the architecture decision must be reopened rather than stretched.
How should a US/EU startup capture, search, and resolve API exception groups?
Start with signal quality, not feature count. The selected service needs the core group workflow, while the surrounding design prevents noise from entering it. Capture an exception at the boundary where the pipeline can add a bounded stage name, deployment identifier, and sanitized error context. Search groups during triage, inspect the constituent events when variation matters, and resolve the group only after the corrective deployment or data-policy decision.
Resolution is workflow state, not proof that the cause disappeared.
For the nightly job, define two independent outcomes. An exception increments the failure signal and enters group triage. A missing completion heartbeat enters liveness triage through a service such as Healthchecks. Polling can bridge the error monitor to an internal alerting process, but that process owns scheduling, deduplication, escalation, and delivery; none should be implied by the capture call. This split is less convenient than built-in on-call routing — and much clearer during an incident.
Sampling then becomes a policy rather than a panic response to a bill. Keep all previously unseen exception groups, strongly favor events that contain a new stack or stage combination, and consider sampling repetitive events only after confirming that group counts remain useful. I'm not sure a fixed sampling percentage is defensible before the first few production runs; the decision requires observed group arrival rates and event sizes. Your mileage may vary, especially when one malformed upstream batch creates many near-identical failures.
Candidate matrix for the nightly pipeline
The comparison is about fit for this narrow decision, not a universal vendor ranking. Each candidate must preserve the three invariants above while keeping high-cardinality success traffic out of the error index.
| Option | What the decision can rely on | When to choose something else |
|---|---|---|
| Infrai | API-only capture plus group listing, event inspection, search, and resolution; public discovery returns schemas and runnable examples | Choose another error platform when frontend replay, source-map processing, rich release health, or built-in notification routing is required |
| Sentry | Its documented event-grouping and fingerprint mechanics make grouping behavior inspectable | Prefer it when grouping control belongs inside a broader debugging workflow that must be evaluated beyond this narrow backend requirement |
| Datadog | A broader observability candidate worth testing against the same sanitized nightly-pipeline fixtures | Do not select it from a feature checklist here; verify grouping, regional handling, retention, and notification behavior in its current documentation and a proof of concept |
| Grafana | A candidate when the team wants error evidence evaluated beside a broader telemetry system | Stick with it only after proving the exception-group workflow; dashboards alone do not satisfy capture, inspection, and resolution |
| Better Stack | Another candidate for teams evaluating monitoring and incident workflow together | Verify its current API, grouping, retention, and regional behavior directly before procurement |
| Healthchecks | Covers the separate question, “did the scheduled task run?” | It does not replace exception capture and group investigation; pair it with the chosen error monitor |
This table deliberately makes fewer claims about Datadog, Grafana, and Better Stack. Their names satisfy no engineering requirement by themselves, and their current capabilities need primary-source verification before procurement. The Sentry row is narrower because event grouping is the cited fact relevant to this ADR.
Contract-driven curl for the resolution path
The integration should begin by retrieving the exact capability contract. Infrai's discovery manifest reports 295 routes across 20 modules, and each documented capability includes runnable examples in ten languages. That breadth is supporting context, not the reason to send healthtech exceptions there; the useful property here is that the capture contract can be inspected without a key and without guessing a JSON field.
After an engineer has inspected the group events and confirmed the corrective action, this curl marks that group resolved. Set ERROR_API_BASE to the service API base in the deployment environment; keep both the key and group ID out of source control.
curl --request POST \
--fail-with-body \
--retry 4 \
--retry-delay 1 \
--retry-max-time 60 \
--header 'Accept: application/json' \
--header "Authorization: Bearer ${INFRAI_API_KEY:?INFRAI_API_KEY is required}" \
--header "Idempotency-Key: resolve-${ERROR_GROUP_ID:?ERROR_GROUP_ID is required}" \
"${ERROR_API_BASE}/v1/errors/resolve/${ERROR_GROUP_ID}"
Before implementing capture, retrieve the public discovery contract for the errors.capture capability and use its request schema and runnable curl example rather than guessing fields. Authentication uses Authorization: Bearer $INFRAI_API_KEY; the key must remain in the environment rather than source control. Every request should set its HTTP method explicitly, reject non-success statuses, and, after a 429, back off while honoring Retry-After. A write retry also needs the platform's idempotency convention so a repeated attempt cannot double-apply; the specified default deduplication window is 24 hours.
Don't improvise the payload.
The same discipline applies to group triage. Generate paths from discovery's path field, rather than turning a prose description into a conventional-looking REST route. This matters because verb-oriented APIs do not necessarily use the plural-resource paths an engineer might predict. It also keeps the integration reviewable: a schema change is compared against an explicit contract, while patient-data exclusion remains a local preflight rule.
Operationally, the poller should remember the last successful query boundary, deduplicate work using stable group identity, and expose its own heartbeat. Those are client responsibilities in this design. The monitor supplies queryable error state; it does not supply phone, SMS, or webhook delivery. There is no need to pretend otherwise.
Acceptance tests and exit conditions
The acceptance test is deliberately small: a sanitized exception enters the expected group, its events can be inspected, an authorized resolution changes the workflow state, and the independent heartbeat detects a deliberately omitted run. Test repetitive exceptions separately to confirm that the group remains useful under noise. Then record event size, group arrival rate, and retained-run count so the byte budget can be recalculated from observed inputs rather than optimism.
I reject full log ingestion as the primary error-tracking path for this workload. Structured logs still have a place for carefully selected operational facts, but the available search filter parameters are not declared in discovery, there is no bulk export or subscription interface, and there is no per-user deletion endpoint. Those constraints are material in healthtech. Retention and cold-storage error codes also do not amount to a configuration surface, so a governance review cannot assume controls that aren't exposed.
I also reject “add tracing now, in case we need it.” The selected surface has no distributed trace query or span tree. That is suitable when the nightly pipeline needs simple grouped exceptions, and unsuitable when root-cause analysis depends on following work across multiple services. In the latter case, adopt an OpenTelemetry-compatible tracing backend after defining a sampling policy and measuring how span cardinality changes storage. Trace identifiers in logs can support correlation, but they do not create a tracing product.
The simple option is not suitable when browser context is the diagnostic unit. Stick with a frontend-centered error platform when Session Replay, source-map reconstruction, crash symbolication, or rich release health is required. Stick with a dedicated on-call platform when routing policies, threshold rules, and escalations are core requirements. Keep Healthchecks beside either choice when silent non-execution is the feared failure.
There is also a governance review trigger: if a deletion request must remove a person's log records, don't route those records into a store without a compatible deletion mechanism. Better still, keep direct identifiers and raw clinical data out of telemetry from the beginning. Data minimization reduces exposure and bytes at once — a rare case where the security and cost arguments point in exactly the same direction.
The ADR should be reopened when frontend failures enter scope, cross-service latency becomes a diagnostic question, the poller becomes an operational burden, or retained error volume breaks the agreed budget. Until one of those conditions occurs, simple exception groups plus a separate heartbeat provide the cleaner signal.
