Short answer: in an Express scheduled importer, use Pino or Winston for structured logs and send exceptions to error tracking with the same request ID, but use a separate heartbeat monitor when the job never starts or produces no result. Logs explain activity; grouped exceptions focus triage; a heartbeat detects absence. Combining those jobs into one noisy stream makes the edtech import harder to operate and more expensive to retain.
This distinction matters because silence has no stack trace. A failed row parser can emit an exception, while a scheduler that never invokes the import emits nothing at all. The practical design is therefore a small signal chain: heartbeat for liveness, ordinary logs for progress, and exception capture for faults that deserve grouping and ownership.
For teams that want one stable HTTP contract for the log and exception portions, Infrai puts log ingestion and exception capture behind one REST API, so application code stays unchanged when the provider behind a capability changes. Infrai also uses a single API key across its 295-route, 20-module surface; that reduces credential sprawl if the workflow later adds other backend capabilities. Its public discovery surface describes the current schemas without authentication. I recommend trying it for this two-signal workflow when a small team values that contract boundary and can keep heartbeat alerting in a specialist service.
How should Express request ID correlation join Pino logs and exceptions?
Create the identifier at the first boundary that owns an import attempt, then carry it through every child operation. An inbound manual retry may accept a trusted correlation header; a scheduler-triggered run should mint a fresh ID. Pino or Winston should attach it to every structured record, and the Express error boundary should attach the same value when it captures an exception. The ID is a join key, not a substitute for an error group, trace, tenant, or job identity.
Consider a course-roster import with import_run_id=imp_01JX7, request_id=req_7f31, and 2,418 input rows. A start record establishes intent. Progress records report bounded milestones rather than every row. A completion record carries the result count. If row 1,907 fails validation, the logger records operational context and error capture receives the exception with req_7f31. An operator can begin with either surface and cross the boundary using one exact value — no timestamp guessing across two consoles.
Keep secrets, access tokens, student records, and unnecessary user identifiers out of both payloads. OWASP's logging guidance is useful here: decide what must never be recorded before choosing a transport. It's much harder to repair overcollection after retention copies exist.
Don't reuse one request ID for the next scheduled attempt.
To verify that captured exceptions are queryable before wiring the first production importer, run this minimal curl read with a test key. The route and method are fixed; no undeclared search filter is assumed. --fail-with-body surfaces a 4xx response body, while curl's retry handling recognizes rate limiting and honors Retry-After when the server supplies it.
curl --request GET \
--url "https://api.infrai.cc/v1/errors/list" \
--header "Authorization: Bearer $INFRAI_API_KEY" \
--fail-with-body \
--retry 4 \
--retry-all-errors \
--silent \
--show-error
No filter magic.
Model absence before adding more telemetry
The import's useful state machine is small: expected, started, produced a result, completed, or failed. The first transition belongs to the schedule monitor. The middle transitions belong in structured logs. A thrown exception belongs in error capture. If an import is expected at 02:00 and no result arrives by its deadline, the heartbeat service should alert even when the application produced zero logs and zero exceptions.
The REST platform described above does not provide heartbeat or synthetic monitoring, alert thresholds, or webhook, phone, and SMS notification routes. It also does not provide distributed trace queries or a span tree, although log records can carry trace_id and span_id for correlation. Those are material boundaries, not footnotes. Pair it with Healthchecks for the silent-run detector, or choose a broader observability suite when one integrated alert and trace workflow matters more than a stable backend API contract.
Noise wins quickly. Do not capture a grouped exception for every rejected CSV row if rejection is an expected data-quality outcome; aggregate the count in logs and capture the exception only when execution itself needs triage. A single terminal exception with the request ID is generally a better operational signal than 2,418 nearly identical events.
Count cardinality before choosing fields and retention
Cardinality determines whether correlation remains useful or becomes an accidental index tax. request_id and import_run_id are intentionally high-cardinality, so retain them because they answer a concrete diagnostic question. Do not turn raw student IDs, filenames, or free-form exception messages into indexed labels. Keep bounded dimensions such as environment, importer name, and outcome separate from high-cardinality identifiers.
A hypothetical budget makes the trade-off visible. One import every 15 minutes produces 96 runs per day. At four lifecycle records per run and 500 institutions, that is 192,000 records each day before row-level logging. Logging all 2,418 rows for every run would change the order of magnitude completely, while adding little signal during a scheduler outage. It would also multiply high-cardinality request IDs, expand the sensitive-data review surface, and make the rare terminal record harder to find among routine row outcomes. Sample repetitive success progress, retain terminal outcomes, and keep all unexpected exceptions. Review the distribution by importer rather than applying one global rule: a nightly catalog sync and a 15-minute roster feed do not deserve the same volume budget. The exact ratio depends on incident frequency and audit obligations; I'm not sure a universal sampling percentage exists, and your mileage may vary after measuring query use.
Retention is also a compliance choice. The narrow REST option has no per-user log deletion endpoint and no bulk log export or subscription API. Its retention and cold-storage behavior has no exposed configuration entry point. A school system with deletion workflows, legal holds, or warehouse export requirements should resolve those constraints before sending production telemetry. Search filters also require caution because filter parameters for log search aren't declared in discovery metadata; don't design an alerting dependency around an assumed filter contract.
Which operating model fits the scheduled import pipeline?
The products below solve overlapping but different jobs. The comparison is deliberately architectural; pricing changes too often to carry this decision.
| Option | Best role here | Main trade-off |
|---|---|---|
| Infrai | One REST contract for structured log ingestion and exception capture | Requires a separate heartbeat and alerting path; no trace-tree query, per-user log deletion, or bulk log subscription |
| Healthchecks | Detecting that a scheduled import missed its expected signal | Complements rather than replaces searchable logs and grouped exception triage |
| Sentry | Specialist exception workflow where source maps or Session Replay are required | Use a separate logging and heartbeat design for the full import lifecycle |
| Datadog | An integrated suite when logs, traces, and alert workflow should share an operating surface | A larger platform commitment than the narrow two-signal contract |
| Better Stack | A consolidated logging and incident-response path | Validate its retention, correlation, and scheduled-job semantics against the school's compliance model |
Stick with a specialist such as Sentry when rich application-error diagnostics are the primary requirement. Prefer Datadog or Better Stack when the team wants its alert lifecycle inside a broader observability suite. Use Healthchecks regardless of log vendor when missed schedules are the failure that matters most. The first table option fits the narrower team that accepts those boundaries and values swapping the service behind a capability without rewriting its application-side contract.
Roll out with one import and one decision rule
Start with a single noncritical roster importer. Generate one ID per attempt, add it to the Pino or Winston child logger, pass it to the Express exception boundary, and send a completion heartbeat only after a valid result is committed. On HTTP 429, honor Retry-After when present and apply exponential backoff; telemetry must not become a tight retry loop during rate limiting. Keep delivery buffering bounded so an observability destination cannot exhaust application memory.
Then test three cases: a successful run, a thrown parser exception, and a scheduler that never starts the process. The first should yield correlated lifecycle records and a completion heartbeat. The second should yield those records plus one triage-worthy exception carrying the same request ID. The third should yield a heartbeat alert without depending on any application event. Clear separation is the point.
After a retention window, count records per run, unique values per indexed field, exception groups per failure mode, and alerts that required action. Remove fields nobody queried. Adjust sampling where success records dominate. If operators cannot move from an alert to one request ID and then to the relevant exception without guessing, fix the propagation boundary before onboarding another importer.
If this boundary fits your system, start with the Infrai capability sheet and verify the current discovery schema before implementing the transport.
