Media Rollback Evidence: Cron Worker Error Tracking, Retry Failure Capture

For a Node.js media pipeline backed by Postgres, a cron worker's background job must preserve more than “the worker crashed.” It needs enough evidence to decide whether a half-published asset should roll forward or roll back.

Short answer: background-job error tracking works when every failed run records structured job context and distinguishes an expected retry from a terminal failure, while a separate heartbeat monitor catches the silent case where the cron job never ran.

That split matters. An exception tracker sees code that ran and failed. It cannot infer that a scheduler never invoked the code. Treat those as two different signals from day one.

The failure envelope is the rollback boundary

Picture the before state. A Node.js worker writes publish failed and a stack trace. The Postgres row says the asset is still processing. BullMQ schedules another attempt. Nobody can quickly tell whether the first attempt uploaded a rendition, changed the database, or stopped before either action. The retry might repair the incident. It might also repeat a side effect.

Now picture the after state. Each attempt emits one failure envelope containing the job name, queue, attempt number, stable payload identifiers, and stack trace. The envelope also says whether another retry is expected. The database keeps the durable workflow state; the tracker keeps the diagnostic evidence. An operator can search for asset asset_8421, compare attempt 2 with attempt 3, and decide whether reverting the release is safer than replaying the job.

That is the diagram in words: cron trigger -> queue -> worker attempt -> durable state change -> structured failure evidence. A heartbeat watches the first arrow. Error tracking watches the worker attempt. Postgres remains the source of truth for what actually committed.

Keep secrets and raw customer content out of that evidence. Payload identifiers are usually enough to reconstruct the path by looking up authorized records later. OWASP's logging guidance explicitly warns against recording access tokens, passwords, sensitive personal data, and other high-risk values. Deletion obligations matter too; if a system cannot delete logs by user, don't copy user profiles into its error context and hope retention policy will solve the problem.

Tiny payloads win.

How should a Node.js cron worker capture background job retry failures?

Use one small wrapper around the work, then let the queue library continue to own retry scheduling. The wrapper below is runnable TypeScript with no tracking SDK. It defines the contract a reporter must receive, emits a structured record, and rethrows the original error so BullMQ or Agenda can apply its configured retry policy.

type FailureEvidence = {
  jobName: string;
  queue: string;
  attemptNumber: number;
  payloadId: string;
  terminal: boolean;
  errorName: string;
  errorMessage: string;
  stackTrace: string;
};

type JobContext = {
  jobName: string;
  queue: string;
  attemptNumber: number;
  maxAttempts: number;
  payloadId: string;
};

type FailureReporter = (evidence: FailureEvidence) => Promise<void>;

const wait = (milliseconds: number): Promise<void> =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

async function reportFailure(evidence: FailureEvidence): Promise<void> {
  const apiKey = process.env.INFRAI_API_KEY;
  const apiBaseUrl = process.env.INFRAI_API_BASE_URL;
  if (!apiKey || !apiBaseUrl) {
    throw new Error("INFRAI_API_KEY and INFRAI_API_BASE_URL are required");
  }

  const idempotencyKey = [
    evidence.queue,
    evidence.jobName,
    evidence.payloadId,
    evidence.attemptNumber,
  ].join(":");

  for (let requestAttempt = 0; requestAttempt < 4; requestAttempt += 1) {
    const response = await fetch(new URL("/v1/errors/capture", apiBaseUrl), {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey,
      },
      body: JSON.stringify(evidence),
    });

    if (response.status === 429 && requestAttempt < 3) {
      const retryAfter = Number(response.headers.get("Retry-After"));
      const delayMilliseconds = Number.isFinite(retryAfter)
        ? retryAfter * 1_000
        : 2 ** requestAttempt * 1_000;
      await wait(delayMilliseconds);
      continue;
    }

    if (!response.ok) {
      const reason = await response.text();
      throw new Error(`Error capture rejected (${response.status}): ${reason}`);
    }

    return;
  }
}

export async function runTrackedJob<T>(
  context: JobContext,
  work: () => Promise<T>,
  reportFailure: FailureReporter,
): Promise<T> {
  try {
    return await work();
  } catch (cause: unknown) {
    const error = cause instanceof Error ? cause : new Error(String(cause));
    const terminal = context.attemptNumber >= context.maxAttempts;

    await reportFailure({
      jobName: context.jobName,
      queue: context.queue,
      attemptNumber: context.attemptNumber,
      payloadId: context.payloadId,
      terminal,
      errorName: error.name,
      errorMessage: error.message,
      stackTrace: error.stack ?? `${error.name}: ${error.message}`,
    });

    throw error;
  }
}

async function example(): Promise<void> {
  await runTrackedJob(
    {
      jobName: "publish-rendition",
      queue: "media-publish",
      attemptNumber: 2,
      maxAttempts: 3,
      payloadId: "asset_8421",
    },
    async () => {
      throw new Error("rendition manifest was rejected");
    },
    reportFailure,
  );
}

example().catch(() => {
  process.exitCode = 1;
});

In BullMQ, build JobContext inside the worker processor: job.name, job.queueName, job.attemptsMade + 1, job.opts.attempts ?? 1, and a stable identifier from job.data supply the five values. In Agenda, build the same contract inside the defined job using the job name, failure count, configured attempt ceiling, and identifier in job data. The exact framework fields stay at the edge; the evidence format does not change.

There is one subtle rule here. Report attempt 2 of 3, but mark it as expected to retry. Report attempt 3 of 3 as terminal. If both become identical red alerts, engineers learn to ignore the first two and may miss the third. The UI should focus on terminal failures while still retaining earlier attempts for reconstruction.

The reporter calls Infrai's verified error-capture route directly. It uses one deterministic key per queue, job, payload, and attempt, so retrying the request cannot double-apply that capture. It also checks the response and backs off on HTTP 429 instead of hammering the API. Use the live discovery response for errors.capture to validate the current request schema before deploying an adapter; the public discovery surface returns the request JSON Schema and runnable TypeScript example without requiring a key.

Don't swallow the exception. The queue needs it.

Run the rollback drill before choosing a tracker

Before comparing products, rehearse one failure on paper. Suppose asset_8421 writes a rendition manifest, then fails before the Postgres transaction records publication. Ask three questions: can the operator find every attempt by the asset identifier, can they see which attempt is terminal, and can they locate the durable commit boundary without reading raw customer content?

If any answer is no, changing vendors won't rescue the rollback. Tighten the evidence contract first. Then repeat the drill for the opposite ordering, where Postgres commits before the remote publication step fails. This second case is deliberately uncomfortable — it exposes whether the worker has a safe compensating action or merely retries an irreversible side effect.

Where each monitoring option fits

The decision is less about a logo and more about the adjacent signals you already operate. Compare the workflow, not a feature-count page.

Infrai's concrete advantage here is operational consolidation: one credential and one bill cover a broad backend surface, while plain HTTP avoids adding another SDK to each Node.js worker. The catch is meaningful. Without built-in alert routing, a team must poll list or search APIs and send its own email, Slack, or webhook notification. It is not suitable as the only monitor for scheduled jobs because it has no heartbeat or synthetic monitoring.

Stick with Sentry when integrated cron monitoring is the deciding constraint. Choose Datadog when the organization already depends on its cross-signal operational view. Rollbar remains a sensible application-error candidate when exception triage is the center of the workflow. Add Healthchecks, or a similar heartbeat tool, whenever a missed schedule is itself an incident.

I'm not sure which tracker will be cheapest for your event volume, and a static article can't settle that; current retention, ingestion, and alerting terms would resolve it. Cost shouldn't decide rollback safety anyway. Evidence completeness and notification ownership should.

Can error tracking prove that the scheduled job ran?

No.

A tracker can capture a thrown error only after worker code begins executing. A deleted cron entry, paused queue, scheduler authentication problem, or deployment that never registered the task can produce no exception at all. The clean pattern is a dead-man's switch: ping a heartbeat service on successful completion, set a grace period, and alert when the ping is late.

For a media workflow, name that heartbeat after the user-visible promise, such as daily-rights-refresh, rather than after a process like worker-7. Then an alert says what may now be stale. The error event answers “why did attempt 3 fail?” The heartbeat answers “did today's run finish?” Those answers belong together during an incident, but they come from different mechanisms.

Alert delivery is another boundary. If the selected error API stores and searches events but does not route notifications, run a small poller against its supported query surface, persist the last processed event identifier, and make notification delivery idempotent. Polling every minute does not mean sending the same Slack message 60 times. Use a stable key made from the error group and terminal attempt, then record delivery before advancing the cursor.

For rollback, the final decision rule can stay crisp: roll forward when a retry is expected and the durable state shows no conflicting side effect; investigate or roll back when the terminal attempt failed after a state-changing step; escalate immediately when the heartbeat is late and there is no attempt evidence. Your mileage may vary for irreversible publishing targets, so document the exact commit boundary for each worker.

References

• https://docs.sentry.io/product/crons/
• https://docs.datadoghq.com/monitors/types/process_check/
• https://grafana.com/docs/grafana/latest/alerting/
• https://betterstack.com/docs/uptime/cron-and-heartbeat-monitor/
• https://docs.rollbar.com/docs/javascript
• https://healthchecks.io/docs/
• https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html
• https://gdpr-info.eu/art-17-gdpr/