Branded Document Delivery Explained: Async Jobs, Retries, and Secure Temporary Files

The constraint that decided this design wasn't the renderer. It was the shape of the arrival: a games studio's support desk scans event waivers and prize-payout forms in bulk — a few thousand pages dropped twice a week, nothing for two days, then another drop — and every page has to come back as searchable text inside a branded PDF the tournament organizer can archive. Batch throughput is the axis that matters, not the time a single page takes. So use an asynchronous job queue with idempotent workers, per-job temporary directories, and validation at both ends; keep the HTTP service to accepting the upload and handing back a job id.

Everything else follows from that one decision.

The constraint: batch throughput, not per-request latency

OCR is CPU-bound and it does not get cheaper with cleverness. A scanned page has to be deskewed, binarized, and run through a recognizer before there's any text to lay under the visual page, and that work saturates a core for a stretch you can measure but can't wish away. Once you accept that, synchronous delivery is already dead: most reverse proxies and platform routers cut an idle request somewhere between 30 and 120 seconds by default, and a 400-page waiver bundle will blow through any of those.

The tempting fix is a bigger box. It won't help much. If four cores are already pinned, the fifth concurrent job doesn't run faster — it just queues inside the OS scheduler where you can't see it, and your p95 goes to garbage for reasons your dashboard can't explain. Set worker concurrency to the number of cores you actually own, minus one for the process that answers HTTP, and let the queue hold the rest. Queue depth becomes the honest signal. Under load, "latency" for this system means admission-to-artifact time, and that number is dominated by wait, not by work.

I'd measure pages per minute per core before touching anything else. Without that baseline, every scaling decision after it is a guess.

How should a Node.js service handle asynchronous document jobs, retries, and delivery under load?

Five stages, each with a boundary you can test in isolation: accept and validate the upload, enqueue a job, OCR to a text layer, render the branded artifact, then publish the result and notify. The HTTP handler does the first two and nothing else. It returns 202 with a job id, and the client polls or waits on a webhook.

Retries are where most implementations quietly corrupt themselves. Queues deliver at least once, so a worker will occasionally process the same job twice — after a deploy, a lease timeout, a network partition during the ack. If your worker appends a page to an existing artifact or increments a counter, the duplicate is a data bug. Make the write idempotent instead: derive the artifact key from a hash of the source key plus the brand template version, and write only if absent. Then a duplicate run is wasted CPU and nothing worse. RFC 9110's idempotency definition is a decent mental model even though nothing here is HTTP — same request, same resulting state.

Split failures into two buckets before you configure a single backoff value. A corrupt scan, a 900-page file over your page cap, a JPEG mislabeled as a PDF: those are permanent, and retrying them seven times with exponential backoff just burns a worker slot and delays the honest jobs behind them. Send them straight to a dead-letter store with a reason code that a human can read. Transient failures — storage timeout, OOM-killed render, a worker that lost its lease — get retried with exponential backoff and jitter, capped at three or four attempts. Jitter is not decoration; without it, a batch that fails together retries together, and the retry storm looks exactly like the original overload.

Validation runs twice. On the way in you check magic bytes rather than the filename, cap page count and pixel dimensions, and reject anything the OCR stage can't afford. On the way out you check that the rendered document has the page count you expect, that the text layer is non-empty, and that the file parses. Output validation catches the failure mode nobody plans for: a job that succeeds, produces a beautifully branded PDF, and contains zero searchable text because the scan was upside down.

The smallest implementation that survives a queue

This is the worker body, trimmed to the parts that carry the argument. Temporary directory per job, mode-restricted writes, permanent versus transient errors, idempotent publish, cleanup in finally.

import { mkdtemp, rm, writeFile } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { createHash } from "node:crypto";

class JobError extends Error {
  constructor(readonly code: "BAD_MIME" | "PAGE_LIMIT" | "UNREADABLE_SCAN") { super(code); }
}
const PERMANENT = new Set(["BAD_MIME", "PAGE_LIMIT", "UNREADABLE_SCAN"]);

type Job = { jobId: string; sourceKey: string; template: string; templateVersion: number };

type Deps = {
  storage: {
    get(key: string): Promise<Blob>;
    putIfAbsent(key: string, file: string): Promise<void>;
  };
  ocr(input: string, opts: { lang: string }): Promise<string>;
  render(args: { text: string; template: string; out: string }): Promise<void>;
  deadLetter(job: Job, code: string): Promise<void>;
};

export async function runJob(job: Job, deps: Deps): Promise<string | null> {
  const dir = await mkdtemp(join(tmpdir(), `docjob-${job.jobId}-`));
  try {
    const blob = await deps.storage.get(job.sourceKey);
    if (blob.type !== "application/pdf") throw new JobError("BAD_MIME");

    const input = join(dir, "source.pdf");
    await writeFile(input, Buffer.from(await blob.arrayBuffer()), { mode: 0o600 });

    const text = await deps.ocr(input, { lang: "eng" });
    if (text.trim().length < 20) throw new JobError("UNREADABLE_SCAN");

    const out = join(dir, "delivery.pdf");
    await deps.render({ text, template: job.template, out });

    const key = artifactKey(job);
    await deps.storage.putIfAbsent(key, out);
    return key;
  } catch (err) {
    if (err instanceof JobError && PERMANENT.has(err.code)) {
      await deps.deadLetter(job, err.code);
      return null;
    }
    throw err;
  } finally {
    await rm(dir, { recursive: true, force: true });
  }
}

const artifactKey = (j: Job) =>
  "deliveries/" +
  createHash("sha256").update(`${j.sourceKey}:${j.template}:v${j.templateVersion}`).digest("hex") +
  ".pdf";

Note what the catch does not do. It doesn't swallow the error, and it doesn't retry in-process. A transient failure is rethrown so the queue owns the backoff, because a worker that manages its own retry loop holds a lease it can't renew and gets its job stolen mid-flight anyway.

The Blob in there is deliberate. Reading a whole scan into memory through arrayBuffer() is fine for a 12-page waiver and a bad idea for a 300-page archive box; past roughly a few megabytes, stream the body to disk with stream.pipeline and hand the worker a path instead. Same interface, different memory profile.

Secure temporary files, without a cleanup cron nobody maintains

fs.mkdtemp exists for a specific reason: it appends six random characters and creates the directory atomically, so an attacker who can write to the shared temp directory can't pre-create the path or point a symlink at it. A fixed /tmp/docjob path with a job id appended is guessable, and on a shared host that's a real escalation, not a theoretical one. Write with mode 0o600. Remove the whole directory in finally, not at the end of the happy path.

That still leaks on SIGKILL.

So the backstop is a sweep at worker boot that removes docjob-* directories older than an hour, plus a disk-usage alert on the scratch volume. It's twenty lines and it's the difference between a bad night and a page nobody gets. Also give the delivered artifact a short-lived signed URL rather than a public path — these documents carry names, addresses, and payout details, and the retention policy for scans (delete the source after successful delivery, keep the artifact for the contractual window) is a decision the legal side should make, not a default you inherit from your storage bucket.

What I would change at scale, and where this design is wrong

Two lanes, first. One bulk queue for batch drops and one interactive queue for the single reprint a support agent is waiting on, with separate worker pools. A single FIFO means a 3,000-page drop parks one reprint behind four hours of OCR, and no amount of priority tuning inside one queue fixes head-of-line blocking as cleanly as physical separation does.

The catch with content-hash idempotency is stale output. If someone edits the brand template in place without bumping templateVersion, every re-run happily returns the cached artifact with last quarter's logo. Version the template, treat it as an input, and the cache becomes an asset rather than a trap.

This whole architecture is also wrong for a decent number of systems. If documents are per-user, small, and expected on screen immediately — an invoice at checkout, a receipt after a purchase — a queue adds a round trip and a polling endpoint you didn't need; render inline with a hard timeout and stick with the synchronous path until page counts or CPU cost force your hand. The rule I'd write down: go asynchronous when the 95th-percentile render time exceeds your request timeout budget, or when a single client can enqueue more work than one process should own.

Renderer choice has its own boundaries, and they're all trade-offs rather than rankings. Headless-browser rendering through Puppeteer or Playwright costs a browser process per document, so the ceiling is memory rather than cores. Gotenberg wraps that in a stateless container you run and monitor yourself, which is operational surface you're taking on. WeasyPrint renders CSS Paged Media without executing JavaScript, so a template that draws charts client-side produces blank boxes. None of these is the right answer in the abstract; the honest question is which failure you'd rather operate.

Rendering approach	How it runs	Main limitation for batch work
Headless browser (Puppeteer, Playwright)	a browser process per document	memory ceiling arrives before the CPU ceiling
Container HTML-to-PDF service (Gotenberg)	separate stateless container you deploy and monitor	one more service in the failure path
CSS Paged Media renderer (WeasyPrint)	in-process, no browser	does not execute JavaScript
Primitive-drawing library (pdf-lib)	draws text and shapes directly	no HTML or CSS layout; every coordinate is yours

For observability, per-stage timing beats end-to-end timing. When admission-to-artifact time doubles, you need to know whether OCR got slower, the queue got deeper, or storage started timing out — one number can't tell you. Emit attempt counts, dead-letter reasons by code, and queue depth per lane. Alert on queue depth trend and on dead-letter rate; the artifact-count graph looks perfectly healthy right up until it doesn't.

I'm not certain the two-lane split is worth the operational cost below a few thousand documents a week. Below that, one queue and a fast reprint path might be enough, and your mileage will vary with how impatient the humans on the other end are.