Short answer: put customer identity verification on a durable job queue, validate the document before rendering, keep temporary files private and short-lived, and measure p95 latency separately for verification, rendering, and retries. For a media team redacting personal data from a document, a lower render cost is useless if the output loses text or layout that an editor must inspect.
The decision changed when I treated fidelity as a release gate instead of a nice-to-have. A fast raster pass can hide a name in a screenshot while leaving the selectable text in the PDF. A perfect render can also make a queue unusable during a traffic spike. The service needs both a correctness contract and a latency budget.
Build log: define the redaction contract before the queue
Start with a document manifest. It should contain a random job ID, the authenticated customer ID, the source object reference, requested redaction classes, and an expiry time. It should not contain a filename supplied by the customer, an email address, or the raw identity value you plan to remove.
Validation is cheap compared with rendering. Reject unsupported media types, oversized inputs, missing redaction policy, and expired manifests before any worker opens a file. Keep the accepted set explicit. For example, a PDF with a declared page count of 200 is not automatically safe to process: the worker must still enforce a page and pixel budget after parsing.
Fail closed.
Here is a small TypeScript boundary. It deliberately accepts a storage key rather than a path, and it returns a discriminated result that the queue can record.
type RedactionClass = "name" | "address" | "account-number";
type Manifest = {
jobId: string;
customerId: string;
objectKey: string;
classes: RedactionClass[];
expiresAt: number;
};
type Validation =
| { ok: true; manifest: Manifest }
| { ok: false; code: "expired" | "unsupported-policy" | "bad-key" };
const allowedClasses = new Set<RedactionClass>([
"name",
"address",
"account-number",
]);
export function validateManifest(input: unknown, now = Date.now()): Validation {
if (!input || typeof input !== "object") return { ok: false, code: "bad-key" };
const value = input as Partial<Manifest>;
if (typeof value.jobId !== "string" || !/^[a-zA-Z0-9_-]{16,80}$/.test(value.jobId)) {
return { ok: false, code: "bad-key" };
}
if (typeof value.customerId !== "string" || typeof value.objectKey !== "string") {
return { ok: false, code: "bad-key" };
}
if (!Array.isArray(value.classes) || value.classes.length === 0 ||
value.classes.some((item) => !allowedClasses.has(item as RedactionClass))) {
return { ok: false, code: "unsupported-policy" };
}
if (typeof value.expiresAt !== "number" || value.expiresAt <= now) {
return { ok: false, code: "expired" };
}
return { ok: true, manifest: value as Manifest };
}
The identity check belongs at the trusted ingress, before this manifest is created. The worker receives the resulting customer ID and an authorization decision, not a second copy of identity evidence. That split limits how much sensitive data enters logs and retry payloads.
How should asynchronous jobs, retries, validation, and secure temporary files meet a latency budget?
Use a state machine, not a boolean called done. A useful sequence is accepted, verifying, rendering, checking-output, complete, and rejected. Store an attempt number and a next-eligible timestamp with every job. A retry must be safe to run twice: the output key includes the job ID and the renderer writes to a temporary object before an atomic publish.
The worker below shows the shape without tying it to a queue vendor. downloadToPrivateTemp must create a file with owner-only permissions; removeTemp must run in finally, including validation failures. The AbortSignal carries the remaining job deadline so a slow render does not consume the whole queue.
type Job = { manifest: Manifest; attempt: number; deadline: number };
type Services = {
verifyCustomer: (customerId: string, signal: AbortSignal) => Promise<boolean>;
downloadToPrivateTemp: (objectKey: string, signal: AbortSignal) => Promise<string>;
renderRedacted: (path: string, classes: RedactionClass[], signal: AbortSignal) => Promise<Uint8Array>;
checkOutput: (bytes: Uint8Array, classes: RedactionClass[]) => Promise<boolean>;
publishAtomically: (jobId: string, bytes: Uint8Array) => Promise<void>;
removeTemp: (path: string) => Promise<void>;
};
export async function runJob(job: Job, services: Services): Promise<"complete" | "rejected"> {
const remaining = job.deadline - Date.now();
if (remaining <= 0) throw new Error("deadline-exceeded");
const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), remaining);
let tempPath: string | undefined;
try {
const authorized = await services.verifyCustomer(job.manifest.customerId, controller.signal);
if (!authorized) return "rejected";
tempPath = await services.downloadToPrivateTemp(job.manifest.objectKey, controller.signal);
const bytes = await services.renderRedacted(tempPath, job.manifest.classes, controller.signal);
const valid = await services.checkOutput(bytes, job.manifest.classes);
if (!valid) throw new Error("output-validation-failed");
await services.publishAtomically(job.manifest.jobId, bytes);
return "complete";
} finally {
clearTimeout(timer);
if (tempPath) await services.removeTemp(tempPath);
}
}
Do not retry every exception. Retry network timeouts and temporary storage responses with exponential backoff and jitter; reject malformed manifests, failed authorization, and output-validation failures. Cap attempts, then move the job to a review queue with a reason code. Your mileage may vary on the cap because a one-page document and a 500-page scan have different render budgets, but the policy must be explicit.
An idempotency record closes the duplicate window. Before publishing, compare the job ID and a content hash with the prior result. If they match, return the existing result. If they differ, retain the new object under a versioned key and require review. Never let a retry overwrite an editor's already-approved redaction.
That last rule sounds obvious. It is easy to violate when a retry handler has permission to write directly to the final object.
Measure fidelity and render cost as separate signals
Latency under load is a distribution, not an average. Record queue wait, identity-verification duration, download duration, render duration, output-check duration, and total age at completion. Track p50 and p95, plus the count of jobs that exceed their deadline. Averages conceal a queue that is fine for nine customers and unusable for the tenth. When a load test mixes a two-page transcript with a 500-page scan, the long tail often comes from memory pressure and worker contention rather than from the identity provider itself; that is why each stage needs its own timer and why queue age belongs on the same dashboard as render duration. Set a deadline per class of input, then make the admission controller reject work that cannot finish inside the remaining budget. I don't tune concurrency from a single happy-path run, because it says nothing about the queue after a burst of retries, a cold worker, or a larger-than-usual image. Measure those cases explicitly and keep the fixture mix in the test report.
Fidelity needs its own checks. Keep a fixture set with selectable text, embedded fonts, rotated pages, transparent overlays, and scanned images. For each fixture, assert that forbidden strings are absent from extracted text and that the page geometry remains within a defined tolerance. Also inspect a small visual sample; pixel comparisons alone miss a redaction rectangle that shifted but still looks statistically similar.
I once assumed that shrinking a preview would make the system faster. It did, but the OCR pass then missed a low-contrast account number. The useful benchmark was not “milliseconds per page”; it was recall of the forbidden fields at each render scale, alongside p95 render time. Keep that benchmark in CI and run a load test with the same mix of short and long documents.
Temporary files deserve observability too. Emit a job ID, byte count, and lifetime in seconds, never the source path or identity value. Alert on a lifetime above the deletion SLA and on disk usage near the worker limit. A private directory and restrictive permissions reduce exposure; encryption at rest and automatic expiry reduce the blast radius if a host or backup is later inspected.
What should change at scale, and when is this design the wrong fit?
Separate verification workers from render workers once either stage saturates independently. Give rendering a bounded concurrency pool because it consumes CPU and memory; give verification a pool sized for its upstream service limit. Admission control should return a queued status quickly instead of allowing unbounded in-process promises. A small dashboard showing queue age, p95 by stage, retry reasons, and temporary-byte totals is more actionable than a single “success rate” number.
The catch is operational weight. This design is not suitable when a user needs a synchronous decision in a few hundred milliseconds, when documents are tiny and ephemeral enough that a queue adds more latency than it removes, or when your team cannot operate private scratch storage and a review queue. In those cases, stick with a synchronous, memory-only path for a tightly bounded input, or choose a managed document workflow whose compliance and retention controls you can verify. Do not pretend the asynchronous architecture is free.
It is also a poor fit when the redaction policy cannot be expressed as testable classes and output assertions. Human review may be the correct boundary for ambiguous handwriting or context-dependent names. The engineering win is knowing that boundary before traffic arrives.
Further reading
- MDN, “Blob API”: https://developer.mozilla.org/en-US/docs/Web/API/Blob
- OWASP, “File Upload Cheat Sheet”: https://cheatsheetseries.owasp.org/cheatsheets/File_Upload_Cheat_Sheet.html
- W3C, “Trace Context”: https://www.w3.org/TR/trace-context/
