Designing an OCR document pipeline on serverless AWS

Document OCR is easy to demo and hard to run. The demo takes a PDF, calls an OCR service, and prints the fields. The production version has to handle a ten-page scan uploaded from a phone, a form filled in by hand, a field read with 61 percent confidence, and a user who needs to know whether their document was accepted. I have built this on serverless AWS for a platform used by a large number of students, and this is the shape that held up.

The pipeline, end to end

1. Upload straight to S3. The browser gets a pre-signed URL from an API and uploads the file directly. Nothing passes through a Lambda, so a 25 MB scan does not hit a payload limit, and the API only records that an upload was requested.
2. S3 event to EventBridge to Step Functions. The object-created event starts a Step Functions execution per document. The state machine is the only place that knows the order of steps, which is what makes the pipeline readable six months later.
3. Validate before OCR. A short Lambda checks the file type, size and page count, and rejects what the OCR service would reject anyway. This step saves money and gives the user a fast, specific error.
4. Asynchronous OCR. Amazon Textract's synchronous calls have page and size limits, so anything that can be multi-page goes through the asynchronous API: StartDocumentAnalysis with an SNS notification channel, and the state machine waits on a task token until the completion message arrives. GetDocumentAnalysis is paginated, so the collector loops on NextToken until it has the whole result.
5. Extract and normalise. Raw OCR output is blocks and geometry. A Lambda maps it to the fields the product cares about, normalises dates and amounts, and records a confidence score per field, not per document.
6. Decide: accept, review, or reject. Per-field thresholds, set from measured data, route each document. Everything above the threshold is accepted automatically. Anything below goes to a human review queue with the page image and the proposed value side by side. Anything unreadable goes back to the user with a reason.
7. Store and notify. Results land in DynamoDB keyed by document ID, the state machine emits a completion event, and the application notifies the user.

Confidence thresholds are measured, not guessed

The single most important number in the system is the per-field threshold, and the mistake is to pick one number for every field. A date of birth read wrong is a different kind of error from a middle initial read wrong.

We built a golden set: a few hundred documents labelled by hand, run through the pipeline, with the OCR value and confidence recorded for each field. From that you can plot, for each field, accuracy against confidence, and pick the threshold where accuracy reaches the level the business accepts. Some fields needed; some could never be trusted automatically and always went to review.

The golden set then becomes a regression test. Every change to the extractor, every change to a prompt or a model version, runs against it before deploy.

The human review loop is part of the architecture

A pipeline that cannot route to a person is a pipeline that will silently accept wrong data. The review queue is not an exception path; it is a first-class step with its own metrics: queue depth, time to review, and the share of reviewed values that the reviewer changed. That last number is the feedback that tunes the thresholds. If reviewers accept 99 percent of what they see for a field, the threshold for that field is too conservative.

Reviewer corrections are written back to the same DynamoDB record with who changed what and when. Nothing is overwritten.

Sensitive data decides half the design

Documents of this kind carry personal data, so the pipeline is built around not leaking it:

• The upload bucket and every downstream bucket are encrypted with KMS and block public access.
• S3 lifecycle rules delete the raw uploads after a fixed window. Keep the extracted fields, not the scans, unless there is a legal reason to keep the scans.
• Lambdas log document IDs and confidence scores, never field values. A single log line with a value in it is a data leak that lives for the retention period.
• Reviewers see pages through short-lived pre-signed URLs, and the review application never stores the image.

Failure modes that were not in the diagram

• Duplicate uploads. Users re-upload when the first upload is slow. The document ID is derived from the user and a content hash, so the second upload short-circuits instead of running OCR twice.
• OCR service throttling. Textract has account-level concurrency limits. The state machine uses a retry with backoff on the start call, and a concurrency limit on the step keeps a bulk upload from starving everyone else.
• Stuck executions. A completion notification that never arrives leaves the execution waiting on its task token. Every wait state has a heartbeat timeout, and a timed-out execution goes to the review queue with a "needs re-run" flag instead of hanging forever.
• Partial results. A twelve-page document where page nine failed is still useful. Page-level status is stored, and the decision step treats a missing required page as "review," not "reject."

What I would keep and what I would change

I would keep Step Functions as the backbone. The visual execution history alone has saved hours of debugging, and the task-token pattern makes the asynchronous OCR call look synchronous in the state machine. I would keep the golden set; it is the only reason we could change the extractor with confidence.

I would move the confidence thresholds into configuration from the start rather than code, because they change more often than anything else. And I would build the review metrics on day one, because the pipeline is only as good as the feedback loop that tunes it.