System Design: PDF Processing Pipeline

A capstone system design walkthrough — designing a system that ingests, parses, transforms, and extracts structured data from PDF documents at scale — covering the ingestion and job queue, the multi-stage extraction pipeline (text, layout, tables, OCR for scanned pages), handling malformed and adversarial files safely, idempotent and resumable processing, human-in-the-loop review for low-confidence extractions, and the specific correctness, security, and throughput demands that make PDF processing a uniquely messy system design problem.

Table of Contents

1. Introduction
2. Why PDF Processing Is a Different Kind of Hard
3. The Core Domain Model
4. The Document Store and Job Log: Immutable Inputs, Replayable Pipeline
5. Idempotency and Exactly-Once Processing Per Document
6. The Multi-Stage Extraction Pipeline
7. The Document Processing State Machine
8. Isolating Untrusted Input: Sandboxing and Parser Security
9. OCR and the Confidence Problem
10. Human-in-the-Loop Review
11. Handling Failure, Retries, and Poison Documents
12. Data Security and Compliance
13. Consistency, Availability, and the CAP Trade-off for a Pipeline
14. Scaling the System
15. Observability for a Document Processing Pipeline
16. Common Pitfalls
17. Quick Reference Table
18. Conclusion

Introduction

A PDF processing pipeline takes the general system design vocabulary covered in this series' System Design guide — job queues, worker pools, blob storage, state machines — and applies it to an input format that is, in practice, far less well-behaved than its specification suggests: PDFs in the wild come from decades of different producers, span genuinely scanned images to machine-generated text to deliberately obfuscated or malformed files, and a pipeline built assuming "the PDF spec is followed" will fail constantly in production. This guide walks through designing such a system end to end, drawing directly on this series' Event-Driven Architecture, Background Services, Data Pipeline, and Security guides, each of which turns out to be load-bearing infrastructure for processing PDFs reliably and safely at scale, rather than optional architectural polish.

Upload → Document Store (blob) → Job Log (source of truth) → Stage 1: Classify → Stage 2: Extract (text/layout/OCR)
                                                                                        ↓
                                                              Stage 3: Structure/Validate → Human Review (if low confidence)
                                                                                        ↓
                                                                              Output Store + Downstream Consumers

1. Why PDF Processing Is a Different Kind of Hard

The input format is adversarial by nature, not just messy

Most systems covered in this series can assume input roughly conforms to a schema, with validation catching genuine edge cases. A PDF processing pipeline can't make that assumption at all: the format is a container that can legally hold embedded fonts, JavaScript, forms, encrypted content, deeply nested objects, and can be produced by hundreds of different tools with varying (and sometimes deliberately broken) spec compliance — some fraction of input, especially from untrusted or adversarial sources, is actively malformed or crafted to exploit parser bugs. This is why sandboxing (Section 7) and defensive parsing get as much design attention in this guide as extraction accuracy does.

The same document can require wildly different processing paths

A born-digital PDF with a text layer: text extraction is fast, cheap, and highly accurate.
A scanned image of the same kind of document: requires OCR, is slower, and is
  probabilistic rather than exact — the SAME downstream schema, arrived at very differently.

Unlike most ingestion pipelines in this series where one processing path handles all input reasonably well, PDFs bifurcate sharply into digital-native and scanned/image-based documents (and frequently mix both within a single file), which is precisely why this guide treats classification (Section 5) as a first-class early pipeline stage rather than an afterthought — routing each document, and even each page, down the cheapest path that will actually work.

Extraction confidence is not binary, and the pipeline must know that

A critical, freeing realization for the design that follows: a PDF processing pipeline, in the overwhelming majority of real-world designs, does not need to guarantee perfect extraction on every document — it needs to know, and expose, how confident it is in each extraction, and route the genuinely uncertain fraction to a human (Section 9) rather than silently propagating a wrong answer downstream with the same confidence as a clean, machine-generated extraction. This mirrors the "know what you don't know" discipline covered in this series' Machine Learning Systems guide, applied here to structured extraction rather than a classification task.

2. The Core Domain Model

Modeled with DDD, per this series' companion guide

public record DocumentId(Guid Value);
public record JobId(Guid Value);

public enum DocumentStatus { Uploaded, Classified, Extracting, ExtractionComplete, NeedsReview, Reviewed, Failed }

public class DocumentJob // the AGGREGATE ROOT, per this series' DDD guide
{
    public JobId Id { get; }
    public DocumentId DocumentId { get; }
    public DocumentStatus Status { get; private set; }
    public IReadOnlyList<PageResult> PageResults { get; private set; }
    private readonly List<JobEvent> _domainEvents = new();

    public void CompleteExtraction(IReadOnlyList<PageResult> results)
    {
        if (Status != DocumentStatus.Extracting)
            throw new InvalidOperationException($"Cannot complete extraction from status {Status}");
        PageResults = results;
        Status = results.Any(r => r.Confidence < ConfidenceThreshold)
            ? DocumentStatus.NeedsReview
            : DocumentStatus.ExtractionComplete;
        _domainEvents.Add(new ExtractionCompletedEvent(Id, Status));
    }
}

This directly applies this series' DDD guide's aggregate pattern — DocumentJob is the aggregate root, enforcing its own state transitions (extraction can't be "completed" from a state that was never extracting) rather than trusting every caller to check status before mutating it, and raising domain events at exactly the points those transitions genuinely occur.

Separating the document from the job that processes it

public record StoredDocument(DocumentId Id, string BlobUri, string Sha256Hash, long SizeBytes, DateTimeOffset UploadedAt);
public record PageResult(int PageNumber, ExtractionMethod Method, string ExtractedText, double Confidence, IReadOnlyList<TableRegion> Tables);

As covered in this series' DDD guide's aggregate-sizing discussion, keeping the immutable StoredDocument (the raw bytes, hashed and stored once) separate from DocumentJob (the mutable processing state, potentially re-run or reprocessed) means a document can be reprocessed — a new extraction model, a corrected classification rule — without ever touching or re-uploading the original bytes, which is exactly the separation Section 3's replayability depends on.

3. The Document Store and Job Log: Immutable Inputs, Replayable Pipeline

Why the raw document must be stored once and never modified

❌ Extracting text and discarding the original PDF: no path to re-extract with a better
   model or a bug fix; the original evidence of what was actually processed is gone.
✅ Store the raw PDF bytes, immutably, in blob storage — every pipeline stage reads from
   it but never modifies it.

A PDF processing pipeline needs the original bytes to remain available and untouched for the lifetime of the system — reprocessing (a new extraction model version, a fix to a parsing bug, a customer dispute about what a document actually said) is common enough that treating the original upload as immutable, content-addressed storage (keyed by a hash of its bytes, per this series' Blob Storage guide) is a foundational decision, not an optimization.

The job log as the append-only backbone for pipeline state

CREATE TABLE pipeline_job_log (
    sequence_id BIGINT PRIMARY KEY,
    job_id UUID NOT NULL,
    document_id UUID NOT NULL,
    stage VARCHAR NOT NULL,            -- Classify, Extract, Structure, Review
    event_type VARCHAR NOT NULL,       -- Started, Completed, Failed, Retried
    payload JSONB NOT NULL,
    created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

In practice this table's role is usually filled by a distributed log or durable job queue (Kafka, or a workflow engine's own event store) rather than a bare relational table — every stage transition for every document is first durably appended, before the next stage begins. This gives a durable, replayable record (rebuild a document's full processing history, or resume a job interrupted mid-pipeline, from where it left off), and a backbone for downstream consumers via the outbox/CDC pattern, directly echoing this series' Event-Driven Architecture guide's discussion of avoiding dual-write inconsistency between "advance pipeline state" and "publish the event."

Derived extraction output is always recomputable, never the sole record

The structured output (extracted fields, tables) is a DERIVED projection of the pipeline
  run against the immutable original document — it can always be regenerated by
  re-running the pipeline (or a newer version of it) against the same stored bytes.

Per this series' Caching and Materialized View discussions, treating extracted output as a derived, regenerable projection — rather than the only record of what a document contains — is what makes model upgrades, bug fixes, and audits tractable: nothing is ever "lost" that can't be recovered by reprocessing the original, unmodified input.

4. Idempotency and Exactly-Once Processing Per Document

Why this matters even more once you add retries and parallel workers

As covered throughout this series' RabbitMQ, Kafka, and Event-Driven Architecture guides, every job queue provides at-least-once delivery, and a worker crashing mid-extraction is a routine, expected occurrence at scale, not an edge case — an un-idempotent pipeline stage means a retried job either duplicates output (the same table extracted twice into a downstream system) or, worse, corrupts partially-written results, which is precisely why idempotency is this guide's single most emphasized property.

Content-addressed deduplication at ingestion

var hash = Sha256(fileBytes);
var existing = await _documentStore.FindByHashAsync(hash);
if (existing is not null)
{
    return existing.DocumentId; // the SAME document was already uploaded and processed — no reprocessing
}

Hashing the uploaded bytes and checking for an existing document with the same hash, before ever starting a pipeline run, is the first and cheapest idempotency guarantee in the system — per this series' Deduplication pattern discussion, it prevents wasted extraction work on a document that's already been processed, which matters considerably at volume given how often the same document (a resubmitted form, a duplicated batch upload) genuinely does recur.

Idempotency at every stage the document passes through

Classify stage: idempotent — re-running classification on the same document yields the same result,
  safely overwritable.
Extract stage: writes are keyed by (job_id, stage, page_number) with a uniqueness constraint,
  so a retried extraction can't produce duplicate page results.
Downstream publish: consumers must ALSO be idempotent against redelivery, per this series'
  Event-Driven Architecture guide.

Idempotency needs to be enforced at every hop, not just at ingestion — each stage's write should have a database or storage-level constraint preventing a duplicate result for the same (job_id, stage) from ever being written twice, since at pipeline scale "we'll just be extra careful about retries" is not an acceptable substitute for structural, enforced guarantees at every layer.

5. The Multi-Stage Extraction Pipeline

Classification: routing each document (and each page) down the cheapest viable path

public ExtractionMethod ClassifyPage(PdfPage page)
{
    if (page.HasEmbeddedTextLayer() && page.TextLayerCoversPage())
        return ExtractionMethod.DirectTextExtraction; // fast, cheap, near-perfect accuracy
    if (page.IsPrimarilyImage())
        return ExtractionMethod.Ocr;                  // slower, probabilistic (Section 8)
    return ExtractionMethod.HybridTextAndOcr;          // some pages mix both within one document
}

As covered in this series' Data Pipeline guide's routing patterns, classifying each page before extracting it — rather than running every page through the same, most-expensive path "just in case" — is a direct application of the "know which path is hot, optimize for it" discipline echoed throughout this series' Caching guide: the large majority of born-digital pages should never touch OCR at all.

Text and layout extraction

Direct text extraction pulls the embedded text layer along with its POSITIONING —
  layout (columns, headers, reading order) matters as much as the raw characters for
  correctly reconstructing a document's actual structure, not just its words.

A PDF's text layer, extracted naively, often loses reading order (columns interleaved incorrectly, headers/footers mixed into body text) — per this series' Document Parsing guide's discussion, layout-aware extraction uses each text element's bounding box and font metadata to reconstruct genuine reading order and structural elements (headings, paragraphs, lists) rather than a flat, unordered stream of characters.

Table extraction as its own specialized sub-stage

Tables require detecting the table's boundaries, then its row/column grid, THEN
  extracting cell content correctly aligned to that grid — a meaningfully different
  problem from general text extraction, usually handled by a dedicated model or library.

Tables are common enough, and different enough from prose extraction, to warrant their own dedicated sub-stage (per this series' Specialized Model Integration guide) — detecting table regions, inferring the grid structure, then extracting and aligning cell content, since naively running general text extraction over a table region reliably scrambles row/column alignment.

Structuring and validating output against an expected schema

public ValidationResult Validate(ExtractedDocument doc, JsonSchema expectedSchema)
{
    var result = _schemaValidator.Validate(doc.StructuredOutput, expectedSchema);
    // required fields present? types correct? plausible value ranges (per this series' Data Quality guide)?
    return result;
}

The final pipeline stage validates extracted, structured output against an expected schema — per this series' Data Quality and Contract Testing guides, this catches both extraction errors (a date field that didn't parse) and document-level surprises (an entirely unexpected document type routed into the wrong pipeline) before output reaches downstream consumers, feeding low-confidence or failed validations into human review (Section 9).

6. The Document Processing State Machine

An explicit, enumerable set of states and legal transitions

Uploaded → Classified → Extracting → ExtractionComplete → (Reviewed if flagged)
                              ↓
                          Failed

As covered in Section 2's DocumentJob aggregate, a document's processing lifecycle is a small, explicit state machine — and the aggregate's own methods are what enforce that only legal transitions are ever possible, throwing rather than silently succeeding if called out of order (completing extraction on a job that was never marked as extracting, for instance).

Why an explicit state machine matters more here than for most pipelines

Given this guide's emphasis on retries and worker crashes being routine (Section 4), having every legal and illegal state transition explicitly enumerated and enforced by the aggregate itself — rather than scattered conditional checks across worker code — is precisely the kind of rigor this series' DDD guide argues pays for itself most clearly in domains with genuinely high retry and concurrency rates, and pipeline processing fits that description closely.

Resumability: picking up a job exactly where it left off

public async Task ResumeAsync(JobId jobId)
{
    var job = await _repository.GetByIdAsync(jobId);
    var completedStages = await _jobLog.GetCompletedStagesAsync(jobId); // per Section 3's replayable log
    var nextStage = _pipeline.GetNextStage(completedStages);
    await nextStage.ExecuteAsync(job);
}

Because every stage transition is durably logged (Section 3), a worker crash mid-pipeline doesn't require restarting a document from scratch — resuming means reading the log to determine the last completed stage and continuing from there, which matters considerably at scale, since re-running an expensive OCR stage unnecessarily on every transient worker restart would waste substantial compute.

7. Isolating Untrusted Input: Sandboxing and Parser Security

Why PDF parsing is a genuine attack surface, not a theoretical concern

The PDF spec permits embedded JavaScript, forms, and deeply nested/recursive object structures.
Parser vulnerabilities (buffer overflows, XML entity expansion in embedded metadata,
  decompression bombs in embedded streams) are a recurring, real vulnerability class
  across PDF libraries, not a hypothetical one.

As covered in this series' OWASP Top 10 and Secure File Handling guides, a PDF processing pipeline that accepts uploads from external or semi-trusted sources must treat every file as potentially adversarial — this isn't a theoretical concern specific to this guide's caution; it reflects a genuine, recurring vulnerability class across PDF-parsing libraries that a production pipeline needs to design around structurally, not patch reactively.

Sandboxed, resource-bounded parsing

Every parsing operation runs in an isolated sandbox (a container or a dedicated,
  network-isolated worker process) with hard CPU, memory, and wall-clock time limits —
  per this series' Container Security guide's isolation discipline, a malformed or
  adversarial PDF can crash or exhaust ITS sandbox without affecting any other job.

Per this series' Container Security and Resource Isolation guides, running the extraction stage's parsing logic in a tightly sandboxed, resource-bounded environment — separate from the orchestration and storage layers, with no outbound network access and strict memory/CPU/time ceilings — contains the blast radius of a malicious or pathological file to that one job, rather than risking the whole pipeline's stability on the correctness of a third-party parsing library against arbitrary untrusted input.

Explicit handling of embedded active content

if (document.ContainsEmbeddedJavaScript() || document.ContainsEmbeddedFiles())
{
    // per this series' Secure File Handling guide: strip or explicitly flag, never execute
    document.StripActiveContent();
}

Embedded JavaScript and embedded files are legitimate PDF features with legitimate uses, but a processing pipeline has no reason to ever execute them — per this series' Secure File Handling guide, active content is stripped or explicitly flagged before any further processing, never executed, regardless of how the file claims to want it used.

8. OCR and the Confidence Problem

Why OCR output is fundamentally probabilistic, unlike a text layer

Direct text extraction from a text layer: exact, deterministic — the characters
  ARE the characters the document contains.
OCR on a scanned image: a MODEL'S BEST GUESS at what characters are present,
  with an associated confidence score that varies by character, word, and region.

This distinction, more than almost any other design choice in this guide, is what makes PDF processing genuinely different from a typical ETL pipeline — per this series' Machine Learning Systems guide's discussion of probabilistic outputs, OCR results must always carry their confidence score downstream alongside the extracted text, never presented with the same certainty as a direct text-layer extraction.

Per-field and per-region confidence, not just a whole-document score

public record OcrResult(string Text, double Confidence, BoundingBox Region);
// a single low-confidence region (a smudged signature field) shouldn't hide behind
// an otherwise-high overall document confidence average

A single document-level confidence score can mask a genuinely important, localized problem — per this series' Data Quality guide's granularity discussion, tracking confidence per extracted field or region (not just averaged across the whole document) is what lets Section 9's review routing target the specific part of a document that actually needs human eyes, rather than sending an entire otherwise-clean document to review over one unclear field.

Image preprocessing to improve OCR accuracy before it ever runs

Deskewing, contrast normalization, and noise reduction applied to scanned pages
  BEFORE OCR, per this series' Image Processing guide, measurably improve OCR
  accuracy on real-world scanned documents (crooked scans, poor lighting, low resolution).

As covered in this series' Image Processing guide, a modest preprocessing stage — deskewing, contrast normalization, denoising — ahead of the OCR model itself is a low-cost, high-leverage step that measurably reduces the volume of low-confidence extractions reaching Section 9's review queue, since a meaningful fraction of OCR errors trace back to scan quality rather than the OCR model itself.

9. Human-in-the-Loop Review

Routing on confidence, not on document type alone

A document is routed to review when ANY extracted field's confidence falls below
  threshold — not just when the document TYPE is generally known to be error-prone.
Routing by type alone either over-sends (wasting reviewer time on clean documents
  of a "risky" type) or under-sends (missing a genuinely low-confidence field in a
  normally-reliable type).

Per this series' Workflow Engine guide's routing patterns, review routing driven by Section 8's actual per-field confidence — rather than a coarse document-type heuristic — targets reviewer attention at the specific extractions that actually need it, which matters for the same reason Section 10's precision/recall trade-off matters in a surveillance system: reviewer time is a limited, expensive resource that a poorly-targeted routing rule wastes.

The review interface surfaces exactly what's uncertain, not the whole document

public record ReviewTask(JobId JobId, IReadOnlyList<FlaggedField> FieldsNeedingReview, string DocumentPreviewUri);

A reviewer's task is scoped to the specific low-confidence fields (with the surrounding document shown for context, per this series' UX-for-review-workflows discussion), not a request to re-verify an entire document from scratch — this keeps review throughput high and lets the same reviewer capacity cover meaningfully more documents than a "review everything" policy would allow.

Reviewer corrections feed back into the pipeline, not just the single job

A reviewer's correction resolves THIS document's job — but aggregated corrections,
  over time, are exactly the labeled data that improves the underlying extraction
  model (per this series' MLOps guide's feedback loop discussion), the same way
  analyst dispositions feed back into detection tuning in a surveillance system.

As covered in this series' MLOps guide, reviewer corrections are valuable training signal beyond resolving the individual job — systematically capturing them (with appropriate consent/data handling per Section 11) closes the loop between human review and model improvement, gradually reducing the fraction of documents that need review in the first place.

10. Handling Failure, Retries, and Poison Documents

Retrying transient failures without retrying deterministic ones forever

if (exception is TransientStorageException)
{
    await _retryPolicy.RetryWithBackoffAsync(job); // per this series' Resilience/Polly guide
}
else if (exception is MalformedDocumentException)
{
    await MoveToDeadLetterAsync(job); // retrying won't help — the document itself is the problem
}

Per this series' Resilience guide's distinction between transient and deterministic failures, a storage timeout genuinely warrants a retry with backoff, but a parsing failure caused by the document itself (Section 7's malformed or adversarial input) will fail identically on every retry — routing these differently prevents a poison document from being retried indefinitely and wasting worker capacity that healthy jobs need.

Dead-lettering and quarantine for documents that can't be safely processed

A document that repeatedly fails parsing, or is flagged by sandboxing (Section 7) as
  structurally suspicious, is moved to a QUARANTINE store — not silently dropped,
  not endlessly retried — with enough metadata for a human to decide what to do with it.

As covered in this series' Dead Letter Queue pattern discussion, a document that can't be safely or successfully processed after bounded retries is moved to an explicit quarantine state, preserving the original bytes and the failure history, rather than either silently discarding it (losing a potentially important document) or endlessly retrying it (wasting capacity indefinitely on a job that will never succeed).

Circuit breaking around a specific failing extraction dependency

If the OCR service itself is degraded or unavailable, a circuit breaker (per this
  series' Resilience guide) stops routing new jobs to it and instead queues them,
  rather than every in-flight job piling up retries against a service that's currently down.

Per this series' Resilience guide, wrapping calls to any external or specialized extraction dependency (an OCR service, a table-extraction model) in a circuit breaker prevents a degraded dependency from cascading into pipeline-wide backlog growth — jobs needing that specific stage queue cleanly and resume once the dependency recovers, rather than every worker independently retrying against a service that's already struggling.

11. Data Security and Compliance

Documents frequently contain sensitive data the pipeline itself never asked for

Uploaded documents — invoices, contracts, medical forms, ID scans — routinely contain personally identifiable or otherwise sensitive information incidental to the document's stated purpose. The practical strategy mirrors this series' Secret Management and Data Privacy guides' least-privilege and data-minimization principles: access to raw documents and extracted output is scoped narrowly, sensitive extracted fields (SSNs, account numbers) are handled per this series' Data Classification guide's tagging and access-control patterns, and retention of raw documents is bounded deliberately rather than kept indefinitely "just in case."

Encryption at rest and in transit, without exception

// Blob storage configured for encryption at rest, per this series' Secret Management guide;
// TLS enforced on every internal pipeline hop, not just the external upload endpoint

Every principle covered in this series' Secret Management and Transport Security guides applies directly here: documents and extracted output encrypted at rest, TLS enforced on every hop including internal pipeline-to-pipeline calls, and access credentials for storage and extraction services rotated and never hardcoded.

Audit logging of access, distinct from pipeline processing logs

logger.LogInformation("Document {DocumentId} accessed by {UserId} for {Purpose}", documentId, userId, purpose);

As covered in this series' Structured Logging and OWASP Top 10 guides, every access to a stored document or its extracted output needs to be logged with enough context (who, what, when, why) to support both regulatory audit requirements and forensic investigation after an incident — kept as a distinct audit trail from the pipeline's own operational logs (Section 14), since the two serve genuinely different purposes and different retention requirements.

12. Consistency, Availability, and the CAP Trade-off for a Pipeline

Why the write path (job state, extraction results) favors consistency, while throughput elsewhere doesn't have to

As covered in this series' System Design guide's CAP theorem discussion, the pipeline's job state — what stage a document is in, what its extraction results are — needs strong consistency within a job: two workers racing to claim and process the same job concurrently, per Section 4, is exactly the kind of duplicate-effect bug idempotency exists to prevent. But the pipeline as a whole can, and should, tolerate individual jobs failing or queueing under load rather than the system attempting perfect, uninterrupted availability for every job at every moment.

Where eventual consistency is deliberately, explicitly scoped in

Job state transitions (Section 6) → strong consistency required within a job, no compromise
A downstream "documents processed today" DASHBOARD → eventual consistency, a few seconds, is fine
Search indexing of extracted text → eventually consistent, updated asynchronously from the log

Not every part of the system needs the same bar — job state transitions do, but downstream, read-only projections (dashboards, search indexes over extracted content) can and should tolerate the eventual consistency this series' Event-Driven Architecture and CQRS discussions describe generally, since a search index being briefly stale carries none of the risk a duplicated or corrupted extraction result does.

13. Scaling the System

Applying this series' System Design guide's building blocks, with pipeline-specific emphasis

Worker pool scaling (per this series' Background Services guide): extraction workers scale
  horizontally and independently per stage — OCR workers (CPU/GPU-heavy) scale separately
  from text-extraction workers (lightweight), matching resource profile to workload
Queue-based decoupling (per this series' RabbitMQ/Kafka guides): each pipeline stage reads
  from its own queue, so a slow OCR stage backing up doesn't block classification or
  text extraction from proceeding on other documents
Blob storage for documents (per this series' Object Storage guide): scales independently
  of compute; large or high-page-count documents don't strain the same storage tier as
  small ones differently, since blob storage scales roughly uniformly per object

Every technique from this series' System Design guide applies here, with the caveat that each one needs to be evaluated against this guide's stage-specific resource profiles (Section 5) before being applied — OCR and table extraction are meaningfully more expensive than direct text extraction, and scaling them identically wastes capacity on the cheap path or starves the expensive one.

Batching for throughput on genuinely expensive stages

OCR and ML-based table extraction benefit substantially from batched inference
  (per this series' ML Systems guide's batching discussion) — grouping several pages'
  worth of work into one model invocation amortizes fixed inference overhead, at the
  cost of a small, bounded increase in per-document latency.

Per this series' ML Systems guide, batching inference calls on the pipeline's most expensive stages meaningfully improves throughput per unit of compute, trading a small amount of added latency (waiting briefly to fill a batch) for substantially better resource utilization at scale — a trade-off well worth making for stages that aren't on a synchronous, user-waiting critical path.

14. Observability for a Document Processing Pipeline

Every guide in this series' observability trio, applied with pipeline-specific stakes

Structured logs (per this series' Structured Logging guide): every stage transition,
  every failure and retry, every quarantine decision — with document and job IDs for correlation
Distributed tracing (per this series' Distributed Tracing guide): tracing a single document's
  journey from upload through every stage to final output — essential for diagnosing why
  a specific document is stuck, slow, or produced unexpected output
Metrics (per this series' Prometheus/Grafana guide): per-stage throughput and latency,
  OCR/extraction confidence distribution over time, review queue depth (Section 9),
  quarantine rate (Section 10) — the aggregate health signals an operations team watches continuously

Every technique from this series' observability guides applies directly, with one pipeline-specific addition worth stating explicitly: the confidence distribution of extractions over time (Section 8) is itself a critical health metric here — a gradual drift toward lower average confidence often signals a genuine problem (a new, poorly-supported document format appearing in the input mix, a preprocessing regression) well before it shows up as an obvious failure spike.

Alerting on pipeline-health and extraction-quality symptoms

# Per this series' Prometheus/Grafana guide's symptom-based alerting principle
rate(pipeline_stage_failures_total{stage="ocr"}[5m]) / rate(pipeline_stage_attempts_total{stage="ocr"}[5m]) > 0.10

A sudden spike in a specific stage's failure rate, a growing review queue backlog, or a rising quarantine rate are exactly the kind of user-facing (or reviewer-facing) symptoms this series' Prometheus/Grafana guide argues alerts should be built around — per-stage granularity matters here specifically because an aggregate, pipeline-wide success rate can easily hide one struggling stage (say, a specific document type breaking table extraction) until backlog has already grown substantially.

15. Common Pitfalls

Quick Reference Table

Conclusion

A PDF processing pipeline takes every general system design technique covered throughout this series and applies it to an input format that is fundamentally less trustworthy and more heterogeneous than most systems are designed to assume — because a meaningful fraction of real-world PDFs are malformed, adversarial, or simply require a completely different processing path than the "clean" case a naive design would optimize for. The design that actually holds up under that reality rests on a small number of non-negotiable foundations: an immutable, content-addressed document store and replayable job log; idempotency enforced at every stage a document passes through; classification-first routing that sends each page down the cheapest viable path; sandboxed parsing that treats every input as potentially adversarial; and honest, per-field confidence tracking that routes genuine uncertainty to a human rather than propagating a wrong answer with false confidence.

Nearly every architectural pattern covered elsewhere in this series shows up here in service of that bar — DDD's aggregates enforcing a resumable job lifecycle, Event-Driven Architecture's idempotent, queue-decoupled stages, Container Security's isolation discipline applied to untrusted parsing, MLOps' feedback loop between human review and model improvement, and the full observability trio watching over both pipeline health and extraction quality itself. PDF processing is, in that sense, less a distinct discipline from everything else in this series than the place where its cumulative lessons about immutability, idempotency, defensive input handling, and honest uncertainty matter more visibly, and more unforgivingly, than almost anywhere else.

Found this useful? Feel free to star the repo, open an issue with corrections, or share the malformed-PDF-that-crashed-a-worker story that turned out to matter far more than a clean extraction ever should.