I Searched 'attention' in Attention Is All You Need. The PDF Gave Me 23 Hits, Not 91

I run a visa agency in Bali. The other half of my week goes into Apify Actors, and one of them is a document reader: give it PDF, Word, Excel, PowerPoint, EPUB, HTML or CSV files and it returns one flat row per document with the Markdown, the plain text, the tables, the metadata and — if you ask — retrieval chunks that remember which heading they came from.

Boring category. Everyone has a PDF-to-text tool. I built mine because I needed every format to come back with the same field names, and I kept it because of one bug I did not expect to find: the single most-read PDF in machine learning extracts as gibberish, and nothing about the output looks broken.

The measurement

On 19 October 2026 I read arxiv.org/pdf/1706.03762 — Attention Is All You Need, 15 pages, produced by pdfTeX-1.40.25 — with pdfplumber's word grouping at the default word spacing, and then at a tighter one. Same file, same library, same page loop. Two tolerances.

                   default (x_tolerance 3.0)   tight (2.0)
words, 15 pages                      1,923          6,002
characters                          34,260         38,339
lines carrying a glued token      337 of 609      2 of 609

Three point one times the words. Here is what that looks like on page 1, same eight lines, both readings:

default:
  Abstract
  Thedominantsequencetransductionmodelsarebasedoncomplexrecurrentor
  convolutionalneuralnetworksthatincludeanencoderandadecoder. Thebest
  performing models also connect the encoder and decoder through an attention
  mechanism. We propose a new simple network architecture, the Transformer,
  basedsolelyonattentionmechanisms,dispensingwithrecurrenceandconvolutions
  entirely. Experiments on two machine translation tasks show these models to
  besuperiorinqualitywhilebeingmoreparallelizableandrequiringsignificantly

tight:
  Abstract
  The dominant sequence transduction models are based on complex recurrent or
  convolutional neural networks that include an encoder and a decoder. The best
  performing models also connect the encoder and decoder through an attention
  mechanism. We propose a new simple network architecture, the Transformer,
  based solely on attention mechanisms, dispensing with recurrence and convolutions
  entirely. Experiments on two machine translation tasks show these models to
  be superior in quality while being more parallelizable and requiring significantly

Look at which lines survived. Lines 4, 5 and 7 are perfect in both readings. Lines 2, 3, 6 and 8 are one word each. That is the detail that makes this dangerous: it is not a broken file, not a broken page, not a broken paragraph. It is per line, interleaved, inside one abstract. Twenty-four of page 1's forty-four lines are affected and twenty are clean. Any eyeball check, any "does this look like English" assertion, any first-200-characters log line will pass.

The cause is in the file, not the parser. TeX justifies a line by shrinking the inter-word space, and on a tightly set line the gap between two words becomes smaller than the gap a letter-spacing table says a space must be. The extractor is doing exactly what it was told: no space wider than threshold, so no word break. The longest token I got out of the acknowledgements section is 101 characters:

implementingtensor2tensor,replacingourearliercodebase,greatlyimprovingresultsandmassivelyaccelerating

Why this is worse than it looks

Glued text is not "slightly messy text". It is text with the word boundaries deleted, and every downstream tool you own is built on word boundaries. Full-document whole-word search over the two readings:

A keyword index built on the default reading finds a quarter of the mentions of the word in the title. It finds positional zero times in the paper that introduced positional encoding. Not an error — a 0 that looks like a legitimate answer.

Embeddings degrade the same way, less visibly. basedsolelyonattentionmechanisms is a token sequence no sentence in your corpus shares, so the chunk containing it lands somewhere unrelated in vector space. You do not get an error; you get recall that is quietly worse than your eval set suggests, because your eval set was probably written from the clean version of the document.

And then the part I enjoyed:

cl100k_base tokens, default reading: 10,344
cl100k_base tokens, tight reading:    9,685

The broken text has 11% fewer characters and 7% more tokens. Deleting the spaces makes the file smaller and the bill bigger, because the tokenizer has to chop Thedominantsequencetransduction into word fragments it has no clean entry for. You pay a premium for the version a model cannot read.

The fix, and why it is a per-page retry

The obvious move — always use the tighter spacing — is wrong, and I can show it with the counter-example. IRS Form W-9, produced by Adobe Designer 6.5, first six pages:

words at default: 6,278
words at tight:   6,278
page 1 output:    byte-identical

A normally set document does not care. But documents with genuinely wide tracking do: lower the threshold far enough on display type or a letter-spaced heading and you start splitting single words into pieces, which breaks search in the opposite direction. There is no global tolerance that is right for every PDF, because the correct value is a property of how that page was typeset.

So the Actor measures instead of guessing. Per page:

def glue_ratio(text: str) -> float:
    """Share of tokens long enough to be several words run together."""
    tokens = text.split()
    if not tokens:
        return 0.0
    return sum(1 for token in tokens if len(token) > 25) / len(tokens)

If a page's glue ratio clears 5%, the page is read a second time at the tighter spacing, and whichever reading has fewer run-together tokens wins. Page 1 of the Transformer paper scores 0.200 by default and 0.002 tight, so the retry wins. Page 1 of the W-9 scores 0.000, the retry never runs, and nothing is risked. Twenty-five characters is a deliberately conservative threshold: real English words that long are rare enough that a legitimate Donaudampfschiffahrtsgesellschaft cannot move a page's ratio past 5% on its own.

This is not one unlucky paper. Both other LaTeX documents I checked, first ten pages each:

If your RAG pipeline ingests arXiv, this is not an edge case. It is your corpus.

What a run looks like

Pages 1–3 of the same paper, through Document Text Extractor:

{
  "urls": ["https://arxiv.org/pdf/1706.03762"],
  "includeMarkdown": true,
  "chunkForRag": true,
  "pageRange": "1-3"
}

{
  "fileName": "1706.03762",
  "format": "pdf",
  "fileSizeBytes": 2215244,
  "pageCount": 15,
  "pagesProcessed": 3,
  "title": null,
  "author": null,
  "creator": "LaTeX with hyperref",
  "producer": "pdfTeX-1.40.25",
  "wordCount": 1330,
  "charCount": 8929,
  "chunkCount": 11,
  "tableCount": 0,
  "ocrPageCount": 0,
  "status": "ok",
  "warnings": []
}

From the API, with the page range doing the cost control — the pricing is $0.0035 per document plus $0.00015 per page, so pageRange is the knob that matters on long files:

curl -X POST "https://api.apify.com/v2/acts/lergassy~document-text-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"urls":["https://arxiv.org/pdf/1706.03762"],"chunkForRag":true,"pageRange":"1-3"}'

Three things in that output are worth more than the word count.

title and author are null. The paper is called Attention Is All You Need; the PDF's metadata Title field is an empty string, and so is Author. pdfTeX writes those only if the document asked it to, and most papers do not. The title exists on page 1 as the largest font on the page and nowhere else, which is why PDF headings have to be inferred from font size relative to body text — there is no structure in the file to read. Do not build a library index on PDF metadata titles.

tableCount is 0 on pages that contain a table. The Transformer paper's tables are drawn with TeX rules rather than as ruled grids, and pdfplumber's table detector wants lines. Word, Excel, PowerPoint and CSV tables come back exact because the structure is in the file; PDF table detection is good and not perfect, and the honest advice is to check tableCount against what you see rather than assume.

Chunk charCount is not len(text). With the default 120-character overlap, chunk 2 reports charCount: 1187 and carries 1,309 characters — the overlap prefix is prepended to text but not counted in charCount, which measures the chunk's own body. Useful if you want to know how much new material a chunk adds; wrong if you are budgeting a context window. Use the length of the string for that.

Chunk 1 of that run is also a 134-character fragment of the author block — names and Gmail addresses — filed under headingPath: ["Attention Is All You Need", "Abstract"]. Title pages are the worst case for font-size heading inference: author lines, affiliations and footnote markers all sit at sizes between the title and the body. It is a real chunk with a wrong label, and on a title page I have not found a heuristic that beats it.

When not to use this

If every document you handle is a scan, the per-page OCR fee ($0.01) is the whole bill and a dedicated OCR service may price better at volume. If you need signature fields, form field values or redaction awareness, this returns text and tables, not form semantics. If your files live behind a login or in Google Drive share links, fetch them yourself and pass base64 — a Drive preview URL returns HTML, and you will get a corrupt-file row for it. I confirmed that one by accident today: a documentation PDF link I grabbed served an HTML error page with a .pdf name, and the parser correctly refused to call it a PDF.

And if you only read one family of documents, measure your own glue ratio before you trust anyone's word count — including mine. The check is four lines of Python, and on a LaTeX corpus it is the difference between 1,923 words and 6,002.