Document Ingestion and Parsing Pipelines

An annual report puts its segment results in a table that breaks across two pages. A naive text extractor emits the row labels as one run of lines, the numbers as another, and the continuation page's header in the middle of both. Every downstream stage works perfectly on this output: the chunker splits it cleanly, the embedder encodes it faithfully, the retriever returns it for the right query. The generator then reports an operating margin that belongs to a different segment, and no metric in the retrieval stack flags anything.

Ingestion is the stage that decides what text exists. Retrieval quality is bounded by it, and its errors are silent because the output is still text.

Why PDF is hard

A PDF is a program for placing glyphs at coordinates on a page. It carries no paragraphs, no reading order, no notion that two runs of text form a table cell, and for a scanned page no text at all, only an image. Everything a language model needs, headings, lists, column order, table structure, captions, has to be reconstructed.

Reconstruction breaks into sub-problems. Layout analysis finds regions and labels them: text, title, table, figure, header, footer. Text recognition reads the embedded text layer where one exists and runs OCR where it does not. Table structure recognition recovers rows, columns and spanning headers. Reading order serialises the regions so two columns are not interleaved. Each is a learned model or a heuristic with its own error rate, and errors multiply along the chain.

Two architectures

The pipeline approach runs a specialised model per sub-problem. Docling, for instance, combines a layout model trained on DocLayNet with the TableFormer structure model and runs efficiently on commodity hardware (Auer et al., 2024, Docling Technical Report, arXiv:2408.09869). DocLayNet itself is 80,863 manually annotated pages across 11 region classes, drawn from diverse sources such as financial reports, manuals and patents, because models trained only on scientific articles degrade on anything else (Pfitzmann et al., 2022, DocLayNet, KDD 2022, arXiv:2206.01062).

The end-to-end approach gives a vision-language model the page image and asks for markdown. Nougat did this for academic papers, including LaTeX for equations (Blecher et al., 2023, Nougat, arXiv:2308.13418). olmOCR fine-tuned a 7B VLM on 260,000 pages and reported converting a million pages for about $176, against more than $6,240 for the same volume through GPT-4o (Poznanski et al., 2025, olmOCR, arXiv:2502.18443). At a 5-million-page corpus that is roughly $880 versus more than $31,000, which changes whether a full reparse is a routine operation or a budget request.

Which one wins, and where

Parsers are scored against ground-truth markdown, commonly with normalised edit distance

where \(\mathrm{Lev}\) is the Levenshtein distance between the predicted text \(\hat{y}\) and the reference \(y\), so 0 is perfect and 1 is unrelated.

The evidence does not support either camp unconditionally. In the first release of OmniDocBench, pipeline tools led overall: MinerU scored 0.058 text edit distance on English pages against 0.144 for GPT-4o. General VLMs generalised better on slides and handwritten notes, where the pipeline's layout model misread handwriting as figures, but also hallucinated text on rotated pages and merged multi-column layouts (Ouyang et al., 2024, OmniDocBench, CVPR 2025, arXiv:2412.07626). Newer VLM parsers claim to have closed the gap, and leaderboard positions have moved with each model release, so any comparison is dated the day it is run.

The two families fail differently, and that matters more than the averages. A pipeline that misses a table loses content visibly: a gap, a garbled row. A VLM that misreads one produces fluent, plausible markdown with a wrong number in it. The first is caught by a reviewer skimming output; the second is not. A different route skips parsing entirely by retrieving over page images, as in visual document retrieval, at the price of losing text you can filter, cite or diff.

Incremental re-ingestion

A corpus changes daily, and parsing everything nightly does not scale. The workable design keys every derived artefact by what produced it:

A document is reprocessed only when its content hash or one of the version components changes, and only from the first stage whose key changed. Storing the parsed markdown as a durable intermediate is what makes that possible: an embedding-model upgrade re-embeds existing chunks without reparsing, and a chunking change re-chunks without OCR. Deletions need explicit tombstones, because a document that disappears from the source generates no event, and its chunks otherwise stay retrievable indefinitely.

When it breaks

Empty text layers pass validation. Some PDFs carry an invisible text layer of garbage from an earlier OCR pass, or none at all. A pipeline that trusts the text layer returns nothing or nonsense with no error. Checking extracted characters per page against page area catches most of these.

Boilerplate pollutes every chunk. Running headers, footers and page numbers repeated on 300 pages become 300 near-duplicate retrieval candidates unless layout analysis removes them.

Digits are where OCR costs most. A misread letter rarely changes meaning; a misread digit in a financial table does, and edit distance weights them equally. Domain audits should sample numeric cells specifically.

Per-document routing beats one parser. Born-digital pages with simple layouts extract cheaply from the text layer; scans, forms and dense tables justify a model. Routing by page type keeps cost near the cheap path while spending on the pages that need it, and it requires a classifier that is itself one more stage to evaluate.