Users report the assistant gets numbers wrong when answering from documents containing tables. Diagnose.
Show the full answer Hide the answer
What the interviewer is testing
Whether you recognise that ingestion decisions bound retrieval quality, and that no downstream component recovers destroyed information.
The diagnosis
Fixed-size chunking split the table. A chunk containing rows without the header row has numbers with no column meaning. The model receives "4,200 | 3,100 | 12%" with no indication of what those figures represent, and produces a plausible attribution — which is wrong.
Related failures: a table split mid-row so figures are orphaned; a table separated from the caption that gives it context; and merged cells or multi-level headers flattened into ambiguity.
No embedding model, reranker or prompt recovers information that was destroyed at ingestion.
The fix
Structure-aware extraction. Detect tables during ingestion and handle them as units — never split a table across chunks, and if a table exceeds the chunk size, repeat the header rows in each part.
Serialise tables in a form the model reads reliably, such as markdown, with the caption and surrounding context attached.
Contextual enrichment: prepend each chunk with document title, section path and a short description of what the document is. A chunk reading "the limit is 40 hours" is useless alone and usable as "Employment Policy > Overtime > Weekly limits".
Retrieve small, expand to the parent section before sending to the model — accurate matching with sufficient context.
The broader point
Chunking is the least glamorous part of a RAG system and the one that most determines its ceiling. Teams optimise embeddings and rerankers while leaving a fixed 512-character split in place.
What a strong answer adds
For numerically critical content, considering whether retrieval is the right mechanism at all. Structured data belongs in a database queried deterministically, with the model generating the query rather than reading the numbers from prose. A model reading figures out of a retrieved table will occasionally misread them, and for financial or regulatory figures that is not acceptable at any error rate.
Common weak answers
Increasing the chunk size, which reduces the frequency without fixing the cause. Switching embedding models.