Documentation That Survives an Audit
The properties that separate documentation an assessor accepts from documentation they discount, why generated beats written, and how to structure a technical file so it stays true as the system changes.
Two organisations produce technical files of similar length for similar systems. One is accepted; the other prompts a request for evidence that the described process actually operated. The difference is a small number of structural properties, and they are cheaper to build in than to retrofit.
The properties
Traceable to a system version. A document describing "the model" is describing whichever version the author had in mind. A document naming the model version, dataset version and code revision it describes can be checked against the artefact, and a reader can tell whether it is current.
Generated where generation is possible. Evaluation results, component versions, data lineage and configuration should be emitted by the pipeline, not transcribed. Transcription introduces error and, more importantly, breaks the link between the document and the system, which is exactly what an assessor is probing.
Dated and attributed. Each section carries who produced it and when. A section written eighteen months ago is not thereby wrong, and it is a different kind of claim from one regenerated last week, and the reader is entitled to know which.
Consistent with other artefacts. The technical file, the model card, the registry entry and the monitoring dashboard should agree. Inconsistency between them is the most common finding in a technical review, because the documents are maintained by different people on different cycles.
Honest about gaps. A section saying "this was not evaluated, because X" is stronger than a section that omits the question. Omission reads as an oversight; a stated gap reads as a decision, and it can be assessed as one.
Structure
The arrangement that ages well separates the stable from the volatile. A stable core describes purpose, scope, architecture, intended use, risk assessment and design decisions, and changes rarely. Generated appendices carry evaluation results, versions, configuration and metrics, and are regenerated on each release. A change log records material changes and their review.
This means a release regenerates the appendices automatically and touches the core only when the design changed, which is the only arrangement under which documentation keeps pace with a system that ships weekly.
When it breaks
Documentation is owned by whoever has time. Without a named owner per document and a trigger for review, everything drifts at the rate of the fastest-changing component. Ownership is the control; the format is not.
The audit asks about the process, not the document. An assessor's follow-up is usually "show me the last three times this review ran". A perfect document describing a process that runs once is a description of an intention, and the records of its operation are what substantiate it.
Volume substitutes for substance. A four-hundred-page file is not more credible than a forty-page one and is harder to keep true. Length correlates with retrospective assembly, and assessors read it that way.
Translation and accessibility are afterthoughts. Documentation intended for affected people, as opposed to for assessors, has to be readable by them: plain language, the right languages, and available where the decision is communicated rather than on a policy page nobody visits.
12 flashcards for this concept
Click a card to reveal the answer.