System Cards and Documenting the Whole Pipeline

A model card describes a model. Users interact with a system: a model plus retrieval, prompts, filters, tools, thresholds and fallbacks, several of which change more often than the model does. Documenting only the model describes the component least responsible for what users experience.

What a system card adds

Architecture. What components exist and how a request flows through them, including which model versions, what retrieval sources, what safety filters, and what happens on failure. A reader should be able to locate where a given behaviour originates.

Composed behaviour. How the parts interact, particularly where one compensates for another. A filter catching outputs a model produces is a different system from a model that does not produce them, and the distinction matters for what happens if the filter is removed or bypassed.

Evaluation of the system, not the components. End-to-end results on realistic inputs, including the retrieval-plus-generation path rather than each in isolation. Component metrics can all look good while the composition fails.

Human oversight. Where people are in the loop, what they see, what authority they have, and what the expected review rate is. This is the section regulators ask about and the one most often written aspirationally.

Operational envelope. Rate limits, expected load, latency targets, and what degrades first under stress. Documented degradation behaviour is what allows a downstream consumer to plan for it.

Known failure modes with examples. Concrete observed failures, not generic caveats.

Keeping it current

The reason system documentation goes stale is that it is a separate artefact from the system. The techniques that work all reduce that separation.

Generate what can be generated: component versions, evaluation results and configuration come from the registry and the evaluation pipeline rather than being transcribed. Store the prose next to the code so it is reviewed in the same change. Version the document with the system so a reader can retrieve the description that matches a given release. And trigger review on material change rather than on a calendar, since a quarterly review of a weekly-changing system is always describing something that no longer exists.

When it breaks

Documentation lags fastest where change is fastest. Prompts and retrieval configuration change constantly and are the least documented parts of most systems, so the description is most wrong about the components that most determine behaviour.

Internal and external versions diverge. An externally published card is necessarily less detailed, and maintaining two documents means one is stale. Deriving the public version from the internal one, mechanically, avoids the divergence.

Aspirational human oversight is a documented control that does not exist. Writing that a human reviews outputs, when the review rate is two percent and the reviewer has no authority to override, records a control the system does not have. That is worse than recording no control, because it is relied upon.

Third-party components are opaque. A system built on a hosted model can document how it uses it and cannot document how it works. Stating the boundary explicitly, and what is therefore unknown, is more useful than describing the component vaguely.