Audit Logs as Assurance Evidence

An auditor arrives nine months after an incident and asks what the system did. Every answer available is a reconstruction, and the quality of that reconstruction was fixed long before the question was asked, by whoever decided which fields to write and how long to keep them. Record-keeping is the one assurance control that cannot be retrofitted.

European law puts this in the statute rather than in guidance. Article 12 requires high-risk systems to technically allow for the automatic recording of events over their lifetime, and Article 19 requires providers to keep those logs for a period appropriate to the intended purpose and at least six months, unless other Union or national law provides otherwise, to the extent the logs are under their control (EU AI Act, Article 19). Deployers carry a matching retention duty. Notably, the regulation sets a floor and does not specify a format, a schema, or a maximum, which means the engineering question is wide open and the compliance question is narrow.

Four properties that make a log evidence

Telemetry is written to help an on-call engineer in the next hour. Evidence is written to withstand a hostile reading in two years. The difference comes down to four properties.

Completeness against a question. A log is not complete in the abstract; it is complete with respect to the questions it must answer. Start from the claims in the safety case and the defeaters that would void them, then derive the fields. For a decision system this usually means the input as received, the feature values actually used, the model version, the output, the thresholds applied, and the human action taken afterwards. The feature vector matters more than the raw input, because the raw input plus a since-changed pipeline does not reproduce the decision.

Integrity. An auditor has to believe the log was not edited after the fact. Append-only storage, write-once retention policies, hash chaining or periodic signed digests all work; an application-writable table in the production database does not. The cost is real: tamper-evident storage removes the ability to correct a mistake, so corrections become new records rather than edits.

Reconstructability. A record of outputs without the code, weights, prompts and configuration that produced them supports a claim about what happened and no claim about why. This is why record-keeping is coupled to artefact versioning: the log must name an immutable identifier for every component, and that identifier must still resolve when someone dereferences it.

Admissible provenance. Timestamps from a synchronised source, a clear account of what is sampled and what is exhaustive, and a documented retention schedule. An auditor who cannot tell whether an absence of records means nothing happened or means sampling dropped it will treat the silence as uninformative.

The sampling tension

Full capture is the only honest default for the fields that bear on a safety claim, and it is sometimes unaffordable. A generative system logging complete prompts and completions at high volume generates storage and, more importantly, a sensitive data liability that grows every day it is retained.

The workable split is by field rather than by request: log the full decision-relevant metadata for every call, and sample the raw content. That preserves the ability to compute rates, distributions and drift exhaustively, while keeping the expensive and sensitive payloads to a representative subset. State the sampling rate in the documentation, because an undocumented sampling rate silently invalidates every count computed from the log.

When it breaks

Retention floors are not retention targets. Six months is shorter than the time it takes many harms to surface, and shorter than most litigation. Deriving the schedule from the statutory minimum rather than from the questions you expect to be asked is how organisations end up with logs that expired one month before the complaint.

Logging collides with data protection, and the AI Act says so. Article 19 subordinates its own retention rule to Union law on personal data. The resolution is engineering, not drafting: pseudonymise at write time, separate identifiers from decision records, and keep the join key under tighter access control and a shorter clock than the records themselves.

Logs record what the system did, not what it would have done. Operational records support claims about realised behaviour. A safety case claiming the system never produces a class of output needs evidence from adversarial testing, because absence in a log over a benign traffic distribution is weak evidence about behaviour under an attacker.

An unread log is not a control. Record-keeping satisfies an obligation only when something consumes it: alerting, periodic review, or an auditor. The most common finding in control testing is a log that exists, retains correctly, and has never once been queried, which establishes the control was designed and never operated.