Third-Party AI Vendor Risk Management

The 2023 Foundation Model Transparency Index scored ten major developers on 100 disclosure indicators, covering data, labour, compute, capabilities, risks and downstream use, and found an average of 37 out of 100 (Bommasani et al., 2023, The Foundation Model Transparency Index, arXiv:2310.12941). The 2024 follow-up reached 58, but mostly because developers submitted reports specifically for the index, and opacity about data access, copyright status and downstream impact persisted (Bommasani et al., 2024, The 2024 Foundation Model Transparency Index, arXiv:2407.12929). That is the information environment a procurement team works in. The organisation buying the model stays accountable for decisions it makes with the model, and it will never see most of what it would need to assess the model the way it assesses one built in-house.

This concept is about the commercial and governance relationship. Technical intake of downloaded weights and datasets (pinning, formats, backdoor probing) is covered in vetting-third-party-models-and-datasets on the supply-chain track.

Accountability does not transfer

Every serious regime says the same thing in different words. The US banking agencies' Interagency Guidance on Third-Party Relationships: Risk Management (June 2023) organises oversight around a lifecycle of planning, due diligence and selection, contract negotiation, ongoing monitoring and termination, scaled to the criticality of the activity (OCC Bulletin 2023-17). The 2026 model risk guidance adds that vendor models, even when the vendor withholds code, data or methodology, still need validation of "conceptual soundness, design, development data, and performance," along with ongoing outcomes analysis. In the EU, DORA has applied to financial entities' ICT third-party arrangements since 17 January 2025, and Article 25(4) of the AI Act requires the provider of a high-risk system and anyone supplying it a model, tool or component to specify by written agreement "the necessary information, capabilities, technical access and other assistance" the provider needs to comply.

The public sector reached the same conclusion from the buyer's side. OMB memorandum M-25-22 (April 2025) told US federal agencies to avoid vendor lock-in through data and model portability terms, and to write contracts that "permanently prohibit the use of non-public inputted agency data and outputted results to further train publicly or commercially available AI algorithms" absent explicit consent.

What due diligence can learn, and what it cannot

Split the questions by whether the answer can be checked.

Checkable by you: performance on your task data, behaviour under your adversarial probes, latency and cost under your load.

Attestable by the vendor: training-data provenance, evaluation methodology, incident history, subprocessors, security certifications. Treat these as claims that carry contractual weight, not as evidence. A certificate such as ISO/IEC 42001 tells you a management system exists within a stated scope; it tells you nothing about this model on your data.

Unknowable in practice: the full training corpus, and when the model behind an API alias will change.

The third category is why the contract matters more for AI than for most software. It is the only instrument that converts an unknowable into a notification obligation.

The controls that make a hosted model governable

Version pinning and change notice. Contract for dated model versions, a minimum deprecation notice period, and advance notice of material behaviour changes. Without these, every provider release is an unvalidated change to your production system.

A canary set with a statistical trigger. Keep a fixed evaluation set and run it on a schedule. Suppose it has 500 cases, with a 92% pass rate last month and 86% this month. The pooled rate is \(\bar p = 0.89\), and the two-proportion test statistic is

well past a two-sided 1% threshold. A six-point drop on 500 cases is real, and it is the kind of change Chen, Zaharia and Zou documented between two releases of the same named GPT-4 service (arXiv:2307.09009). A 1-point drop on the same set would not be detectable, which tells you how large the set has to be for the risks you care about.

Data-use and retention terms. Whether inputs and outputs can train future models, how long logs persist, where they are processed, and who the subprocessors are.

Incident and audit rights. Notification windows for security and safety incidents, cooperation with your regulators, and enough evidence access for your own conformity or validation duties. Under the AI Act a downstream provider cannot compile its technical file without upstream information, so the dependency is legal as well as practical.

Exit. Portability of fine-tuned artefacts, prompts and evaluation data, and a tested fallback model. Termination is the lifecycle stage organisations plan least and need most urgently when a vendor deprecates a model at short notice.

When it breaks

Questionnaire inflation. Questionnaires grow AI sections that vendors answer with marketing copy, and unverifiable answers get the same weight as checkable ones.

Concentration hidden behind diversity. Three "different" vendors built on the same foundation model share one failure mode. DORA's oversight of critical ICT providers exists because concentration risk is invisible at the level of a single contract.

Leverage asymmetry. A mid-sized buyer rarely gets bespoke change-notice or audit terms from a frontier lab. Practitioners disagree on the answer: some argue for accepting standard terms and investing in independent monitoring, others for keeping high-stakes uses on models the organisation can host and pin. The first is cheaper; the second is the only one that fully controls versioning.

Validation of the wrong artefact. Teams validate the model and forget the vendor's surrounding system (safety filters, system prompts, retrieval layers) that the vendor can change independently. The thing to pin and monitor is the full endpoint behaviour, not the model name.