Inference Trust Boundary

A support product adds a feature that drafts replies, and the implementation is one HTTPS call to a model provider. On the diagram it looks like every other third-party call: a box, an arrow, done. The difference is what is in the payload. An ordinary integration sends fields that somebody classified when the schema was written. A prompt sends free text that was assembled at runtime from a ticket body, a knowledge-base excerpt and whatever the customer pasted in, and the classification that travelled with the database column is lost at the moment of concatenation.

The inference trust boundary is the crossing drawn explicitly, labelled with the categories of data that go over it and what happens to them on the other side.

Why it matters

Obligations attach to the payload class and the dwell time, not to the existence of the call, and both are invisible in a box-and-arrow drawing. At 30 days of abuse-monitoring retention - a common provider default - an erasure request received on day one is not fully satisfiable until day 31 unless the contract provides a deletion path into that buffer. The published response time in the privacy notice has to accommodate that, or it is wrong.

The breach arithmetic changes too. A compromise of the provider on day 20 exposes 30 days of accumulated ticket text, not the single request that happened to be in flight. And a provider holding customer content is a processor, which brings a data processing agreement, a subprocessor-list entry and, in many enterprise contracts, advance notice to your own customers before you add one.

Implementation patterns

• Label the crossing with three things: the data categories in the prompt, the retention on the far side, and the jurisdiction the inference runs in.
• Draw the assembly step as its own process box. The prompt builder is where unclassified content enters, and it is the only place a redactor can sit.
• Record the redactor's miss rate if there is one. A redaction step changes the residual risk from "all ticket text" to "whatever the redactor misses", and that is a number, not a yes or no.
• Mark the training-reuse flag separately from retention. They are different contractual terms and the diagram should not conflate them.
• Treat retrieval context as part of the payload. A retrieval step that injects internal documents sends those documents over the same crossing, and teams routinely diagram only the user's question.
• Add logs and traces as crossings of their own. Prompt and completion text captured by a tracing agent is a second copy with its own retention, frequently longer than the provider's.

Industry example

The shape is best described as an archetype, because the contractual terms differ per provider and change often - but it has been observed in production repeatedly since 2023: a mid-size support-desk vendor adds a reply-drafting feature, passes the ticket thread to a hosted model, and the privacy review signs off on a diagram showing a new external box. Six weeks later a customer exercises a deletion request, and the team discovers that the provider's abuse-monitoring buffer holds the content for a further 30 days and that the observability stack captured prompts as span attributes with a 400-day retention. Neither path was on the diagram, both were on the crossing, and the longer of the two was the internal one.

Failure scenarios

• Silent payload growth. The prompt gains a retrieved document or a user profile in a later release; the diagram still shows the original call and the privacy review was done against it.
• The telemetry copy. A tracing agent captures request bodies on error, exporting prompt text to a log store in another region with a retention far longer than the provider's.
• Flag-enabled exposure. The feature ships behind a flag, the diagram is marked "prototype", and the first real customer account is enabled without a re-review.
• Residency assumed from the vendor's headquarters rather than from the inference region, which for many providers is configurable and defaults elsewhere.
• Erasure that cannot be executed. A deletion request is accepted with a 30-day SLA while the downstream buffer has no deletion API at all.

Trade-offs

Drawing and maintaining this boundary costs real effort, and it adds labels to a diagram whose readability is already the constraint. Every label is one more thing that can go stale, and a crossing annotated with retention terms from last year's contract is worse than an unlabelled one, because it is believed. The honest mitigation is a provenance line - what this was generated or checked from, and when - rather than a promise to keep it current.

Against that, this is the one annotation that converts a privacy review from an opinion into a check. A reviewer with the categories and the dwell time can answer the erasure, residency and breach-scope questions from the drawing in a minute; without them the review becomes a meeting with the vendor's contract open.

When not to use it

For an internal prototype running on synthetic or already-public data, the labels are overhead and the diagram's job is to show the shape. The annotation earns its place the day real customer content can reach the call - which for a flagged feature is the day the flag is enabled for one account, not the day it ships generally. And where the model runs inside your own infrastructure and the content never leaves your trust zone, there is no crossing to label; the relevant artefact is then a retention note on the inference log, not a boundary.

Interview question

Q: Your team wants to add a hosted model to summarise customer conversations. Walk me through what you add to the data-flow diagram, and tell me which single fact you would refuse to sign off without.

What a strong answer covers: the prompt-assembly process box, the labelled crossing with categories and retention, the jurisdiction, the training-reuse flag as a separate item, and the telemetry path as a second crossing. The fact to refuse without is the retention on the far side together with whether a deletion path exists into it, because that single pair determines whether the company's published erasure commitment is achievable. A strong answer also says what it would not demand - a SOC 2 report answers how competently the provider operates controls and says nothing about what you sent or how long it is kept - and names the review trigger that catches payload growth in a later release.

Quick check

Quiz: A new box and arrow are added for a model provider. Which two labels does the privacy review actually need on the arrow? The data categories crossing it and the retention on the far side; everything else follows from those two.

Flashcard: Why does a 30-day provider retention change your published erasure SLA? A deletion request on day one cannot be fully satisfied until day 31 unless the contract provides a deletion path into the abuse-monitoring buffer.