Contract Coverage Gap
also called Contract Blind Spot, Unasserted Field Surface
The share of a provider's response that a consumer's code reads but its contract never asserts on - the exact surface a provider can delete while every contract check stays green.
A provider wants to remove a field. The broker says no consumer asserts on it. The field is deleted, and six weeks later a finance export has a blank column.
Nothing malfunctioned. A generated consumer contract is a derivative of the consumer's tests, not of the consumer's code, and consumer-driven contract testing quietly assumes those two sets are equal. They never are. The difference between them is the contract coverage gap, and it is the surface on which the practice gives a confident wrong answer rather than no answer.
The gap is not a bug in any tool. It is the price of deriving a machine-checkable artefact from a human-written test suite, and it is invisible because both directions of the check report success: the consumer's build is green, and the provider's verification is green because there is nothing left to verify.
Why it matters
Consumer-driven contracts exist to grant a permission that nothing else grants: the right to remove things from a long-lived API. Backward-compatibility rules only ever let a schema grow. Contracts let a provider prove that a field is unused and delete it. That proof is the whole value proposition, and the coverage gap is the amount by which the proof is wrong.
The failure it produces is the worst-shaped one available: silent, delayed and confidence-increasing. Deleting a consumer test improves every signal in the pipeline, because a verification suite can only fail on expectations that exist. Detection typically runs weeks, because a missing JSON field deserialises to null in most client libraries rather than raising.
Implementation patterns
- Diff the published contract on every publish. An expectation that disappeared is a reviewable event, treated like a deliberate API change. One comparison in the publish step, and it catches the case on day zero rather than day sixty.
- Derive coverage from the deserialisation code, not from tests. Walk the consumer's response types or client code to enumerate the fields it reads, intersect with the fields the contract asserts, and publish the ratio alongside the contract. Test coverage cannot be the measure here, because the test suite is the thing that was wrong.
- Ground removals in production telemetry. The provider logs field-level access per consumer and refuses to remove any field read in the last 30 days. This is the only check that observes what is used rather than what someone wrote a test about, and it is the one to build first if only one gets built.
- Fail closed on the consumer side. A deserialiser that rejects a missing required field converts a blank column into a loud error at the first request, moving detection from weeks to minutes.
- Expire contracts. A contract from a consumer version that has not been deployed for 90 days should stop constraining the provider, or the contract set ratchets into the full response surface and stops narrowing anything.
Industry example
The pattern is visible in any estate running a Pact-style broker with a deployment compatibility gate. The tooling answers "is this version verified against the versions currently deployed", which is exactly the right question, and it answers it from the recorded expectations. Teams that add field-level access logging on the provider routinely find that the set of fields consumers actually read is larger than the union of contract expectations - the practical consequence is that the "safe to delete" list is shorter than the broker says, and the difference is not small.
Failure scenarios
- The deleted test. The only assertion touching a field is removed during a cleanup, the contract shrinks, and the provider deletes the field at the next tidy-up.
- The consumer that never tested the error path. Contracts cover the 200 response and say nothing about the 409 the consumer's retry logic depends on, so the provider changes the error body freely.
- The field read only in a rarely-exercised branch - a reconciliation job, a monthly export, an admin screen - which is precisely the code least likely to have a contract-generating test.
- Semantic drift inside a covered field. The contract asserts a string is present. The provider changes what the string means. Every check passes.
Trade-offs
| Choose | Gains | Pays |
|---|---|---|
| Measure and close the gap | Removals become genuinely safe; deprecation stops being guesswork | Static analysis of consumer code, a coverage metric to maintain, provider-side access logging |
| Live with the gap | No extra machinery; the practice works for most changes | A rare, silent, weeks-late data defect that nobody traces back to a deleted test |
| Abandon contracts for a compatibility registry | No consumer cooperation needed; works for unknown consumers | The schema can only grow, forever |
The honest position is that the gap is tolerable for additive change and intolerable for removals. Measure it before you rely on a contract set to authorise a deletion, and not before.
When not to use it
Do not build coverage measurement for a two-team API with a handful of fields deployed together - a conversation is cheaper than static analysis, a metric and a review gate. The gap also stops mattering if the provider simply never removes fields, which is a legitimate policy for an API with a small and slow-moving surface. The metric earns its cost exactly when the provider intends to use contracts as permission to delete. If nobody is deleting, measuring the gap is instrumentation of a decision that is not being made.
Interview question
Q: Your broker says no consumer asserts on the legacy_status field and a team wants to remove it. What would you check before signing that off, and how would you make the check automatic?
What a strong answer covers: the contract is generated from consumer tests, so absence of an assertion is not absence of use · check contract history for a recent shrink and ask what changed in that consumer's tests · get field-level read telemetry from the provider over at least 30 days, covering monthly and quarterly jobs · a deprecation window with logging before deletion rather than deletion behind a green check · automating it as a publish-time contract diff plus an access-log gate · and the recognition that the cheapest permanent fix is consumers that fail closed on missing required fields.
Quick check
Quiz: A consumer deletes a test and its contract shrinks. Which pipeline signal goes red? None - every check improves, because a verification suite can only fail on expectations that exist.
Flashcard: What does a consumer contract actually cover? - Only the response surface the consumer's tests exercise, not what its code reads; the difference is the surface a provider can delete while everything stays green.