A partner integration issues PATCH requests with JSON Merge Patch semantics. On Tuesday one partner's nightly sync cleared the phone number on 40,000 customer records. The requests were valid, your service applied them correctly, and no error was returned anywhere. What happened, and what do you change?
Show the full answer Hide the answer
The trigger
In JSON Merge Patch, null means "delete this member". The partner's client serialised its internal model directly, and fields that were simply unset in that model serialised as null rather than being omitted.
Absent and null are different values in the protocol and identical concepts in most object models, which is where the defect lives. No developer decided to delete 40,000 phone numbers; a serialiser did, by default, correctly.
Why it propagated
There was nothing to catch. The request was well-formed, the semantics are documented and were applied faithfully, and the service returned 200 to each one. No component in the path can distinguish "delete the phone number" from "I had no phone number to send", because at the wire level they are the same bytes.
Why detection lagged
Deletions are invisible in success metrics. Request rate normal, error rate zero, latency flat. The only signal that would have fired is a business metric — the count of customer records with a populated phone number, per partner, per day — and that class of metric is exactly what most platforms do not have.
The structural fix versus the tempting local fix
The tempting fix is to tell the partner to stop sending nulls. It works until the next partner, and it makes a protocol hazard into a relationship problem.
The structural fixes, in order:
- Prefer JSON Patch for partner-facing partial updates.
{"op": "remove", "path": "/phone"}is an explicit instruction that no serialiser produces by accident. The verbosity is the feature. - If merge semantics must stay, require an explicit field mask. The caller lists the fields it intends to change; anything not on the list is untouched regardless of what the body contains. This inverts the default from destructive to inert, and it is the pattern adopted in several large API design guides for exactly this reason.
- Rate-limit destructive effect, not requests. Reject a batch whose net effect clears a field on more than 5% of a partner's records in production, and require an explicit override header to proceed. It costs a little friction on legitimate bulk corrections, which is a price worth paying once. This is the control that catches the next variant you have not thought of, which is the only kind of control worth building after an incident like this.
- Alert on populated-field counts per partner per day. Cheap, and it turns a silent deletion into a page.
The general lesson
In any partial-update protocol the most dangerous value is the one that means "remove", because it is also the default serialisation of "I have nothing here". Every partial-update API has this shape somewhere. The only real question is whether removal is explicit or inferred, and inference is what makes an accident possible.
When this is the wrong lesson
For an internal API where both sides are one team's code, the serialiser is known, the model is known, and merge patch is entirely fine — field masks there are ceremony that buys nothing. The hazard arrives with the second organisation, because that is the point at which you stop being able to reason about how the request was produced.