Deletion Safety Threshold

A synchronisation system reads a source of truth and applies the difference. An empty or truncated response from the source is indistinguishable from a source in which everything was deleted, and a system that applies the difference faithfully will remove every record.

The threshold is a simple guard: if a sync would delete more than some proportion of the population, stop and require confirmation.

Why it matters

This is among the highest-consequence, lowest-cost controls in integration engineering. The failure is total rather than partial — every user in a customer's tenant deactivated, every product delisted, every record removed — and it is caused not by a bug in your logic but by a transient fault in someone else's system.

Recovery is possible only if you kept the prior state, which is a second design requirement that follows from the first.

Implementation patterns

• A proportional threshold, not an absolute one, since a directory of fifty and one of a hundred thousand need different limits. Typically a small single-digit percentage, tuned per connection where churn is genuinely high.
• Halt and notify rather than proceeding. The correct behaviour is to pause the connection, alert its owner and preserve the current state — not to apply and then attempt to undo.
• Distinguish absence from deletion. An empty page, a truncated response, an authentication failure and a genuinely empty directory must be different code paths. Treating a failed fetch as an empty result is the root cause, and the threshold is the backstop for when that distinction fails.
• Soft-delete first, so the reversal is a flag rather than a restore from backup.
• Retain the previous sync state, so recovery does not depend on the source system being available.
• Quarantine malformed records and continue. A single unparseable record must not fail the run, which is what the naive implementation does — but a large proportion of malformed records should trigger the same halt as a large proportion of deletions.

Industry example

Enterprise-integration platforms such as WorkOS synchronise identity from customer-operated directories that vary enormously in reliability. A customer's identity provider returning an empty result during maintenance, or an expired credential producing an authorisation error the connector maps to "no users", is a routine occurrence — and without a threshold the consequence is every employee of that customer losing access.

The same shape governs product-catalogue synchronisation from supplier feeds, inventory synchronisation, and any pipeline where an upstream fault can look like an intentional removal.

Failure scenarios

• No threshold, so a source-side fault becomes a total data loss for a tenant.
• Threshold expressed absolutely, so it is either useless for large tenants or blocks routine change for small ones.
• Auto-remediation instead of halt, which attempts to undo a change already applied downstream.
• Hard deletes, making recovery a restore rather than a flag flip.
• The alert going to a queue nobody watches, which converts a halt into an outage of a different kind.

Trade-offs

A threshold produces false positives: a genuine large restructuring — a customer offboarding a division — triggers a halt and requires manual confirmation. That is friction for a real operation, and it is the correct trade, because the cost of the false positive is a support interaction and the cost of the false negative is a customer's entire workforce losing access.

The tuning question is only the level, and it should be per-connection where a customer's churn genuinely warrants it — with the override recorded rather than configured silently.

Interview question

"Your directory sync connector receives an HTTP 200 with an empty user list from a customer's identity provider. Describe everything that happens next in a well-designed system, and everything that happens in a naive one."