A workforce platform must integrate with hundreds of external systems with wildly different reliability. What integration architecture scales, and what does not?
Show the full answer Hide the answer
What does not scale
Point-to-point integration written per partner inside the product. Each is bespoke, each has its own retry and error handling, each fails differently, and the count grows linearly while the maintenance grows faster because every product change must be validated against all of them.
Also does not scale: one shared worker pool for all integrations, which means the slowest or most broken partner consumes capacity everyone needs.
What scales
- A connector abstraction with a declared capability set per partner. The platform speaks one internal model; each connector translates. Capabilities are declared honestly — supported, approximated, unsupported — because flattening real differences produces values that are uniformly available and quietly wrong.
- Per-connector isolation: its own workers, queue, concurrency limit, retry policy derived from that partner's observed behaviour, and its own circuit breaker. One policy across hundreds of heterogeneous partners is guaranteed wrong for nearly all of them.
- Per-connector scheduling, since a fifty-record source and a hundred-thousand-record source need different cadences and different strategies.
- Checkpointing, so a sync interrupted at 80% resumes rather than restarting.
- Schema tolerance: reject and quarantine a malformed record, continue the run. A single bad record must not fail the whole sync, which is exactly what the naive implementation does.
- Reconciliation independent of the event stream, because change events from external systems are lost, arrive out of order, and are sometimes simply not sent.
The safety control that must not be omitted
A deletion threshold. An external system returning an empty result because of an error is indistinguishable from one where everything was removed, and a sync that faithfully applies the difference will deprovision an entire customer's workforce.
Refusing to process a deletion batch above a proportion of the population without confirmation is not optional — its absence has produced the most damaging incidents in this category.
The organisational consequence
A capability and reliability matrix per connector is a product feature, not internal documentation. Customers need to know what their systems support before they build on it, and support engineers need to know what is known-unreliable before they investigate.