A grocery platform integrates with retail partners whose inventory data varies in freshness, completeness and reliability. How should the integration be designed so one partner's bad data does not degrade the whole platform?

The core requirement

Every failure must be scoped to one partner. A partner's broken feed must not block other partners' updates, and must not silently publish degraded data for that partner either. Both failure modes are unacceptable and they pull in opposite directions, which is what makes the design interesting.

The design

1. Per-partner pipelines with independent state. Separate ingestion, validation, transformation and publication per partner, each with its own status, freshness metric and alerting. Shared infrastructure, isolated failure.

2. Last-known-good as the fallback. When a partner's feed fails validation, hold that partner at its previous state and alert. Do not publish partial data; do not blank the catalogue. This single decision prevents the worst outcome — a truncated upload silently deleting a retailer's entire catalogue, causing items to vanish from search and orders to fail at fulfilment.

3. Gross anomaly detection before load. A feed with dramatically fewer rows than usual, or with prices an order of magnitude different, is almost certainly broken rather than intentional. This check catches more real incidents than any schema validation.

4. Separate pipelines by change rate. Catalogue changes daily and tolerates hours of lag. Inventory changes continuously and stale availability directly causes failed orders and substitutions. Merging them into one nightly batch — the common initial design — makes the fast-changing data as slow as the slow.

5. Explicit provenance on every record. Which partner, which feed run, which file, what time. Without it, nothing downstream can be debugged or reprocessed, and a data-quality question becomes unanswerable.

6. Partner-visible data quality reporting. Show partners their own feed health, rejection reasons and freshness. This converts a support burden into self-service and, more importantly, gives the partner an incentive and a means to fix their side.

The consistency problem that cannot be engineered away

Inventory is asynchronously reported and physically mutable. A shopper picks the last item on a shelf one second after the platform sold it. No amount of pipeline quality fixes that, because the discrepancy originates in the physical world.

So the architecture must accept it as a permanent condition and design the product around it:

• Safety stock — do not sell the last units of an item, sized per partner by observed reliability.
• Substitution as a first-class flow, not an error path, with customer preferences captured up front.
• Reconciliation at pick time, where the physical truth is discovered, with a defined customer experience for a shortfall.
• Per-partner reliability scoring feeding both the safety-stock parameter and the commercial relationship.

The transferable lesson

In B2B integration, data quality is a property of the partner, not of your pipeline, and it varies by orders of magnitude across partners. The architecture's job is to make that variance a parameter — per-partner thresholds, freshness expectations, safety margins and reliability scores — rather than a source of platform-wide incidents.

Systems that assume uniform partner quality are calibrated for the best partner and broken by the worst.