A data-ingestion platform runs hundreds of connectors, each with different APIs, rate limits, failure modes and schemas. How should isolation, retries, scheduling, checkpointing, backfills and schema evolution be designed?

Isolation, which is the foundation

Per-connector workers, queues, concurrency limits and error handling. A slow or broken source must not consume capacity that other syncs need — and with hundreds of heterogeneous sources, at any moment several are degraded.

One retry and timeout policy across all of them is guaranteed to be wrong for nearly all of them: a source that normally answers in 300ms and one that takes 8 seconds cannot share a timeout.

Scheduling and checkpointing

• Per-connector schedules and strategies. A fifty-record source and a hundred-million-record source need different cadences and different approaches — full refresh versus incremental.
• Checkpointing at a granularity that makes resumption cheap. A sync interrupted at 80% must resume, not restart. For a large source that is the difference between an hour and a day, and interruptions are routine rather than exceptional.
• State stored per connector and per stream, so one stream's failure does not reset another's position.

Retries and failure

• Per-source retry policy derived from that source's observed behaviour, with a budget expressed as a fraction of requests so retry load is bounded during a broad failure.
• A circuit breaker per source, tripping on latency as well as errors — a source that is slow but responding consumes a worker for the full timeout and returns nothing.
• Disable and notify after a sustained failure, rather than retrying indefinitely against something that is not coming back.

Schema evolution

The hardest part. Sources add columns, rename them, change types and remove them without notice.

• Additive changes propagate automatically; anything else requires a decision.
• A type change or a removal should quarantine rather than fail or silently coerce. Silent coercion is the worst option — it succeeds, and the data is wrong downstream with nothing indicating it.
• Record the schema history, since a downstream consumer needs to know when a column's meaning changed.

Schema tolerance and backfills

Reject the record, quarantine it, continue the sync. A single malformed record must not fail the whole run, which is what the naive implementation does.

Backfills need their own rate limit and their own queue, because a backfill running at full speed against a source's rate limit will exhaust the quota that incremental syncs need — starving the freshness the platform promises.