A 43-second network partition caused GitHub over 24 hours of degraded service in 2018. How does a 43-second event become a day-long incident?

The case, as publicly reported

On 21 October 2018, routine maintenance replacing failing optical equipment caused a 43-second loss of connectivity between GitHub's US East Coast network hub and its primary US East Coast data centre.

Their MySQL topology automation (Orchestrator) responded as configured: it promoted a primary in the US West Coast data centre. When connectivity returned 43 seconds later, the East Coast cluster had already accepted writes that the West Coast had not, and the West Coast was now taking writes of its own.

The result was two databases each holding writes the other did not have. GitHub declined to automate a resolution — automated reconciliation risked data loss — so they ran in a degraded read-mostly state while restoring from backups and manually reconciling. Full recovery took over 24 hours.

Why a 43-second event lasts a day

Because the cost is not the outage, it is the divergence. Once two primaries have accepted conflicting writes, there is no correct automatic merge — the system cannot know which version of a row is intended. Recovery becomes a data-reconciliation project measured in hours or days, regardless of how brief the triggering event was.

This is the asymmetry that makes automated cross-region failover dangerous: failing over is fast, un-failing over is not.

The design questions it raises

1. Should failover be automatic across regions? Automation is right when it is faster than humans and the failure signal is unambiguous. Across regions, neither holds: a 43-second blip is indistinguishable from a real regional failure at the moment of decision, and the cost of being wrong is asymmetric. A common conclusion after this incident was to require human confirmation for cross-region promotion while keeping in-region failover automatic.

2. Was the detection window right? Failing over on 43 seconds of loss means any transient event triggers a promotion. Longer thresholds trade a slower response to real failures for far fewer spurious ones — and given the asymmetry, that trade is usually correct.

3. Was there fencing? The deposed primary continued accepting writes. Fencing tokens, or a quorum that a partitioned minority cannot achieve, are what prevent the divergence rather than detecting it afterwards.

4. What is the cross-region write latency? Synchronous replication would have prevented divergence at the cost of adding inter-region latency to every write. That is a legitimate design choice with a real price, and it should be made deliberately rather than discovered during an incident.

What a strong answer adds

The principle: prefer failing to unavailable over failing to inconsistent, for any system where reconciliation is expensive. Availability can be recovered in minutes; consistency, once lost, may not be recoverable at all. GitHub chose degradation over data loss during recovery, which is the same principle applied under pressure.