GitHub 2018: 43 Seconds of Partition, 24 Hours of Recovery
also called GitHub October 2018 Incident
A 43-second network partition triggered an automated cross-region database failover, and reconciling the resulting divergence took over 24 hours.
What happened
On 21 October 2018, planned maintenance replacing failing 100G optical equipment caused a 43-second loss of connectivity between GitHub's US East Coast network hub and its primary US East Coast data centre.
Their automated MySQL failover system, seeing the primary as unreachable, promoted a West Coast cluster to primary. During the brief window, writes were accepted in both locations. When connectivity returned, the two clusters had diverged.
The outage was not the 43 seconds. It was the 24 hours and 11 minutes of degraded service that followed, while GitHub reconciled the divergent data and restored consistency without losing customer writes.
The design tension
The automation did exactly what it was configured to do. The failure was in the configuration's assumptions: a failover threshold short enough to trigger on a transient partition, combined with a topology where the promoted replica was thousands of kilometres away, and no fencing to prevent the original primary from continuing to accept writes.
This is the classic split-brain shape. With two candidate primaries and no quorum arbiter able to distinguish "the other side is down" from "I cannot see the other side", both sides can reach a defensible but incompatible conclusion.
The transferable lesson
The cost of a wrong failover is usually far higher than the cost of a slow one. Reconciling divergent writes across a database with GitHub's write volume is a bespoke, careful, manual operation — there is no tool that does it, and there is no way to rush it.
Three specific design points follow:
Set failover thresholds against the duration of transient network events, which are far more common than genuine host failure. Failing over after 43 seconds means failing over for every brief blip.
Fence the deposed primary. A failover that does not positively prevent the old primary from writing has not completed; it has created two primaries.
Consider whether automatic cross-region promotion is wanted at all. Automatic failover within a region and a deliberate human decision for cross-region is a defensible position, and many organisations adopt it after an incident of this shape.