Split Brain
A partition in which two subsets of a cluster each believe they are authoritative, accepting conflicting writes that cannot afterwards be reconciled.
The most damaging distributed systems failure, because unlike an outage it does not announce itself — both halves appear healthy, both serve traffic, and the damage is discovered when the partition heals and the two histories cannot be merged.
It happens whenever leadership is decided by a mechanism that a partition can satisfy twice: a health check that each side passes from its own vantage point, a timeout-based failover with no coordination, or a two-node cluster where each node concludes the other has failed.
The prevention is majority quorum. A partition can produce at most one majority, so at most one side can elect a leader; the minority side must refuse to serve writes even though it is running perfectly well. That refusal is the design working as intended, and it is the part that gets overridden under pressure by an operator who sees a healthy node declining traffic.
The mechanisms that support it in practice: fencing so a deposed leader's writes are rejected by storage; STONITH-style isolation in infrastructure clusters; and witness or arbiter nodes to create an odd count where the topology is naturally even.
The design rule that follows: any manual failover procedure must include a step that positively prevents the old primary from returning to service, and any automatic one must be quorum-based. "Bring the old primary back and reconcile later" is how data loss becomes permanent.