Failover Orchestration
The sequence of detection, decision, promotion and traffic redirection that moves service from a failed component to a healthy one.
Four stages, and each has its own failure modes.
Detection. Health checks must distinguish a failed component from a network partition between the checker and the component. Checking from a single vantage point produces false positives; checking from several and requiring agreement reduces them at the cost of latency.
Decision. Automatic failover is fast and can trigger wrongly; manual failover is deliberate and adds human latency that may exceed the RTO. The common compromise is automatic within a region, manual across regions — because a cross-region failover is expensive to reverse.
Promotion. A replica becomes primary. With asynchronous replication this accepts data loss of whatever had not replicated, and there must be an explicit decision about whether to wait or proceed.
Redirection. Clients must reach the new primary — via DNS (slow, subject to caching), a load balancer (fast), or client-side discovery.
The two things that break real failovers: failback is harder than failover and is almost never rehearsed, and split brain — two components both believing they are primary, which requires fencing or quorum to prevent, not merely careful sequencing.