Your database primary is unreachable from the monitoring system but is still serving some clients. Do you fail over?

What is being tested

Whether you recognise a partition, and whether you know that failing over without fencing produces a worse outcome than the fault.

The situation

A network partition and a dead primary look identical from the observer's side. The critical difference here is stated in the question: the primary is still serving some clients, which means it is alive and accepting writes.

Failing over now creates split brain — two primaries, both accepting writes, producing divergent histories that cannot be automatically merged. Reconciling that afterwards is manual, expensive, and sometimes impossible, because two clients may have been told contradictory things about the same record.

Split brain is worse than the outage you were trying to prevent, because an outage ends and diverged data does not.

What must be true before you can fail over safely

Fencing. The old primary must be prevented from writing, not asked to stop. It cannot be asked; you cannot reach it. Mechanisms:

• Quorum. The isolated primary cannot achieve a majority and therefore steps down on its own. This is why odd-numbered clusters and a witness in a third location exist — a two-node arrangement can never establish which side is authoritative.
• Fencing tokens. A monotonically increasing token issued on each promotion; the storage layer rejects writes carrying a stale token. This works even when the old primary is unreachable, because the rejection happens at the resource, not at the node.
• Forcible isolation. Power off the old primary, or revoke its network access or credentials at the infrastructure layer.

Without one of these, do not fail over. The correct action is to keep investigating and to accept the partial outage while doing so.

The decision framework

1. Establish whether it is the primary or the observer that is isolated. Check from multiple independent vantage points. A monitoring system in one zone reporting a primary in another as unreachable is at least as likely to be a monitoring problem.
2. Check whether clients are succeeding. They are, per the question. That is strong evidence the primary is healthy and your visibility is not.
3. If fencing is available, fence, verify, then promote.
4. If it is not, do not promote. Restore observability, or isolate the primary at the infrastructure layer first.

The general lessons

False-positive failovers cause outages. Failing over unnecessarily means you caused an outage to fix one that did not exist. Detection needs multiple independent observers, a quorum before acting, and a deliberate delay so transient blips do not trigger it.

Automate where the cost of being wrong is low — a stateless instance behind a load balancer — and keep a human in the loop where the consequence is data divergence. There is no shame in manual database failover, provided the manual path is practised, documented and fast. The worst option is automation nobody trusts, disabled during the incident by someone who then performs an unrehearsed procedure under pressure.

Failback is harder than failover and is almost never rehearsed. Reconciling what was written during the split, and re-establishing replication in the correct direction, is where the real complexity lives.