A managed database performs a routine minor-version upgrade with a 40-second failover. The application returns errors throughout. Whose problem is this?
Show the full answer Hide the answer
What is being tested
Whether you understand that managed services move operational work but do not remove the application's responsibility for its own resilience.
The reasoning
Failover is a documented, expected behaviour of every managed database. The provider's contract is that failover happens and completes, not that it is invisible. An application that treats a connection as permanently valid will break during any failover — planned upgrade, hardware replacement, or a genuine failure.
The application needs:
- Connection pool health checking, so dead connections are detected and discarded rather than handed to a request.
- Retry on transient connection errors, with backoff and jitter, distinguishing a connection failure (safe to retry) from a query failure (may not be).
- Idempotent writes where a retry could duplicate a side effect.
- Short connection timeouts, so a request fails fast and retries rather than hanging for the full failover duration.
- DNS caching disabled or bounded, because the endpoint moves and some runtimes cache resolution for the process lifetime — traffic then goes to the old address indefinitely rather than for 40 seconds.
- Graceful degradation for read paths that could serve cached or replica data during the window.
What is legitimately the provider's side
You can and should:
- Choose a maintenance window aligned with your low-traffic period.
- Know the documented failover time and treat it as a design input, not a surprise. If 40 seconds exceeds your RTO for this service, that is a valid reason to self-manage or to choose a different topology — but it must be decided, not discovered.
- Ask whether a read-write splitting proxy would let reads continue during a write-side failover.
The broader point
This is the general shape of the managed-services trade: you buy operational capacity and you inherit the provider's failure modes. Both halves are real. The mistake is not adopting managed services — that is usually correct, because the premium is typically less than a fraction of an engineer's time. The mistake is adopting one without reading its failure behaviour and designing for it.
"Managed" means somebody else does the operations, not that the operations do not happen. The same applies to backups: some managed services retain snapshots briefly, some retain them longer, and almost none are restore-tested by you. That test is still yours to run.