intermediate 2 min answer

A service fetches a feature flag value and a database credential from central services on every request. What is wrong?

amazoncontrol-planeavailabilitycaching
Show the full answer Hide the answer

What the interviewer is testing

Whether you recognise a control-plane dependency in a data plane, which is one of the most common hidden availability problems.

What is wrong

The service has adopted the availability of two management systems.

Control planes — configuration services, secret managers, flag services, service registries — are more complex and lower-volume than data planes, and they are consistently the less reliable half. Study almost any major cloud outage and the failure is in a control plane while the data planes kept running for whatever was already provisioned.

The arithmetic: if the flag service offers 99.9% and the secret manager 99.9%, this service cannot exceed about 99.8% regardless of its own quality. It has also added two network round trips of latency to every request, and it will fail catastrophically during exactly the incidents where those central services are stressed.

The fix

Cache both locally with a long fallback.

Flag values: fetch on start-up, refresh in the background on an interval, and continue using the last known value indefinitely if refresh fails. A flag evaluation must never fail the request, and the default when nothing is available must be safe and local.

Credentials: fetch at start-up and on rotation, cache for the credential's lifetime, and refresh ahead of expiry rather than on demand.

This converts two hard dependencies into soft ones. The service continues operating correctly with the configuration it already has when the control plane is unavailable — which is static stability.

What a strong answer adds

Applying the same test across the estate: if the provisioning and configuration APIs were unavailable right now, what would stop working? The honest answer usually includes autoscaling as a capacity plan, disaster recovery that provisions the secondary at failover time, and service discovery that fails closed.

Every item on that list is a data plane depending on a control plane at exactly the moment the control plane is least likely to be there.

Common weak answers

Adding retries, which does not help when the dependency is down. Moving the flag service closer, which reduces latency and not the dependency.