A grocery marketplace of the kind Instacart operates replaces static database passwords with dynamic credentials leased for 15 minutes from a secrets broker. What has it gained, what has it paid, and when does that bill arrive?

What is gained, quantified

The value of a stolen credential falls to its remaining lease. A static password exfiltrated today is useful until somebody notices and rotates it, which published breach analyses measure in weeks. A 15-minute lease is useful for an average of seven minutes, and no human action is required to make that true - which is the real gain, because the control does not depend on anyone remembering anything.

Two smaller gains matter: revocation becomes expiry, so there is no "did we actually invalidate it" question; and every lease is an audit record tying a database session to a workload identity and a time, which is what an auditor asks for and what static shared passwords can never provide.

What is paid

• The broker joins the critical path for new connections. Not for existing ones, which is the saving grace, but a broker outage means no new connections anywhere - so a deploy, a scale-out or a pod restart during the outage produces instances that cannot reach the database. The broker's availability must now exceed the database's.
• Connection pools churn. A pool authenticated with an expiring credential must rebuild connections as leases roll over. If every pod took its lease at the same moment, every pod renews at the same moment: 400 pods with 5 pools each is 2,000 leases, and synchronised expiry turns a steady 2 per second into a 2,000-request spike against the broker and a simultaneous reconnect storm against the database. Jitter the lease lifetime by 10 to 20% or build the spike in deliberately.
• The database accumulates identities. Dynamic credentials usually mean a real user or role per lease. Those must be reaped, and a broker that crashes mid-revoke leaves orphans - a slow leak that eventually hits a role limit or makes the user table itself a performance problem.
• Cold-start latency. Each new connection now costs a broker round trip plus credential creation, often tens to low hundreds of milliseconds. It is invisible for a warm pool and painful for short-lived workloads such as serverless functions, which create connections constantly.
• Secret zero moves rather than disappearing. Something must authenticate the workload to the broker. Done properly that is a platform-issued workload identity; done badly it is a long-lived token in an environment variable, and the whole exercise has relocated the problem.

When the cost becomes visible

At the first broker incident, and at the first traffic peak that coincides with mass lease renewal. Both arrive after the project is declared finished, which is why the lease jitter and the broker's own availability target belong in the design rather than in a later retrospective.

How to keep the option to reverse

Keep a sealed break-glass static credential with its own alert on any use, documented in the runbook and rotated on a schedule. Keep the lease lifetime a configuration value, so it can be raised from 15 minutes to 8 hours during a broker incident instead of becoming an outage. And measure the lease failure rate as a first-class service level indicator, because it is now a dependency of every deployment.

When this is the wrong move

The decision rule: dynamic credentials are worth it when the credential grants access to data whose exposure is material and the workload population is dynamic. For a handful of long-lived services reading a non-sensitive store, a well-managed static credential with a tested rotation pipeline is proportionate, and it does not put a broker in front of every connection.