Operating Capacity Budget

A five-engineer team's design calls for a relational database, a search index, a cache, a message broker and a time-series store. Each choice is individually defensible. The design is still wrong, for a reason unrelated to any of the five.

A stateful system holding production data carries duties that do not shrink with traffic: version upgrades on the vendor's calendar, backup verification, a restore rehearsal somebody has actually run, capacity headroom, a failure runbook, a retention policy. Budget 0.3 to 0.5 of an engineer continuously per system - so five systems is 1.5 to 2.5 engineers of a five-person team before a line of product code.

Why it matters

Operating attention is the resource that actually runs out, and unlike headcount or cloud spend it appears on no budget line. A team that adds a fourth stateful system files no request; it simply becomes slower at everything, and the slowdown is blamed on complexity in general. Making it a number turns the conversation from aesthetic to arithmetic: "that is our fourth system and the budget is two, so tell me which one it replaces."

Implementation patterns

• State the budget with its derivation so it survives its author: "two systems, from 0.4 engineer-years of standing attention each against a five-person rotation."
• Make one store do several jobs, with isolation inside it. Give each workload its own connection pool and statement timeout so an analytical query cannot starve the order path. That isolation is most of what the extra systems were buying.
• Use the database-as-queue pattern properly: SELECT ... FOR UPDATE SKIP LOCKED, in PostgreSQL since 9.5 (2016), makes a claim a row update, retries a column and the dead-letter queue a WHERE clause.
• Write the threshold that buys the next slot before the pressure arrives, as a measurement: a search engine when relevance fails on a labelled query set rather than when latency is disliked; a cache when read p95 misses target after indexes are fixed.

Industry example

The shape small teams converge on is one relational store carrying rows, a SKIP LOCKED job queue, full-text search and partitioned event tables, with the second slot in reserve. Teams writing this up since the mid-2010s consistently give an operational reason rather than a technical one: not that the specialised systems are worse at their jobs, but that the team cannot run them well. The counter-example teaches the same lesson - a streaming platform serving tens of millions of concurrent viewers has a dedicated system for nearly every job, because each one has a team and the standing attention is somebody's whole role. The budget is a claim about a specific team's capacity, not about technology, which is why copying a large organisation's component list is the error it prevents.

Failure scenarios

• The silent fourth system. A cache added for one endpoint, never in a design document, now holding state the application depends on and nobody on the rotation can restore.
• The unrehearsed restore. Backups exist for five systems and the drill has been run on one - and the restore that matters involves the store that is also the queue, where recovery means replaying work rather than reinstating rows.
• Blast radius concentration in the single-store design: one runaway query touches search, the queue and the order path at once, and five end-of-support calendars would have produced a near-continuous upgrade obligation instead.

Trade-offs

What it buys: diagnostic speed that compounds, one upgrade calendar, one restore drill, and a rotation whose members have seen this system's failure modes often enough to recognise them in minutes.

What it pays: a capability gap you have chosen. Full-text search in a relational store is adequate rather than good; a database-backed queue has no replay for many independent consumers; partitioned tables are not a time-series engine. You are trading capability for operability on purpose, and the honest design document says so.

When not to use it

When the constrained capability is the product. If relevance is what customers buy - a marketplace, a documentation search - the search engine is the first slot and something else collapses into the database. The budget sets a count, not which systems.

When the team has outgrown the derivation. Above roughly 15 engineers with two rotations the arithmetic gives a different number. Redo the calculation on headcount changes rather than treating it as a principle.

Interview question

Q: "A five-engineer team proposes five stateful systems and you tell them the budget is two. Justify the number, then tell me what happens when search quality genuinely is not good enough."

What a strong answer covers: deriving the number from standing attention against rotation size rather than asserting it; showing the single-store design with per-workload pools and statement timeouts as the isolation mechanism; and having written the relevance threshold in advance.

Quick check

Quiz: Why does a budget of two stateful systems follow from a five-person team? Each load-bearing system costs roughly 0.3 to 0.5 of an engineer continuously in upgrades, backup verification, restore rehearsal, headroom and runbooks, so five consumes half the team.

Flashcard: Which four jobs can one relational store absorb? - Transactional rows, a SKIP LOCKED work queue, full-text or trigram search, and partitioned time series retired by partition drop.