Contention Blind Spot

A payments change passes in its per-PR environment, passes staging, and twenty minutes after the production deploy checkout p99 goes from 300 ms to 9 s with lock waits piling up on one table. The environment ran the same image and the same migration. Nothing was skipped.

The change moved a row lock earlier in the transaction so it was held across a network call. With one request in flight that is invisible: the lock is held for milliseconds and released. With forty concurrent workers on a narrow key range, the wait queue grows, each waiter holds a connection, the pool saturates, and everything sharing that pool degrades. Correctness never changed; the duration of a critical section did.

This is not a gap in test coverage. It is a gap in what the environment is capable of showing. A single-user environment with a few hundred seeded rows has no second transaction to contend with, no key skew to concentrate contention, and no table large enough for the query planner to change its mind.

Why it matters

Ephemeral per-change environments have become the standard answer to shared-staging queues and drift, and they are a genuine improvement. They also reproduce exactly the parity axes that are easy to copy — image, schema, configuration — and score close to zero on the two axes where the expensive defects live: concurrency and data shape. A team that treats a green ephemeral pipeline as evidence about production behaviour has made a category error, and the bill arrives as a latency incident twenty minutes after a deploy rather than as a failed test.

The economics favour closing it: the defects in this class are among the most expensive to diagnose in production, because the symptom (p99 latency) is several layers from the cause (a lengthened critical section) and the change that caused it looks innocuous in review.

Implementation patterns

• A production-shaped data subset. Not full volume: anonymised data that keeps row counts within an order of magnitude on the tables that matter and preserves skew in the top keys. The hot key is the entire point, so a uniform synthetic dataset reproduces the blind spot faithfully.
• A 60-second concurrency smoke test at 10 to 20 requests per second against the ephemeral environment, with pass conditions on lock wait time, pool utilisation and p99 rather than on correctness. It costs a minute of pipeline time and it is the only check that can fail for this reason.
• Assert on critical-section duration. Instrument transaction and lock-hold times in tests, and fail when a hot path's hold time regresses by more than a set factor. This catches the mechanism even at low concurrency.
• Keep one long-lived environment with production-scale data for the cases a subset cannot represent — a nightly job over a full table, a plan regression that needs real statistics.
• Shadow real traffic at a small percentage against the new version for the services where this class of defect is business-critical, which is the only method that reproduces real key distributions.

Industry example

The public record for this is generic rather than branded, which is itself informative: incidents of this shape are reported as "slow after deploy" and rarely written up, because the postmortem action is usually "add a load test" and the environment's structural limitation goes unnamed. The recurring production pattern is a change that adds work inside an existing transaction — an added call, a retry, an extra query, a new index maintained on write — and passes every functional check because the assertions are about outcomes. The lesson holds regardless of stack: concurrency is a property of the environment, and an environment with one user has set it to one.

Failure scenarios

• The lengthened critical section, as above: latency collapse under normal load minutes after a deploy.
• Connection-pool exhaustion, where a slightly slower query multiplied by concurrency consumes the pool and unrelated endpoints on the same pool fail first, sending responders to the wrong service.
• A plan flip. A query that used an index on 200 rows scans a 40M-row table in production, and nothing in the pipeline ever compiled it against real statistics.
• Races and double-processing, where two workers handle the same item because the environment only ever ran one worker.
• False confidence in the fix. The hotfix is verified in the same blind environment, so a regression in the same class ships again.

Trade-offs

Adding a data subset and a concurrency check costs pipeline minutes, a data-anonymisation job to maintain, and a new class of flaky failure to triage: a 15-requests-per-second check on shared infrastructure will occasionally fail for reasons unrelated to the change. Teams that will not fund the triage should not add the gate, because a check that is routinely re-run without investigation is worse than none — it teaches the team to ignore the one signal that can see this defect.

When not to use it

If the service has no shared mutable state — a stateless transformation, a read-only API in front of a cache — concurrency parity buys little, and correctness tests plus a production canary are enough. Similarly, for very low-traffic internal tools, production concurrency is one, so the environment is already faithful. Invest here when a service has a hot table or a shared pool and a deploy cadence fast enough that a canary alone would let a latency regression reach everyone before anyone looks.

Interview question

Q: A change passes its per-PR environment and staging and causes a latency incident in production twenty minutes after deploy. Leadership asks you to make staging more production-like. What do you say?

What a strong answer covers: naming the missing axes (concurrency and data shape) rather than accepting "more production-like"; explaining why a bigger shared staging environment reintroduces queues and drift while still running one user at a time; the two cheap additions and where they run; the production alert that attributes faster than p99 (lock wait time and pool saturation, alerted on rate of change after a deploy); and the limit of the approach, since a subset cannot reproduce every plan regression.

Quick check

Quiz: Why can a per-PR environment not catch a lock-contention regression? — Because the defect needs concurrent transactions and skewed data to appear at all, and a one-user environment with seeded rows supplies neither.

Flashcard: What two axes of parity do ephemeral environments systematically miss, and what closes them cheaply? — Concurrency and data shape; a production-shaped data subset plus a 60-second smoke test at 10 to 20 requests per second asserting lock waits and pool use.