By around 2011 Etsy was deploying to production roughly 25 times a day, having published its deployment tool (Deployinator) and its metrics collector (StatsD). There was no separate multi-day regression phase before a release. What replaced it, what did the approach cost, and where would copying it be a mistake?

The situation they were in

A regression phase exists to answer one question: is it safe to release? It answers it by batching changes, testing the batch, and releasing it. That works, and it has a structural property: the more changes in a batch, the harder it is to attribute a problem to one of them, and the longer the gap between writing code and learning whether it was right.

Etsy's move was to attack the batch size rather than the testing. At 25 deploys a day, a deploy contains roughly one change, which changes what evidence is needed to be confident in it.

What replaced the phase

Three things, and the order matters:

1. Measurement first. StatsD exists because the model requires that the effect of a change is visible within seconds. Counters and timers became cheap enough that engineers instrumented freely, and graphs were the acceptance criterion: deploy, watch the relevant lines, and know. This is the load-bearing piece — continuous deployment without immediate observation is just faster releasing.
2. Deploy as a routine act. A one-button tool used by everyone, many times a day, removes the ceremony that makes deployment rare and therefore dangerous. Frequency is what makes each deploy small, and smallness is what makes diagnosis easy.
3. Separating deploy from release. Code goes to production behind flags and is turned on for a fraction of traffic. The "test" of a risky change is a percentage ramp with the graphs open, which tests something a staging environment structurally cannot: real traffic, real data volumes, real user behaviour.

What it cost them

An engineering culture where every engineer is responsible for their change in production, which is a hiring and training cost, not a tooling one. A flag system that accumulates dead flags unless someone removes them. And a very high bar for the quality of metrics: the whole model rests on the graphs being trustworthy, so bad instrumentation is a safety problem rather than an inconvenience.

It also does not remove testing. It moves the emphasis to fast tests that run before the deploy and observation that runs after it, and gives up the kind of confidence that only a long exhaustive pass provides.

Where copying it would be a mistake

• When a change cannot be observed quickly. A batch pipeline whose correctness is visible tomorrow morning, a model retrained weekly, a data migration: the feedback loop the model depends on does not exist, and the regression phase is doing real work.
• When the consequence of a bad minute is not recoverable. Medical devices, payment ledgers, anything embedded on a customer's hardware, anything where a released artefact cannot be turned off. Etsy could roll forward in minutes; a firmware update cannot.
• When release requires an external gate. A regulated change process with documented approval is not removed by deploying more often; it is answered by separating deploy from release, so the approval gates the flag flip rather than the artefact.
• When you copy the frequency without the instrumentation. This is the common failure: an organisation adopts continuous deployment as a target number of deploys per day, without the graphs or the flags, and has simply removed its safety net. The deploys were the visible part; the measurement was the enabling part.

What a strong answer adds

That this is the same argument as small pull requests, told at the level of releases: the dominant variable in diagnosing a failure is how many things changed. Everything Etsy built — the deploy tool, the metrics collector, the flags — exists to drive that number towards one and make the consequence of a wrong one visible in seconds.

Common weak answers

• "They replaced testing with monitoring." They did not remove the fast tests that run before a deploy. They removed the batching, and added a second source of evidence after it.
• "Continuous deployment means no QA." The verification work moved into instrumentation, flag design and production analysis, which is more specialised rather than less.
• "We deploy 30 times a day too." Deploy count without per-change observability measures activity, not safety, and it is the metric most easily gamed by splitting one change into six.