A mobile regression suite fails about one run in five, and almost always on app launch rather than on an assertion. The team adopts a library that retries only the launch step instead of the whole test and the flakiness nearly disappears. What have they bought and when does the bill arrive?

What is gained

The gain is not "fewer red builds" - a blanket retry achieves that too. The gain is that the retry is scoped to the step that is environmentally unreliable, so the assertion still fails loudly.

Whole-test retry re-runs everything, including the check that would have caught a real defect. A test that fails intermittently because of a genuine race in the product passes on the second attempt and the build goes green, which is how a retry policy launders defects into production. Step-level retry keeps the laundering surface down to one named step.

eBay published a Swift package for exactly this, Targeted Auto Retry, which retries individual steps rather than tests. A step is declared as a main action, a success condition and an optional reset step; if the success condition fails, the main action is retried, and after repeated failures the whole test fails. eBay reports that mobile teams adopting it saw up to 99% of flakiness eliminated across their regression suites - a self-reported figure from their own engineering blog, and a plausible one, because app launch dominates mobile flake.

The arithmetic explains the urgency. At a 20% per-run failure rate, a team running the suite eight times a day sees it red more often than green, and above roughly a few percent a suite stops being a signal and starts training people to re-run.

What is paid

The retried step stops being measured. Launch reliability and launch duration were being sampled hundreds of times a day by this suite; after the change, a launch that fails half the time looks identical to one that never fails. If launch degrades in the product - a slow migration on first open, a flaky cold-start dependency - the suite is now the last place you would notice.

eBay's own post is explicit about the condition: use it only after exhausting attempts to fix the flakiness directly, and do not apply it where the step has a real intermittent bug, because the retry hides it.

When the bill arrives

Not at adoption. It arrives the first time a production incident traces to the behaviour inside a retried step, and the postmortem finds that the suite had been observing it successfully for months and reporting nothing.

Two mechanisms keep the option open:

• Emit retry counts as a metric and alert on the trend. A step retried 2% of the time is noise; the same step at 30% is a product regression. Without this number the policy is unfalsifiable.
• Run one scheduled job per day with retries disabled and treat its failure rate as the real flake measurement. That restores the signal you sold without restoring the interruptions.

When this is the wrong tool and the simpler alternative wins

When the flakiness is in your own test code, which is most of the time: a fixed sleep, a shared fixture, an assertion racing an animation. Fix it. A retry turns a 3-second problem into a 9-second problem and every test in the suite pays the latency, whereas the fix is usually one wait-for-condition.

Retry is the right tool for a step you do not own and cannot make deterministic - launching a simulator, acquiring a device, resolving a vendor's sandbox. The test for whether you qualify: can you name the mechanism that makes this step unreliable? If not, you are not scoping a retry, you are hiding a bug.