advanced 2 min answer

Every release needs a two-day manual regression pass by four people, the team ships fortnightly, and leadership wants weekly releases. Walk me through where you start.

regressionrelease-strategyrollbackrisk-based-testingjetbrains
Show the full answer Hide the answer

What the interviewer is testing

Whether you automate the ritual or replace it. The weak move is to automate the 400-step script, which encodes the manual process, costs a year, and produces a slow suite that fails for the same reasons the manual pass did. The strong move is to ask what the pass is actually buying and to buy it more cheaply.

The clarifying questions that change the answer

  • "What did the last ten regression passes find?" If they found four defects and three were in the same two flows, you have your automation backlog and the evidence that the rest is ceremony.
  • "How long does a rollback take, and when was one last performed?" This usually reframes the problem entirely.
  • "What is the cost of a defect reaching production for one hour?" A payments platform and an internal reporting tool need different answers.
  • "Is the manual pass finding regressions, or finding that environments differ?"

A strong answer's arc

  1. Measure the yield. Two days by four people is 64 person-hours per release; fortnightly, that is most of an engineer. State it in those terms, because the conversation is about cost.
  2. Fix the reverse before the forward. A release that can be rolled back in five minutes needs far less pre-release certainty than one that cannot. Feature flags, a versioned schema with expand-and-contract migrations, and a rehearsed rollback usually buy more risk reduction per week of effort than any test suite.
  3. Automate the top flows only. Twenty to thirty journey tests typically cover the flows where the manual pass found anything, and they can run in minutes rather than days.
  4. Move the remainder to risk-based sampling, rotating which areas are examined by a human each release, so coverage is broad over time without gating every release.
  5. Catch the rest after release: synthetic journeys on production, a staged rollout with guardrail metrics, and an alert on the business outcome rather than the CPU.
  6. Retire the script deliberately. Each automated flow removes steps from the manual document, and the document shrinking is the measure of progress.

Common weak answers

  • "Record and replay the manual script." Produces the most brittle suite in the industry, because recorded selectors and timings encode a session rather than an intent.
  • "Hire more QA engineers." Doubles the cost of the constraint.
  • "Mandate 80% coverage." Unrelated to the failure mode, and it converts a delivery problem into a metric-gaming problem.

What a strong answer adds

The organisational layer. The regression pass exists because someone was burnt and nobody has been given permission to stop doing it, so the plan needs a named risk owner, a defined trial (one release with the reduced pass plus production verification), and a rollback for the change itself. The build infrastructure matters too: a team in the mould of JetBrains, shipping tooling used by other developers, will need a longer compatibility matrix than a web product, and the strategy should say which risks it is deliberately not covering.