intermediate 3 min answer

A marketplace in Flipkart's mould freezes all deployments for its six-day annual sale. On day three a pricing defect is discovered that under-charges a category by about 8%, costing a material amount per hour. There is no pre-agreed exception path. Walk through what you do now, and what you change before next year's sale.

flipkartdeployment freezechange managementflash salerollback
Show the full answer Hide the answer

What is being tested

Whether you understand what a freeze actually is. A freeze is a bet that no change will be needed during the window, and the bet is being lost right now. The useful question is not whether to break the freeze but which mechanism has the smallest risk, and that depends on what was built before the freeze began.

The reasoning

Rank the available actions by blast radius, not by how much they feel like a deployment.

1. Is the price a configuration value or a flag? If the pricing rule is data, changing data is not a deployment and may not be covered by the freeze at all. This is the first question because it is the cheapest answer and it is frequently available and forgotten.

2. Can the category be disabled rather than fixed? Removing the affected category from the sale stops the loss immediately, costs revenue on that category, and risks nothing else. For a quantifiable loss per hour, the comparison is arithmetic, and a reversible action that stops the bleeding while a fix is prepared is usually right.

3. If code must ship, ship the narrowest possible change through the narrowest possible path. One service, one line, a canary at 1% of traffic with a defined abort, and a rollback rehearsed before the deploy. The risk of a deployment during peak is real and it is mostly a function of scope.

The call

Disable the category first, then fix properly with a narrow canary. It stops the loss within minutes with near-zero risk, and it converts an emergency deployment into a planned one. If the category is too large a share of sale revenue to disable, then the narrow canary is the answer and the loss continues for the 30 to 60 minutes it takes to do it carefully.

What it costs

Breaking a freeze has a cost beyond the technical risk: it is precedent. The next person who wants to ship during a freeze will cite this, and the freeze's authority depends on exceptions being rare and visible. So the exception must be recorded with the reason, the authoriser and the measurement that justified it.

What to change before next year

A rehearsed freeze exception path, which is the deliverable:

  • Scope defined in advance. Which services may ship during a freeze, with what maximum diff size, through which pipeline. "One service, one canary step, automatic abort" is a specification.
  • A named authoriser reachable during the window, with a deputy.
  • Rehearsed before the window opens. Run the exception path on a no-op change the week before the sale. An exception path that has never been executed is a plan, not a capability, and the freeze window is the worst possible time to discover its first bug.
  • Flags and pricing as data, so the most likely emergency needs no deployment at all.
  • A kill switch per category and per feature, tested, because disabling is almost always the lowest-risk first move.

When this is the wrong answer

When the freeze is externally imposed and breaking it has consequences beyond the technical. A regulator-approved change window, a contractual commitment to a partner, or an exchange's trading calendar are not engineering decisions, and the answer there is to disable, degrade and absorb the loss until the window closes. Know which kind of freeze you are in before you touch anything.

Common weak answers

  • "Freezes are an anti-pattern, ship continuously." True as a long-term position and useless today. It also ignores why the freeze exists: during the single highest-revenue week, the expected cost of a bad change is at its maximum.
  • "Get approval and deploy the fix." Skips the cheaper, safer options and treats a deployment as the only instrument available.
  • "Wait until the sale ends." A defensible answer only once the hourly loss has been compared against the risk of the narrow change, with numbers. Said without that comparison it is avoidance.