advanced 2 min answer

A brokerage must deploy without disrupting customers who may be mid-transaction. Which release strategy fits, and what does it require?

growwreleasecanarydrainingstateful
Show the full answer Hide the answer

The strategy

Rolling deployment with connection draining and progressive traffic shift, plus feature flags to separate deployment from release.

The essential properties:

  • Draining: an instance being replaced stops receiving new requests and completes in-flight ones before terminating. Without this, every deployment kills a proportion of in-flight transactions, which for a financial operation means an ambiguous outcome rather than a retryable error.
  • Backwards-compatible changes only, since two application versions run simultaneously by design during a rolling deployment. Every schema change goes through expand-and-contract; every API change is additive.
  • Deployment separated from release. Code is deployed dark and enabled by a flag, so the risky moment is a flag change that can be reverted in seconds rather than a deployment that takes minutes.
  • Progressive enablement with automated halt conditions, triggered by error rate, latency or a business metric rather than by a human noticing.

What "in-flight" means here and why it is harder

A web request completes in milliseconds. A financial operation may span an external call whose outcome is unknown, and terminating an instance mid-call leaves a payment or an order in an indeterminate state.

So draining must account for the operation's duration, not the request's — which means either waiting long enough, or having a durable record from which the operation resumes on another instance. The second is strictly better and is what durable workflow execution provides.

The scheduling constraint

A change freeze during market hours, which is not a branching strategy but a deployment window. It coexists with trunk-based development and continuous integration: changes merge continuously and deploy outside the window.

The freeze must have an exception process, since an urgent fix during the window is sometimes necessary and an absolute freeze produces either an unfixed problem or an unrecorded violation.

What canary analysis must look at

Not average latency, which hides everything. Error rate by endpoint, latency percentiles, and business metrics — order success rate, payment completion rate — segmented by customer tier and by region.

A canary that looks healthy on averages while failing a small but important segment is the specific failure this analysis exists to catch.