A payment platform is mid-incident: transactions are partially failing, the cause is unclear, and several teams are investigating. What structure makes the response effective?
Show the full answer Hide the answer
The structure
- A single incident commander who does not debug. Their job is coordination, decisions and keeping the response moving. The most common failure is that the most knowledgeable engineer becomes both commander and investigator, and then does neither well.
- A communications lead, separate, handling merchants, support, leadership and status pages. In a payments incident this is a substantial job on its own — merchants are making their own decisions based on what you tell them.
- Investigators, working specific hypotheses with explicit ownership, so two people are not chasing the same theory while a third is unexamined.
- A scribe or a shared timeline, because reconstructing what happened afterwards from memory produces a worse postmortem and, in a regulated payments business, an inadequate regulatory report.
The priority order that must be explicit
Mitigate, then diagnose. Restoring service takes precedence over understanding the cause, and teams routinely invert this because understanding is more satisfying. If a rollback, a failover to another provider, or disabling a feature restores service, do it before finding out why.
In payments there is a critical qualification: mitigation must not create financial ambiguity. Failing over mid-transaction, or restarting a service holding in-flight authorisations, can turn a service incident into a reconciliation incident that takes weeks to resolve. The mitigation must consider the state of money in flight, which is what distinguishes this domain from a general web outage.
The decisions that need to be pre-made
- Who can declare an incident — the answer must be "anyone", or declaration is delayed by hierarchy.
- Who can authorise a mitigation with a financial consequence — halting settlement, disabling a payment method, failing over to a more expensive provider.
- What the merchant communication says at each severity, drafted in advance, because writing it under pressure produces either silence or a statement that is later wrong.
- When to stop. An incident that is mitigated but not understood should be closed as an incident and continued as an investigation, or the response team stays engaged for hours after the customer impact ended.
The measure of a good response
Not the duration, which depends mostly on the nature of the failure. Time to mitigation, and whether the actions taken were reversible. A response that restored service in ten minutes by an irreversible action that caused a reconciliation problem is worse than one that took thirty minutes and left the ledger intact.