The large takeovers documented in this guide went around the login and through a recovery path: a help-desk reset, a synced second factor, a text message. This guide reconstructs how production systems and identity products design that path, what the postmortems and open bug trackers say went wrong, and which choices flip with the value of the account.
The problem without the technology's name: a person who can no longer prove who they are by the usual means must be able to get back in, and a stranger who claims to be that person must not.
Authentication gets the engineering attention: phishing-resistant keys, device binding,
risk scoring. Recovery gets a form. That asymmetry is the subject of this guide, and the
evidence for it is unusually consistent. When Okta described the pattern behind the 2023
casino intrusions, it did not describe a stolen password or a broken cryptographic check. It
described callers who tried to
convince service desk personnel to reset all Multi-factor Authentication (MFA) factors
for highly privileged users
(Okta, 2023-08-31).
When Retool described its own breach, the second factor had not been defeated; it had been
sitting in a place the attacker could reach by controlling one Google account
(Retool, 2023-09-13).
When Reddit explained a 2018 intrusion, the weak step was a text message
(Krebs, 2018-08-01).
Each of these is a recovery or enrolment path wearing the costume of a login.
I am calling the underlying principle the recovery ceiling: an account is exactly as
strong as the weakest path that can change what the account trusts, not as strong as the
path most users take. The sources do not use that phrase. The FIDO Alliance comes closest in
its 2019 recovery paper: The entire ecosystem is only as strong as the weakest link
(FIDO Alliance, 2019).
What this guide adds is the evidence for where that weakest link actually sits in production
systems, and the conditions under which each fix is worth its cost.
The tightest recovery policies in the corpus are not accidents of immaturity. GitHub
states that if you lose both your two-factor credentials and your recovery methods,
you have permanently lost access to your account
; Apple says its recovery wait
is fixed
and that contacting Apple Support can't help shorten the wait
. Both
have deliberately removed the human override, which is the exact component the 2023
intrusions exploited. The only published cost of that choice is Google's: take away the
easy channels and success falls from over 80% to 14%. Nobody has published what the strict
policy costs in permanently locked-out legitimate users.
Scope. This guide covers recovery and reset of consumer and workforce accounts: lost authenticators, forgotten credentials, and the support channel that backs them. It does not cover phishing-resistant login itself, session theft after login, or credential stuffing. It does not cover customer-identity verification for regulated onboarding beyond what NIST says about re-proofing. Machine credentials are the subject of an earlier dig on machine identity expiry and renewal.
No organisation publishes its recovery system whole. The shape below is assembled from standards, vendor documentation and the failures; the reconstruction is marked where it is mine.
Read across NIST, FIDO, GitHub, Apple, Google and Okta and the same six stages recur, in the same order, even where the products differ. First, a trigger: the user (or a support agent acting for them) states that they cannot authenticate. Second, a gate that decides which proofs are acceptable for this account. Third, the proof itself. Fourth, a hold, sometimes zero seconds and sometimes days. Fifth, the action, which in the better designs is rebinding an authenticator rather than opening a session. Sixth, the aftermath: notification, session revocation, audit, and reverting whatever a hijacker changed.
NIST SP 800-63B-4 is the closest thing to a public specification of the proof stage. It
lists four recovery classes (saved recovery codes, issued recovery codes, recovery
contacts, and repeated identity proofing) and sets concrete parameters: a saved code carries
at least 64 bits from an approved random bit generator
, is stored hashed, and is
replaced on use; issued codes expire in 10 minutes by text or voice, 24 hours by email, and
21 days by US postal mail; a service must allow at least two recovery addresses
(NIST SP 800-63B-4, Sec. 4.2).
The same document fixes the aftermath: In all cases, account recovery SHALL cause a
notification
to the subscriber, to at least two notification addresses where available.
That last requirement is the one most commonly missing from the failure write-ups below.
FIDO's 2019 guidance puts reduction of recovery before recovery: encourage every user to enrol a second authenticator so that a lost phone never reaches the recovery flow at all. Google's Advanced Protection is its named example, requiring two security keys.
Runs this way at: FIDO Alliance, GitHub (passkey, security key, recovery codes, in its recovery docs).
Apple's recovery takes several days or longer
, a confirmation email arrives within
72 hours, and Apple states the wait cannot be shortened by support. Google says an
Advanced Protection recovery takes a few days
. A delay is a control only if the
rightful owner learns of it in time, which is why NIST insists on notification.
Runs this way at: Apple, Google Advanced Protection.
The Keycloak reset flow is the counter-example: when OTP reset is switched off, a password reset in the same browser logs the user in with no OTP. The proposed fix flips a default so the reset ends at a fresh login. Recovery should change what the account trusts, then make the user authenticate against the new state.
Evidence: keycloak #12759, #39447.
Social and third-party recovery keeps recurring. Facebook published a Delegated Recovery specification for letting a user's account at another provider recover their account (the repository was archived in September 2020), and Google shipped recovery contacts in October 2025 for locked-out personal accounts.
Runs this way at: Facebook (archived), Google.
Microsoft and CISA both describe the help desk as a primary route. CISA: threat actors
use social engineering to convince IT helpdesk personnel to reset passwords and/or MFA
tokens
. Microsoft's recommendations include scoping help-desk privilege so it
excludes administrative operations over Global Administrators.
Evidence: CISA AA23-320A, Microsoft.
Google's measurement paper describes adding a remission step to its flow after seeing hijackers change settings and delete mail and contacts: restoring deleted content, removing added content, and resetting options. It also found users preferred content restoration as an optional last step.
Evidence: Bursztein et al., 2014.
What none of the public sources documents is how the stages are wired together inside a real service: where the gate's risk decision lives, whether the help-desk tooling calls the same API as the self-service flow, how the hold is stored and cancelled. Those are the parts the incidents say matter most, and the public record is silent. The open-source identity servers, Keycloak and Ory Kratos, are the only places the wiring is visible, and their issue trackers are as informative about recovery as any postmortem (section 4).
Five forks. For each: what the evidence shows organisations chose, what they turned down, and the condition that flips the answer.
visual verification.
The 2023 casino intrusions are the case against the human override, but they are weaker
evidence than they look. MGM's own filing gives a financial figure and a description of the
outcome; the details of the call come from secondary reporting. Caesars' filing says the
cause was a social engineering attack on an outsourced IT support vendor
(Caesars 8-K),
which tells you the override had been outsourced to people with no stake in the account.
Okta told TechTarget that Caesars was one of four victims it tracked between 29 July and 19
August 2023, with MGM a fifth, and that it was relying on partners for attribution
(TechTarget, 2023-09-19).
The corroborated claim is narrow: a help-desk-reset pattern was in use against several
organisations in the same weeks. The transferable decision is also narrow: if a human can
reset a factor, the strength of the factor is the strength of that human's verification
procedure.
not nearly as secure as we would hope.
single-factor-authenticationwithout administrators noticing and gave no central control to disable it.
| Decision | Chosen | Rejected | Because | Evidence |
|---|---|---|---|---|
| Human override | None, or via the same gate | Reset on caller facts | Help-desk resets were the entry in the 2023 campaign | CISA, 2023 |
| Proof channel | Codes, second device, contacts | Secret questions alone | Weaker than passwords; forgotten by about 40% | Bonneau et al., 2015 |
| Session after reset | Fresh sign-in | Auto-login | Reset link bypassed OTP | keycloak #12759 |
| Timing | Fixed wait plus notice | Instant email link | Email-only access disabled MFA on many sites | Amft et al., 2023 |
| Factor backup | Several device-bound | Synced under the login | Sync root became the real factor | Retool, 2023 |
Six entries in four failure classes. Two are incidents with published write-ups by the affected organisation, two are reported through filings and reporting, and two are defects still open in identity software.
The classes matter more than the entries. Class A, the human override: a person with reset power is persuaded. Class B, the recovery root: a second factor is stored under an account that is itself recoverable. Class C, the interceptable channel: the proof travels over something an attacker can read or redirect. Class D, the recovery that is also a login: the flow grants a session or counts as a factor, so it can be used to skip the controls it was meant to restore.
consistent patternof calls to IT service desks asking to reset every MFA factor of privileged users, followed by sign-in through anonymising proxies and use of compromised super-administrator accounts to grant privileges. CISA adds that early calls
learn what steps are neededbefore the reset call.
SMS-based authentication is not nearly as secure as we would hopeand did not say how the codes were obtained. Reddit's original post was not retrievable here, so this is as reported by Krebs.
some internal policies were not followed; the article reports no specific policy change.
does not treat them as the same factorand suggests requiring another backup method such as lookup codes.
No postmortem in the corpus describes a legitimate user permanently locked out by a strict policy, and none reports the support cost of a verified human recovery. GitHub and Apple tell you what they chose; neither publishes how many people it affects. The failure class that the chosen design trades for takeover is therefore unmeasured in public, and an organisation adopting the strict policy is running the first experiment.
Recovery rates and incident costs, each with context, date and source. The recovery-rate figures are the oldest and the most reusable.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Recovery success, SMS code | 81% | Users who supplied a phone number | 2014 | IMC '14 | |
| Recovery success, secondary email | 75% | Paper also gives 74.57% for email; about 5% of recovery mails bounced | 2014 | IMC '14 | |
| Recovery success, secret questions or manual review | 14% | Users with neither phone nor secondary email | 2014 | IMC '14 | |
| Secret answers not recalled | about 40% | English-speaking US users, millions of attempts | 2015 | WWW '15 | |
| Fake answers given to look "harder to guess" | 37% | Google survey | Of those who admitted faking answers | 2015 | WWW '15 |
| Manual hijacking rate | 9 per million users per day | Cases observed 2011 to 2014 | 2014 | IMC '14 | |
| Sites reviewed / hands-on tested | 1,303 / 71 | Academic study | Many allowed MFA bypass with only email access; counts not given in the abstract | 2023 | Amft et al. |
| Saved recovery code entropy | at least 64 bits | NIST | Hashed at rest, replaced on use | 2025 draft | SP 800-63B-4 |
| Issued code lifetimes | 10 min text, 24 h email, 21 days US post | NIST | Contiguous US postal; 30 days elsewhere | 2025 draft | SP 800-63B-4 |
| Apple recovery confirmation | within 72 h; wait "several days or longer" | Apple | Wait fixed, support cannot shorten | 2026 | Apple Support |
| MGM incident impact | about $100M; under $10M one-time | MGM Resorts | Adjusted Property EBITDAR, September 2023 | 2023 | 8-K |
| Retool customer accounts taken over | 27 | Retool | Cloud customers only | 2023 | Retool |
| Age of open Keycloak bypass issue | about 4 years 4 months | Keycloak | 27 June 2022 to 23 October 2026 (derived) | 2026 | #12759 |
What to plan against. Measured: Google's three channel success rates and the memorability of secret answers, all from the operator's own logs but all more than a decade old and from one consumer population. Specified: NIST's code sizes and lifetimes, which are requirements, not observations. Claimed: nothing here is a vendor performance claim, which is itself unusual for this topic. Derived: the age of the Keycloak issue is my arithmetic from the issue's creation date.
The unit economics, as far as they exist. Two variables dominate. The first is the lockout rate: the share of legitimate users per year who reach recovery and cannot complete it. Google's 14% figure applies only to users who gave no phone or secondary email, and no source says how many that is. The second is the loss per takeover, where MGM's roughly $100 million is the largest figure in the record and is a company-level cost, not a per-account one. An illustrative calculation, not a measurement: if a service with a million accounts loses 0.5% a year to unrecoverable lockout and the strict policy converts those to permanent losses, that is 5,000 users a year; whether that is cheaper than the expected takeover loss depends on a figure no source provides. The model excludes help-desk labour, insurance, and regulatory notification costs.
The three recovery-rate figures are Google 2014 and the secret-question study is 2015; mobile-first sign-in, passkeys and SIM-swap prevalence have all moved since. The 14% figure also mixes secret questions with manual review, so it cannot be read as the success rate of questions alone (the 2015 paper gives 44% to 61% depending on country for questions). The hijacking rate counts only manual hijacks Google detected.
Every source behind this page, graded. Filter by kind. Where a primary source was blocked from this environment, the card says so.
First-person incident report: SMS lure, fake portal, voice call, device enrolment in Okta, and the effect of Google Authenticator cloud sync on the second factor. 27 cloud customers affected.
Describes the service-desk MFA-reset pattern across customers, the attacker's use of super-administrator accounts and a second identity provider, and a long list of mitigations.
Open issue with maintainer discussion from 2022 to 2026, a workaround, and a proposed commit that makes forced re-authentication the default.
Umbrella issue: one flow resets every credential type, MFA reset adds a credential instead of replacing it, and lost-device users must use "Forgot password".
Report that recovery's email code and a second email code satisfy two-factor assurance. Open, with one reply suggesting an additional backup method.
A protocol letting an application delegate account recovery to an account at another provider. Archived read-only in September 2020; reference implementations exist, including a Ruby one built by GitHub.
Four recovery classes with code entropy, lifetimes, contact and address rules, and a mandatory notification on every recovery.
Two-step strategy: reduce recovery needs with multiple authenticators, and re-run identity proofing when that fails. Edited by staff from Yahoo Japan, Visa and Amazon.
States the approximate $100 million September impact and under $10 million one-time costs. Does not describe the intrusion method.
Attributes suspicious activity to a social-engineering attack on an outsourced IT support vendor; earliest event 7 September 2023.
Government advisory documenting help-desk password and MFA resets, SIM swapping, and the call-to-learn-the-process step. Revised July 2025.
Details resets of passwords and MFA by tricked administrators, SIM swaps, added MFA methods as persistence, and scoping advice for help-desk roles.
Relays Reddit's disclosure: SMS 2FA, a June 2018 intrusion at hosting providers, and the statement that SMS is not as secure as hoped. Reddit's own post was blocked from this environment.
Step-by-step account of the chained recovery attack, with Apple's response that internal policies were not followed.
Okta's statement that Caesars was one of four tracked victims between 29 July and 19 August and MGM a fifth, with attribution left to partners.
Reports Google's October 2025 launch of recovery contacts for locked-out personal accounts. The page gives no limits or lifetimes.
Measurement of manual hijacks, including channel success rates and the added remission step. Read from the paper PDF.
Millions of recovery attempts on personal knowledge questions: weak, forgotten, and falsified.
1,303 sites reviewed, 71 tested by hand; many allowed MFA to be disabled with only email access, and help pages often did not match behaviour.
Dean Saxe's "iron triangle" of access continuity, privacy and security for judging recovery mechanisms. Abstract only; no recording was reviewed.
Treats recovery as privileged access and lists post-reset cleanup, token lifecycle and rate limiting. Abstract only; no recording was reviewed.
Lists recovery codes, passkeys, security keys, a fallback number and verified devices; states support cannot restore a protected account.
Fixed waiting period, 72-hour confirmation, support unable to shorten.
States that Advanced Protection tightens account recovery and that verification takes a few days.
Seven rungs. The line between toy and real is rung 4, where a delay and an independent notification make the flow resistant to someone who passes the proof.
List every way a user, a support agent or another system can change a password, a factor or a contact address in your product, and what each requires.
Done when: you have a table of paths with the weakest one marked. Teaches: the recovery ceiling is a property of the table, not of your login page.
Generate codes with at least 64 bits, store them hashed, throttle verification, and issue a replacement when one is used.
Done when: a used code fails on replay and a new one is shown once. Teaches: a recovery code is a password you are not allowed to reuse.
Build the reset flow so it rebinds the credential and then requires full sign-in, including the second factor, from a clean session.
Done when: an automated test that resets in one browser tab and opens the app in the same browser lands on the login page. Teaches: the Keycloak bypass in miniature.
Delay the rebinding by a fixed window, send an alert to every registered address except the one that received the code, and include a cancel link that works without logging in.
Done when: a simulated attacker who passes the proof is stopped by a cancel from the owner. Teaches: a delay is only a control if someone is told.
Replace the reset button with a tool that issues a recovery request through the same gate, adds an out-of-band confirmation to the account holder's registered channel, and cannot act on administrator accounts.
Done when: no support action changes a factor without a gate record. Teaches: the human is a client of the recovery API, not a bypass of it.
For each second factor in your organisation, find where it is stored and which account can recover that store, including synced authenticator apps and password managers.
Done when: every factor has a named recovery root and an owner. Teaches: Retool in your own estate.
Run a vishing exercise against the help desk, an email-takeover exercise against the reset flow, and a legitimate-lockout drill; log success by path.
Done when: you can state the legitimate recovery rate and the attacker pass rate per path. Teaches: the trade-off in section 3, with your own data.
The queries that found the material above, so the search can be extended when this page goes stale. Vocabulary learned along the way is in the last block.
"service desk" "reset" MFA social engineering incident report 2023Retool postmortem Google Authenticator cloud sync account takeoverReddit 2018 security incident SMS two-factor intercepted"social engineering attack on an outsourced IT support vendor" 8-KMGM Resorts 8-K October 2023 cybersecurity issue Adjusted Property EBITDARAA23-320A Scattered Spider help desk reset"account recovery" Google Bursztein Bonneau secret questions SMS success ratemulti-factor authentication recovery deployments evaluation 1,303 websitesNIST 800-63B-4 account recovery saved recovery codes recovery contactsrepo:keycloak/keycloak is:issue reset credentials OTP bypassrepo:ory/kratos is:issue recovery AAL2"recovery flow" "account takeover" is:issue label:security"recovery contacts" OR "delegated recovery" OR "recovery key" account security"permanently lost access" 2FA recovery policy support cannot restore"remission" OR "post-reset" session revocation account hijacking recoveryThe queries that did not work are instructive. Searching for a named MGM or Twitter root-cause report returns only secondary coverage, because neither company published one. The NYDFS report on the 2020 Twitter intrusion would have been the best regulator-grade source here; its page returned 404 from this environment, so Twitter is absent from the incident list rather than cited second-hand.