Every system that stores a username, an email address, a package name or a file path has quietly chosen an equivalence relation: which strings count as the same name. This guide reconstructs that choice from Spotify's 2013 account hijack, GitHub's and Django's 2019 password-reset collisions, two Git CVEs seven years apart, a pair of CPython CVEs, the npm typosquatting record and the standards that were rewritten afterwards. A reader leaves able to specify the equality contract for an identifier, test it, and say where in a system it has to be enforced.
The problem, stated without any technology's name: two pieces of a system must agree on whether two strings refer to the same thing, and they usually agree only by accident.
Why this topic, and why now: in August 2025 Ryan Barnett stood on the Black Hat USA stage and demonstrated, again, filters bypassed because a check ran before a normalization step (Barnett, 2025). Jonathan Birch had shown the same antipattern against URL parsers at the same conference six years earlier (Birch, 2019), and Spotify had written the definitive postmortem six years before that. A class of bug that keeps reappearing for twelve years is not a knowledge gap. It is an architecture gap: the decision about what counts as the same name is made implicitly, in several places, by library defaults, and nobody owns it.
The scope here is identifier equality: account names, email addresses used as identifiers, package and module names, repository paths and hostnames. The guide covers how that relation is chosen, stored, enforced and changed. It deliberately does not cover full-text search collation, display-only text rendering, or internationalised email transport (SMTPUTF8), each of which has different constraints and a different failure record.
The fix the standards bodies eventually wrote down is the workaround Spotify shipped in an
afternoon in 2013. Spotify stopped trusting its canonicalizer and refused any username
X unless X equalled canonical_username(X). Four years later the
IETF's nickname profile, RFC 8266, told every implementer that its own rules are "not an
idempotent procedure for all code points" and that they must re-apply them until the output
stops changing, rejecting the input if it has not settled after three extra passes. Go's
implementation runs the pipeline four times
(x/text precis).
The standard did not make canonicalization safe; it made the fixed-point check mandatory.
The second surprise is how much of the defensive code in the corpus is an emulation of
somebody else's equality. Git ships a hand-written list of the sixteen code points macOS's
HFS+ silently ignores, and the hashed NTFS 8.3 short name of .gitmodules
(gi7eba), because the repository and the filesystem it is checked out onto
disagree about which paths are the same file
(utf8.c,
path.c).
Read the incidents side by side and every one of them is the same shape. A name moves through several stages (typed by a person, checked at registration, stored, looked up, acted on, handed to something downstream such as a filesystem, a resolver or a mail server), and at least two of those stages decide sameness differently. I am calling this equality skew: the sources call it a canonicalization bug, a case-mapping collision, a normalization antipattern, typosquatting and CWE-180, but the mechanism is the same, and naming it is what lets an architect look for it on a diagram before it ships.
Who has solved it, as opposed to patched it: the clearest production designs in the record are the Go module system, which refuses to let any storage layer decide case (x/mod); the Python packaging index, whose equality is one ASCII regex specified in PEP 503 and reused verbatim in PEP 685; crates.io, which widened equality to fold hyphen and underscore (RFC 940); Discord, which split one identifier into a strict handle and a free display name in 2023; and Chrome's IDN display policy, which is the only system in the corpus that decides sameness the way a human eye does.
The common shape across the systems that stopped having this class of incident, with the points where they diverge.
Laid side by side, the systems that fixed the problem converge on six components. Not one of
them got there by choosing a better normalization form. They got there by separating the
jobs that a single lower() call had been quietly doing at once: deciding which
characters are allowed, deciding which strings are the same key, deciding which strings are
too similar to coexist, and deciding which representation an action is performed on.
The cheapest equality function is the one that has very little to compare. PEP 503 names
may only use ASCII letters, digits and .-_; Go module paths are ASCII-only "today",
and the x/mod source lists the Unicode questions (Kelvin sign, combining marks) it has not
answered yet as the reason. Discord's 2023 handles are lowercase letters, digits, period
and underscore. Spotify's 2013 design accepted Unicode it could not canonicalize, which is
the failure in one sentence.
One function computes the key, it is pinned to a Unicode version, and its output is tested for idempotence at write time. Twisted's nodeprep still declares "We require Unicode version 3.2" in the source; RFC 8266 makes the repeat-until-stable loop normative; Spotify made X == canon(X) a registration rule. The order inside the function matters: Django's comparison normalizes before it folds, per UTR #36.
Equality is enforced by the store, not by application code that remembers to call the
normalizer. crates.io compares names case-insensitively so RUSTC-SERIALIZE cannot
be uploaded beside rustc-serialize, and RFC 940 extended the same key to fold
- and _. PEP 503 makes the normalized name the URL itself, and tells
clients they "MUST request the normalized URL" rather than rely on redirects.
A second relation, broader than identity, blocks registrations that a human would confuse. npm refuses names that differ from an existing one "in punctuation only"; Chrome compares a hostname's UTS #39 skeleton against top domains and shows punycode on a hit. The gate's output is always "no", never "these are the same account". Merging on a fuzzy relation is how Spotify and GitHub lost accounts.
After a match, everything downstream uses the stored value. Django's 2019 fix says it plainly: the token "will be sent to the email address on record rather than the submitted address". The secondary accounts of GitHub's fix describe the same move. This one rule neutralises an entire class of collision even when the comparison is still too loose.
When a name is handed to a system with its own equality (a filesystem, a resolver), the
producer must reject anything the consumer would alias. Git emulates HFS+ and NTFS on
every platform at fsck and push time; CPython's _checknetloc runs
the IDNA normalization in advance to see whether a separator would appear; Go escapes
uppercase as !a so no case-folding store can merge two modules.
Where the systems diverge is on how much of the human relation to bring into the machine.
Chrome brings almost all of it, because its users read hostnames with their eyes and have
nothing else to go on. PyPI and crates.io bring a sliver (case and separators). Go brings
none: it would rather force github.com/!azure/azure-sdk-for-go onto disk than
let a filesystem guess. The divergence is not taste. It tracks who reads the identifier:
machines that copy and paste can live with a strict relation; people who retype from memory
cannot, which is exactly the reason Discord gives for retiring discriminators and casing.
One more component is missing from almost every system and present in none of the published designs explicitly: a version number on the canonicalizer. Twisted pins Unicode 3.2; Python's standard library on this machine reports Unicode 14.0.0; the confusables file grew by 210 mappings between Unicode 16 and 17. Neither account says so, but a canonical key computed under one table and compared under another is equality skew across time rather than across components, and the only defence is to store which version produced each key so that a re-canonicalization can be planned rather than discovered.
Six forks, each taken differently by at least two systems in the record.
canonical_username column.get_users still does over a candidate set..GIT and git~1, PEP 685's "MUST raise an error" when two extras normalize alike./Foobar/ to /foobar/), merging is acceptable only when the target is resolved through the strict key and PEP 503's rule holds: clients must not depend on it.core.protectHFS, core.protectNTFS).| Decision | Chosen | Rejected | Because | Flips when | Evidence |
|---|---|---|---|---|---|
| Repertoire | ASCII for machine identifiers | Unicode with stringprep | Case folding is not a pairwise relation outside ASCII | Users type names in their own script | Go x/mod |
| Where equality lives | Stored key, unique index | Compare-time normalization only | Store enforces it for every code path | Relation will change and cannot be backfilled | RFC 940 |
| Near-collisions | Refuse the newcomer | Merge into the existing account | A merge on a wide relation is an identity transfer | Never, for security decisions | npm 2017 |
| Action target | Record found by lookup | Raw submitted input | Neutralises collisions even with a loose comparator | No credible case found | Django 3.0.1 |
| Downstream equality | Model strictest consumer | Trust the platform | The checker and the filesystem disagree | Names never leave the minting system | Git 2.2.1 |
| Changing the relation | Normalize and migrate existing names | Apply only to new names | Grandfathered names reopen the gap | Old names are frozen and read-only | Checkmarx 2023 |
The last row needs the argument shown, because the sources disagree. npm in December 2017 chose to apply the lowercase rule only to new names, which is cheap and breaks nobody; Checkmarx found in May 2023 that the 3,815 grandfathered uppercase names had left 1,900 packages open to a lowercase twin until npm patched it. PEP 685 took a middle route for extras: writers must emit normalized names from now on, readers "SHOULD warn" on legacy ones, and the authors measured the blast radius first (73 clashing extra names across PyPI, three among valid names) before deciding grandfathering was safe. Discord took the expensive route and began moving every account to a new handle. The context that explains the split is whether the old names are still writable. npm's were, and that is why the gap was exploitable; PEP 685's only change is how tools read existing metadata, which is why measuring first was enough.
Ten published incidents, grouped by which pair of stages disagreed. The design rule in each card is the part to carry into a review.
The narrowest and most instructive class. One function, applied twice, gives two answers. Nothing else in the system needs to be wrong.
nodeprep.prepare was idempotent, like lower(), so it could be reapplied safely as a name moved between services.The commonest class, and the one with the cleanest fix. A lookup matches on a loose relation, then the action is carried out on the raw input, which is not the thing that matched.
ı compared equal to the victim's after case conversion ("ı".upper() is "I"), and the reset token was mailed to the typed address, which the attacker controlled (per secondary accounts, at a mail domain accepting that character).The name is fine where it was checked, and becomes something else when a filesystem, a URL resolver or a mail server applies its own folding.
.Git/config, legal on Linux, "would have clobbered .git/config" when checked out on a case-insensitive filesystem. HFS+ also ignores zero-width joiners and bidi controls (.git), and NTFS maps git~1 to .git..git/config on checkout of a hostile repository (CVE-2014-9390); the release note describes only the clobbering, and the severity follows from config being able to name commands Git runs (my inference). Found by the Mercurial maintainers (Matt Mackall, Augie Fackler), whose tool shared the flaw.fsck and on push "on all platforms" so hosting servers on Linux refuse them too. Git hard-codes the 16 HFS+-ignored code points and the hashed NTFS short names, and its own comment calls the emulation "not complete by any means".urlsplit could find the host by splitting on ASCII separators, independent of what IDNA would later do to the string.a/c, so a netloc that contains no slash when parsed gains one when resolved. Reported by Jonathan Birch (Microsoft) and Panayiotis Panayiotou on 16 February 2019 as CVE-2019-9636. The first fix skipped the userinfo before @; Red Hat's Riccardo Schirone reported the bypass on 3 June 2019 as CVE-2019-10160.urlsplit before fetching it; fixes in five branches, twice._checknetloc pre-applies NFKC ("IDNA uses NFKC equivalence, so normalize for this check") and raises if any of /?#@: appears.Here every component agrees, and the identifier is still wrong, because the person reading it uses a wider relation than any code does, or because the relation was changed while old names stayed live.
crossenv (for cross-env) posted environment variables to an attacker host.xn--80ak6aa92e.com, all Cyrillic, which Chrome, Firefox and Opera displayed as apple.com. A single-script label passed the mixed-script filter.network.IDN_show_punycode.Read across the four classes and one pattern dominates. Not one of the ten fixes was "use a better normalization form". Django did change its comparator, but it also stopped trusting it. The structural fixes were all about placement: check the fixed point at write time (Spotify, RFC 8266), act on the record (Django, GitHub), enforce at the point of entry to the shared namespace (Git's push-time check, npm's registration gate), and treat a rule change as a migration over existing keys (npm 2023, PEP 685). An architect reviewing a design should ask where each of those four placements is, not which Unicode form the team picked.
The detection gap deserves its own sentence. Spotify learned of its bug from a forum post; GitHub from a bounty hunter; Git from a competing project's maintainers; npm from a developer's tweet; the uppercase gap from a vendor's research blog six years after it opened. No account in this corpus describes equality skew being found by monitoring. That is an Open finding in the sense the evidence rules use: nobody has published a production detector for it, and a reader should assume their own system would also learn about it from the outside.
There is no cost curve for this topic; the planning numbers are sizes of the relation, sizes of the exposure, and how long fixes took. Each is dated.
| Metric | Value | At | Context | As of | Source |
|---|---|---|---|---|---|
| Code points where NFKC(lower(x)) is not idempotent | 629 | Derived | Map-then-normalize order, as in stringprep profiles; Python 3.11.17, Unicode 14.0.0; single code points only | 2026-10 | Script in section 7 |
| Code points where NFKC(casefold(x)) is not idempotent | 637 | Derived | Same run; folding first does not help | 2026-10 | Script in section 7 |
| Code points where casefold(NFKC(x)) is not idempotent | 0 | Derived | Django's UTR #36 order; single code points only, strings not exhaustively tested | 2026-10 | Script in section 7 |
| Non-ASCII code points that fold to one ASCII letter or digit | 998 | Derived | casefold(NFKC(x)); each is a candidate alias for an ASCII identifier | 2026-10 | Script in section 7 |
| UTS #39 confusable mappings | 6,355 / 6,565 / 6,712 | Unicode | Versions 16.0.0, 17.0.0, 18.0.0 development file; the visual relation grows every year | 2024 to 2026-08 | confusables.txt |
| Fixed-point limit | 3 extra passes | IETF | Then reject; Go runs 4 passes total | 2017-10 | RFC 8266 via x/text |
| npm names with uppercase, grandfathered | 3,815 | npm | Of which 1,900 could be impersonated by a lowercase twin | 2023-05 | Checkmarx |
| Typosquat packages removed | about 40 | npm | crossenv: about 700 downloads, at most 50 real installs, about 2 weeks live | 2017-08 | npm |
| Extra-name clashes under PEP 503 normalization | 73 (3 valid) | PyPI | Across all extras on PyPI, measured before standardizing | 2022-03 | PEP 685 |
| Friend requests failing on discriminator or casing | almost half | Discord | Company's stated reason for strict lowercase handles | 2023-05 | Discord |
| Time between the first and second CPython fix | about 3.5 months | CPython | 2019-02-16 report to 2019-06-03 bypass report; five branches patched each time | 2019 | PSRT database |
| Git release lines patched for CVE-2021-21300 | 14 | Git | v2.17.6 to v2.29.3 plus v2.30.2 | 2021-03 | Git notes |
| HFS+ ignorable code points Git emulates | 16 | Git | Hard-coded in next_hfs_char, self-described as incomplete | 2026-10 | utf8.c |
| Languages with working Trojan Source exploits | 12 | Cambridge | 19 organisations notified | 2021 to 2023 | USENIX 2023 |
The four derived rows are mine, computed on this machine's Python, and they are bounded: they test each code point alone, not sequences, so they undercount strings that only misbehave in combination (RFC 8266's diaeresis-plus-space case is one). They will change with the Unicode version; rerun the script against the interpreter your service actually uses. Checkmarx's figures are a vendor's research and come from secondary retrieval; the npm counts differ by three between sources. No source publishes the operating cost of running a skeleton gate or a fixed-point check, and I have not estimated one: both are string operations at registration time, and the published systems treat them as free.
What to plan against: if your identifiers accept anything beyond ASCII, the relation you are choosing has on the order of a thousand aliases for your ASCII namespace alone, and about six hundred inputs on which the most common pipeline order disagrees with itself. Those are the test cases. If they do not appear in the test suite, the design has not been tested.
Every source behind this page, graded. Filter by kind. GitHub-hosted sources were fetched directly; the rest came through search retrieval, as the ledger notes.
The canonical write-up of a non-idempotent canonicalizer: one reset, three accounts, and a fix by fixed-point check at registration.
Case-mapping collision in password reset; the fix changes the comparator and, more importantly, mails the address on record.
Researcher's disclosure of the same class at GitHub; details here rest on secondary summaries because the page was unreachable from the research sandbox.
About 40 lookalike packages removed; crossenv exfiltrated environment variables for two weeks before a developer noticed.
.Git, git~1 and .git all alias .git on someone's filesystem; Git now checks on push on every platform.
Sixteen hard-coded code points that HFS+ ignores, with the maintainers' own note that the emulation is incomplete.
NTFS trailing dots and spaces, 8.3 short names up to ~4, and hashed short-name prefixes such as gi7eba for .gitmodules.
Case-insensitive filesystems plus symlinks plus delayed checkout gave RCE on clone, seven years after the 2014 fix.
CVE-2019-9636 and its bypass CVE-2019-10160: a host check that had to emulate IDNA's NFKC and got the emulation wrong once.
The shipped fix: normalize with NFKC first, then refuse if a separator appears.
NFKC then casefold, per UTR #36; the reset mail reads the stored address.
The library at the centre of the Spotify incident: map, then NFKC, then checks, pinned to Unicode 3.2.
Quotes RFC 8266's admission of non-idempotence and implements the fixed-point loop.
Go encodes uppercase as !a so no case-insensitive store can merge modules, and keeps Unicode out until it can define normalization.
6,565 visual-confusable mappings, up from 6,355; the 18.0.0 development file has 6,712.
The standard that makes the repeat-until-stable rule normative and caps it at three extra passes.
One regex defines package identity; clients must request the normalized URL.
Seven years after PEP 503, extras still had no equality; the fix reuses PEP 503, errors on collisions and measured 73 clashes first.
crates.io identity folds case and, after this RFC, hyphen versus underscore; the compiler sees a separate crate name.
Script-mixing rules, whole-script confusables and a skeleton match against top domains decide when to show punycode.
Compiler and reviewer disagree about token identity via bidi controls and homoglyphs; 12 languages, 19 organisations notified.
Builds a homoglyph database (SimChar) beyond Unicode's confusables list and measures IDN homographs in the wild.
Black Hat USA 2019. URLs that parse as one host and resolve as another; reported in Edge, .NET, Python, Java, Office 365 and Gmail.
Black Hat USA 2025. The same validate-before-canonicalize class against WAFs, six years after HostSplit.
Names that differ from an existing one in punctuation only are refused; the gate rejects, it does not merge.
3,815 grandfathered uppercase names; 1,900 open to a lowercase twin. Vendor research.
Independent account of the grandfathering gap and npm's patch.
Independent account of crossenv with slightly different package counts.
Split one identifier into a strict lowercase handle and a free display name, citing failed lookups on casing and discriminators.
All-Cyrillic apple.com shown as Latin in three browsers because the filter only caught mixed scripts.
Secondary account of the GitHub dotless-i collision; disagrees with others on upper- versus lower-casing.
Six rungs. The first two take an evening; the line from toy to real is crossed at rung four.
Write canon = lambda s: unicodedata.normalize("NFKC", s.lower()) and apply it
to ᴮᴵᴳᴮᴵᴿᴰ once and twice. Then loop over every code point and count where
canon(canon(c)) != canon(c); swap the order to NFKC-then-casefold and count again.
Done when: you get BIGBIRD then bigbird, and two counts that differ (629 versus 0 on Python 3.11). Teaches: order inside the canonicalizer is a security property.
For one identifier in your own system, write a one-page spec: allowed repertoire, the canonical function and its Unicode version, the wider registration relation, and which downstream systems receive the name.
Done when: a reviewer can answer "are these two strings the same account?" from the page without reading code. Teaches: the relation is a decision, not a library default.
Add a canonical-key column with a unique index and a canon_version column.
Reject at write if the fixed point fails. Backfill existing rows and list the collisions
before you enforce, as PEP 685 did.
Done when: inserting Foo_Bar beside foo-bar (or your relation's equivalent) fails at the database. Teaches: equality enforced by convention is not enforced.
Audit every flow that matches on a name and then acts: password reset, invitations, ownership transfer, webhooks. Each must take its destination from the matched row, never from the request.
Done when: a test submits victim@gıthub.com and asserts the token goes to the stored address or nowhere. Teaches: the cheapest fix in the corpus closes the commonest class.
Compute a UTS #39 skeleton (or an npm-style punctuation-and-case-stripped form) at registration and refuse on collision with an existing skeleton. Store the confusables data version beside it.
Done when: registering a Cyrillic lookalike of an existing name is refused, and a new confusables release produces a report of newly colliding names rather than silence. Teaches: human equality is a second relation with its own lifecycle.
For each place a name is written (object keys, file paths, URLs, mail), build a test matrix that materializes names on a case-insensitive filesystem (a macOS or Windows CI runner) and through your URL parser, and asserts no two accepted names alias.
Done when: the matrix fails on .Git-style and U+2100-style inputs before your validator is fixed, and passes after. Teaches: your equality is the union of your consumers'.
The queries that found the material above. The vocabulary that unlocked this topic was "idempotent", "case mapping collision", "fixed point" and "skeleton".
"case mapping collision" password resetcanonicalization "not idempotent" account takeover"compared equal" "lower-cased" CVE password resettyposquatting registry postmortem "differ only" punctuation OR caserepo:git/git is_hfs_dotgit OR is_ntfs_dotgit"apply the rules repeatedly until the output string is stable""normalized name" PEP comparison MUST raise an error"case-fold equivalent" "Kelvin" module pathBlack Hat Unicode normalization "validate before canonicalize"IDN homograph skeleton UTS 39 measurement paper"Trojan Source" homoglyph identifier compiler warning"Turkish locale" toLowerCase bug site:github.com is:issue"case-insensitive" collision "unique index" migration usernameNFKC casefold identifier comparison UTR 36 2.11.2