A release goes out and checkout fails for users in one market only - Turkey. The code path is identical for everyone and no market-specific logic was changed. The error is a failed comparison against a string constant. What happened?
Show the full answer Hide the answer
The trigger
Somewhere in the path, a string is normalised for comparison using a locale-sensitive case conversion, then compared against an ASCII constant. In Turkish, case conversion of the letter I does not behave the way every other Latin-script locale behaves: uppercase I lowercases to a dotless ı, and lowercase i uppercases to a dotted İ.
So a check that lowercases an input and compares it to "identity", "id", "invoice" or any other token
containing an I produces "ıdentity" under a Turkish locale and fails. No exception is thrown. The comparison
is simply false.
Why it propagated
The defect is invisible in review and invisible in tests, because both run in the developer's locale. The code reads correctly. It is correct, for every locale the team has ever run it in.
It propagates further because case normalisation is idiomatic and appears everywhere: header names, enum parsing, feature-flag keys, file extensions, protocol tokens, sorting keys. One of them is enough, and the one that breaks is usually deep in a shared utility rather than in market-specific code.
Why detection lagged
Monitoring is aggregated globally. Turkey is a small share of traffic for most products, so a complete failure in one market is a small dip in the global success rate - well inside normal variance. The alert threshold that would catch it is per-market, and almost nobody has one until after the first incident.
The structural fix, versus the tempting local fix
The tempting fix is to special-case Turkish. That leaves every other locale-sensitive behaviour unaddressed, and there are more: Azerbaijani has the same I behaviour; German ß uppercases to two characters, changing string length; several locales sort differently from code-point order.
The structural fix is a rule that can be enforced:
- Never use locale-sensitive operations for machine comparison. Use the locale-invariant form for identifiers, tokens and protocol values, and reserve locale-aware operations for text shown to a human.
- Add a lint rule, because this is exactly the class of defect that rules catch reliably and reviews do not.
- Run a pseudo-locale in continuous integration - a locale that exercises case conversion, 40% text expansion and bidirectional layout - so the assumption is broken on every build rather than in production. The cost is roughly 2 days to set up and a slower pipeline; the alternative is finding it in a market.
- Alert on success rate per market, which also catches payment provider failures, regional CDN problems and regulatory blocks.
The general lesson
Correctness that depends on ambient environment state is correctness you have not tested. The locale is one such variable; the time zone, the system clock, the default character encoding and the default currency are others, and each has produced the same shape of incident: code that works everywhere it has been run, and fails where it has not.
When this is over-engineering
A product serving a single market in a single language does not need a pseudo-locale pipeline. Choose the lint rule as soon as a second locale exists, and add the pseudo-locale pipeline only when the second market does - the rule is cheap and catches most of it, and the infrastructure is not worth its maintenance until someone is shipping to a market nobody on the team speaks.