Evidence ledger
26 sources
Checked 02 Sep 2026
Evidence ledger
One row per claim in A cache miss is a protocol, not an event: who published it, what grade it carries, when it was written, when the link was last checked, and the quote or figure it rests on. Nothing in the guide is cited from memory, so anything not in this table is not in the guide.
One row per claim. Checked dates are the day the source was retrieved in this research session.
Method note. This session ran behind an egress proxy that allowed direct fetches only to github.com, raw.githubusercontent.com and pkg.go.dev. GitHub-hosted sources (memcached protocol.txt, groupcache, rails, twitter/cache-trace, golang/go issues, internetarchive/xfetch) were fetched in full. Every other source was retrieved through search-engine retrieval in this session: the quoted sentences below were returned verbatim from the live pages by the search layer, several confirmed by exact-phrase queries. No quote is from memory. Links were recorded as returned by the search engine; the verifier could not re-resolve hosts outside the proxy allowlist, which is noted in the guide's footer.
| # | Org | Title | Tier | Published | Checked | URL | Claim I take from it | Supporting quote or figure |
|---|---|---|---|---|---|---|---|---|
| 1 | More Details on Today's Outage | postmortem | 2010-09-23 | 2026-09-02 | https://engineering.fb.com/2010/09/23/uncategorized/more-details-on-today-s-outage/ | The error-handling path around the cache, not the bad config value, sustained the outage: clients treated DB errors as invalid cache entries and deleted the key, re-triggering the miss | "every time a client got an error attempting to query one of the databases it interpreted it as an invalid value, and deleted the corresponding cache key" | |
| 2 | More Details on Today's Outage | postmortem | 2010-09-23 | 2026-09-02 | https://engineering.fb.com/2010/09/23/uncategorized/more-details-on-today-s-outage/ | Amplification: the fix-it queries overwhelmed the DB cluster | "that cluster was quickly overwhelmed by hundreds of thousands of queries a second"; "even after the original problem had been fixed, the stream of queries continued" | |
| 3 | More Details on Today's Outage | postmortem | 2010-09-23 | 2026-09-02 | https://engineering.fb.com/2010/09/23/uncategorized/more-details-on-today-s-outage/ | Recovery required stopping all traffic to the DB cluster, i.e. turning off the site | "The way to stop the feedback cycle was quite painful ... we had to stop all traffic to this database cluster, which meant turning off the site" (as summarized in retrieval; feedback-cycle wording confirmed by exact-phrase search) | |
| 4 | Data Center Knowledge | Technical Details of Facebook Outage | blog | 2010-09 | 2026-09-02 | https://www.datacenterknowledge.com/outages/technical-details-of-facebook-outage | Duration and severity: ~2.5 hours, worst outage in over four years | "down or unreachable for approximately 2.5 hours, marking the worst outage the company had experienced in over four years" |
| 5 | Slack | Slack's Incident on 2-22-22 | postmortem | 2022 | 2026-09-02 | https://slack.engineering/slacks-incident-on-2-22-22/ | A routine Consul upgrade rolled 25% of the fleet; restarted memcached nodes rejoined empty, dropping hit rate at peak traffic | "on February 22, a PBR step updated Consul on 25% of the fleet"; "the node that leaves the service catalog gets replaced by Mcrib. The new cache node will be empty" |
| 6 | Slack | Slack's Incident on 2-22-22 | postmortem | 2022 | 2026-09-02 | https://slack.engineering/slacks-incident-on-2-22-22/ | The misses landed on the most expensive query shape: a cross-shard scatter query for GDM membership | "a significant portion of the cache was unavailable, requiring most users to query every shard"; "it is quite inefficient to find the set of users that are members of a particular GDM because you have to query every shard in the datastore" |
| 7 | Wikimedia | Incidents/2020-02-04 app server latency | postmortem | 2020-02 | 2026-09-02 | https://wikitech.wikimedia.org/wiki/Incidents/2020-02-04_app_server_latency | A partial hit-ratio dip, not a cache wipe, was enough: Babel WANCache hit ratio fell to ~52% and p75 app-server latency went from ~300 ms to 1500-4000 ms | "From 16:03 to 16:12 UTC ... the WANObjectCache hit ratio for Babel keys dropped to about 52%"; "75th percentile latency increasing from the usual ~300 ms to 1500-4000 ms"; each miss-driven API request "hung for ten seconds waiting for a response before timing out" |
| 8 | memcached | doc/protocol.txt (meta commands) | source | master, checked | 2026-09-02 | https://github.com/memcached/memcached/blob/master/doc/protocol.txt | Mainline memcached now carries lease semantics: N/R vivify-and-win flags, W/X/Z win/stale/lost responses, targeted at dog-piling | "Used to help with so called \"dog piling\" problems with recaching of popular items"; "a client may be informed they have \"won\" ownership of a cache item"; "This is used when the act of recaching an item can cause undue load on another system (CPU, database accesses, time, and so on)" |
| 9 | memcached | doc/protocol.txt (meta delete) | source | master, checked | 2026-09-02 | https://github.com/memcached/memcached/blob/master/doc/protocol.txt | Invalidation can mark-stale instead of delete: md with I flag bumps CAS and serves stale with X flag until one client recaches |
"Instead of removing an item, this will give the item a new CAS value and mark it as stale ... the client will be supplied an 'X' flag to show the data is stale and needs to be recached" |
| 10 | Google (groupcache) | golang/groupcache README | source | 2013 | 2026-09-02 | https://github.com/golang/groupcache | Coalescing is the founding feature: one load per replicated process set, multiplexed to all callers; super-hot keys are auto-mirrored | "coordinates cache fills such that only one load in one process of an entire replicated set of processes populates the cache, then multiplexes the loaded value to all callers"; "supports automatic mirroring of super-hot items to multiple processes" |
| 11 | Google (groupcache) | singleflight/singleflight.go | source | 2013 | 2026-09-02 | https://github.com/golang/groupcache/blob/master/singleflight/singleflight.go | The whole mechanism is ~one mutex, one map, one WaitGroup; duplicates wait and share one result | "Package singleflight provides a duplicate function call suppression mechanism." First caller executes; duplicates wg.Wait() and receive the same result |
| 12 | Rails | activesupport/lib/active_support/cache.rb | source | main, checked | 2026-09-02 | https://github.com/rails/rails/blob/main/activesupport/lib/active_support/cache.rb | Serve-stale-while-one-regenerates is a one-parameter API: race_condition_ttl bumps the expiry so one process regenerates while others read the stale value | "an expired value can be reused while a new value is being generated ... preventing multiple processes from simultaneously regenerating the same entry (also known as the dog pile effect)"; "If the first process errors out while generating a new value, another process can try ... after the extended time window has elapsed" |
| 13 | twitter/cache-trace | source | 2020-03 traces | 2026-09-02 | https://github.com/twitter/cache-trace | Production trace scale: 54 clusters released, 2.8 TB compressed / 14 TB raw, CC-BY | "2.8 TB in compressed form, 14 TB uncompressed" | |
| 14 | Go project | golang/go issue #53427 | source | 2022-06, open | 2026-09-02 | https://github.com/golang/go/issues/53427 | singleflight is load-bearing enough to argue generics over, yet still lives in x/sync, not the standard library; the proposal remains open | Proposal for "singleflight.TypedGroup[K, V]" etc.; issue open with no attached PRs as of check date |
| 15 | Scaling Memcache at Facebook (NSDI '13) | paper | 2013-04 | 2026-09-02 | https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf | Leases bound the stampede: token issued at most once per 10 s per key; peak DB query rate fell from 17K/s to 1.3K/s | "memcached server regulates the rate at which it returns tokens ... return a token only once every 10 seconds per key"; "Without leases, cache misses resulted in a peak database query rate of 17K/s; with leases, the peak database query rate was reduced to 1.3K/s" (figures via retrieval of paper summaries; corroborated across three independent write-ups) | |
| 16 | Scaling Memcache at Facebook (NSDI '13) | paper | 2013-04 | 2026-09-02 | https://www.usenix.org/system/files/conference/nsdi13/nsdi13-final170_update.pdf | Scale context: over a billion requests per second, trillions of items | "processing over a billion requests per second and storing trillions of items" | |
| 17 | Twitter / CMU | A large scale analysis of hundreds of in-memory cache clusters at Twitter (OSDI '20) | paper | 2020-11 | 2026-09-02 | https://www.usenix.org/conference/osdi20/presentation/yang | TTL, not eviction, often defines the working set; many production cache workloads are write-heavy; 153 clusters, 80 TB of traces analysed | "TTL is an important and sometimes defining parameter of cache working sets"; "collected production traces from 153 in-memory cache clusters at Twitter, sifting through over 80 TB of data"; "many are far more write-heavy ... than previously shown" |
| 18 | Vattani, Chierichetti, Lowenstein | Optimal Probabilistic Cache Stampede Prevention (VLDB 2015) | paper | 2015 | 2026-09-02 | https://www.vldb.org/pvldb/vol8/p886-vattani.pdf | XFetch: each reader independently recomputes early with probability rising near expiry, time-shifted by deltabetalog(rand()); exponential variate proven optimal; no coordination, no tuning | "extremely simple to implement and requires no parameter tuning"; recompute when now - delta * beta * log(rand()) >= expiry, beta default 1.0 |
| 19 | IETF (M. Nottingham) | RFC 5861, HTTP Cache-Control Extensions for Stale Content | adr | 2010-05 | 2026-09-02 | https://datatracker.ietf.org/doc/html/rfc5861 | stale-while-revalidate and stale-if-error standardize serve-stale for HTTP caches | "allows a cache to immediately return a stale response while it revalidates it in the background, thereby hiding latency (both in the network and on the server) from clients"; stale-if-error "improves availability" |
| 20 | Wikimedia | Memcached for MediaWiki (wikitech design page) | adr | current, checked | 2026-09-02 | https://wikitech.wikimedia.org/wiki/Memcached_for_MediaWiki | Purge is a SET, not a DELETE: tombstones (~11 s hold-off) prevent instant repopulation with stale data; interim values (~1 s) absorb the regeneration stampede during the tombstone window | "WANCache formulates its purge as a SET operation that stores a placeholder value known as a 'tombstone'"; "deleting keys via the DELETE command would mean ... the same key could be re-populated immediately with the same stale value that was just deleted"; "To avoid a recompute stampede, maybe-stale values are stored as an 'interim value' in a sister key" |
| 21 | Thundering Herds & Promises (Nick Cooper) | blog | 2019-04 | 2026-09-02 | https://instagram-engineering.com/thundering-herds-promises-82191c8af57d | Cache the in-flight computation, not the value: concurrent misses find the promise and wait on one backend request; motivated by empty-cache cluster turn-up | "we cache a Promise that will eventually provide the value" (exact-phrase confirmed); on miss "instead of going immediately to the backend we create a Promise and insert it into the cache" | |
| 22 | DoorDash | Avoiding Cache Stampede at DoorDash (Zohaib Hassan) | blog | 2018-08 | 2026-09-02 | https://careersatdoordash.com/blog/avoiding-cache-stampede-at-doordash/ | An L1 miss inside one process is already a stampede against L2/DB; solved with a coroutine debounce returning a shared Deferred | "solve cache stampede caused by a L1 cache miss, resulting in parallel duplicate reads to L2 or DB"; "The debounce method immediately returns Deferred |
| 23 | Discord | How Discord Stores Trillions of Messages | blog | 2023-03 | 2026-09-02 | https://discord.com/blog/how-discord-stores-trillions-of-messages | Coalescing moved into a dedicated Rust data-service tier; consistent-hash routing by channel ID makes coalescing effective against hot partitions | "The first incoming request triggers a worker task ... Any subsequent requests for the same data check for an active worker and subscribe to its result instead of issuing a new query"; "For each request to their data service, they provide a routing key. For messages, this is a channel ID" |
| 24 | Netflix | Cache warming: Agility for a stateful service | blog | 2018-12-04 | 2026-09-02 | https://netflixtechblog.com/cache-warming-agility-for-a-stateful-service-2d3b1da82642 | Never expose a cold replica: replica warmer copies data to new replicas, instance warmer refills replaced nodes | "a replica warmer mechanism to copy data from existing replicas to new ones, and an instance warmer mechanism to bootstrap a node after replacement using data from other replicas" |
| 25 | Netflix | Cache warming: Leveraging EBS for moving petabytes of data | blog | 2021-11-26 | 2026-09-02 | https://netflixtechblog.medium.com/cache-warming-leveraging-ebs-for-moving-petabytes-of-data-adcf7a4a78c3 | Warming is petabyte-scale infrastructure: dumpers write to multi-attach EBS, populators stream into new replicas | "Multi-attached EBS volumes are attached to both source nodes and populator nodes so that populators can read data written by a dumper in a pipelined manner" |
| 26 | Madhur Ahuja | Close look at proxy_cache_lock and proxy_cache_use_stale in Nginx | blog | 2016-12-25 | 2026-09-02 | https://www.madhur.co.in/blog/2016/12/25/proxy-cache-lock-nginx.html | Coalescing without serve-stale converts a stampede into a latency wall: waiters block until the one origin fetch completes | "if you have set proxy_cache_lock to on, all the requests which will arrive at Nginx while the cache is being updated will essentially wait" |
| 27 | nginx | ngx_http_proxy_module documentation | vendor | current, checked | 2026-09-02 | https://nginx.org/en/docs/http/ngx_http_proxy_module.html | proxy_cache_lock admits one populator per key; proxy_cache_use_stale updating serves stale during refresh; both default off | "only one request at a time will be allowed to populate a new cache element"; the updating parameter "permits using a stale cached response if it is currently being updated" |
| 28 | Varnish Software | Under the hood (docs) / 10 Varnish Cache mistakes | vendor | current, checked | 2026-09-02 | https://docs.varnish-software.com/book/what-is-varnish/under-the-hood/ | Coalescing has a failure mode: if the response turns out uncacheable, waiters are re-sent one at a time (request serialization); hit-for-pass exists to bypass the waiting list | "If the backend response is uncacheable (with TTL = 0), the waiting requests can't reuse that response, so Varnish sends the first request from the waiting list to the backend again ... waiting requests one by one, a side effect called request serialization"; "Items on the hit-for-miss or hit-for-pass cache will bypass the waiting list to avoid serialization" |
| 29 | Fastly | Request collapsing (documentation) | vendor | current, checked | 2026-09-02 | https://www.fastly.com/documentation/guides/concepts/edge-state/cache/request-collapsing/ | At CDN scale, coalescing is two-level: fetches for an object are focused through a single cache node, which queues so one request escapes to origin | "identify multiple simultaneous requests for the same resource, and make just one backend fetch for it"; "All fetches for a particular object are focused through a single cache node ... only one request actually escapes the data center to go to your origin" |
| 30 | Strange Loop / Netflix | Caching at Netflix: The Hidden Microservice (Scott Mansfield) | talk | 2016-09 | 2026-09-02 | https://www.thestrangeloop.com/2016/caching-at-netflix-the-hidden-microservice.html | Scale of the cache tier as a service: ~30 million requests/s at peak, hundreds of billions of objects, tens of thousands of memcached instances (speaker's abstract figures) | "production EVCache deployments routinely handle upwards of 30 million requests/sec, storing hundreds of billions of objects across tens of thousands of memcached instances" |
| 31 | Internet Archive | Preventing Cache Stampede with Redis and XFetch (Jim Nelson, RedisConf17) + test harness | talk | 2017-05 | 2026-09-02 | https://github.com/internetarchive/xfetch | Production adopter of the VLDB paper; harness comparing fetch/locked/xfetch/xlocked concludes plain fetch and lock-only do not scale | "locked and fetch are susceptible to cache stampede, congestion collapse, and starved workers. They do not scale well."; "xlocked's results are the best of the four strategies in that it has zero cache misses and no simultaneous recomputes" |