Edge Cache and CDN Platform  ·  View 06 of 29  ·  2 · People and journeys

Journey — Draining a PoP

Network on-call takes Frankfurt out for a router upgrade and brings it back. The two tense moments are the withdrawal and the return.

Editable source SVG draw.io All views
Network on-call FRA router upgrade Goal — Upgrade Frankfurt's routers without users noticing Trigger — Scheduled 02:00 UTC change window Done when — No error spike; FRA at 90% byte hit ratio in 4 h 1 · Plan 2 · Drain 3 · Withdraw ◆ moment of truth 4 · Maintain 5 · Return ◆ moment of truth 6 · Warm What they do Checks n−1 headroom Starts the drain Withdraws prefixes Upgrades routers Releases the PoP Watches refill System behind it Capacity report Envoy drain pop-guard Neighbour PoPs Purge log replay Cold-mode fetch cap How it feels Relaxed Alert Tense Where it hurts AMS near ceiling at peak Long video sessions Flows break on withdraw Purges missed while away Refill hits the shield What the platform gives Headroom shown per PoP Bounded 120 s window Drain first, then withdraw Serves only when caught up Refill rate-limited Journey — Network On-call Takes Frankfurt Down and Brings It Back v 1.0 · owner Edge Architecture · date 2026-09

The trough, and the answer

  • Withdrawal breaks every connection still open, because anycast moves packets and not sessions. The drain comes first: Envoy sends GOAWAY and closes QUIC connections, new connections go elsewhere, and in-flight work gets 120 seconds.
  • Coming back is riskier than leaving. A PoP that missed nine hours of purges would serve stale prices the moment it announces, so it stays silent until it has replayed the log and confirmed its bundle is current.

Decisions it forces

  • Headroom is checked before the change, not during it. The capacity report shows whether AMS, CDG and LHR can absorb FRA at that hour's peak.
  • A returning PoP starts in cold mode with its shield fetch concurrency capped at 25% until its byte hit ratio passes 80%, so its refill does not become the shield's incident.
  • Long video sessions are expected to reconnect. Players resume with a byte-range request, which the next PoP answers from cache.

Numbers

  • Target of 90% of steady byte hit ratio within 4 hours of returning. The first drill measures it rather than assuming it.
  • BGP convergence of about 30 seconds after withdrawal, measured by external probes.