Edge Cache and CDN Platform  ·  View 14 of 29  ·  4 · Data

Access Records — From Request to Offload and Cost

One structured record per request, shipped at least once when the pipeline is healthy and dropped with a count when it is not.

Editable source SVG draw.io All views
Emit Traffic Server log one record per request Envoy access log TLS · protocol · token Ship Vector 20 GB disk buffer Drop counter gap counted, never blocks Buffer Kafka per core DC · 3 days Store ClickHouse 30 days hot Parquet on Ceph 400 days cold Derive Hit ratio + offload property · PoP · path Cost model per TB · per miss Act Grafana owner dashboards Chargeback monthly export buffer full Access Records — From One Request to Offload and Cost Application we own Security / platform Risk / gap Queue / topic Data store External / third party failure / alternate About 7.8 billion records a day at steady state, roughly 470 GB compressed. Delivery never waits for any of it. v 1.0 · owner Edge Observability · date 2026-09

Decisions

  • Vector writes to a bounded disk buffer and drops when it is full. A logging outage never slows delivery, and the size of the gap is itself a metric.
  • Kafka sits between the PoPs and ClickHouse so that an analytics outage or schema change is replayed from three days of buffer, not lost.
  • Each record carries the configuration version that served it. A regression can be tied to a commit without guessing from timestamps.

What a record holds

  • Property, rule, key hash, cache status, object size, bytes served, upstream tier, TTFB, total time, PoP, client country and ASN, protocol, and config version. The key hash, never the key, so a URL with personal data in it is not copied into analytics.

Numbers

  • About 7.8 billion records a day at 90,000 requests per second steady state, about 470 GB a day compressed. 30 days hot is about 28 TB with two replicas; 400 days cold is about 190 TB of Parquet.
◀  The Cache Key Contract All views Flash Crowd — Thousands of Misses, One Origin Fetch  ▶
Solution Architecture v1.0 · Edge Architecture · 2026-09 · 29 views · open source, fully on-premises14-access-record-pipeline