Audit Log Service  ·  View 17 of 40  ·  4 · Data

Data Flow — Commit to Answerable

What happens to one record between the acknowledgement and the moment somebody can find it, with the volumes at each hop.

Editable source SVG draw.io All views
Accepted JSON Record p50 1.2 KB · p99 8 KB Commit Receipt event_id + offset Buffered Ordered Partition 240 partitions Zone Replication 3 AZ · isr 2 Sealed and chained Parquet Segment zstd · ~4 : 1 Merkle Root ≤ 60 s window Signed Checkpoint every 15 min Locked Immutable Object locked policy · ZRS Tier Schedule hot 30 d → cool Indexed Query Projection no payload bytes ClickHouse Insert p95 60 s from commit Reconciliation count + root match Answerable Point Query 30 d · p95 2 s Aggregate Query 13 mo · p95 30 s Archive Job Trino · ≤ 12 h rebuild source beyond the index Data Flow — Accepted Record to Answerable Query Application we own Queue / topic Data store Security / platform batch Volume: 2.2 billion events/day, 2.6 TB raw, 650 GB compressed. The index carries no payload bytes — an investigator who needs before/after values pays a segment fetch for them, which keeps the dominant marginal cost proportional to the fields people actually filter on. v 1.0 · owner Data & AI Global Practice · date 2026-09

Decisions

  • The index carries no payload bytes. An investigator who needs before/after values pays a segment fetch for the page they are reading, which keeps the dominant marginal cost proportional to the fields people filter on.
  • Sealing and indexing consume the buffer independently, so index lag never delays custody and custody never delays the query path.
  • Reconciliation is part of the flow, not an afterthought: every sealed segment's record count and root are compared against what the index holds, and the result advances the completeness bound queries report.

Volumes

  • 2.2 billion events/day, 2.6 TB raw, ~650 GB compressed at roughly 4:1 with zstd and per-column encoding.
  • p50 record 1.2 KB, p99 8 KB, hard cap 256 KB.
  • 240 partitions; the largest tenant produces 35% of total volume and is given a dedicated partition set so its bursts cannot displace anyone else's.

Targets

  • Commit to queryable p95 ≤ 60 s, p99 ≤ 5 min. Point query over 30 days p95 ≤ 2 s. Aggregate over the online window p95 ≤ 30 s. Archive retrieval accepted immediately, results within 12 h.
◀  Segment, Chain and Certificate All views Retention and Tiering  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 40 views · Microsoft Azure and open source17-commit-to-queryable