Audit Log Service
A shared audit log for the whole estate: it accepts security- and compliance-relevant events from every application and administrative surface, holds them so that nobody — including its own operators and the subscription's administrators — can alter or remove one inside its retention period, and answers investigative and regulatory questions against thirteen months of that history. Read the set in order. Act 1 fixes the boundary and the plane separation everything else follows from; act 2 says who the record is actually for and where each of them is currently let down; acts 3 and 4 give the structure and say which store is the truth and which is merely convenient; act 5 is what happens when the thing is used; act 6 is running and paying for it; act 7 is why it is safe, with the operators inside the threat model rather than outside it. One decision governs the whole set: the write path holds no authority over a record's lifetime, and no principal exists anywhere that can both write a record and remove it.
1 · Context and scope What sits inside the boundary, who feeds it, who checks it, and the four-plane separation that the rest of the architecture is a consequence of.
01
Producers — write only
Business Applications
410 services
Admin Consoles
privileged actions
Platform Services
IAM, payments, HR
Azure Activity Log
cloud control plane
People it serves
Platform Operator
runs it, reads nothing
Investigator
read within scope
Compliance Officer
read, export, hold
Legal Hold Officer
segregated role
External assurance
External Auditor
trusts nothing
Regulator
evidence on demand
Independent Witnesses
RFC 3161 TSA · Rekor
Audit Log Service
capture · custody · disposal
Custody held elsewhere — deliberately
Confidential Ledger
separate subscription
Managed HSM
non-exportable keys
Microsoft Entra ID
workload identity
operates
queries
exports
places holds
POST events
POST events
stream
diagnostics
verifies offline
signed bundles
anchors heads
chain heads
signs heads
authenticates
Audit Log Service — System Context
External / third party
Person or role
Data store
Security / platform
synchronous
event / async
Out of scope: application and operational logging, SIEM detection and correlation, and the business records themselves. Regulators receive evidence through the auditor's bundle, not through an interface of their own.
v 1.0 · owner Data & AI Global Practice · date 2026-09
System Context
Who writes to the audit log, who reads it, who checks it — and which custody is deliberately held by somebody else.
HTML page
SVG
draw.io
03
Capture plane — may create a record, may never remove one
Boundary
Ingest API
write only
Validator
schema + secrets
Hand-off
Durable Buffer
no delete verb
Quarantine
rejections, audited
Custody plane — may retain and expire, may never create a record
Seal and chain
Sealer
append only
Chain Service
hash forward
Segment Store
locked retention
Expire
Retention Engine
extend, never shorten
Legal Hold Registry
legal role only
Disposal Service
certificate or nothing
Query plane — derived, authoritative for nothing
Projection
Indexer
rebuildable
Search Index
disposable
Access
Query API
tenant scoped
Archive Query
async jobs
Evidence plane — may read and prove, may never write a record
Proof
Verifier
recompute chain
Proof Service
inclusion, absence
Delivery
Export Service
watermark, cap
Offline Verifier
no service needed
Managed HSM
key custody team
Confidential Ledger
checkpoint custody
Microsoft Entra ID
identity team
sealed hand-off
projection
signed head
sign, never export
Four Planes — No Principal Can Both Write and Remove
Application we own
Queue / topic
Data store
External / third party
Security / platform
event / async
synchronous
Four planes, four managed identities, four Entra groups, no membership overlap. The capture plane holds no delete verb in its data-access layer; the custody plane holds no insert verb for records.
v 1.0 · owner Data & AI Global Practice · date 2026-09
Four Planes and the Rule Between Them
The critical design decision drawn as a boundary: which plane may create a record, which may retain and expire one, and which may only prove things about them.
HTML page
SVG
draw.io
2 · People and journeys Who the record is for, what each of them is trying to get done, and the three journeys that decide whether this service is worth having.
04
The people the record is for
Investigator
38 named, scoped
Goal — Show me every privileged action this
account took last March, and tell me honestly
whether I am seeing all of them.
Core journeys
Trace an account
view 05
Scope a breach window
Compliance Officer
12 named
Goal — Give me something I can hand to a
regulator that they can check without ever
asking us a question.
Core journeys
Produce an evidence bundle
Certify a disposal
External Auditor
annual and ad hoc
Goal — Prove to me that the record could not
have been altered, without me having to trust
you or your operators.
Core journeys
Verify a bundle offline
view 06
Challenge a gap in a range
The people who run it
Platform Operator
SRE, on call 24×7
Goal — Hold ingest above 99.99% without ever
being able to read a single customer's audit
record.
Core journeys
Absorb a 4× burst
Rebuild the index
Security Engineer
detection and response
Goal — Be told within fifteen minutes if anyone
has touched a segment, and that includes me.
Core journeys
Answer a divergence alert
Review export anomalies
Legal Hold Officer
segregated, 4 named
Goal — Freeze everything touching this matter
today, and be certain nothing expired overnight
while I slept.
Core journeys
Place and lift a hold
Prove nothing was lost
The systems that write to it
Business Application
410 services, one SDK
Goal — Emit my audit events without your outage
becoming my outage.
Core journeys
Onboard an event class
view 07
Survive an audit outage
Admin Console
privileged surfaces
Goal — Record the privileged action I just
performed, including the one that was refused.
Core journeys
Record a denied action
Azure Control Plane
Activity Log export
Goal — Land every subscription-level change in
the same record as the applications, not in a
portal of its own.
Core journeys
Export control-plane events
The Service Itself
reads, exports, holds
Goal — Audit my own use into myself, so nobody
reads this log without leaving a record of
having read it.
Core journeys
Write a read record
Write a key-use record
The machines that act without being asked
Rolling Verifier
full corpus / 7 days
Goal — Walk the whole chain every week and name
the first divergence before anyone else finds
it.
Core journeys
Complete a sweep
Retention Scheduler
nightly window
Goal — Dispose of exactly what expired, nothing
that is held, and leave a signed certificate
behind.
Core journeys
Run a disposal window
Index Reconciler
continuous
Goal — Find the record that exists but cannot be
found, before an auditor does.
Core journeys
Repair a divergence
Cross-Region Replicator
continuous, one way
Goal — Keep the second region inside RPO 0 while
holding no right to delete anything in the
first.
Core journeys
Replicate segments
Who the Record Is For, and What Each of Them Gets to Do
Person or role
Journey / task
External / third party
Security / platform
Four groups, and the fourth is the one usually left off. A weekly verifier and a nightly disposer have SLAs, failure paths and blast radius of their own; leaving them off the cast list is how they end up with no owner.
v 1.0 · owner Data & AI Global Practice · date 2026-09
Actors and Their Core Journeys
Fourteen actors in four groups, each with the goal they actually arrive with — including the four that are machines.
HTML page
SVG
draw.io
3 · Structure The layering rule, the deployable units inside each plane, every interface in and out, and the shape of the hash chain that makes tampering provable.
09
Capture plane — namespace audit-ingest · managed identity ingest-mi
envoy-gateway
Envoy · mTLS · 12 pods
ingest-api
Rust axum · 24 pods
admission
schema · secrets · size
quota-governor
OPA · per tenant
quarantine-writer
rejections, audited
Durable buffer — Strimzi on dedicated node pool
kafka-broker
12 × D16s v5 · RF
3
kraft-controller
3 pods, 3 zones
tiered-storage
6 h local, 72 h Blob
Custody plane — namespace audit-custody · managed identity custody-mi
sealer
Go · 32 pods
chain-service
Go · 6, leader per shard
checkpoint-signer
Go · 3 pods
compactor
chain-preserving
retention-engine
Go · 3 pods
disposal-service
Go · 2 pods
Query plane — namespace audit-query · managed identity query-mi
indexer
Go · 24 pods
clickhouse
16 × E16ds v5
query-api
Go · 12 pods
reconciler
Go · 4 pods
trino
archive, autoscaled
Evidence plane — namespace audit-evidence · managed identity evidence-mi
verifier
Go · 16 pods
proof-service
Go · 6 pods
export-service
Go · 6 pods
Azure Blob Storage
immutable containers
Confidential Ledger
separate subscription
Managed HSM
per-tenant keys
PostgreSQL Flexible
registry, holds
Valkey Cluster
dedup, quota
acks=all, isr 2
consume
PUT once
append head
consume
Container Architecture — Deployable Units (C4 L2)
Interface / broker
Application we own
Queue / topic
Security / platform
Data store
synchronous
event / async
Five namespaces, five managed identities, five Entra groups with no shared membership. Blob data-plane roles are granted per plane: the sealer may create a blob and no more; nothing anywhere holds Storage Blob Data Owner.
v 1.0 · owner Data & AI Global Practice · date 2026-09
Container Architecture (C4 L2)
Every deployable unit, its technology and its scale, grouped by the plane whose identity it runs under.
HTML page
SVG
draw.io
11
Assemble — a segment is closed at most 60 seconds after it opens
Read
Partition Reader
ordered, committed
Segment Grouping
tenant · date · class
Write
Parquet Writer
zstd · per-column
Seal Trigger
60 s or 256 MB
Chain — hashing forward so that removing anything invalidates everything after
Hash
Leaf Hashing
SHA-256, canonical
Segment Merkle Root
RFC 6962 style
Predecessor Link
prev_root in header
Sign
Checkpoint Builder
15 min, all tenants
HSM Signature
non-exportable
TSA Countersign
RFC 3161
Lock — immutability is a storage property, not an application rule
Object
Immutable Container
locked time-based
Encryption Scope
per-tenant CMK
Version-Level Lock
no overwrite
Record of it
Segment Registry
PostgreSQL, derived
Lifecycle Tiering
hot → cool → archive
Expire — acting only on records this plane did not create
Retention Class
13 mo · 7 yr · 10 yr
Hold Check
tenant, actor,
resource
Crypto-Shred
destroy segment DEK
Disposal Certificate
signed, re-ingested
Tombstone Node
chain stays verifiable
Confidential Ledger
chain heads only
signed head
Custody Plane — Seal, Chain, Lock, Expire
Application we own
Decision point
Security / platform
External / third party
Data store
synchronous
The lock is set by a locked time-based retention policy on the container. Once locked, the policy can be extended and cannot be shortened, deleted or bypassed — not by the subscription owner, not by a support ticket, and not by the plane that wrote the object.
v 1.0 · owner Data & AI Global Practice · date 2026-09
Custody Plane
Assemble, chain, lock, expire — the four things that happen to a record after nobody can change it any more.
HTML page
SVG
draw.io
4 · Data Which tier is the system of record and which is disposable, the canonical record, the model behind a proof, and where the tenant boundary sits in storage.
5 · Runtime Ingest and what the acknowledgement promises, sealing and signing, querying and the record a query leaves, verification, export, disposal, legal hold, an audit-plane outage and an index rebuild.
21
Kafka Buffer
Sealer
Immutable Blob
Chain Service
Managed HSM
Timestamp Authority
Confidential Ledger
1. poll committed offsets, ordered
2. group by tenant · date · event class
3. close at 60 s or 256 MB, whichever first
4. PUT segment — create-only permission
5. object digest + immutable-until
6. merkle_root, prev_root, object digest
7. extend this tenant's head
8. sign checkpoint over all tenant heads
9. signature — key never leaves the HSM
10. timestamp request over checkpoint root
11. RFC 3161 token
12. append signed head
13. ledger transaction receipt
14. key-use event, back into the log
Seal, Chain, Sign, Publish — One 15-Minute Checkpoint
If the HSM is unreachable, messages 1 to 7 continue and 8 onwards pause: ingest never fails because sealing cannot proceed. The unsigned interval is visible in verification as pending, never as a break.
v 1.0 · owner Data & AI Global Practice · date 2026-09
Seal, Chain, Sign, Publish
One fifteen-minute checkpoint end to end, including what happens when the key service is unreachable.
HTML page
SVG
draw.io
6 · Operations Where it runs, how it survives a region, how a change to the rules is approved, what is watched, what it costs, and the operator's loop over a system they cannot read.
29
Azure · West Europe
Availability zone 1
AKS Ingest Pool
3 × D16s v5
Kafka Brokers
4 of 12
ClickHouse Shards
replica set A
Custody Pool
sealer, chain
Availability zone 2
AKS Ingest Pool
3 × D16s v5
Kafka Brokers
4 of 12
ClickHouse Shards
replica set B
Custody Pool
sealer, chain
Availability zone 3
AKS Ingest Pool
3 × D16s v5
Kafka Brokers
4 of 12
ClickHouse Shards
replica set C
Custody Pool
sealer, chain
Zone-redundant by construction — no single zone holds the only copy
Blob Storage ZRS
immutable containers
PostgreSQL Flexible
zone-redundant HA
Managed HSM
3-zone pool
Front Door + WAF
global anycast
Private Endpoints
in-estate producers
Separate subscription, separate administrators, separate break-glass
Confidential Ledger
chain heads
Checkpoint Signing Key
own HSM pool
Ledger Admin Group
no segment access
Microsoft Entra ID
global tenant
Timestamp Authority
third party
seal
signs heads
Deployment — One Region, Three Availability Zones
Security / platform
Queue / topic
Data store
Interface / broker
External / third party
event / async
synchronous
Loss of one zone removes a third of ingest capacity and no data: Kafka keeps isr ≥ 2, ClickHouse keeps two replicas, and the segment store is zone-redundant beneath the immutability policy. Producers see nothing but a brief drop in a connection pool.
v 1.0 · owner Data & AI Global Practice · date 2026-09
Deployment — One Region, Three Zones
What runs where, what is zone-redundant by construction, and which subscription deliberately is not ours.
HTML page
SVG
draw.io
30
West Europe — active
Ingest Plane
25 k/s steady
Custody Plane
sealing, chaining
Query Plane
full index
Segment Store
locked, ZRS
North Europe — warm standby
Ingest Plane
20% warm · RTO 15 min
Custody Plane
idle, resumes chain
Query Plane
lagging · RTO 4 h
Segment Replica
locked at destination
Continuous replication — one way, no delete right at the destination
Object Replication
segments · RPO ≈ 0
MirrorMaker 2
buffer · 72 h
ClickHouse Replica
rebuildable anyway
PostgreSQL Replica
registry, holds
Custody independent of both regions
Confidential Ledger
own subscription
Timestamp Authority
outside the estate
Published Public Key
verifier needs no API
Public Witness Log
Rekor · phase 3
locked on arrival
signed heads
Two Regions, and a Custody Chain That Belongs to Neither
Application we own
Data store
Queue / topic
External / third party
Security / platform
event / async
synchronous
A region failover changes which region seals; it changes nothing about what has already been sealed. The chain continues in the second region from the last checkpoint the ledger holds, and the ledger is in neither region's subscription — which is what stops a regional administrator from being able to
rewrite both sides of the evidence.
v 1.0 · owner Data & AI Global Practice · date 2026-09
Two Regions, and Custody That Belongs to Neither
Active-passive failover, one-way replication, and the reason the checkpoint ledger sits outside both regions.
HTML page
SVG
draw.io
7 · Assurance Trust zones, identity, separation of duties, key custody, what each insider attempt actually meets, and every named failure mode with its required handling.
35
Internet — assumed hostile
Off-Estate Producer
partner systems
Auditor Workstation
offline verification
Published Key + Algorithm
deliberately public
Perimeter — holds no key
Front Door + WAF
rate limit, TLS 1.3
Private Endpoint
in-estate default path
Envoy mTLS
SVID or Entra token
Capture — write only
Ingest API
ingest-mi
Validator + Detector
reject or accept
Kafka Buffer
no delete API exposed
Custody — no standing access
Sealer + Chain
custody-mi
Segment Store
create-only role
Retention Engine
extend only
Evidence and query
Query API
query-mi, tenant scoped
Verifier
read-only on segments
Export Service
capped, watermarked
Key and checkpoint custody
Managed HSM
non-exportable
Confidential Ledger
append only
Custody Admin Group
disjoint membership
TLS 1.3 + workload token
principal, tenant, write scope
consume, in-cluster mTLS
read-only SAS-free RBAC
sign · Crypto User only
signed bundle, out of band
Trust Zones — Every Crossing Authenticated, Every Zone Narrower
External / third party
Security / platform
Interface / broker
Application we own
Queue / topic
Data store
synchronous
event / async
Zones narrow left to right and every crossing is mutually authenticated, in-cluster included. The perimeter can see a record in flight and can read none at rest; the custody zone has no standing human access at all; and the last edge leaves unauthenticated on purpose, because the bundle is built to be
verified by someone holding no credential of ours.
v 1.0 · owner Data & AI Global Practice · date 2026-09
Trust Zones
Six zones of decreasing exposure, every crossing authenticated — and the one edge that deliberately is not.
HTML page
SVG
draw.io
38
Signing — one key makes a checkpoint believable
Managed HSM Pool
own subscription
EC P-384, Non-Exportable
FIPS 140-3 L3
Crypto User: Signer Only
one workload identity
Rotation with Overlap
old key still verifies
Encryption — one key per tenant, one data key per segment
Tenant KEK
500 keys in HSM
Segment DEK
wrapped, per object
Blob Encryption Scope
bound to the prefix
Shred = Destroy DEK
instant, irreversible
Publication — what an outsider needs and gets
Public Verification Key
published, versioned
Verification Algorithm
specified, open
TSA Certificate Chain
third-party root
Witness Anchor
phase 3
What no principal holds, by construction
Exportable Private Key
no export path
Standing HSM Admin
PIM, quorum, audited
Sign and Delete
no such principal
wraps
public half
Key Custody — What Nobody Holds Is the Design
Security / platform
Application we own
External / third party
Risk / gap
synchronous
Every use of the signing key is itself an audit event in this log, which means the log records the operations that make the log believable. A rotation overlaps so old checkpoints keep verifying, and needs two approvals recorded as events before it can begin.
v 1.0 · owner Data & AI Global Practice · date 2026-09
Key Custody
Three kinds of key, what the outside world is given, and the combinations nobody holds.
HTML page
SVG
draw.io
40
Absorbed — the producer sees nothing
Ingest Timeout
retry + idempotency key
Duplicate Delivery
one record, 24 h window
Zone Loss
isr ≥ 2, transparent
Broker Restart
producer backoff
Degraded — in the declared order, never the reverse
Key Service Down
ingest on, sealing pauses
Index Loss
Trino path, SLO suspended
Region Loss
RTO 15 min / 4 h
Sealer Backlog
buffer absorbs 72 h
Contained and made visible — the gap is itself a record
Producer Outage
spool 1 h, gap marker
Poison Record
quarantine, no head-of-line
Clock Skew
flagged beyond 5 min
Seal Gap
re-sealed, never discarded
Incident — never handled automatically
Insider Deletion Attempt
prevented, alerted
Segment Divergence
Sev-1, no auto-repair
Acknowledged, Never Sealed
Sev-1 defect, RPO 0
Checkpoint Signature Fails
key or ledger compromise
unless roots differ
Named Failure Modes and Their Required Handling
Application we own
Risk / gap
failure / alternate
Index divergence is repaired from segments and is routine; segment divergence is repaired by nobody and is an incident. The distinction is the whole architecture in one line: the derived thing may be rebuilt, and the thing it was derived from may not be touched.
v 1.0 · owner Data & AI Global Practice · date 2026-09
Named Failure Modes
Sixteen failures in four categories, sorted by what the service is permitted to do about each.
HTML page
SVG
draw.io
Open svg/<view>.svg or drawio/<view>.drawio in draw.io Desktop or at app.diagrams.net to edit. The SVG carries the diagram inside it, so it is both the picture and the source. This folder is self-contained — copy it whole and every link still resolves.