Audit Log Service  ·  View 10 of 40  ·  3 · Structure

Ingest Plane

Every check that happens before an event can be accepted — and the absence of any path that changes one.

Editable source SVG draw.io All views
Arrival Network Private Endpoint in-estate producers Front Door + WAF off-estate only Identity mTLS Termination Envoy Token Validation Entra · SPIFFE Tenant Assignment from identity only Admission — every check before the record can be accepted Shape Schema Validation versioned, Apicurio Vocabulary Check controlled verbs Size Cap 256 KB, reject Content Secret Detector Presidio + rules PAN / Credential reject, never redact Governance Idempotency Check 24 h window Tenant Quota token bucket Class Declaration open or closed Commit — the acknowledgement is issued here and nowhere earlier Durable write Kafka Producer acks=all, isr ≥ 2 Partition Router tenant · resource Promise Commit Receipt event_id + offset received_at Stamp service clock Rejection — a refusal is itself an auditable event Structured Error field and reason Quarantine Topic never blocks a partition Quarantine Segments same retention lock oversized secret found durable Ingest Plane — Everything That Happens Before the Acknowledgement Interface / broker Security / platform Application we own Decision point Data store Queue / topic failure / alternate event / async Nothing here transforms the record. Validation and detection can only accept or reject; there is no redaction path, because a record that had to be edited to be storable was never a faithful account of the action. v 1.0 · owner Data & AI Global Practice · date 2026-09

Decisions

  • There is no redaction path. Validation and detection may accept or reject, nothing else: a record that had to be edited to be storable was never a faithful account of the action.
  • tenant_id is assigned from the authenticated producer identity and is never read from the payload. A producer that claims another tenant is rejected, not trusted.
  • The secret detector runs before commit, not after. A record that cannot be deleted must not contain something that must be rotated.

Latency budget

  • Token validation with cached JWKS ≈ 0.3 ms; schema validation ≈ 0.4 ms; secret scan over 1.2 KB with a compiled rule set ≈ 0.2 ms; dedup round trip ≈ 1 ms.
  • The remainder of the 25 ms p50 is the Kafka acks=all round trip across three zones. That is where the budget actually goes.
  • In-estate producers reach a private endpoint directly; only off-estate traffic pays for Front Door.

Risks

  • A rejection is a refused audit record, which is itself information. It goes to a quarantine topic and is sealed under the same retention lock — but a producer whose events are all being rejected has a silent gap until someone reads the rejection metrics.
  • The dedup window is Valkey, and Valkey is ephemeral. Losing it produces duplicate records within a 24 h window: visible, reconcilable by idempotency key, and deliberately preferred over putting a durable store on the hot path.
◀  Container Architecture (C4 L2) All views Custody Plane  ▶
Solution Architecture v1.0 · Data & AI Global Practice · 2026-09 · 40 views · Microsoft Azure and open source10-ingest-plane