Audit Log Service  ·  View 20 of 40  ·  5 · Runtime

Synchronous Ingest

Thirteen messages from a producer's call to a durable commit — and a precise statement of what message 11 promises.

Editable source SVG draw.io All views
Producer SDK Envoy Gateway Ingest API Entra / SPIRE Valkey Dedup Kafka Buffer Sealer 1. POST /v1/events — batch of 500 2. validate workload token (mTLS bound) 3. principal + tenant_id 4. forward with observed identity 5. schema, vocabulary, 256 KB cap 6. secret scan — reject, never redact 7. SETNX idempotency_key, TTL 24 h 8. new (or duplicate: return first event_id) 9. produce · acks=all · isr ≥ 2 10. offset durable in 3 zones 11. 202 + commit receipt (event_id, offset) 12. consume within the seal window 13. seal receipt queryable ≤ 60 s later Synchronous Ingest — What the Acknowledgement Actually Promises Question 3 answered. The acknowledgement at message 11 promises the record is durable across a zone loss and will be sealed; it does not yet promise the object is under a retention lock. That gap is bounded at 60 seconds, measured, and alerted — an acknowledged record that never reaches a segment is a Sev-1 defect, not a gap. v 1.0 · owner Data & AI Global Practice · date 2026-09

Question 3 answered

  • The acknowledgement is given at the durable buffer. It promises the record survives the loss of an availability zone and will be sealed; it does not promise the object is already under a retention lock.
  • Acknowledging only at object commit would make the promise exact and push ingest latency into seconds, breaking the 150 ms p99 and, with it, the fail-closed classes that depend on a synchronous answer.
  • The gap is bounded, measured and alerted: sealing within 60 s, and an acknowledged record that never reaches a segment is a Sev-1 defect with RPO 0, not an accepted gap. Buffer administration carries no delete verb and no human standing access.

The receipt contract

  • The commit receipt returns event_id and partition offset at acknowledgement; the seal receipt, queryable within 60 s, returns segment_id and the checkpoint that covers it.
  • A producer that needs custody-grade certainty polls for the seal receipt. Most do not, and the two-stage receipt is what lets them make that choice knowingly.
  • Duplicate submissions inside 24 h return the first event_id rather than creating a second record — a retry after a timeout is not a second account of one action.

Risks

  • Between acknowledgement and seal there is a window in which a sufficiently privileged Kafka administrator could in principle destroy data. Mitigated by removing topic-delete from every role, PIM-gating broker administration and alerting on any use of it; the 60 s bound is what makes the exposure small rather than absent.