Incident Management Platform  ·  View 16 of 34  ·  4 · Data

Alert Ingest — Untrusted In, Normalised Out, Nothing Lost

Two transports into one normalised stream, with the raw payload travelling beside every normalised alert and every failure kept and counted.

Editable source SVG draw.io All views
Sources Alertmanager grouped upstream Vendor email SMTP Edge HAProxy edge quota · 429 Postfix inbound per-integration address Gateway Verify HMAC · replay window Normalise CEL · schema check Buffer Raw and normalised one message, both Quarantine schema failures Consume Incident engine per-integration consumer Quarantine monitor threshold · SEV4 Keep Raw payloads ClickHouse · 90 d Integration owner low-severity incident HTTPS SMTP parsed valid invalid sink over threshold Alert Ingest — Untrusted In, Normalised Out, Nothing Lost External / third party Interface / broker Security / platform Application we own Queue / topic Risk / gap Decision point Data store Person or role synchronous event / async failure / alternate batch Source and receive timestamps are both kept and never reconciled. A malformed payload is kept and counted, never rejected silently. v 1.0 · owner Reliability Architecture · date 2026-09

Decisions

  • Normalisation is a per-integration CEL expression, versioned and evaluated in a sandbox with a time and memory limit. A mapping is data that the integration owner can change without a deploy, and a bad mapping cannot hang the gateway (ADR-10).
  • The raw payload and the normalised alert are one message. Normalisation logic will be wrong for some integration, and keeping both together is the only way to find out which, and to re-normalise the last 90 days when the mapping is fixed.
  • Source and receive timestamps are both kept and never reconciled. Ordering uses receive time; the difference is reported per integration, because a source clock that drifts is a finding, not a nuisance.

Targets

  • 2,000 alerts a second sustained per site, 20,000 a second for 120 seconds. Deduplication on source, source key and fingerprint within a 24-hour window.

Risks

  • Email ingest is the least structured and the most likely to be spoofed. Each integration gets its own address and a sender allowlist, and email-sourced alerts cannot propose SEV1.
◀  Incidents, Coverage and Evidence — Data Model All views Coverage Snapshot — Resolved Ahead, Verified Twice, Read in One Lookup  ▶
Solution Architecture v1.0 · Reliability Architecture · 2026-09 · 34 views · open source, fully on-premises16-ingest-pipeline