Webhook Reliability
The delivery, retry, ordering, verification and replay concerns that separate a production webhook system from a fire-and-forget HTTP POST.
Webhooks invert the usual dependency: your system now makes outbound calls to endpoints you do not control, that may be slow, down, or misconfigured, and whose failure is your operational problem.
What a robust implementation requires. Asynchronous dispatch through a queue, so a slow consumer never blocks the business transaction that triggered the event. Retries with exponential backoff and jitter, over a period long enough to survive a consumer's outage — hours to days — with a dead letter path afterwards. Signature verification so the consumer can confirm authenticity, using an HMAC over the payload with a shared secret and a timestamp to prevent replay. Idempotency, since retries mean duplicate delivery is certain, so every event carries a stable identifier. And a replay and inspection interface, because consumers will lose events and will ask.
The properties to state explicitly in the documentation, since they surprise integrators: delivery is at-least-once, ordering is not guaranteed — a retried event arrives after later ones — and consumers must therefore be idempotent and should reconcile against a query API rather than treating the event stream as authoritative.
The security concern that is easy to miss on the sending side: consumer-supplied URLs are an server-side request forgery vector, so outbound targets must be validated against internal address ranges and the metadata endpoint.