Visibility Timeout

The mechanism that makes at-least-once delivery work without a persistent connection. A consumer receives a message; it disappears from the queue for the visibility timeout; if the consumer deletes it before the timeout expires, it is gone; if not, it reappears for someone else.

Getting the duration wrong causes the two most common queue bugs. Too short and a consumer still legitimately processing a slow message has it redelivered to a second consumer, so the work happens twice concurrently — which is the source of duplicate charges, duplicate emails and duplicate records, and which is invisible in testing because test messages are fast. Too long and a message whose consumer crashed sits invisible for that whole duration before anyone retries it, adding directly to recovery time.

The right value is derived from the tail of processing time, not the average, and it should be extended dynamically for long operations where the client library supports it (a heartbeat that extends visibility while work continues).

And the corollary that never goes away: because redelivery is normal rather than exceptional, consumers must be idempotent.