At 03:40 every insert into the events table starts failing with an out-of-range error for the id column. The table holds about 2.1 billion rows, nothing was deployed, reads are unaffected. What happened, and what is the recovery while traffic continues?

The trigger

A 32-bit signed integer tops out at 2,147,483,647. The sequence behind the id column reached it, and every subsequent insert fails. The row count in the question is the giveaway: the table is at the limit of its key type, not of its storage, its index or its machine.

This is one of the few production failures with an exact, known arrival date, computable at any time from the sequence's current value and the insert rate - and it is also one of the most frequently hit, because the default integer width in a schema written years earlier is rarely revisited.

Why nobody saw it coming

Nothing degrades first. Insert latency is flat at 2.1 billion rows just as it was at 2 billion, and then every write fails at once. There is no gradual signal to alert on unless you create one, and the obvious dashboards - disk, CPU, replication lag, slow queries - all look normal right up to the failure.

The recovery, in order

1. Stop the bleeding. If the id's only job is uniqueness, the sequence can be reset to the negative range: a 32-bit signed integer also holds values down to -2,147,483,648, which is another 2.1 billion values available immediately with an ALTER SEQUENCE and no table change. It is ugly and it buys the time the real fix needs. Verify first that no application code, foreign system or URL scheme assumes a positive id.
2. Add a new 64-bit column alongside, nullable, with no default. This is a metadata-only change on a modern Postgres.
3. Backfill in batches, bounded by time and row count, with a sleep between batches so vacuum and replication keep up. At a few thousand rows per batch, 2.1 billion rows is days of wall-clock time - plan for it rather than discovering it.
4. Keep the two columns in step with a trigger or a dual-write for new rows during the backfill.
5. Build the unique index on the new column concurrently, then swap: switch the primary key and the sequence default inside one short transaction, and drop the old column later, once nothing references it.

The tempting local fix and why it hurts

ALTER TABLE ... ALTER COLUMN id TYPE bigint is one line, and on a table this size it rewrites the whole table under an exclusive lock - hours of total unavailability for that table, plus every foreign key referencing it. It is the right move on a small table and a severe outage on this one.

The structural fix

Use 64-bit keys by default. The extra 4 bytes per row costs single-digit gigabytes at this scale, and 9.2 × 10^18 values is not a limit anyone reaches. The exceptions are narrow lookup tables where the width genuinely matters in an index that is read billions of times.

When this is the wrong answer

If the id has semantic meaning outside the database - printed on documents, embedded in partner integrations, used in a sharding function - a type change is a contract change and the negative-range trick may be the only safe move at all, followed by a versioned external identifier rather than a column migration.

The alert that would have caught it

Sequence headroom as a ratio: last_value / max_value for every sequence, alerting at 70%. It is a cheap query over the catalogue, it runs in seconds, and it converts the one failure class with a computable due date into a ticket rather than an incident.