Publish-Subscribe

Definition

A publisher sends to a topic; subscribers register interest and receive copies. The publisher does not know the subscriber set. The central design question is where the fan-out happens.

Fan-out strategies

Fan-out on write (push). When the message is published, write a copy into each recipient's inbox or push it to each connected client. Reads are then trivial. Cost is proportional to the number of recipients and paid at write time.

Fan-out on read (pull). Store the message once; each recipient assembles their view when they look. Writes are cheap; reads are expensive and repeated.

Hybrid. Push for the common case, pull for the pathological one. This is what most large systems converge on, because the distribution of recipient counts is extremely skewed: nearly all messages go to a handful of people and a tiny number go to hundreds of thousands.

Industry example

Slack's real-time message delivery makes the trade concrete. A message in a three-person direct message must reach three connected clients. A message in a company-wide channel with 100,000 members must reach every one of them that is currently online, within a second, and must also be available when the others reconnect.

Two distinct problems live inside that. Delivery to connected clients is a live fan-out over persistent connections, bounded by how many sockets a server can hold and how quickly it can write to them. Durability for disconnected clients is a storage and catch-up problem: a client that reconnects after two hours must efficiently learn what it missed without replaying everything.

The architectural consequences: connections are sharded so that the members of one channel are reachable from a bounded set of servers; presence — who is online — becomes its own high-write, low-value-per-write subsystem that must be allowed to be approximate; and huge channels are treated as a special case rather than as the general one, because designing everything for the 100,000-member channel would make the three-person case absurdly expensive.

Failure scenarios

• The celebrity problem. One publisher with millions of subscribers makes write-time fan-out catastrophic. Pull for those; push for everyone else.
• Presence treated as strongly consistent. Enormously expensive, and nobody cares if the green dot is three seconds stale.
• Slow subscribers blocking publication. A subscriber that cannot keep up must be buffered boundedly and then dropped or disconnected, not allowed to apply backpressure to everyone.
• No catch-up protocol. Clients reconnect and either miss messages or re-download everything, causing a thundering herd after any network blip. Reconnection storms after a partial outage are a classic self-inflicted second outage.

Trade-offs

Push gives low read latency and predictable read cost, and it multiplies write cost by the subscriber count. Pull gives cheap writes and unpredictable, repeated read cost. The right answer depends on the distribution of subscriber counts, not the average — and the average is particularly misleading here.

Interview question

"Design message delivery for a chat product with channels ranging from 2 to 200,000 members. Where does the fan-out happen, and does the answer change with channel size?"