LinkedIn Professional Network  ·  View 12 of 30  ·  4 · Data

Event Backbone

Who publishes, who reacts, and why a publisher never knows its consumers.

Editable source SVG draw.io All views
Publishers · event · partition key on the arrow Identity + Profile UserCreated · ProfileUpdated Social graph ConnectionCreated Content PostCreated · Liked Messaging MessageSent Jobs JobApplied Tracking views · clicks The log Kafka 100k+ topics · Avro Consumers Feed fan-out PostCreated Search indexers entity changes ATC notifications graph · content · jobs Recommendations tracking · graph Pinot tracking Anti-abuse (CASAL) signups · posts member pair author conv_id job member lag ≤ 5 s lag ≤ 10 s lag ≤ 1 s minutes lag ≤ 1 min lag ≤ 5 s LinkedIn — Event Backbone Application we own Queue / topic Data store Security / platform event / async No consumer is known to its publisher; adding a consumer never needs a publisher release. v 1.0 · owner Data Architecture · date 2026-09

Decisions

  • Kafka is the system's memory. Every state change and every member action becomes an event, and consumers are added without touching publishers
  • Schemas are Avro in a registry, with compatibility checked at build time: a breaking schema change fails CI
  • Change events come from change capture (Brooklin), not from services publishing after a commit, so a crash between the write and the publish cannot lose an event

Reading the arrows

  • On the publisher side, each arrow is labelled with the topic's partition key: member, pair, author, conversation or job, as section 15 of the requirement asks
  • On the consumer side, each arrow is labelled with that consumer's lag target. Lag is the SLO a stream consumer is paged on
  • The Avro schema registry and Brooklin (change capture, cross-colo mirroring) sit beside Kafka; they are drawn on views 08, 11 and 19

Published numbers

  • 7T messages a day, 100k+ topics, 4,000+ brokers (LinkedIn, 2019)
  • Brooklin replaced Kafka MirrorMaker for cross-colo mirroring, shrinking hundreds of mirror clusters to about a dozen (LinkedIn, 2019)

Risks

  • Tracking events dwarf business events, so they run on separate clusters. A tracking surge cannot delay a MessageSent
  • Order holds only within a partition. Consumers that need order key by member or conversation
  • Every topic also lands in HDFS hourly through Gobblin; that path is drawn on views 10 and 11, not here
◀  Data Flow — Online, Nearline, Offline All views Core Data Model  ▶
Solution Architecture v1.0 · LinkedIn's own stack in LinkedIn-run colos · Enterprise Architecture · 2026-0912-event-backbone