LinkedIn Professional Network  ·  View 10 of 30  ·  4 · Data

Data Architecture and Ownership

Record, derived, log and lake: which data is backed up and which is rebuilt.

Editable source SVG draw.io All views
Systems of record · replicated and backed up, never rebuilt Espresso Profiles member URN key Posts + comments author key Connections canonical pair key Conversations conversation key MySQL Credentials adaptive hashes Applications idempotent writes Credits + billing ledger Ambry Media originals immutable blobs Derived online · rebuildable from the log or the lake Graph and search LIquid 270B edges Galene base + live index Feed FollowFeed actor timelines Feed inboxes Venice hybrid Serving Venice recs · features Couchbase cache · TTL Pinot member analytics Log and lake · the replay source for everything above Log Kafka 7 days + compacted Lake HDFS + Iceberg OpenHouse tables Feature store Feathr DataHub lineage · ownership CDC Gobblin LinkedIn — Data Architecture and Ownership Data store Queue / topic Security / platform event / async batch Anything in the middle boundary can be rebuilt; only the top boundary needs backups. v 1.0 · owner Data Architecture · date 2026-09

Decisions

  • Only the top boundary is backed up. Everything below it is rebuilt from Kafka or the lake, which costs less than backing it up and restores faster
  • A connection is written once, keyed by the ordered pair of members; each member's adjacency list in LIquid is derived from that row
  • Credentials, applications and credits live in MySQL because they need multi-row transactions; everything else member-facing lives in Espresso

Retention

  • Kafka keeps event topics for 7 days and keeps change topics compacted indefinitely
  • The lake keeps data for years, governed by DataHub ownership and purpose tags
  • Deleting a member purges the record stores and sends tombstones through the log, so every derived store forgets them too

Risks

  • A derived store whose source topic was not compacted cannot actually be rebuilt, which makes it a backup gap. DataHub lineage is checked for every derived store
  • Member deletion is expensive in the lake, where Iceberg partitions have to be rewritten
◀  Integration Catalogue All views Data Flow — Online, Nearline, Offline  ▶
Solution Architecture v1.0 · LinkedIn's own stack in LinkedIn-run colos · Enterprise Architecture · 2026-0910-data-architecture-zones