LinkedIn Professional Network  ·  View 11 of 30  ·  4 · Data

Data Flow — Online, Nearline, Offline

The same data moving at three speeds, and the rule for which speed computes what.

Editable source SVG draw.io All views
Capture Transport Compute Serve Online · milliseconds Rest.li request Direct call D2 client-side LB In-process model Quasar scoring Response p95 ≤ 300 ms Nearline · seconds Change capture Brooklin Kafka tracking + changes Samza jobs stateful Venice hybrid nearline writes Offline · hours Gobblin ingest HDFS + Iceberg Spark on Azkaban daily DAGs Venice push versioned bulk swap hourly features Data Flow — Online, Nearline, Offline Interface / broker Application we own Queue / topic Data store batch synchronous Same data, three speeds: the online tier never computes what nearline or offline can precompute. v 1.0 · owner Data Architecture · date 2026-09

Why three tiers

  • Online answers in milliseconds but can only look things up. Offline can compute anything but is hours stale. Nearline sits between, seconds stale with bounded compute
  • Each result is computed in the cheapest tier that meets its freshness need: People You May Know offline, feed features nearline, privacy online
  • Venice unifies the output: batch pushes and nearline writes land in the store the online tier reads

Published numbers

  • Venice: 200+ hybrid datasets with nearline writes; about 500 Voldemort read-only stores migrated by 2018 (LinkedIn, 2022)
  • Galene: base index rebuilt offline, e.g. weekly, with live updates in seconds (LinkedIn, 2014)

Risks

  • Two paths computing the same feature can disagree (training/serving skew). Feathr defines a feature once for both
  • A late batch push can overwrite fresher nearline data; hybrid stores order the two by version
◀  Data Architecture and Ownership All views Event Backbone  ▶
Solution Architecture v1.0 · LinkedIn's own stack in LinkedIn-run colos · Enterprise Architecture · 2026-0911-online-nearline-offline