LinkedIn Professional Network  ·  View 19 of 30  ·  5 · Runtime

Search — Indexing and Query

An offline base index, a live buffer fed in seconds, and a federated query that personalises by graph distance.

Editable source SVG draw.io All views
Source Transport Build Index Serve Base index · weekly HDFS snapshots Azkaban DAG Index build Hadoop · Spark Base index shards Galene · Lucene Live updates · seconds Espresso · MySQL Brooklin → Kafka Doc builder Samza Live update buffer in memory Query · ≤ 300 ms Member query API layer Rewrite + spell query tagging Typeahead prefix index Federated broker 7 verticals Personalised ranker graph distance scatter · gather Search — Indexing and Query Data store Application we own Queue / topic Person or role Interface / broker synchronous Each searcher serves base shard plus live buffer; the base index is rebuilt so deletes never accumulate. v 1.0 · owner Search Engineering · date 2026-09

Decisions

  • Galene, on Lucene: a base index rebuilt offline plus an in-memory live buffer fed by change capture. Each rebuild purges deleted documents and rebalances shards
  • One federated broker queries seven verticals (people, jobs, companies, posts, groups, schools, skills) and blends them by query intent
  • Personalisation uses graph distance: a second-degree connection outranks a stranger with the same name

Numbers

  • Search p95 under 300 ms; typeahead under 100 ms (assumed)
  • Indexing is eventually consistent, within seconds; the base index is rebuilt weekly (LinkedIn, 2014)
  • Semantic job search now adds embedding-based retrieval on GPUs (LinkedIn, 2025)

Risks

  • Privacy must be enforced twice: opted-out profiles are never indexed, and viewer-dependent visibility is checked at query time
  • If Galene is down, search degrades. Profile, feed and apply never call it
◀  Message Delivery — Ordered, Online and Offline All views Notifications — Generate, Decide, Deliver  ▶
Solution Architecture v1.0 · LinkedIn's own stack in LinkedIn-run colos · Enterprise Architecture · 2026-0919-search-indexing-and-query