Architecture Decision Record — LinkedIn Professional Network

Thirty-three decisions that make up this architecture. Everything else on these thirty views is convention, and convention needs no defending. Each record is written to teach as well as to record. It opens with the forcing question and the context, then the decision, then what LinkedIn actually did and when. It weighs the credible alternatives, some rejected and some right for anyone who is not LinkedIn, and states what the choice buys and costs, the conditions that would flip it, and the transferable lesson. The bias throughout is LinkedIn's own stack. Where LinkedIn open-sourced a system, the open-source project is named. Where the system is internal, the portable equivalent is named. Where LinkedIn has not published how it does something, the record says so and the decision is this design's own.

Evidence. LinkedIn facts come from LinkedIn's engineering blog, Apache project records and LinkedIn's newsroom, and each record carries its dated sources. Figures are as published on those dates, not current counts. Three items are widely reported but were not traced to a primary LinkedIn source for this record: the Leo monolith, the 2011 Project Inversion freeze and the 2012 password breach. They are marked where used.

How to read a record

• Question: The forcing question: why a decision was needed at all.
• Context: The requirement, the scale and the constraint that make it hard.
• Decision: What this architecture does, stated so it can be checked.
• What LinkedIn did: The published history, dated, or a note that nothing was published.
• Options weighed: Chosen, rejected, deferred, or right elsewhere, with the reason for each.
• Consequences: What the choice buys and what it costs, both kept visible.
• Choose differently when: The conditions that would flip the decision for your system.
• Lesson: The principle that transfers to systems that are not LinkedIn.

Decision map

Foundations: The two things that must never change, so everything else can.

• ADR-01 · Hold identifiers and API schemas fixed; let service boundaries move
• ADR-02 · Every entity is identified by a typed URN, never by mutable data

Edge and API: How a request reaches a colo, and how services agree on contracts.

• ADR-03 · Azure Front Door at the global edge; Apache Traffic Server pins members to a colo
• ADR-04 · Rest.li resources between services; GraphQL generated from them for clients
• ADR-05 · D2: client-side service discovery and load balancing

Identity and privacy: Who the caller is, what they may see, and how secrets stay secret.

• ADR-06 · Own the identity service: adaptive hashing, risk-based MFA, short-lived tokens
• ADR-07 · Authorize at the data: central service ACLs plus a member-privacy check in every read path
• ADR-08 · Workload identity, mTLS, envelope encryption and short-lease secrets

Data: Systems of record, derived serving stores, and the lake.

• ADR-09 · Espresso is the primary store for member-facing entities
• ADR-10 · Sharded MySQL where money and irreversible state need multi-row transactions
• ADR-11 · Venice serves all derived key-value data
• ADR-12 · Couchbase caches hot entities, invalidated from the change log
• ADR-13 · LIquid serves the graph from memory; an Espresso edge store records it
• ADR-14 · Galene search: an offline base index, a live buffer, federated verticals
• ADR-15 · Ambry holds media; clients upload directly on signed URLs; the CDN delivers
• ADR-16 · Apache Pinot for analytics shown to members
• ADR-17 · Offline platform: HDFS + Iceberg via OpenHouse, Spark, Gobblin, Azkaban, DataHub

Streams: The log, change capture, and nearline compute.

• ADR-18 · Kafka is the central log for every change and every action
• ADR-19 · Change capture with Brooklin, never dual writes
• ADR-20 · Samza runs the nearline jobs

Product systems: Feed, messaging, notifications, jobs and relevance.

• ADR-21 · Hybrid fan-out: push for normal authors, pull for high-degree ones
• ADR-22 · Ranking is separate from storage, and content is decorated last
• ADR-23 · Messages keyed by conversation, ordered by one sequencer per conversation
• ADR-24 · Real-time delivery over Server-Sent Events, with presence and push fallback
• ADR-25 · Concourse generates notifications; ATC decides; neither is on a core path
• ADR-26 · Applications are transactional; job recommendations are derived
• ADR-27 · One ML platform (Pro-ML) for feed, search, jobs and people recommendations

Operations: Retries, colos, releases, observability and degradation.

• ADR-28 · Every retryable write takes an idempotency key
• ADR-29 · Four active colos, members pinned, and draining a colo as routine
• ADR-30 · Deploy dark, judge by canary, release by ramp
• ADR-31 · Observe member impact: SLO alerts, business anomaly detection, overload shedding
• ADR-32 · A written graceful-degradation contract, per capability

Assurance: Abuse, fraud and moderation.

• ADR-33 · Layered abuse defence writing one auditable decision log

Technology by capability

What each capability is built from, where that technology came from, and what to use if you are not LinkedIn. Every row links to the record that argues it. Origin labels: LinkedIn · open source means LinkedIn built it and released it; LinkedIn · internal means LinkedIn built it and has not released it; This design means the requirement is met by a pattern rather than a named LinkedIn system.

The decisions, and the alternatives that lost

Foundations

The two things that must never change, so everything else can.

ADR-01 · Hold identifiers and API schemas fixed; let service boundaries move

Status: Accepted · Shown on views: 02, 07, 13

How do we start small without re-platforming when the network reaches a billion members?

Context. The requirement asks for a platform that can grow from a monolith into a globally distributed system without changing its core data and API contracts. Service boundaries are the thing most likely to be wrong early, and the thing most expensive to change once clients depend on them.

Decision. Two things are permanent contracts from day one: typed URN identifiers (ADR-02) and schema-defined Rest.li resources (ADR-04). Deployable units may start coarse, one per domain group, and split later behind the same resources. Clients never learn which process serves a resource.

What LinkedIn actually did. LinkedIn began as a single Java application, widely reported as Leo, and decomposed over years into hundreds of services. In 2011 it paused feature work for about two months (Project Inversion, widely reported) to rebuild its build and release tooling, because the decomposition had outgrown it. Rest.li, D2 and URNs are what allowed services to split without clients noticing.

What it buys

• Clients are unaffected when a service is split or merged
• Teams can own resources before they own processes
• A modular monolith is a legitimate starting point

What it costs

• Schema discipline and compatibility checks are required from the first commit
• Coarse early services have a larger blast radius
• Some resource boundaries will still prove wrong and need versioning

Choose differently when. If the product is unproven and one team builds it, start with a modular monolith that uses the same URNs and resource schemas. Split only when a team, a scaling profile or a failure domain demands it.

Lesson. Stable identifiers and schemas are the architecture. Service boundaries are an implementation detail you should be able to move.

Sources: Rest.li: RESTful service architecture at scale (LinkedIn, 2013)

ADR-02 · Every entity is identified by a typed URN, never by mutable data

Status: Accepted · Shown on views: 13, 14, 16

What identifies a member when their name, email, vanity URL and employer all change?

Context. The requirement asks for a stable user ID independent of mutable profile data. Members change their email, their custom URL, even their name; a company is renamed; a post is edited. Anything keyed on those values breaks references across the estate.

Decision. Every entity gets a typed URN, such as urn:li:member:123 or urn:li:post:456, assigned once, never reused and never derived from data a user can change. Every reference across services, events and stores is a URN. The vanity URL is a unique but mutable attribute that resolves to a URN.

What LinkedIn actually did. LinkedIn uses typed URNs of the form urn:li:: throughout its services and its public APIs, where they are visible to any developer.

What it buys

• References never break on a rename
• Deletion propagates as URN tombstones through the log
• Feed and search can store references and resolve content at read time

What it costs

• Every read has to hydrate URNs into content
• URN parsing and validation sit in every service's client library
• Merging two duplicate accounts needs an explicit alias mapping

Choose differently when. A small system on one database can use plain surrogate keys. The rule that matters is still the same: never key on anything a user can edit.

Lesson. Never key on something a user can change.

Edge and API

How a request reaches a colo, and how services agree on contracts.

ADR-03 · Azure Front Door at the global edge; Apache Traffic Server pins members to a colo

Status: Accepted · Shown on views: 02, 22, 27

Where do TLS termination, caching, DDoS absorption and colo selection happen?

Context. Members are everywhere; the colos are in a few places. TLS handshakes over long distances dominate first-byte latency. Each member's data is served from a home colo, and operators need a lever that moves members between colos in minutes.

Decision. Azure Front Door terminates TLS close to the member, applies WAF and bot rules, and caches static assets and media. Inside each colo, an Apache Traffic Server tier pins each member to their home colo with a signed cookie. Re-pinning is how a colo is drained for maintenance, for disaster recovery or for a load test.

What LinkedIn actually did. In 2017 LinkedIn described 13 GeoDNS PoPs running ATS with a plugin that pins a user to a data centre through a signed cookie, and TrafficShift, which uses that pinning to drain data centres. In 2020 it moved from its own 19 PoPs to Azure Front Door's 165+ and reported up to 25% faster median page load. ATS remains the proxy tier.

What it buys

• Shorter TLS round trips worldwide
• DDoS absorbed at provider scale
• Colo drain is a routine, rehearsed operation

What it costs

• A dependency on one edge provider
• Two proxy hops, each needing its own observability
• Cookie-based pinning must be re-established when a member is moved

Choose differently when. If you serve from one region, a CDN plus a regional load balancer is enough. If regulation demands a sovereign edge, run your own and accept fewer PoPs.

Lesson. Keep reaching the member fast (the edge) separate from sending them to the right data centre (the traffic tier). They fail, and change, for different reasons.

Sources: TrafficShift: load testing at scale (LinkedIn, 2017) · LinkedIn scales edge with Azure Front Door (LinkedIn, 2020)

ADR-04 · Rest.li resources between services; GraphQL generated from them for clients

Status: Accepted · Shown on views: 02, 08, 09, 23

How do hundreds of services and three client platforms agree on contracts that can evolve?

Context. The requirement asks for versioned, authenticated, observable APIs. Hundreds of teams change services every day, and web, iOS and Android each want pages shaped differently. Hand-written REST drifts; client-specific endpoints multiply.

Decision. Services expose schema-defined Rest.li resources, with backward compatibility checked in CI (ADR-30). Protocol Buffers is the wire encoding between services. Product clients query a GraphQL layer whose types are generated from the Rest.li schemas, so a client can shape a page without a new endpoint and the types cannot drift from the services.

What LinkedIn actually did. Rest.li was open-sourced in 2013. In 2023 LinkedIn moved Rest.li's wire encoding from JSON to Protocol Buffers and reported up to 60% lower latency for services with large payloads. From 2022 to 2023 it adopted GraphQL for integrations and then for product APIs, generating GraphQL types from Rest.li schemas, with no central gateway.

What it buys

• A breaking change fails the build, not a client
• Binary encoding cut latency and CPU on large payloads
• Clients iterate on pages without mid-tier releases

What it costs

• An in-house framework to maintain alongside the industry standard
• Schema tooling is on every team's critical path
• GraphQL queries need cost limits so one query cannot fan out to hundreds of calls

Choose differently when. Greenfield today: gRPC and Protocol Buffers internally, with a GraphQL or BFF edge, is the same design with a larger ecosystem.

Lesson. The protocol matters less than having one schema language with compatibility checks in the build.

Sources: Rest.li (LinkedIn, 2013) · How LinkedIn adopted a GraphQL architecture (LinkedIn, 2023)

ADR-05 · D2: client-side service discovery and load balancing

Status: Accepted · Shown on views: 07, 08, 11

How does a call find a healthy instance without a load balancer on every hop?

Context. A single feed request fans out to dozens of mid-tier calls, and the estate makes millions of calls a second. A central load balancer adds a network hop, a capacity plan and a failure point to every one of them.

Decision. Instances register in a discovery registry. The D2 client library in every caller selects instances with latency- and health-aware balancing, and degrades hosts that misbehave. No load balancer sits between services inside a colo.

What LinkedIn actually did. D2 ships with Rest.li and originally used a ZooKeeper registry. LinkedIn has since described a next-generation discovery system that uses Kafka for writes and xDS/gRPC for reads, replacing the ZooKeeper registry (covered by InfoQ, February 2026).

What it buys

• Lowest possible per-call latency
• No shared balancer to overload
• Health signals come from real callers

What it costs

• Balancing logic lives in every client library and language
• The registry becomes critical infrastructure
• Client library upgrades are needed to change balancing behaviour

Choose differently when. In a polyglot estate, or where the platform team cannot ship client libraries, a sidecar mesh gives the same behaviour without per-language code.

Lesson. At high fan-out, every central hop is paid on every call. Put the decision in the client and keep the registry off the request path.

Sources: Scalable multi-language service discovery at LinkedIn

Identity and privacy

Who the caller is, what they may see, and how secrets stay secret.

ADR-06 · Own the identity service: adaptive hashing, risk-based MFA, short-lived tokens

Status: Accepted · Shown on views: 27, 28

How do 1.2 billion members sign in safely without making every sign-in painful?

Context. The requirement lists registration, email and phone authentication, social login, MFA, sessions, recovery and deletion, with strong consistency for account changes. At this scale credential stuffing never stops, and an account takeover is a reputational event.

Decision. Credentials live in sharded MySQL, hashed with argon2id using a per-member salt and a pepper held in KMS. MFA offers passkeys and TOTP, with SMS as a last resort, and is required by risk: a new device, an unusual IP or unusual velocity. Sessions use a short-lived signed access token plus a rotating refresh token bound to the device; revocations travel as Kafka events to every colo. The same service is an OAuth 2.0 and OIDC provider for third-party apps, and accepts SAML for enterprise seats.

What LinkedIn actually did. LinkedIn has not published a named identity system. The widely reported 2012 breach exposed unsalted SHA-1 password hashes (6.5 million were posted that year, and a set of about 117 million surfaced in 2016). It is the textbook case for adaptive, salted hashing.

What it buys

• A leaked hash table is expensive to crack
• Most sign-ins have no friction; risky ones are challenged
• A revocation reaches every colo in seconds

What it costs

• A security-critical service to staff and audit
• Risk scoring can lock out legitimate travellers
• SMS as a factor is defeated by SIM swaps, so it is not offered on privileged seats

Choose differently when. Below tens of millions of users, or without a dedicated identity team, buy a managed CIAM and spend the effort on product.

Lesson. Hash for the breach you will eventually have.

ADR-07 · Authorize at the data: central service ACLs plus a member-privacy check in every read path

Status: Accepted · Shown on views: 16, 27, 29

How is privacy enforced when member data is copied into caches, search indexes and feeds?

Context. The requirement lists public, connections-only and private content, blocks, company and recruiter permissions, and applicant privacy, all enforced on the server. Member data is copied into Galene, FollowFeed, Venice and Couchbase, and each copy is another place a check can be forgotten.

Decision. There are two layers. Service-to-service authorization answers whether service X may call resource Y, through a central ACL service whose decisions are cached in each caller. Member privacy answers whether this viewer may see this field of this member. It is a policy library that every Rest.li resource returning member data calls, and decoration calls again, using the block list (both directions), graph degree and per-field visibility settings. A blocked viewer receives 404, not 403.

What LinkedIn actually did. LinkedIn described a central ACL service in 2019. It has a Rest.li API, is managed through Nuage, is enforced inside each application with local caching, and sends access logs through Kafka. Member-level privacy enforcement has no single published system; this layer is this design's.

What it buys

• A new copy of member data inherits the check at read time
• A block cannot be discovered by probing
• Every recruiter and admin read is auditable

What it costs

• Every read path pays for the block and degree lookups, so both are cached
• The library has to ship in every service, and a test must assert each resource calls it
• Settings change faster than derived stores update, so privacy is applied at read, never precomputed

Choose differently when. On one database, row-level security plus gateway checks suffice. Relationship-based authorization services (SpiceDB, OpenFGA) are the managed form of this pattern.

Lesson. Authorize at the data, not only at the door. Every derived copy of member data is another door.

Sources: Authorization at LinkedIn's scale (LinkedIn, 2019)

ADR-08 · Workload identity, mTLS, envelope encryption and short-lease secrets

Status: Accepted · Shown on views: 27

What stops a compromised host or a leaked config file becoming a data breach?

Context. The requirement lists encryption in transit and at rest, secret management, audit logging and least-privilege service access. With hundreds of services and thousands of hosts, network position is a weak proof of identity.

Decision. Every service has a workload identity and presents a short-lived certificate: all service-to-service traffic is mTLS. Data at rest is encrypted with per-dataset data keys, wrapped by master keys that never leave KMS or an HSM. Secrets are fetched at start-up on short leases and never written to config. Access and admin actions are logged to Kafka and land in immutable storage.

What LinkedIn actually did. LinkedIn has published its ACL service and the Kafka-carried access logs (2019). It has not described its KMS or secrets tooling publicly, so that part is the requirement's design.

What it buys

• A stolen config file holds nothing reusable
• Keys can be rotated without re-encrypting all data
• Every access leaves an immutable record

What it costs

• Certificate issuance and rotation is critical infrastructure
• KMS latency and availability sit on start-up paths
• Audit volume is large and needs its own retention policy

Choose differently when. On a cloud, use the provider's workload identity, KMS and secret manager instead of running your own.

Lesson. The perimeter is the workload's identity, not the network it sits on.

Sources: Authorization at LinkedIn's scale (LinkedIn, 2019)

Data

Systems of record, derived serving stores, and the lake.

ADR-09 · Espresso is the primary store for member-facing entities

Status: Accepted · Shown on views: 08, 10, 13

Where do profiles, posts, comments, connections, conversations and jobs live?

Context. These entities are read far more than written, partition naturally by a key (member, author, conversation), need schema evolution without downtime, and need a change stream so that search, feed, graph and cache can follow them.

Decision. Espresso stores them as schema-defined documents, hash-partitioned by key. Transactions are allowed within one partition, and each partition has a timeline-ordered change stream that Brooklin captures (ADR-19). Apache Helix manages partition placement and failover.

What LinkedIn actually did. LinkedIn introduced Espresso publicly in 2015 as its distributed document store. It has MySQL/InnoDB storage nodes, stateless HTTP routers, Helix for cluster management, change capture first through Databus and later through Brooklin, Avro documents and hash partitioning. At the time it served about 30 applications, including Member Profile, InMail and parts of the homepage. It replaced Oracle for those uses, and it is not open source.

What it buys

• Resharding and schema evolution are routine
• Every change is published without application code
• Reads scale with replicas

What it costs

• Transactions stop at the partition boundary
• An internal database must be staffed and developed
• Cross-partition queries go to derived stores, never to Espresso

Choose differently when. Outside LinkedIn, choose Vitess for relational workloads, or a managed document store with change streams (Cosmos DB, DynamoDB) for document workloads.

Lesson. A primary store's most important feature is its change log, because everything derived hangs off it.

Sources: Introducing Espresso (LinkedIn, 2015)

ADR-10 · Sharded MySQL where money and irreversible state need multi-row transactions

Status: Accepted · Shown on views: 10, 13, 17

Which data needs the strongest consistency, and where does it live?

Context. The requirement asks for stronger consistency on authentication, account changes, job applications, connection state and financial or ad-related operations. Espresso's transactions stop at a partition. A job application touches the member, the job and the idempotency key, and an InMail send debits a credit.

Decision. Credentials, job applications, InMail credits and billing live in sharded MySQL with semi-synchronous replication inside the colo. Unique constraints enforce idempotency keys and one application per member per job, and state transitions run in one transaction. Connection state stays in Espresso, where the ordered-pair key makes it a single-row compare-and-set (ADR-13).

What LinkedIn actually did. MySQL is widely used at LinkedIn; Espresso's own storage engine is MySQL. Which domains sit on plain MySQL here is this design's decision.

What it buys

• Money and applications are correct by construction
• Retries are made safe by the database, not by application code
• Mature tooling for backup and point-in-time recovery

What it costs

• Resharding MySQL is a planned project
• Cross-colo failover of these domains needs a reconciliation step
• Two primary-store technologies to operate

Choose differently when. If a domain truly needs serializable writes across regions at once, distributed SQL earns its latency.

Lesson. Use the strongest consistency only where money or irreversible state is involved, and there, make it truly strong.

ADR-11 · Venice serves all derived key-value data

Status: Accepted · Shown on views: 10, 11, 15

Where do recommendations, features and feed inboxes live, which are computed rather than entered?

Context. People You May Know, job recommendations, ranking features and feed inboxes are recomputed in bulk by offline jobs and refreshed nearline by stream jobs. Writing terabytes back into the primary store would crush it, and a bad batch must be reversible.

Decision. Venice holds every derived key-value dataset. An offline push creates a complete new version that is swapped in atomically, and the previous version is kept for rollback. Hybrid stores also accept nearline writes from Samza, ordered after the batch by version.

What LinkedIn actually did. Project Voldemort (open-sourced 2009) served read-only, Hadoop-built stores such as People You May Know. About 500 of those use cases had moved to Venice by 2018. LinkedIn open-sourced Venice in 2022: batch pushes from the Hadoop grid, 200+ hybrid datasets with nearline writes, serving People You May Know and Jobs You May Be Interested In, and backing the Feathr feature store.

What it buys

• A bad push is rolled back by swapping versions
• The system of record never sees derived traffic
• Every derived dataset can be rebuilt from the lake

What it costs

• Storage for the previous version of every dataset
• Batch and nearline writers must agree on versioning
• Data is only as fresh as the last push plus the nearline stream

Choose differently when. Outside LinkedIn, Venice itself is open source. Cassandra or ScyllaDB with bulk loading, or Redis for small derived sets, are the usual alternatives.

Lesson. Derived data deserves its own store: versioned, bulk-swappable, rebuildable, and never in the system of record.

Sources: Open-sourcing Venice (LinkedIn, 2022) · Serving batch-computed data with Voldemort (LinkedIn, 2012)

ADR-12 · Couchbase caches hot entities, invalidated from the change log

Status: Accepted · Shown on views: 08, 10, 26

How does a 200 ms profile read survive decoration fan-out without overloading Espresso?

Context. Decorating one feed page hydrates dozens of URNs, and profiles, company pages and popular posts are read millions of times. The requirement asks for event-driven invalidation where practical, and for transactional truth never to depend on a cache.

Decision. Couchbase clusters cache decorated profiles, companies, popular posts, block lists and session lookups. Invalidation is driven by change events from Brooklin, with a TTL as a backstop. The cache is never the only copy of anything, and a miss always falls through to the system of record.

What LinkedIn actually did. LinkedIn has used Couchbase as its caching layer since 2012; in 2018 it reported 10M+ queries a second across 300+ clusters.

What it buys

• Profile reads meet 200 ms at p95
• Espresso is sized for misses, not for fan-out
• Invalidation follows every write, including writes from jobs

What it costs

• A miss storm after a cluster loss is the feed's most common latency incident
• Invalidation lag means a few seconds of staleness
• Another fleet to operate

Choose differently when. Outside LinkedIn, Redis/Valkey or memcached. Keep the rule regardless: invalidate from change events, not from the writer.

Lesson. Invalidate from the change log, not from the writer. The writer forgets; the log does not.

Sources: Evolution of Couchbase at LinkedIn (LinkedIn, 2018)

ADR-13 · LIquid serves the graph from memory; an Espresso edge store records it

Status: Accepted · Shown on views: 10, 17, 29

How do we answer who is connected, mutual connections and second degree for 1.2B members in milliseconds?

Context. The requirement lists these queries and asks for strong consistency on connection state. Second-degree queries touch thousands of edges each, and are run on nearly every profile view, search result and privacy check. High-degree nodes create hot partitions.

Decision. Connection and follow edges are written to Espresso, keyed by the ordered pair of members (for connections) or by follower and followee. This is the strongly consistent truth. LIquid ingests the edge change log and answers traversal queries from memory: connected, mutual count, second degree, who follows A. Follower counts for high-degree accounts are precomputed, and their adjacency is replicated rather than served from one partition.

What LinkedIn actually did. LinkedIn described LIquid in 2020 as a new in-memory graph database with a relational, Datalog-style query model. In 2023 it reported 270 billion edges and 2 million queries a second at 99.99% availability, with People You May Know graph queries falling from over 1 s to under 50 ms on average.

What it buys

• Privacy checks and ranking can afford graph features on every request
• Connection state is never ambiguous
• The graph index can be rebuilt from the edge log

What it costs

• Memory-resident at enormous scale is expensive hardware
• LIquid lags the edge store by seconds, so the two members involved read the edge store just after a change
• An internal database to staff

Choose differently when. Outside LinkedIn, keep adjacency lists in a key-value store and batch-compute second-degree candidates. Use a sharded graph engine (JanusGraph, TigerGraph) if traversal is the product.

Lesson. Keep the graph's truth in a transactional store and its traversal in memory. They have opposite requirements.

Sources: How LIquid connects everything (LinkedIn, 2023)

ADR-14 · Galene search: an offline base index, a live buffer, federated verticals

Status: Accepted · Shown on views: 19, 26

How does search stay fast, fresh and personal across people, jobs, companies and posts?

Context. The requirement asks for full-text search with typo tolerance, filters, ranking, autocomplete and personalised results, with asynchronous, eventually consistent indexing. Incrementally updated indexes accumulate deletes and skew, and each vertical ranks differently.

Decision. Each vertical has a Galene index. A base index is rebuilt offline on the grid and swapped in, and an in-memory live update buffer fed by change capture through Samza covers the gap. A federated broker queries the verticals in parallel and blends by intent. A personalised ranker uses graph distance and member features. Privacy is applied at index time (opted-out profiles are never indexed) and at query time (viewer-dependent visibility). Semantic, embedding-based retrieval is a further candidate source for jobs.

What LinkedIn actually did. LinkedIn described Galene in 2014: a Lucene-based architecture with a base index built offline with Hadoop MapReduce and rebuilt, for example, weekly, plus a live update buffer and a snapshot index. It replaced earlier systems, including the Cleo typeahead (open-sourced 2012). In 2025 LinkedIn described LLM-based semantic job search with embedding retrieval on GPUs.

What it buys

• A rebuild purges deletes and rebalances shards
• Freshness in seconds without incremental decay
• Graph-aware ranking

What it costs

• Two pipelines, offline and live, must agree
• The weekly rebuild is a large recurring batch job
• An internal engine to staff

Choose differently when. Outside LinkedIn, use OpenSearch or Elasticsearch, or Vespa where ranking is the product.

Lesson. Rebuild the index from scratch on a schedule. Incremental-only indexes rot.

Sources: Did you mean Galene? (LinkedIn, 2014) · Semantic search at scale for LinkedIn's Hiring Assistant (LinkedIn, 2025)

ADR-15 · Ambry holds media; clients upload directly on signed URLs; the CDN delivers

Status: Accepted · Shown on views: 10, 21

How do images, video, documents and resumes get in and out without passing through services?

Context. The requirement asks for large objects in object storage rather than the primary database, direct uploads on temporary signed URLs, processing (images, video, malware scanning) and CDN delivery.

Decision. The media service issues signed, time-limited, resumable upload URLs, and clients write chunks straight to Ambry. Originals are immutable; renditions are derived and can be regenerated. A malware scan and a content classifier gate serving. Public media goes through the CDN; resumes and documents are served only on short-lived authorised URLs, never through the public CDN.

What LinkedIn actually did. LinkedIn open-sourced Ambry in 2016 as an immutable, active-active, multi-data-centre object store, accompanied by a SIGMOD 2016 paper.

What it buys

• Services never carry media bytes
• Uploads resume after a dropped connection
• Renditions can be regenerated for new devices

What it costs

• Signed-URL issuance and expiry must be right, or media leaks
• The scanning pipeline sits between upload and visibility
• Two delivery paths, public and authorised

Choose differently when. Outside LinkedIn, use any object store plus a CDN. Keep private documents off the public CDN either way.

Lesson. Keep bytes out of your services: hand the client a signed URL.

Sources: Ambry on GitHub (LinkedIn, 2016)

ADR-16 · Apache Pinot for analytics shown to members

Status: Accepted · Shown on views: 10, 24

How do Who Viewed My Profile, post analytics and recruiter dashboards answer in under a second?

Context. Analytics shown to members is a product feature with consumer concurrency and a latency SLO. Hadoop answers in hours, and operational stores must never run analytical queries.

Decision. Pinot tables ingest tracking and business events from Kafka and serve member-facing analytics: profile views, post analytics, company page and recruiter dashboards. ThirdEye anomaly detection runs on the same data (ADR-31).

What LinkedIn actually did. LinkedIn built Pinot in 2013 and open-sourced it in 2015, naming Who Viewed My Profile as a use case. It became an Apache top-level project in 2021.

What it buys

• Member analytics fresh within a minute
• No analytical load on operational stores
• The same tables power anomaly detection

What it costs

• Another stateful cluster to run
• Tables are pre-shaped for known queries
• Privacy rules (such as anonymised viewers) must be applied in the serving layer

Choose differently when. For internal dashboards with low concurrency, ClickHouse or a warehouse is simpler.

Lesson. Analytics shown to members is a product feature with a latency SLO, not a report.

Sources: Open-sourcing Pinot (LinkedIn, 2015)

ADR-17 · Offline platform: HDFS + Iceberg via OpenHouse, Spark, Gobblin, Azkaban, DataHub

Status: Accepted · Shown on views: 10, 11

Where do training, BI, A/B analysis and batch recommendations run?

Context. The requirement separates operational systems from large-scale analytics, and feeds a data platform used for analytics, recommendation training, A/B testing, BI and fraud detection.

Decision. Gobblin lands every Kafka topic into HDFS hourly. Tables are Apache Iceberg, governed by OpenHouse. Spark jobs run on Azkaban schedules. DataHub records ownership, lineage and purpose, and that lineage is how the platform knows which derived stores can be rebuilt. No analytical query ever runs against an operational store.

What LinkedIn actually did. All of these are LinkedIn projects. Gobblin (2014) became an Apache top-level project in 2021. Azkaban is open source. DataHub (open-sourced 2020) succeeded WhereHows. OpenHouse (open-sourced 2024) is a control plane for Iceberg tables.

What it buys

• Open table formats keep engines replaceable
• Lineage makes rebuildability auditable
• Training data comes from the same log as serving data

What it costs

• A large Hadoop estate to operate
• Hourly landing means BI is hours fresh
• Member deletion requires rewriting Iceberg partitions

Choose differently when. Outside LinkedIn, a lakehouse on object storage (Iceberg with Spark or Trino, Airflow and DataHub) is this exact design in the cloud.

Lesson. Never let analytics touch the operational store. Give it its own copy, taken from the log.

Sources: Open-sourcing OpenHouse (LinkedIn, 2024)

Streams

The log, change capture, and nearline compute.

ADR-18 · Kafka is the central log for every change and every action

Status: Accepted · Shown on views: 11, 12

How do fan-out, indexing, notifications, recommendations, analytics and abuse detection learn about what happened?

Context. The requirement asks for a durable event bus and for services to avoid synchronous dependencies where an asynchronous workflow is enough. There are at least six classes of consumer, and more will come.

Decision. Every state change (through CDC, ADR-19) and every member action (tracking) is an event on Kafka. Schemas are Avro in a registry with build-time compatibility checks. Tracking and business events run on separate clusters. Each colo has local and aggregate clusters, with Brooklin mirroring between them. Event topics keep 7 days; change topics are compacted.

What LinkedIn actually did. Kafka was created at LinkedIn by Jay Kreps, Neha Narkhede and Jun Rao and entered the Apache incubator in 2011. In 2019 LinkedIn reported 7 trillion messages a day across 100+ clusters, 4,000+ brokers, 100,000+ topics and 7 million partitions.

What it buys

• A new consumer is a deployment, not a negotiation
• Derived stores can be rebuilt by replay
• Notifications and indexing never sit on the write path

What it costs

• Order holds only within a partition
• Schema governance is mandatory
• Running Kafka at this scale is a platform team of its own

Choose differently when. With low volume and few consumers, a queue is simpler. Managed Kafka (Confluent, MSK, Event Hubs) removes the platform burden.

Lesson. A log you can replay turns every derived store into a cache you can rebuild.

Sources: Apache Kafka at 7 trillion messages a day (LinkedIn, 2019)

ADR-19 · Change capture with Brooklin, never dual writes

Status: Accepted · Shown on views: 10, 12, 19

How does a committed write reliably become an event?

Context. A service that writes to its store and then publishes to Kafka can crash between the two steps, losing the event or publishing one for a write that rolled back. Every derived store depends on these events being exactly the committed changes.

Decision. Services write only to their store. Brooklin tails each store's change log (Espresso, MySQL) and publishes the committed changes to Kafka in commit order. Brooklin also mirrors Kafka between colos.

What LinkedIn actually did. LinkedIn open-sourced Databus, its first change-capture system, in 2013. Brooklin (open-sourced 2019) replaced Databus for Espresso and Oracle change capture. It also replaced Kafka MirrorMaker, reducing hundreds of mirroring clusters to about a dozen, and it carries 2T+ messages a day.

What it buys

• Derived stores cannot silently diverge
• Services contain no publishing code
• One tool covers CDC and cross-colo mirroring

What it costs

• Events look like database rows, so a domain-event layer is sometimes needed on top
• Schema changes in the store surface in the stream
• CDC lag becomes an SLO

Choose differently when. If you cannot read the database log (for example, some managed databases), use a transactional outbox.

Lesson. Publish from the database's log, not from the application. The log is the only thing that saw every commit.

Sources: Open-sourcing Brooklin (LinkedIn, 2019) · Open-sourcing Databus (LinkedIn, 2013)

ADR-20 · Samza runs the nearline jobs

Status: Accepted · Shown on views: 11, 15, 20

What computes fan-out, index documents, features and notifications in seconds?

Context. Nearline work sits between the request path and the batch grid: stateful (counts, joins, windows), seconds stale, and continuous. It must restart without losing state and without an external database on the hot path.

Decision. Fan-out, search document building, feature updates, ATC and Concourse run as Samza jobs next to Kafka. Local state is backed by changelog topics, so a restarted task rebuilds its state from Kafka.

What LinkedIn actually did. Samza was created at LinkedIn, entered the Apache incubator in 2013 and became a top-level project in January 2015. ATC runs on Samza (LinkedIn, 2018).

What it buys

• Stateful jobs restart without a database
• Nearline lag is measurable per job
• One execution model for every nearline workload

What it costs

• A smaller community than Flink
• Changelog topics add Kafka storage
• Rebuilding a large state store after a failure takes time

Choose differently when. Outside LinkedIn, choose Apache Flink.

Lesson. Local state plus a changelog makes stateful stream jobs restartable without an external database.

Sources: Apache Samza graduates (ASF, 2015)

Product systems

Feed, messaging, notifications, jobs and relevance.

ADR-21 · Hybrid fan-out: push for normal authors, pull for high-degree ones

Status: Accepted · Shown on views: 14, 15, 16

How does a post reach its audience without either millions of writes or a very slow read?

Context. The requirement asks for both fan-out-on-write and fan-out-on-read, with high-follower accounts on read. Pushing a post by an author with millions of followers means millions of writes; pulling for a member who follows thousands means thousands of reads per page.

Decision. Authors under a follower threshold (assumed 10k, tunable) fan out on write: Samza makes one write per follower into a capped Venice inbox. High-degree authors never fan out; their posts are indexed in FollowFeed by actor and pulled at read time. The feed broker merges the inbox, FollowFeed, recommended candidates and ads, and dedupes by post URN.

What LinkedIn actually did. LinkedIn's published FollowFeed (2016) is pull-based: timelines keyed by actor and content type across 720 partitions, with first-pass ranking on the index nodes and second-pass ranking, dedupe and diversity at the broker. The write-time inbox is added here because the requirement asks for the hybrid.

What it buys

• Bounded write cost for any author
• Bounded read cost for any reader
• Storage strategy can change without touching ranking

What it costs

• Two storage paths to operate and keep consistent
• Authors crossing the threshold must move lanes without losing or duplicating posts
• Fan-out lag under a spike is a new SLO

Choose differently when. If most members follow few actors and reads dominate, pure pull is simpler, and LinkedIn runs it.

Lesson. Push makes reads cheap and pull makes writes cheap. Draw the threshold where the cost curves cross, and move it with data.

Sources: FollowFeed (LinkedIn, 2016)

ADR-22 · Ranking is separate from storage, and content is decorated last

Status: Accepted · Shown on views: 14, 16, 26

How can ranking models change daily without rewriting feed storage, and how do edits, deletes and privacy take effect immediately?

Context. The requirement asks that ranking and recommendation evolve independently of feed storage, and that a recommendation failure degrade to a chronological or trending feed. Posts are edited and deleted; privacy settings change.

Decision. Feed stores hold URNs only. The broker runs the stages in order: gather candidates, filter (privacy, blocks, moderation, already seen), second-pass rank with features from Venice, apply policy (diversity, ad positions), then decorate. Decoration hydrates URNs into content at read time, re-checking privacy. If the ranker fails, the feed falls back to chronological network content plus trending.

What LinkedIn actually did. FollowFeed splits ranking between the index nodes (first pass) and the broker (second pass). Pro-ML's Quasar executes models (LinkedIn, 2019).

What it buys

• Relevance teams ship without storage changes
• Deletes and privacy are correct by construction
• Degraded mode is a stage bypass, not a rewrite

What it costs

• Decoration fan-out at read time needs caching
• The latency budget is spread across stages that each need deadlines
• Policy and model interact and must be tested together

Choose differently when. A small or strictly chronological feed can sort by time and skip ranking entirely.

Lesson. Store references, not rendered content. Late binding makes deletes, edits and privacy correct by construction.

Sources: FollowFeed (LinkedIn, 2016) · Scaling machine learning productivity: Pro-ML (LinkedIn, 2019)

ADR-23 · Messages keyed by conversation, ordered by one sequencer per conversation

Status: Accepted · Shown on views: 13, 18

How are messages ordered, stored once and safe to retry?

Context. The requirement asks for message ordering, history, offline delivery and delivery and read status, with messages partitioned by conversation. Clocks on phones and servers disagree, and mobile clients retry.

Decision. Messages are stored in Espresso keyed by (conversation_urn, seq). One writer per conversation assigns seq as last + 1, so order is total within a conversation. The client supplies a client_msg_id, unique per conversation, so a retry cannot create a second message. The sender is acknowledged after the durable append. Each member's inbox (conversations sorted by last activity) is a derived index keyed by member.

What LinkedIn actually did. Espresso served InMail when LinkedIn introduced it in 2015. LinkedIn has not published its current messaging partitioning, so the conversation-keyed layout is this design's.

What it buys

• Ordering is a property of the data, not of delivery
• Retries are harmless
• Edits and deletes are one write

What it costs

• The sequencer is a per-conversation hot spot for very busy groups
• Inbox listings are eventually consistent
• Group size must be capped

Choose differently when. For very large broadcast channels, per-recipient copies and relaxed ordering are the better trade.

Lesson. Order is something you assign in one place, not something you discover from clocks.

Sources: Introducing Espresso (LinkedIn, 2015)

ADR-24 · Real-time delivery over Server-Sent Events, with presence and push fallback

Status: Accepted · Shown on views: 18, 26

How does a message reach an online recipient in under 500 ms, and an offline one at all?

Context. The requirement allows WebSocket or an equivalent for real-time delivery. Holding hundreds of millions of idle connections is a different scaling problem from serving requests, and every proxy between member and colo must carry the connection.

Decision. Delivery front-ends built on Play and Akka hold one SSE stream per connected device. Presence, from heartbeats on those streams, decides between delivering over SSE and falling back to push through APNs or FCM. The client sends its messages and receipts as ordinary HTTP requests, and receipts flow back through Kafka.

What LinkedIn actually did. In 2016 LinkedIn described instant messaging on Play and Akka actors with SSE, holding hundreds of thousands of persistent connections per machine. It chose SSE because it is plain HTTP. Its real-time presence platform was described in 2018.

What it buys

• Works through existing HTTP infrastructure
• Connection holding scales separately from request serving
• Offline recipients are reached by push

What it costs

• SSE is one-way, so sends are separate requests
• Reconnect storms after a front-end failure need jittered backoff
• Presence is only an approximation

Choose differently when. Collaborative editing, games and other high-rate two-way traffic justify WebSocket.

Lesson. Pick the transport your whole path already understands.

Sources: Instant messaging at LinkedIn (LinkedIn, 2016) · LinkedIn's real-time presence platform (LinkedIn, 2018)

ADR-25 · Concourse generates notifications; ATC decides; neither is on a core path

Status: Accepted · Shown on views: 20, 26

Who decides whether, when and how a member hears about something?

Context. The requirement lists ten notification types across in-app, push and email, with member preferences and creation in under 1 s. Every team wants its notification sent; members leave when they get too many.

Decision. Concourse and service events generate candidates nearline from Kafka. ATC, the single decider, applies preferences, frequency caps, deduplication and aggregation, and chooses the channel. It writes the in-app notification and badge count, and dispatches push and email with retries, backoff and a dead-letter queue. A notification past its usefulness window is dropped. Nothing waits on this pipeline.

What LinkedIn actually did. In 2018 LinkedIn described Air Traffic Controller (ATC), built on Samza, which handles routing, frequency limits, channel selection and aggregation. It described Concourse, which generates personalised content notifications in near real time, the same year.

What it buys

• Members see fewer, better notifications
• A provider outage delays alerts but never fails an action
• Aggregation turns fifty likes into one line

What it costs

• ATC is a shared dependency every product team negotiates with
• Badge counts lag by seconds
• Caps can suppress something a member wanted

Choose differently when. A small product can use a managed notification service (SNS, FCM topics). The rule of one owner for the attention budget still applies.

Lesson. Someone must own the member's attention budget. Make it a service.

Sources: Air Traffic Controller (LinkedIn, 2018)

ADR-26 · Applications are transactional; job recommendations are derived

Status: Accepted · Shown on views: 04, 13, 19

How do jobs stay both correct (the application) and personal (the recommendation)?

Context. The requirement asks for strong consistency on job applications, personalised recommendations based on skills, experience, location, industry, preferences and search behaviour, and applicant privacy.

Decision. Job postings live in Espresso and are indexed into Galene through CDC, so closing a job removes it within seconds. Applications live in MySQL with a unique (member, job) constraint, an idempotency key and a state machine that emits status events to ATC. Recommendations are computed offline and nearline into Venice, re-ranked online, and extended by semantic retrieval. Applicant data is visible only to the member and the job's authorised recruiters.

What LinkedIn actually did. Venice serves Jobs You May Be Interested In (LinkedIn, 2022). LinkedIn described LLM-based semantic job search with embedding retrieval on GPUs in 2025.

What it buys

• No duplicate or lost applications
• Status silence is designed out
• Recommendations can change daily without touching applications

What it costs

• Two stores for one domain
• Recommendation freshness is bounded by the push cadence
• Applicant privacy must be enforced in recruiter search too

Choose differently when. This split holds on any stack. Only the store names change.

Lesson. Split the part that must never be wrong from the part that must be fresh and personal.

Sources: Open-sourcing Venice (LinkedIn, 2022)

ADR-27 · One ML platform (Pro-ML) for feed, search, jobs and people recommendations

Status: Accepted · Shown on views: 11, 16, 25

How do dozens of relevance teams ship models without each building a platform?

Context. The requirement asks that recommendation logic consume behavioural events and feature data without coupling to transactional services, and that ranking evolve independently. Each team building its own features, training and serving multiplies cost and skew.

Decision. Features are defined once in Frame and Feathr and served from Venice. Training runs on Spark and TonY through a unified training service. Models are registered in a model repository, executed by Quasar inside the ranking services, and ramped by T-REX. Health assurance watches drift and online/offline skew. Tracking events close the loop as the next model's labels.

What LinkedIn actually did. In 2019 LinkedIn described Pro-ML: a modelling DSL, a Unified Training Service (Azkaban and Spark), a Model Repository, Quasar, ReMix, Frame and a health-assurance layer. TonY (2018) and Feathr (2022) are open source. DARWIN (2022) is its JupyterHub-based workbench.

What it buys

• Training/serving skew is designed out
• Any team can ramp a model safely
• Feature reuse across products

What it costs

• A platform team on every relevance team's critical path
• Platform abstractions lag new model types (for example LLMs)
• Feedback loops need reserved exploration traffic

Choose differently when. A single ranking team can use a lighter stack: a feature store, an experiment tracker and a model server.

Lesson. Define a feature once. Training/serving skew is the most expensive bug in applied ML.

Sources: Scaling machine learning productivity: Pro-ML (LinkedIn, 2019)

Operations

Retries, colos, releases, observability and degradation.

ADR-28 · Every retryable write takes an idempotency key

Status: Accepted · Shown on views: 17, 18, 26

How do automatic retries avoid duplicate connections, applications, messages and posts?

Context. The requirement asks for retries, exponential backoff and idempotent APIs, and gives POST /connections with an Idempotency-Key as its example. Mobile networks drop responses, so retries are certain.

Decision. POST /connections, /applications, /posts and message sends accept an Idempotency-Key (client_msg_id for messages). The key is stored with a unique constraint next to the created row, and a replay returns the original response. Kafka consumers are idempotent by event ID. Retries use exponential backoff with jitter, and errors are split into retryable and final.

What LinkedIn actually did. This is the requirement's own design. LinkedIn has not published a named system for it.

What it buys

• Retries are always safe
• Clients can retry aggressively on flaky networks
• Duplicate side effects become a bug class that cannot happen

What it costs

• Key storage and TTL management
• Every write API needs the pattern
• A replay must return the original response, which means storing it

Choose differently when. Never. Any system with retries needs this; only the storage of the key varies.

Lesson. Retries are certain. Make duplicates harmless rather than rare.

ADR-29 · Four active colos, members pinned, and draining a colo as routine

Status: Accepted · Shown on views: 22, 26

What survives a zone or a whole-colo loss, and how quickly?

Context. The requirement asks for 99.99% on critical APIs, multi-zone deployment, multi-region DR, and defined RPO and RTO. Failover that is exercised once a year fails when it is needed.

Decision. Four colos all serve traffic. Each member is pinned to a home colo, and within a colo, replicas span fault zones. Stateful stores replicate synchronously in the colo and asynchronously across colos. Draining a colo (re-pinning its members elsewhere) is done on a schedule for load tests and maintenance, so disaster recovery is the same operation. RPO/RTO: fault zone 0 / under 1 minute; colo seconds / under 15 minutes; derived stores rebuilt from the log.

What LinkedIn actually did. In 2017 LinkedIn described four data centres serving member traffic, members pinned by a signed cookie at the ATS tier, and TrafficShift, which drains data centres for load testing and for real incidents. Its full migration to Azure (Blueshift, announced 2019) was paused in 2022. LinkedIn runs its own data centres and uses Azure alongside them.

What it buys

• A colo loss is a rehearsed traffic shift
• Standby capacity serves traffic every day
• Load tests use production traffic

What it costs

• Seconds of asynchronously replicated writes can be lost in a colo failure
• Every colo needs headroom to absorb another's members
• Strongly consistent domains need a written reconciliation playbook

Choose differently when. Below 99.95% targets, a single region across three zones with backup restore to a second region is cheaper and simpler.

Lesson. A failover you do weekly works. A failover you do yearly is a hope.

Sources: TrafficShift (LinkedIn, 2017) · LinkedIn shelved its Azure migration (CNBC, 2023)

ADR-30 · Deploy dark, judge by canary, release by ramp

Status: Accepted · Shown on views: 23, 25

How do hundreds of teams change production daily without members noticing the bad changes?

Context. Services are independently deployable, and relevance teams run many experiments at once. A bad change must be stopped by a machine before a person notices it.

Decision. Schema compatibility, security scanning and integration tests gate the build. Deploys go to canary hosts in one colo, where EKG compares them with control hosts and stops the rollout on regression. Deploys then roll colo by colo. Release is separate: a T-REX ramp from 1% to 100% with guardrail metrics, which can be ramped down in seconds. Schema changes follow expand then contract.

What LinkedIn actually did. LinkedIn described EKG, its automated canary analysis, in 2015. T-REX (formerly LiX and XLNT) ran up to 41,000 concurrent A/B tests on 700M+ members (2020). The 2011 Project Inversion freeze to rebuild release tooling is widely reported.

What it buys

• A regression is stopped automatically
• Features are turned off without a deploy
• Experiments are the normal way to ship

What it costs

• Flags and ramps accumulate and need cleanup
• Overlapping experiments interact
• Canary analysis needs enough traffic to be significant

Choose differently when. Low-traffic products cannot power canary statistics; use staged rollouts with manual verification.

Lesson. Deploying code and releasing behaviour are different acts. Give each its own brake.

Sources: The evolution towards T-REX (LinkedIn, 2020)

ADR-31 · Observe member impact: SLO alerts, business anomaly detection, overload shedding

Status: Accepted · Shown on views: 24, 26

How does anyone know the network is healthy for members, not just for hosts?

Context. The requirement asks for metrics, logs, traces and alerting based on user-facing SLOs rather than infrastructure alone. A drop in feed engagement with every host green is a real outage.

Decision. Services emit RED and saturation metrics to inGraphs. Alerts page through Iris on SLO burn rate, with the rotation held in Oncall. ThirdEye detects anomalies in business metrics in Pinot. Hodor watches queueing and latency and sheds low-priority traffic before a service falls over. Logs travel over Kafka. Traces carry context on every Rest.li hop.

What LinkedIn actually did. inGraphs (metrics dashboards), EKG, ThirdEye (anomaly detection over Pinot), Iris and Oncall (open-sourced 2017) and Hodor (overload detection and remediation, 2022) are all LinkedIn systems. LinkedIn has not published its current tracer; OpenTelemetry is the portable choice.

What it buys

• Fewer, more meaningful pages
• Behavioural outages are caught
• Overload is handled before cascade

What it costs

• SLOs must be defined and maintained for every service
• Anomaly detection needs tuning to avoid false alarms
• Shedding decisions need owners

Choose differently when. Small systems: a hosted observability stack with SLO alerts covers this without building anything.

Lesson. Alert on what members feel. Infrastructure metrics explain an outage; they rarely detect one.

Sources: Hodor (LinkedIn, 2022)

ADR-32 · A written graceful-degradation contract, per capability

Status: Accepted · Shown on views: 16, 20, 26

When a dependency is slow or down, what does each capability do, and who agreed to it?

Context. The requirement gives three examples: recommendations down means a chronological or trending feed, notifications down means core actions still succeed, and search down means profile, feed and apply stay up. Degradation decided during an incident is inconsistent and unrehearsed.

Decision. Every capability has a written behaviour for a slow dependency, a dead dependency and a lost colo, each behind a flag with a named owner. The Rest.li client enforces a timeout and a circuit breaker per dependency. Hodor sheds bot and prefetch traffic first. Core writes never depend on search, recommendations or notifications. Apply fails closed.

What LinkedIn actually did. Hodor (2022) is LinkedIn's overload detection and remediation. The per-capability matrix is this design's form of the requirement's examples.

What it buys

• Nobody improvises during an outage
• Core actions survive every non-critical failure
• Degraded modes are tested, not assumed

What it costs

• Degraded modes interact and need combined testing
• Some degraded modes are visible to members (search off)
• Flags need periodic exercise

Choose differently when. Never. Only the size of the matrix changes with the system.

Lesson. Decide what breaks before it breaks, and write down who agreed.

Sources: Hodor (LinkedIn, 2022)

Assurance

Abuse, fraud and moderation.

ADR-33 · Layered abuse defence writing one auditable decision log

Status: Accepted · Shown on views: 27, 30

How does the network stop fake accounts, spam, fraud and harassment, and prove every decision?

Context. The requirement lists spam, fake accounts, bots, harassment, malicious links, fraudulent jobs, inappropriate content and coordinated abuse. Moderation must combine rules, ML classifiers, user reports and human review, and every decision must be auditable.

Decision. Four lanes: accounts, content, reports and jobs. Each runs signal, detect, decide, act and audit. Registration and login events feed an account risk model with cluster detection. PostCreated and JobCreated feed classifiers. Confident results act automatically and uncertain ones go to a human review queue. Every lane writes the same decision log, which drives appeals and retraining.

What LinkedIn actually did. LinkedIn's anti-abuse application layer is CASAL (2023). It open-sourced its isolation-forest library in 2019. It has described a machine-learned risk score at registration with account-cluster detection (2018), and its content abuse defence (2022).

What it buys

• Every action can be explained and appealed
• Reviewer decisions retrain the models
• Coordinated abuse is visible as clusters

What it costs

• Human review is expensive and emotionally demanding
• False positives hurt legitimate creators and job posters
• Adversaries adapt, so this is never finished

Choose differently when. Small networks can start with rules, reports and a manual queue. Keep the single decision log from day one.

Lesson. An enforcement decision you cannot explain is one you cannot defend. Log every decision the same way.

Sources: CASAL (LinkedIn, 2023) · Isolation forest (LinkedIn, 2019) · Automated fake account detection (LinkedIn, 2018)

The LinkedIn stack, in one table

The LinkedIn systems named on these views: what each one is, when it appeared, and whether you can use it yourself.