Architecture Views

How do we start small without re-platforming when the network reaches a billion members?

Context

The requirement asks for a platform that can grow from a monolith into a globally distributed system without changing its core data and API contracts. Service boundaries are the thing most likely to be wrong early, and the thing most expensive to change once clients depend on them.

Decision

Two things are permanent contracts from day one: typed URN identifiers (ADR-02) and schema-defined Rest.li resources (ADR-04). Deployable units may start coarse, one per domain group, and split later behind the same resources. Clients never learn which process serves a resource.

What LinkedIn actually did

LinkedIn began as a single Java application, widely reported as Leo, and decomposed over years into hundreds of services. In 2011 it paused feature work for about two months (Project Inversion, widely reported) to rebuild its build and release tooling, because the decomposition had outgrown it. Rest.li, D2 and URNs are what allowed services to split without clients noticing.

Options weighed

• ChosenFixed contracts, movable boundaries: Splitting a service becomes a deployment change rather than an API change.
• RejectedMicroservices from day one: Pays the distributed-systems tax (network calls, partial failure, tracing) before the scale that justifies it exists.
• RejectedMonolith sharing one database: Shared tables become implicit contracts that nobody can change without a coordinated release.

Consequences

What it buys

• Clients are unaffected when a service is split or merged
• Teams can own resources before they own processes
• A modular monolith is a legitimate starting point

What it costs

• Schema discipline and compatibility checks are required from the first commit
• Coarse early services have a larger blast radius
• Some resource boundaries will still prove wrong and need versioning

Choose differently when

If the product is unproven and one team builds it, start with a modular monolith that uses the same URNs and resource schemas. Split only when a team, a scaling profile or a failure domain demands it.

What identifies a member when their name, email, vanity URL and employer all change?

Context

The requirement asks for a stable user ID independent of mutable profile data. Members change their email, their custom URL, even their name; a company is renamed; a post is edited. Anything keyed on those values breaks references across the estate.

Decision

Every entity gets a typed URN, such as urn:li:member:123 or urn:li:post:456, assigned once, never reused and never derived from data a user can change. Every reference across services, events and stores is a URN. The vanity URL is a unique but mutable attribute that resolves to a URN.

What LinkedIn actually did

LinkedIn uses typed URNs of the form urn:li:<type>:<id> throughout its services and its public APIs, where they are visible to any developer.

Options weighed

• ChosenTyped URNs: Self-describing references that survive every rename, and that decoration can resolve late.
• RejectedEmail or username as the key: Both change; both are personal data, which turns every foreign key into a privacy liability.
• RejectedUntyped integer IDs: A bare 123 does not say whether it is a member, a company or a post, and it invites cross-type bugs.

Consequences

What it buys

• References never break on a rename
• Deletion propagates as URN tombstones through the log
• Feed and search can store references and resolve content at read time

What it costs

• Every read has to hydrate URNs into content
• URN parsing and validation sit in every service's client library
• Merging two duplicate accounts needs an explicit alias mapping

Choose differently when

A small system on one database can use plain surrogate keys. The rule that matters is still the same: never key on anything a user can edit.

Where do TLS termination, caching, DDoS absorption and colo selection happen?

Context

Members are everywhere; the colos are in a few places. TLS handshakes over long distances dominate first-byte latency. Each member's data is served from a home colo, and operators need a lever that moves members between colos in minutes.

Decision

Azure Front Door terminates TLS close to the member, applies WAF and bot rules, and caches static assets and media. Inside each colo, an Apache Traffic Server tier pins each member to their home colo with a signed cookie. Re-pinning is how a colo is drained for maintenance, for disaster recovery or for a load test.

What LinkedIn actually did

In 2017 LinkedIn described 13 GeoDNS PoPs running ATS with a plugin that pins a user to a data centre through a signed cookie, and TrafficShift, which uses that pinning to drain data centres. In 2020 it moved from its own 19 PoPs to Azure Front Door's 165+ and reported up to 25% faster median page load. ATS remains the proxy tier.

Options weighed

• ChosenManaged global edge + own traffic tier: Provider-scale reach and DDoS capacity at the edge; colo selection stays in LinkedIn's hands.
• RejectedOwn PoP network: LinkedIn ran one and replaced it: 19 PoPs cannot match 165+ on proximity.
• RejectedDNS-only steering to colos: No TLS near the member, and DNS caching makes failover slow and uneven.

Consequences

What it buys

• Shorter TLS round trips worldwide
• DDoS absorbed at provider scale
• Colo drain is a routine, rehearsed operation

What it costs

• A dependency on one edge provider
• Two proxy hops, each needing its own observability
• Cookie-based pinning must be re-established when a member is moved

Choose differently when

If you serve from one region, a CDN plus a regional load balancer is enough. If regulation demands a sovereign edge, run your own and accept fewer PoPs.

How do hundreds of services and three client platforms agree on contracts that can evolve?

Context

The requirement asks for versioned, authenticated, observable APIs. Hundreds of teams change services every day, and web, iOS and Android each want pages shaped differently. Hand-written REST drifts; client-specific endpoints multiply.

Decision

Services expose schema-defined Rest.li resources, with backward compatibility checked in CI (ADR-30). Protocol Buffers is the wire encoding between services. Product clients query a GraphQL layer whose types are generated from the Rest.li schemas, so a client can shape a page without a new endpoint and the types cannot drift from the services.

What LinkedIn actually did

Rest.li was open-sourced in 2013. In 2023 LinkedIn moved Rest.li's wire encoding from JSON to Protocol Buffers and reported up to 60% lower latency for services with large payloads. From 2022 to 2023 it adopted GraphQL for integrations and then for product APIs, generating GraphQL types from Rest.li schemas, with no central gateway.

Options weighed

• ChosenRest.li + generated GraphQL: One schema language, compatibility-checked, serving both service calls and client queries.
• Right elsewheregRPC everywhere: The obvious greenfield choice today; for a Rest.li estate the migration buys little.
• RejectedHand-written REST + JSON: No machine-checked compatibility, so breaking changes are found in production.
• RejectedGraphQL down to every service: Couples service design to client query shapes and makes caching and authorization harder.

Consequences

What it buys

• A breaking change fails the build, not a client
• Binary encoding cut latency and CPU on large payloads
• Clients iterate on pages without mid-tier releases

What it costs

• An in-house framework to maintain alongside the industry standard
• Schema tooling is on every team's critical path
• GraphQL queries need cost limits so one query cannot fan out to hundreds of calls

Choose differently when

Greenfield today: gRPC and Protocol Buffers internally, with a GraphQL or BFF edge, is the same design with a larger ecosystem.

How does a call find a healthy instance without a load balancer on every hop?

Context

A single feed request fans out to dozens of mid-tier calls, and the estate makes millions of calls a second. A central load balancer adds a network hop, a capacity plan and a failure point to every one of them.

Decision

Instances register in a discovery registry. The D2 client library in every caller selects instances with latency- and health-aware balancing, and degrades hosts that misbehave. No load balancer sits between services inside a colo.

What LinkedIn actually did

D2 ships with Rest.li and originally used a ZooKeeper registry. LinkedIn has since described a next-generation discovery system that uses Kafka for writes and xDS/gRPC for reads, replacing the ZooKeeper registry (covered by InfoQ, February 2026).

Options weighed

• ChosenClient-side balancing (D2): No extra hop, and the balancing decision uses the caller's own latency view.
• RejectedCentral L7 load balancer: A hop and a capacity plan on every call, at millions of calls a second.
• DeferredSidecar service mesh (Envoy, Istio): xDS is the direction LinkedIn is moving; a mesh suits a polyglot estate.

Consequences

What it buys

• Lowest possible per-call latency
• No shared balancer to overload
• Health signals come from real callers

What it costs

• Balancing logic lives in every client library and language
• The registry becomes critical infrastructure
• Client library upgrades are needed to change balancing behaviour

Choose differently when

In a polyglot estate, or where the platform team cannot ship client libraries, a sidecar mesh gives the same behaviour without per-language code.

How do 1.2 billion members sign in safely without making every sign-in painful?

Context

The requirement lists registration, email and phone authentication, social login, MFA, sessions, recovery and deletion, with strong consistency for account changes. At this scale credential stuffing never stops, and an account takeover is a reputational event.

Decision

Credentials live in sharded MySQL, hashed with argon2id using a per-member salt and a pepper held in KMS. MFA offers passkeys and TOTP, with SMS as a last resort, and is required by risk: a new device, an unusual IP or unusual velocity. Sessions use a short-lived signed access token plus a rotating refresh token bound to the device; revocations travel as Kafka events to every colo. The same service is an OAuth 2.0 and OIDC provider for third-party apps, and accepts SAML for enterprise seats.

What LinkedIn actually did

LinkedIn has not published a named identity system. The widely reported 2012 breach exposed unsalted SHA-1 password hashes (6.5 million were posted that year, and a set of about 117 million surfaced in 2016). It is the textbook case for adaptive, salted hashing.

Options weighed

• ChosenIn-house identity service: Identity is the product's front door, and per-user pricing at 1.2B members is untenable.
• Right elsewhereManaged CIAM (Entra External ID, Auth0, Cognito): Safer and cheaper below tens of millions of users.
• RejectedLong-lived stateless JWTs: Cannot be revoked before expiry, which is exactly when revocation matters.

Consequences

What it buys

• A leaked hash table is expensive to crack
• Most sign-ins have no friction; risky ones are challenged
• A revocation reaches every colo in seconds

What it costs

• A security-critical service to staff and audit
• Risk scoring can lock out legitimate travellers
• SMS as a factor is defeated by SIM swaps, so it is not offered on privileged seats

Choose differently when

Below tens of millions of users, or without a dedicated identity team, buy a managed CIAM and spend the effort on product.

How is privacy enforced when member data is copied into caches, search indexes and feeds?

Context

The requirement lists public, connections-only and private content, blocks, company and recruiter permissions, and applicant privacy, all enforced on the server. Member data is copied into Galene, FollowFeed, Venice and Couchbase, and each copy is another place a check can be forgotten.

Decision

There are two layers. Service-to-service authorization answers whether service X may call resource Y, through a central ACL service whose decisions are cached in each caller. Member privacy answers whether this viewer may see this field of this member. It is a policy library that every Rest.li resource returning member data calls, and decoration calls again, using the block list (both directions), graph degree and per-field visibility settings. A blocked viewer receives 404, not 403.

What LinkedIn actually did

LinkedIn described a central ACL service in 2019. It has a Rest.li API, is managed through Nuage, is enforced inside each application with local caching, and sends access logs through Kafka. Member-level privacy enforcement has no single published system; this layer is this design's.

Options weighed

• ChosenCentral ACLs + privacy library in every read: Policy is defined once and enforced wherever the data is, including derived copies.
• RejectedChecks at the API gateway only: Internal callers and derived stores bypass the gateway.
• RejectedA remote policy-engine call per request: Adds a network hop to every read; the library with cached inputs gives the same answer locally.

Consequences

What it buys

• A new copy of member data inherits the check at read time
• A block cannot be discovered by probing
• Every recruiter and admin read is auditable

What it costs

• Every read path pays for the block and degree lookups, so both are cached
• The library has to ship in every service, and a test must assert each resource calls it
• Settings change faster than derived stores update, so privacy is applied at read, never precomputed

Choose differently when

On one database, row-level security plus gateway checks suffice. Relationship-based authorization services (SpiceDB, OpenFGA) are the managed form of this pattern.

What stops a compromised host or a leaked config file becoming a data breach?

Context

The requirement lists encryption in transit and at rest, secret management, audit logging and least-privilege service access. With hundreds of services and thousands of hosts, network position is a weak proof of identity.

Decision

Every service has a workload identity and presents a short-lived certificate: all service-to-service traffic is mTLS. Data at rest is encrypted with per-dataset data keys, wrapped by master keys that never leave KMS or an HSM. Secrets are fetched at start-up on short leases and never written to config. Access and admin actions are logged to Kafka and land in immutable storage.

What LinkedIn actually did

LinkedIn has published its ACL service and the Kafka-carried access logs (2019). It has not described its KMS or secrets tooling publicly, so that part is the requirement's design.

Options weighed

• ChosenWorkload identity + mTLS everywhere: Identity comes from a certificate, not an IP address, so lateral movement needs a stolen identity.
• RejectedNetwork segmentation alone: Anything inside the segment is trusted; one foothold is enough.
• RejectedStatic secrets in configuration: They leak through repositories, logs and backups, and never expire.

Consequences

What it buys

• A stolen config file holds nothing reusable
• Keys can be rotated without re-encrypting all data
• Every access leaves an immutable record

What it costs

• Certificate issuance and rotation is critical infrastructure
• KMS latency and availability sit on start-up paths
• Audit volume is large and needs its own retention policy

Choose differently when

On a cloud, use the provider's workload identity, KMS and secret manager instead of running your own.

Where do profiles, posts, comments, connections, conversations and jobs live?

Context

These entities are read far more than written, partition naturally by a key (member, author, conversation), need schema evolution without downtime, and need a change stream so that search, feed, graph and cache can follow them.

Decision

Espresso stores them as schema-defined documents, hash-partitioned by key. Transactions are allowed within one partition, and each partition has a timeline-ordered change stream that Brooklin captures (ADR-19). Apache Helix manages partition placement and failover.

What LinkedIn actually did

LinkedIn introduced Espresso publicly in 2015 as its distributed document store. It has MySQL/InnoDB storage nodes, stateless HTTP routers, Helix for cluster management, change capture first through Databus and later through Brooklin, Avro documents and hash partitioning. At the time it served about 30 applications, including Member Profile, InMail and parts of the homepage. It replaced Oracle for those uses, and it is not open source.

Options weighed

• ChosenEspresso: Partitioned documents on a proven storage engine, with the change log built in.
• RejectedOracle: What LinkedIn moved away from: scale-up cost and licence cost at this volume.
• RejectedHand-sharded MySQL: Resharding and schema changes per shard become the operations team's full-time job, which is exactly what Espresso automates.
• Right elsewhereVitess, Cosmos DB or DynamoDB: The same shape for anyone outside LinkedIn: sharded rows or documents with a change stream.

Consequences

What it buys

• Resharding and schema evolution are routine
• Every change is published without application code
• Reads scale with replicas

What it costs

• Transactions stop at the partition boundary
• An internal database must be staffed and developed
• Cross-partition queries go to derived stores, never to Espresso

Choose differently when

Outside LinkedIn, choose Vitess for relational workloads, or a managed document store with change streams (Cosmos DB, DynamoDB) for document workloads.

Which data needs the strongest consistency, and where does it live?

Context

The requirement asks for stronger consistency on authentication, account changes, job applications, connection state and financial or ad-related operations. Espresso's transactions stop at a partition. A job application touches the member, the job and the idempotency key, and an InMail send debits a credit.

Decision

Credentials, job applications, InMail credits and billing live in sharded MySQL with semi-synchronous replication inside the colo. Unique constraints enforce idempotency keys and one application per member per job, and state transitions run in one transaction. Connection state stays in Espresso, where the ordered-pair key makes it a single-row compare-and-set (ADR-13).

What LinkedIn actually did

MySQL is widely used at LinkedIn; Espresso's own storage engine is MySQL. Which domains sit on plain MySQL here is this design's decision.

Options weighed

• ChosenSharded MySQL for these domains: ACID and unique keys, on the most familiar storage engine in the estate.
• RejectedEspresso for everything: Single-partition transactions cannot debit a credit and record a send atomically when they are keyed differently.
• DeferredDistributed SQL (Spanner, CockroachDB): Pays global-consensus latency for a guarantee members pinned to one colo do not need.

Consequences

What it buys

• Money and applications are correct by construction
• Retries are made safe by the database, not by application code
• Mature tooling for backup and point-in-time recovery

What it costs

• Resharding MySQL is a planned project
• Cross-colo failover of these domains needs a reconciliation step
• Two primary-store technologies to operate

Choose differently when

If a domain truly needs serializable writes across regions at once, distributed SQL earns its latency.

Where do recommendations, features and feed inboxes live, which are computed rather than entered?

Context

People You May Know, job recommendations, ranking features and feed inboxes are recomputed in bulk by offline jobs and refreshed nearline by stream jobs. Writing terabytes back into the primary store would crush it, and a bad batch must be reversible.

Decision

Venice holds every derived key-value dataset. An offline push creates a complete new version that is swapped in atomically, and the previous version is kept for rollback. Hybrid stores also accept nearline writes from Samza, ordered after the batch by version.

What LinkedIn actually did

Project Voldemort (open-sourced 2009) served read-only, Hadoop-built stores such as People You May Know. About 500 of those use cases had moved to Venice by 2018. LinkedIn open-sourced Venice in 2022: batch pushes from the Hadoop grid, 200+ hybrid datasets with nearline writes, serving People You May Know and Jobs You May Be Interested In, and backing the Feathr feature store.

Options weighed

• ChosenVenice: Versioned bulk swaps plus nearline writes, built for exactly this job.
• RejectedWrite derived data into Espresso: Bulk loads would compete with member-facing writes on the system of record.
• RejectedRedis: Memory-bound cost for terabytes, and no atomic version swap.
• RejectedVoldemort read-only: Superseded by Venice at LinkedIn; it had no nearline writes.

Consequences

What it buys

• A bad push is rolled back by swapping versions
• The system of record never sees derived traffic
• Every derived dataset can be rebuilt from the lake

What it costs

• Storage for the previous version of every dataset
• Batch and nearline writers must agree on versioning
• Data is only as fresh as the last push plus the nearline stream

Choose differently when

Outside LinkedIn, Venice itself is open source. Cassandra or ScyllaDB with bulk loading, or Redis for small derived sets, are the usual alternatives.

How does a 200 ms profile read survive decoration fan-out without overloading Espresso?

Context

Decorating one feed page hydrates dozens of URNs, and profiles, company pages and popular posts are read millions of times. The requirement asks for event-driven invalidation where practical, and for transactional truth never to depend on a cache.

Decision

Couchbase clusters cache decorated profiles, companies, popular posts, block lists and session lookups. Invalidation is driven by change events from Brooklin, with a TTL as a backstop. The cache is never the only copy of anything, and a miss always falls through to the system of record.

What LinkedIn actually did

LinkedIn has used Couchbase as its caching layer since 2012; in 2018 it reported 10M+ queries a second across 300+ clusters.

Options weighed

• ChosenCouchbase: A replicated, persistent cache: a node restart does not start cold.
• Rejectedmemcached: No replication, so every restart is a thundering herd onto the store.
• Right elsewhereRedis / Valkey: Equivalent for most teams; choose by operational familiarity.
• RejectedNo cache, read replicas only: Decoration fan-out would multiply Espresso read load several times over.

Consequences

What it buys

• Profile reads meet 200 ms at p95
• Espresso is sized for misses, not for fan-out
• Invalidation follows every write, including writes from jobs

What it costs

• A miss storm after a cluster loss is the feed's most common latency incident
• Invalidation lag means a few seconds of staleness
• Another fleet to operate

Choose differently when

Outside LinkedIn, Redis/Valkey or memcached. Keep the rule regardless: invalidate from change events, not from the writer.

How do we answer who is connected, mutual connections and second degree for 1.2B members in milliseconds?

Context

The requirement lists these queries and asks for strong consistency on connection state. Second-degree queries touch thousands of edges each, and are run on nearly every profile view, search result and privacy check. High-degree nodes create hot partitions.

Decision

Connection and follow edges are written to Espresso, keyed by the ordered pair of members (for connections) or by follower and followee. This is the strongly consistent truth. LIquid ingests the edge change log and answers traversal queries from memory: connected, mutual count, second degree, who follows A. Follower counts for high-degree accounts are precomputed, and their adjacency is replicated rather than served from one partition.

What LinkedIn actually did

LinkedIn described LIquid in 2020 as a new in-memory graph database with a relational, Datalog-style query model. In 2023 it reported 270 billion edges and 2 million queries a second at 99.99% availability, with People You May Know graph queries falling from over 1 s to under 50 ms on average.

Options weighed

• ChosenEdge store + LIquid: Transactional truth and in-memory traversal, each in a system built for it.
• Right elsewhereAdjacency lists in a key-value store: The requirement allows it, and it is enough for first degree; second degree becomes thousands of lookups.
• RejectedA general graph database (Neo4j): Not built for 270 billion edges at 2 million queries a second.
• RejectedPrecompute all second-degree sets: Combinatorial storage, stale the moment anyone connects.

Consequences

What it buys

• Privacy checks and ranking can afford graph features on every request
• Connection state is never ambiguous
• The graph index can be rebuilt from the edge log

What it costs

• Memory-resident at enormous scale is expensive hardware
• LIquid lags the edge store by seconds, so the two members involved read the edge store just after a change
• An internal database to staff

Choose differently when

Outside LinkedIn, keep adjacency lists in a key-value store and batch-compute second-degree candidates. Use a sharded graph engine (JanusGraph, TigerGraph) if traversal is the product.

How does search stay fast, fresh and personal across people, jobs, companies and posts?

Context

The requirement asks for full-text search with typo tolerance, filters, ranking, autocomplete and personalised results, with asynchronous, eventually consistent indexing. Incrementally updated indexes accumulate deletes and skew, and each vertical ranks differently.

Decision

Each vertical has a Galene index. A base index is rebuilt offline on the grid and swapped in, and an in-memory live update buffer fed by change capture through Samza covers the gap. A federated broker queries the verticals in parallel and blends by intent. A personalised ranker uses graph distance and member features. Privacy is applied at index time (opted-out profiles are never indexed) and at query time (viewer-dependent visibility). Semantic, embedding-based retrieval is a further candidate source for jobs.

What LinkedIn actually did

LinkedIn described Galene in 2014: a Lucene-based architecture with a base index built offline with Hadoop MapReduce and rebuilt, for example, weekly, plus a live update buffer and a snapshot index. It replaced earlier systems, including the Cleo typeahead (open-sourced 2012). In 2025 LinkedIn described LLM-based semantic job search with embedding retrieval on GPUs.

Options weighed

• ChosenGalene: Full control of index build, sharding and ranking at LinkedIn's scale.
• Right elsewhereElasticsearch / OpenSearch: The default for almost everyone else.
• RejectedOne index for every entity type: Verticals differ in update rate, ranking features and privacy rules.

Consequences

What it buys

• A rebuild purges deletes and rebalances shards
• Freshness in seconds without incremental decay
• Graph-aware ranking

What it costs

• Two pipelines, offline and live, must agree
• The weekly rebuild is a large recurring batch job
• An internal engine to staff

Choose differently when

Outside LinkedIn, use OpenSearch or Elasticsearch, or Vespa where ranking is the product.

How do images, video, documents and resumes get in and out without passing through services?

Context

The requirement asks for large objects in object storage rather than the primary database, direct uploads on temporary signed URLs, processing (images, video, malware scanning) and CDN delivery.

Decision

The media service issues signed, time-limited, resumable upload URLs, and clients write chunks straight to Ambry. Originals are immutable; renditions are derived and can be regenerated. A malware scan and a content classifier gate serving. Public media goes through the CDN; resumes and documents are served only on short-lived authorised URLs, never through the public CDN.

What LinkedIn actually did

LinkedIn open-sourced Ambry in 2016 as an immutable, active-active, multi-data-centre object store, accompanied by a SIGMOD 2016 paper.

Options weighed

• ChosenAmbry: Built for immutable media blobs, active-active across colos.
• Right elsewhereS3, Azure Blob, GCS, MinIO: Any object store plus a CDN is this design outside LinkedIn.
• RejectedHDFS: Built for large files and batch reads, not millions of small objects served to members.
• RejectedBLOBs in the database: Bloats the system of record and its backups.

Consequences

What it buys

• Services never carry media bytes
• Uploads resume after a dropped connection
• Renditions can be regenerated for new devices

What it costs

• Signed-URL issuance and expiry must be right, or media leaks
• The scanning pipeline sits between upload and visibility
• Two delivery paths, public and authorised

Choose differently when

Outside LinkedIn, use any object store plus a CDN. Keep private documents off the public CDN either way.

How do Who Viewed My Profile, post analytics and recruiter dashboards answer in under a second?

Context

Analytics shown to members is a product feature with consumer concurrency and a latency SLO. Hadoop answers in hours, and operational stores must never run analytical queries.

Decision

Pinot tables ingest tracking and business events from Kafka and serve member-facing analytics: profile views, post analytics, company page and recruiter dashboards. ThirdEye anomaly detection runs on the same data (ADR-31).

What LinkedIn actually did

LinkedIn built Pinot in 2013 and open-sourced it in 2015, naming Who Viewed My Profile as a use case. It became an Apache top-level project in 2021.

Options weighed

• ChosenApache Pinot: Built at LinkedIn for exactly this: user-facing, high-concurrency, fresh.
• Right elsewhereApache Druid: A comparable design and a sound alternative.
• Right elsewhereClickHouse: Excellent for internal analytics at lower concurrency.
• RejectedQuery Hadoop: Hours of latency for a feature members open on their phone.

Consequences

What it buys

• Member analytics fresh within a minute
• No analytical load on operational stores
• The same tables power anomaly detection

What it costs

• Another stateful cluster to run
• Tables are pre-shaped for known queries
• Privacy rules (such as anonymised viewers) must be applied in the serving layer

Choose differently when

For internal dashboards with low concurrency, ClickHouse or a warehouse is simpler.

Where do training, BI, A/B analysis and batch recommendations run?

Context

The requirement separates operational systems from large-scale analytics, and feeds a data platform used for analytics, recommendation training, A/B testing, BI and fraud detection.

Decision

Gobblin lands every Kafka topic into HDFS hourly. Tables are Apache Iceberg, governed by OpenHouse. Spark jobs run on Azkaban schedules. DataHub records ownership, lineage and purpose, and that lineage is how the platform knows which derived stores can be rebuilt. No analytical query ever runs against an operational store.

What LinkedIn actually did

All of these are LinkedIn projects. Gobblin (2014) became an Apache top-level project in 2021. Azkaban is open source. DataHub (open-sourced 2020) succeeded WhereHows. OpenHouse (open-sourced 2024) is a control plane for Iceberg tables.

Options weighed

• ChosenLinkedIn's open-source data stack: Runs in LinkedIn's own colos at exabyte scale, with no per-query cloud bill.
• Right elsewhereA cloud data warehouse: Often right for others; at this scale and in owned colos, the cost model differs.
• RejectedAnalytics on OLTP replicas: One heavy query degrades member-facing latency.

Consequences

What it buys

• Open table formats keep engines replaceable
• Lineage makes rebuildability auditable
• Training data comes from the same log as serving data

What it costs

• A large Hadoop estate to operate
• Hourly landing means BI is hours fresh
• Member deletion requires rewriting Iceberg partitions

Choose differently when

Outside LinkedIn, a lakehouse on object storage (Iceberg with Spark or Trino, Airflow and DataHub) is this exact design in the cloud.

How do fan-out, indexing, notifications, recommendations, analytics and abuse detection learn about what happened?

Context

The requirement asks for a durable event bus and for services to avoid synchronous dependencies where an asynchronous workflow is enough. There are at least six classes of consumer, and more will come.

Decision

Every state change (through CDC, ADR-19) and every member action (tracking) is an event on Kafka. Schemas are Avro in a registry with build-time compatibility checks. Tracking and business events run on separate clusters. Each colo has local and aggregate clusters, with Brooklin mirroring between them. Event topics keep 7 days; change topics are compacted.

What LinkedIn actually did

Kafka was created at LinkedIn by Jay Kreps, Neha Narkhede and Jun Rao and entered the Apache incubator in 2011. In 2019 LinkedIn reported 7 trillion messages a day across 100+ clusters, 4,000+ brokers, 100,000+ topics and 7 million partitions.

Options weighed

• ChosenKafka as the log: Replayable, partitioned and durable, with consumers fully decoupled from publishers.
• RejectedA traditional message queue: Messages disappear once consumed, so there is no replay and nothing to rebuild from.
• RejectedSynchronous calls to each consumer: Every new consumer becomes a publisher release and a new failure dependency.

Consequences

What it buys

• A new consumer is a deployment, not a negotiation
• Derived stores can be rebuilt by replay
• Notifications and indexing never sit on the write path

What it costs

• Order holds only within a partition
• Schema governance is mandatory
• Running Kafka at this scale is a platform team of its own

Choose differently when

With low volume and few consumers, a queue is simpler. Managed Kafka (Confluent, MSK, Event Hubs) removes the platform burden.

How does a committed write reliably become an event?

Context

A service that writes to its store and then publishes to Kafka can crash between the two steps, losing the event or publishing one for a write that rolled back. Every derived store depends on these events being exactly the committed changes.

Decision

Services write only to their store. Brooklin tails each store's change log (Espresso, MySQL) and publishes the committed changes to Kafka in commit order. Brooklin also mirrors Kafka between colos.

What LinkedIn actually did

LinkedIn open-sourced Databus, its first change-capture system, in 2013. Brooklin (open-sourced 2019) replaced Databus for Espresso and Oracle change capture. It also replaced Kafka MirrorMaker, reducing hundreds of mirroring clusters to about a dozen, and it carries 2T+ messages a day.

Options weighed

• ChosenLog-based CDC (Brooklin): The database log saw every commit, in order, and nothing else did.
• RejectedDual writes from the service: A crash between write and publish silently diverges every derived store.
• Right elsewhereTransactional outbox: Correct and portable, but it needs code in every service; CDC gets the same result for free.
• Right elsewhereDebezium: The open-source equivalent outside LinkedIn.

Consequences

What it buys

• Derived stores cannot silently diverge
• Services contain no publishing code
• One tool covers CDC and cross-colo mirroring

What it costs

• Events look like database rows, so a domain-event layer is sometimes needed on top
• Schema changes in the store surface in the stream
• CDC lag becomes an SLO

Choose differently when

If you cannot read the database log (for example, some managed databases), use a transactional outbox.

What computes fan-out, index documents, features and notifications in seconds?

Context

Nearline work sits between the request path and the batch grid: stateful (counts, joins, windows), seconds stale, and continuous. It must restart without losing state and without an external database on the hot path.

Decision

Fan-out, search document building, feature updates, ATC and Concourse run as Samza jobs next to Kafka. Local state is backed by changelog topics, so a restarted task rebuilds its state from Kafka.

What LinkedIn actually did

Samza was created at LinkedIn, entered the Apache incubator in 2013 and became a top-level project in January 2015. ATC runs on Samza (LinkedIn, 2018).

Options weighed

• ChosenApache Samza: Built alongside Kafka at LinkedIn, with local state plus changelog as its core idea.
• Right elsewhereApache Flink: The default choice outside LinkedIn today.
• Right elsewhereKafka Streams: Good for simpler jobs embedded in services.

Consequences

What it buys

• Stateful jobs restart without a database
• Nearline lag is measurable per job
• One execution model for every nearline workload

What it costs

• A smaller community than Flink
• Changelog topics add Kafka storage
• Rebuilding a large state store after a failure takes time

Choose differently when

Outside LinkedIn, choose Apache Flink.

How does a post reach its audience without either millions of writes or a very slow read?

Context

The requirement asks for both fan-out-on-write and fan-out-on-read, with high-follower accounts on read. Pushing a post by an author with millions of followers means millions of writes; pulling for a member who follows thousands means thousands of reads per page.

Decision

Authors under a follower threshold (assumed 10k, tunable) fan out on write: Samza makes one write per follower into a capped Venice inbox. High-degree authors never fan out; their posts are indexed in FollowFeed by actor and pulled at read time. The feed broker merges the inbox, FollowFeed, recommended candidates and ads, and dedupes by post URN.

What LinkedIn actually did

LinkedIn's published FollowFeed (2016) is pull-based: timelines keyed by actor and content type across 720 partitions, with first-pass ranking on the index nodes and second-pass ranking, dedupe and diversity at the broker. The write-time inbox is added here because the requirement asks for the hybrid.

Options weighed

• ChosenHybrid, threshold by degree: Each side handles the case the other is worst at.
• Right elsewherePure pull (FollowFeed): LinkedIn proved it at scale. It is simpler, and right if most members follow few actors.
• RejectedPure push: A high-degree author's post becomes millions of writes and minutes of lag.

Consequences

What it buys

• Bounded write cost for any author
• Bounded read cost for any reader
• Storage strategy can change without touching ranking

What it costs

• Two storage paths to operate and keep consistent
• Authors crossing the threshold must move lanes without losing or duplicating posts
• Fan-out lag under a spike is a new SLO

Choose differently when

If most members follow few actors and reads dominate, pure pull is simpler, and LinkedIn runs it.

How can ranking models change daily without rewriting feed storage, and how do edits, deletes and privacy take effect immediately?

Context

The requirement asks that ranking and recommendation evolve independently of feed storage, and that a recommendation failure degrade to a chronological or trending feed. Posts are edited and deleted; privacy settings change.

Decision

Feed stores hold URNs only. The broker runs the stages in order: gather candidates, filter (privacy, blocks, moderation, already seen), second-pass rank with features from Venice, apply policy (diversity, ad positions), then decorate. Decoration hydrates URNs into content at read time, re-checking privacy. If the ranker fails, the feed falls back to chronological network content plus trending.

What LinkedIn actually did

FollowFeed splits ranking between the index nodes (first pass) and the broker (second pass). Pro-ML's Quasar executes models (LinkedIn, 2019).

Options weighed

• ChosenSeparate stages, late decoration: A model change is a deployment; an edit or delete is visible on the next read.
• RejectedMaterialise the ranked feed at write time: Every model change needs a backfill, and deleted or hidden content lingers.
• RejectedRanking inside the storage layer: Couples model iteration to storage releases.

Consequences

What it buys

• Relevance teams ship without storage changes
• Deletes and privacy are correct by construction
• Degraded mode is a stage bypass, not a rewrite

What it costs

• Decoration fan-out at read time needs caching
• The latency budget is spread across stages that each need deadlines
• Policy and model interact and must be tested together

Choose differently when

A small or strictly chronological feed can sort by time and skip ranking entirely.

How are messages ordered, stored once and safe to retry?

Context

The requirement asks for message ordering, history, offline delivery and delivery and read status, with messages partitioned by conversation. Clocks on phones and servers disagree, and mobile clients retry.

Decision

Messages are stored in Espresso keyed by (conversation_urn, seq). One writer per conversation assigns seq as last + 1, so order is total within a conversation. The client supplies a client_msg_id, unique per conversation, so a retry cannot create a second message. The sender is acknowledged after the durable append. Each member's inbox (conversations sorted by last activity) is a derived index keyed by member.

What LinkedIn actually did

Espresso served InMail when LinkedIn introduced it in 2015. LinkedIn has not published its current messaging partitioning, so the conversation-keyed layout is this design's.

Options weighed

• ChosenPer-conversation sequence: Order is assigned in one place and stored in the key.
• RejectedOrder by timestamp: Clock skew reorders messages, and ties are ambiguous.
• Right elsewhereA copy per recipient mailbox: Wins for huge broadcast lists; for normal conversations, edits and deletes become N writes.

Consequences

What it buys

• Ordering is a property of the data, not of delivery
• Retries are harmless
• Edits and deletes are one write

What it costs

• The sequencer is a per-conversation hot spot for very busy groups
• Inbox listings are eventually consistent
• Group size must be capped

Choose differently when

For very large broadcast channels, per-recipient copies and relaxed ordering are the better trade.

How does a message reach an online recipient in under 500 ms, and an offline one at all?

Context

The requirement allows WebSocket or an equivalent for real-time delivery. Holding hundreds of millions of idle connections is a different scaling problem from serving requests, and every proxy between member and colo must carry the connection.

Decision

Delivery front-ends built on Play and Akka hold one SSE stream per connected device. Presence, from heartbeats on those streams, decides between delivering over SSE and falling back to push through APNs or FCM. The client sends its messages and receipts as ordinary HTTP requests, and receipts flow back through Kafka.

What LinkedIn actually did

In 2016 LinkedIn described instant messaging on Play and Akka actors with SSE, holding hundreds of thousands of persistent connections per machine. It chose SSE because it is plain HTTP. Its real-time presence platform was described in 2018.

Options weighed

• ChosenSSE + HTTP for sends: Plain HTTP through every proxy, CDN and firewall on the path.
• Right elsewhereWebSocket: Right when clients also send at high rate over the same connection.
• RejectedLong polling: A request per message per client multiplies load for no benefit.

Consequences

What it buys

• Works through existing HTTP infrastructure
• Connection holding scales separately from request serving
• Offline recipients are reached by push

What it costs

• SSE is one-way, so sends are separate requests
• Reconnect storms after a front-end failure need jittered backoff
• Presence is only an approximation

Choose differently when

Collaborative editing, games and other high-rate two-way traffic justify WebSocket.

Who decides whether, when and how a member hears about something?

Context

The requirement lists ten notification types across in-app, push and email, with member preferences and creation in under 1 s. Every team wants its notification sent; members leave when they get too many.

Decision

Concourse and service events generate candidates nearline from Kafka. ATC, the single decider, applies preferences, frequency caps, deduplication and aggregation, and chooses the channel. It writes the in-app notification and badge count, and dispatches push and email with retries, backoff and a dead-letter queue. A notification past its usefulness window is dropped. Nothing waits on this pipeline.

What LinkedIn actually did

In 2018 LinkedIn described Air Traffic Controller (ATC), built on Samza, which handles routing, frequency limits, channel selection and aggregation. It described Concourse, which generates personalised content notifications in near real time, the same year.

Options weighed

• ChosenOne central decider (ATC): One place owns the member's attention budget.
• RejectedEach service sends its own: No global frequency cap, so members get flooded.
• RejectedNotify synchronously in the write: Couples a post or connection to push-provider availability.

Consequences

What it buys

• Members see fewer, better notifications
• A provider outage delays alerts but never fails an action
• Aggregation turns fifty likes into one line

What it costs

• ATC is a shared dependency every product team negotiates with
• Badge counts lag by seconds
• Caps can suppress something a member wanted

Choose differently when

A small product can use a managed notification service (SNS, FCM topics). The rule of one owner for the attention budget still applies.

How do jobs stay both correct (the application) and personal (the recommendation)?

Context

The requirement asks for strong consistency on job applications, personalised recommendations based on skills, experience, location, industry, preferences and search behaviour, and applicant privacy.

Decision

Job postings live in Espresso and are indexed into Galene through CDC, so closing a job removes it within seconds. Applications live in MySQL with a unique (member, job) constraint, an idempotency key and a state machine that emits status events to ATC. Recommendations are computed offline and nearline into Venice, re-ranked online, and extended by semantic retrieval. Applicant data is visible only to the member and the job's authorised recruiters.

What LinkedIn actually did

Venice serves Jobs You May Be Interested In (LinkedIn, 2022). LinkedIn described LLM-based semantic job search with embedding retrieval on GPUs in 2025.

Options weighed

• ChosenTransactional apply, derived recommendations: Each half gets the consistency it actually needs.
• RejectedApplications in the document store: Unique-per-member-per-job and idempotency need multi-row guarantees.
• RejectedRecommendations computed on request: Too slow for a page load, and repeats work for every view.

Consequences

What it buys

• No duplicate or lost applications
• Status silence is designed out
• Recommendations can change daily without touching applications

What it costs

• Two stores for one domain
• Recommendation freshness is bounded by the push cadence
• Applicant privacy must be enforced in recruiter search too

Choose differently when

This split holds on any stack. Only the store names change.

How do dozens of relevance teams ship models without each building a platform?

Context

The requirement asks that recommendation logic consume behavioural events and feature data without coupling to transactional services, and that ranking evolve independently. Each team building its own features, training and serving multiplies cost and skew.

Decision

Features are defined once in Frame and Feathr and served from Venice. Training runs on Spark and TonY through a unified training service. Models are registered in a model repository, executed by Quasar inside the ranking services, and ramped by T-REX. Health assurance watches drift and online/offline skew. Tracking events close the loop as the next model's labels.

What LinkedIn actually did

In 2019 LinkedIn described Pro-ML: a modelling DSL, a Unified Training Service (Azkaban and Spark), a Model Repository, Quasar, ReMix, Frame and a health-assurance layer. TonY (2018) and Feathr (2022) are open source. DARWIN (2022) is its JupyterHub-based workbench.

Options weighed

• ChosenOne shared ML platform: Features defined once, one path to production, one set of guardrails.
• RejectedEach team builds its own: Duplicated work, and training/serving skew in every team.
• Right elsewhereKubeflow, MLflow, Feast: The open-source kit outside LinkedIn.

Consequences

What it buys

• Training/serving skew is designed out
• Any team can ramp a model safely
• Feature reuse across products

What it costs

• A platform team on every relevance team's critical path
• Platform abstractions lag new model types (for example LLMs)
• Feedback loops need reserved exploration traffic

Choose differently when

A single ranking team can use a lighter stack: a feature store, an experiment tracker and a model server.

How do automatic retries avoid duplicate connections, applications, messages and posts?

Context

The requirement asks for retries, exponential backoff and idempotent APIs, and gives POST /connections with an Idempotency-Key as its example. Mobile networks drop responses, so retries are certain.

Decision

POST /connections, /applications, /posts and message sends accept an Idempotency-Key (client_msg_id for messages). The key is stored with a unique constraint next to the created row, and a replay returns the original response. Kafka consumers are idempotent by event ID. Retries use exponential backoff with jitter, and errors are split into retryable and final.

What LinkedIn actually did

This is the requirement's own design. LinkedIn has not published a named system for it.

Options weighed

• ChosenClient-supplied idempotency keys: The client knows which attempts are the same intent; the server enforces it.
• RejectedDedupe by content hash: Two identical, legitimate posts would be merged.
• RejectedRely on exactly-once transport: Exactly-once does not exist end to end across a mobile network.

Consequences

What it buys

• Retries are always safe
• Clients can retry aggressively on flaky networks
• Duplicate side effects become a bug class that cannot happen

What it costs

• Key storage and TTL management
• Every write API needs the pattern
• A replay must return the original response, which means storing it

Choose differently when

Never. Any system with retries needs this; only the storage of the key varies.

What survives a zone or a whole-colo loss, and how quickly?

Context

The requirement asks for 99.99% on critical APIs, multi-zone deployment, multi-region DR, and defined RPO and RTO. Failover that is exercised once a year fails when it is needed.

Decision

Four colos all serve traffic. Each member is pinned to a home colo, and within a colo, replicas span fault zones. Stateful stores replicate synchronously in the colo and asynchronously across colos. Draining a colo (re-pinning its members elsewhere) is done on a schedule for load tests and maintenance, so disaster recovery is the same operation. RPO/RTO: fault zone 0 / under 1 minute; colo seconds / under 15 minutes; derived stores rebuilt from the log.

What LinkedIn actually did

In 2017 LinkedIn described four data centres serving member traffic, members pinned by a signed cookie at the ATS tier, and TrafficShift, which drains data centres for load testing and for real incidents. Its full migration to Azure (Blueshift, announced 2019) was paused in 2022. LinkedIn runs its own data centres and uses Azure alongside them.

Options weighed

• ChosenActive-active colos, routine drains: The failover path is exercised every week, so it works when it is needed.
• RejectedActive-passive DR site: Idle capacity that is never proven until the disaster.
• Right elsewhereCloud multi-region: The same design on a cloud; LinkedIn paused its migration.

Consequences

What it buys

• A colo loss is a rehearsed traffic shift
• Standby capacity serves traffic every day
• Load tests use production traffic

What it costs

• Seconds of asynchronously replicated writes can be lost in a colo failure
• Every colo needs headroom to absorb another's members
• Strongly consistent domains need a written reconciliation playbook

Choose differently when

Below 99.95% targets, a single region across three zones with backup restore to a second region is cheaper and simpler.

How do hundreds of teams change production daily without members noticing the bad changes?

Context

Services are independently deployable, and relevance teams run many experiments at once. A bad change must be stopped by a machine before a person notices it.

Decision

Schema compatibility, security scanning and integration tests gate the build. Deploys go to canary hosts in one colo, where EKG compares them with control hosts and stops the rollout on regression. Deploys then roll colo by colo. Release is separate: a T-REX ramp from 1% to 100% with guardrail metrics, which can be ramped down in seconds. Schema changes follow expand then contract.

What LinkedIn actually did

LinkedIn described EKG, its automated canary analysis, in 2015. T-REX (formerly LiX and XLNT) ran up to 41,000 concurrent A/B tests on 700M+ members (2020). The 2011 Project Inversion freeze to rebuild release tooling is widely reported.

Options weighed

• ChosenCanary + ramp, deploy separate from release: Two independent brakes, one on code and one on behaviour.
• RejectedRelease trains: Batches many changes, so every rollback also rolls back unrelated work.
• RejectedDeploy equals release: Every deploy exposes all members at once.

Consequences

What it buys

• A regression is stopped automatically
• Features are turned off without a deploy
• Experiments are the normal way to ship

What it costs

• Flags and ramps accumulate and need cleanup
• Overlapping experiments interact
• Canary analysis needs enough traffic to be significant

Choose differently when

Low-traffic products cannot power canary statistics; use staged rollouts with manual verification.

How does anyone know the network is healthy for members, not just for hosts?

Context

The requirement asks for metrics, logs, traces and alerting based on user-facing SLOs rather than infrastructure alone. A drop in feed engagement with every host green is a real outage.

Decision

Services emit RED and saturation metrics to inGraphs. Alerts page through Iris on SLO burn rate, with the rotation held in Oncall. ThirdEye detects anomalies in business metrics in Pinot. Hodor watches queueing and latency and sheds low-priority traffic before a service falls over. Logs travel over Kafka. Traces carry context on every Rest.li hop.

What LinkedIn actually did

inGraphs (metrics dashboards), EKG, ThirdEye (anomaly detection over Pinot), Iris and Oncall (open-sourced 2017) and Hodor (overload detection and remediation, 2022) are all LinkedIn systems. LinkedIn has not published its current tracer; OpenTelemetry is the portable choice.

Options weighed

• ChosenSLO-based alerting + business anomalies: Pages when members hurt, not when a CPU is busy.
• RejectedInfrastructure threshold alerts: Noisy, and silent during outages that only show up in behaviour.
• Right elsewherePrometheus, Grafana, PagerDuty: The portable equivalent outside LinkedIn.

Consequences

What it buys

• Fewer, more meaningful pages
• Behavioural outages are caught
• Overload is handled before cascade

What it costs

• SLOs must be defined and maintained for every service
• Anomaly detection needs tuning to avoid false alarms
• Shedding decisions need owners

Choose differently when

Small systems: a hosted observability stack with SLO alerts covers this without building anything.

When a dependency is slow or down, what does each capability do, and who agreed to it?

Context

The requirement gives three examples: recommendations down means a chronological or trending feed, notifications down means core actions still succeed, and search down means profile, feed and apply stay up. Degradation decided during an incident is inconsistent and unrehearsed.

Decision

Every capability has a written behaviour for a slow dependency, a dead dependency and a lost colo, each behind a flag with a named owner. The Rest.li client enforces a timeout and a circuit breaker per dependency. Hodor sheds bot and prefetch traffic first. Core writes never depend on search, recommendations or notifications. Apply fails closed.

What LinkedIn actually did

Hodor (2022) is LinkedIn's overload detection and remediation. The per-capability matrix is this design's form of the requirement's examples.

Options weighed

• ChosenPre-agreed contract behind flags: Decided calmly, in advance, by the owners of each capability.
• RejectedAutoscale and hope: Scaling has lead time and hard ceilings, and both are discovered mid-incident.
• RejectedShed randomly at the load balancer: Drops paying recruiters as readily as scrapers.

Consequences

What it buys

• Nobody improvises during an outage
• Core actions survive every non-critical failure
• Degraded modes are tested, not assumed

What it costs

• Degraded modes interact and need combined testing
• Some degraded modes are visible to members (search off)
• Flags need periodic exercise

Choose differently when

Never. Only the size of the matrix changes with the system.

How does the network stop fake accounts, spam, fraud and harassment, and prove every decision?

Context

The requirement lists spam, fake accounts, bots, harassment, malicious links, fraudulent jobs, inappropriate content and coordinated abuse. Moderation must combine rules, ML classifiers, user reports and human review, and every decision must be auditable.

Decision

Four lanes: accounts, content, reports and jobs. Each runs signal, detect, decide, act and audit. Registration and login events feed an account risk model with cluster detection. PostCreated and JobCreated feed classifiers. Confident results act automatically and uncertain ones go to a human review queue. Every lane writes the same decision log, which drives appeals and retraining.

What LinkedIn actually did

LinkedIn's anti-abuse application layer is CASAL (2023). It open-sourced its isolation-forest library in 2019. It has described a machine-learned risk score at registration with account-cluster detection (2018), and its content abuse defence (2022).

Options weighed

• ChosenLayered lanes, one decision log: Each abuse type gets its own detector; all share audit and appeal.
• RejectedRules only: Adversaries learn the rules within days.
• RejectedModels only, no human review: Uncertain cases need judgement, and appeals need a human.

Consequences

What it buys

• Every action can be explained and appealed
• Reviewer decisions retrain the models
• Coordinated abuse is visible as clusters

What it costs

• Human review is expensive and emotionally demanding
• False positives hurt legitimate creators and job posters
• Adversaries adapt, so this is never finished

Choose differently when

Small networks can start with rules, reports and a manual queue. Keep the single decision log from day one.