LinkedIn Professional Network  ·  View 23 of 30  ·  6 · Operations

Delivery Pipeline, Canaries and Ramps

How a change reaches members, and the three things that stop a bad one.

Editable source SVG draw.io All views
Source Multiproduct repo trunk + review Build CI build + tests Artifact repo signed Gates Schema compat Rest.li · Avro Security scan Integration tests Deploy Canary hosts one colo EKG verdict canary vs control Colo by colo rolling deploy Release T-REX ramp 1% to 100% A/B verdict guardrail metrics Fully ramped fail: roll back ramp down Delivery Pipeline, Canaries and Ramps Application we own Data store Decision point failure / alternate Deploy and release are separate acts: code ships dark, a ramp turns it on, and a ramp turns it off. v 1.0 · owner Developer Productivity · date 2026-09

Decisions

  • Rest.li and Avro schema compatibility is a build gate, not a review comment
  • EKG compares canary hosts with control hosts and stops a rollout automatically (LinkedIn, 2015)
  • Release is a T-REX ramp, separate from deploy. T-REX has run up to 41,000 concurrent A/B tests (LinkedIn, 2020)

History worth knowing

  • In 2011 LinkedIn paused feature work for about two months (Project Inversion) to rebuild its release tooling. That is the precedent for treating delivery as architecture (widely reported)
  • LiX and XLNT were the earlier names of the experimentation platform now called T-REX

Risks

  • Ramps interact with each other. T-REX tracks overlapping experiments per member
  • Rolling back data is not rolling back code, so schema changes follow expand then contract
◀  Deployment Topology All views Observability and Operations  ▶
Solution Architecture v1.0 · LinkedIn's own stack in LinkedIn-run colos · Enterprise Architecture · 2026-0923-cicd-and-ramps