Changing policy without redeploying: a decade of Netflix's service stack

The territory

One question runs through this entire decade: how do you change a single rule about how services talk to each other without asking every team to rebuild and redeploy?

State the problem without naming any technology. You operate a few hundred services. You need to change one rule that applies to all of them: how long to wait before giving up on a call, how many times to retry, how many calls may be in flight at once, which instances are eligible to receive traffic. The rule is not owned by any one team. Where do you put it, and what does it cost you to change it?

Netflix answered that question in 2013 by putting the rule in a library, and answered it differently, in stages, over the ten years that followed. Almost none of that later argument was made in a conference talk. It was made in status notices at the top of README files, in issue threads that were closed as not planned, in release notes recording what was dropped, and in the difference between what a repository promises and what its pull request queue shows.

100s of bn

Semaphore-isolated calls per day running through the circuit-breaker library at Netflix

Hystrix wiki, last edited 2017

18 Apr 2016

The day the client-side load balancer was placed in maintenance mode, by commit

Ribbon README history

Netflix modules deleted from Spring Cloud in one release, 27 January 2021

Spring Cloud 2020.0 release notes

Open pull requests on Ribbon, whose README offers to review community work

Checked 2026-09-09

The finding that changed how this guide is written

The stack the industry copied was, by the company's own account, partly not the stack the company ran. Ribbon's status notice lists its own modules one by one, and four of the nine are marked not used; of those that were in production, the notice says "we have wrapped them in a Netflix internal http client and we are not adding new functionality" (Ribbon README, notice added 18 April 2016). An open-sourced library is a snapshot of one company's internal need at one moment, not a product. That reading explains most of what follows.

Figure 1 · Where the policy lived, by year

Each step moves one decision further away from the service binary; the three steps drawn with a heavier gold border, 2016, 2020 and 2024, are the ones where a decision physically left it. Dated from the artefacts themselves: Ribbon README commits, the Hystrix status commit, gRPC proposal A27, Spring Cloud 2020.0 and Zuul 4.0.0.

Diagram source

What this guide covers. The service-to-service integration plane at one company: discovery, load balancing, failure isolation, dynamic configuration, the edge gateway and workflow orchestration, from 2015 to September 2026, plus the downstream record at Spring Cloud and the parallel decision at gRPC. What it does not cover: Netflix's content delivery network, encoding, data platform, machine-learning platform and playback clients, none of which are visible in this evidence base, and the internal systems that replaced the public ones, which are named in status notices but not published.

How it is actually built

Five planes of cross-cutting policy, and the four places any of them can physically live. The architecture question is not which library you use, it is which of the four boxes each plane sits in.

Reading the repositories side by side, the same five planes appear in every generation of the stack, and only their location changes. There is a registry that knows which instances exist; a decision point that chooses one of them for a given call; an isolation mechanism that stops a slow dependency consuming the caller; a configuration plane that carries values that must change without a restart; and a telemetry plane that reports what all of the above are doing.

In the 2013 shape all five were libraries in the same process as the business logic. Eureka held the registry and cached it client-side, refreshed "every 30 seconds", with a lease that expires "in about 90 seconds" if renewals stop (Eureka wiki). Ribbon made the choice of instance. Hystrix bounded the blast radius of a slow dependency with thread pools and semaphores. Archaius carried the values, and its stated purpose is exactly the theme of this guide: "Traditionally applications require a restart whenever configuration changes... Through Archaius, code can have direct access to the most recent configuration without the need to restart" (Archaius README). Servo reported metrics, later replaced by Spectator, whose predecessor now states that it "receives minimal maintenance" (Servo README).

The cost of that shape is legible in a single artefact. Prana, published in 2014, exists only because the shape excludes anyone who is not running a JVM: it "exposes Java based client libraries of various services like Eureka, Ribbon, Archaius over HTTP" (Prana README). A sidecar was already the answer to the library problem two years before the sidecar became an industry pattern, and the same README records what became of it: "The current implementation of this project is not used internally at Netflix." The company solved the multi-language problem a different way.

Figure 2 · The four homes for a cross-cutting rule, and what a change costs in each

The architecture decision is the vertical axis, not the library name. Costs are the deployment consequence of each placement, derived from the artefacts: Ribbon and Prana for the top two rows, Zuul's filter loading for the third, gRPC proposal A27 for the fourth.

Diagram source

Registry, cached at the caller

Every generation keeps a client-side cache of the registry, because the alternative is a lookup on the request path. That cache is the reason a dead instance keeps receiving traffic, and the reason discovery survives a registry outage.

Documented at: Eureka wiki

Decision point, moved twice

In-process in 2013, promised to gRPC interceptors in 2016, and standardised by the wider ecosystem in 2020 as a control-plane subscription rather than a client-side rule set.

Documented at: Ribbon README, gRPC A27

Isolation, from constants to control loops

Thread pools and fixed thresholds gave way to a limit derived at runtime. The successor library equates "a system's concurrency limit to a TCP congestion window" and measures rather than asks.

Documented at: concurrency-limits README

Configuration, the survivor

The one library-era component still being touched in 2026. Its whole reason for existing is that a value must change without a restart, which is the property every survivor in this story shares.

Documented at: Archaius, org listing, updated 29 Jul 2026

Edge gateway, policy as code you can push

Zuul's filters are written to directories that are "periodically polled for changes", then "read from disk, dynamically compiled into the running server". A policy change is a file, not a deployment, which is why the gateway outlived the libraries it once composed.

Documented at: Zuul wiki

Orchestration, adopted then handed back

Conductor was open source from 2016 and discontinued in December 2023; the internal successor, Maestro, is published separately and states that it "schedules hundreds of thousands of workflows, millions of jobs every day".

Documented at: Conductor, Maestro

Two divergence points are worth marking because they are where your system will differ. First, Netflix kept its registry and moved its decision point, while the ecosystem around it kept a thin client and moved both into a control plane; the gRPC proposal adopting xDS is explicit that the appeal is a configuration bus rather than a better balancing algorithm, since "the popular Envoy proxy uses the xDS API for many types of configuration, including load balancing, and that API is evolving into a standard" (A27, last updated 18 March 2020). Second, the isolation plane did not move at all; it stayed in the process and changed character, from constants a human sets to a loop that measures. Those are two genuinely different answers to the same pressure, and section 3 gives the condition that picks between them.

The decisions that matter

Four decisions carry the decade. For each one the sources give the stated reason, which is what lets you derive the condition that flips it.

Decision 1: does the routing decision live in the caller's process?

Chosen, 2016

Move off the in-process load balancer to interceptors on a shared RPC runtime
Stated reasons: "multi-language support and better extensibility/composability through request interceptors"

Rejected

Continuing to invest in the in-process library
Ribbon was already wrapped in an internal client, and half its published modules were unused

Flips when

You have exactly one runtime and fewer services than you have deploy slots in a week
Then the library is cheaper than a control plane, and stays cheaper until the second language arrives

Decision 2: who chooses the number, a human or the algorithm?

Chosen, 2018 onward

Derive the concurrency limit at runtime from measured latency, in the style of TCP congestion control
The Hystrix status notice redirects new work to "adaptive implementations that respond to real-time performance metrics"

Rejected

Statically sized thread pools and stress-test-derived request ceilings
Stated reason: with autoscaling "this value quickly goes out of date and the service falls over by becoming non-responsive"

Flips when

Capacity is fixed and known, and traffic is predictable
A constant is auditable and an adaptive loop is not, which matters where you must explain the limit to a regulator

Decision 3: when internal use stops, do you keep the open-source project alive?

Chosen

Freeze and say so, in the README, with a final version pinned to the last internally used build
Hystrix 1.5.18 was released "to align with the last version of Hystrix used internally at Netflix (1.5.11)"
Conductor: "Effective December 13, 2023, Netflix will discontinue maintenance of Conductor OSS on GitHub"

Rejected

Maintaining a community fork of code the company no longer runs
Also rejected in practice: the offer to review outside pull requests, which the queues show did not happen

Flips when

The project is a protocol or an interface others build against rather than an implementation detail
Interfaces attract forks that stay compatible; implementations attract forks that diverge

Figure 3 · Where to put a cross-cutting rule, as a decision

The first question is not technical. If the rule may wait for the next release train, a library is the cheapest place for it and everything below this line is over-engineering. Derived from the stated reasons in Ribbon's status notice and the concurrency-limits README.

Diagram source

Decision	Chosen	Rejected	Because, as stated	Evidence
Routing decision placement	Interceptors on a shared RPC stack	In-process load balancer	Multi-language support and composability	Ribbon status, 2016
Registry rewrite	Keep the 1.x design	The Eureka 2.0 architecture	Open-source work "is discontinued" and the 2.x artefacts are "use at your own risk"	Eureka wiki, 2018
Isolation mechanism	Adaptive limits from measured latency	Fixed thread pools and thresholds	Autoscaling makes a tuned constant stale	concurrency-limits
Gateway extensibility	Filters polled from disk and compiled at runtime	Redeploying the gateway per policy change	Operational routing changes are needed per customer and per test	Zuul wiki
Gateway async model	CompletableFuture in the filter API	RxJava Observable, the company's own library	Stated as the headline change of the major version	Zuul 4.0.0, 2024
Ecosystem load balancing	xDS subscription from a control plane	The bespoke grpclb protocol	xDS "is evolving into a standard" for configuration, not only balancing	gRPC A27, 2020
Downstream framework support	Remove the Netflix modules outright	Wrapping frozen upstreams indefinitely	Fourteen modules removed in one release, announced thirteen months earlier	Spring Cloud 2020.0
Orchestrator ownership	Hand the project to the community, publish the successor separately	Maintaining Conductor for outside users	Maintenance "discontinued" on a stated date	Conductor README

One decision was written down and then dropped, which is worth more than either outcome alone. Zuul issue #771, "Zuul 3 Improvements", opened 11 April 2020, is a design list that proposes removing the hard dependency on Groovy, dropping RxJava from the API, and making the name resolver and load balancer pluggable so that Eureka and Ribbon are no longer required. It is closed as not planned. Four years later Zuul 4.0.0 shipped the RxJava change on its own. Read together, the two artefacts say something usable: the clean-up plan that removes four dependencies at once does not get scheduled, and the one that removes a single dependency does.

What broke in production

Four failure classes, each reconstructed from threads filed by the people it happened to. Netflix publishes no incident reports for this layer, and that absence is itself part of the finding.

Read the evidence for what it is

There is no first-party postmortem in this corpus. Nothing in the Netflix organisation's public repositories narrates an incident, a duration or a customer impact. What exists instead is the operator's side: issues opened by the engineers who ran this stack elsewhere, and the design changes Netflix made afterwards without saying why. Every entry below is therefore an operator-filed report plus a reconstruction, and it is marked as such. If you need vendor-confirmed incident data for this layer, it does not exist in public.

Figure 4 · The stale-registry failure path, as reported

Notice where the failure is silent: the refresh task stops, and nothing in the request path knows the registry is frozen. Reconstructed from Eureka issue #1510, filed 7 August 2023.

Diagram source

Operator report

Class 1: the detector fails silently and the cache lies

AssumptionA client-side registry cache that refreshes every 30 seconds is at most 30 seconds stale.

What happenedThe client's scheduled refresh was rejected by a saturated two-thread executor. The cache stopped updating, the dependency was redeployed onto new addresses, and every call from that pod failed with "No route to host".

Blast radiusOne pod, all of its outbound calls, until restarted. Reported on eureka-client 1.10.17 with Spring Cloud 3.1.2; the issue was open at the check date.

FixNone published. The structural fix is to treat the age of the cache as a health signal rather than the liveness of the process holding it.

Design ruleAnything that refreshes on a timer needs an exported staleness metric and a readiness check bound to it. A cache with no age is an outage with no alert.

SourceEureka issue #1510, 2023-08-07

Operator report

Class 1b: eviction is slower than the number you configured

AssumptionSetting lease expiry to 90 seconds and the eviction timer to 9 seconds evicts a dead instance in about 90 seconds.

What happenedAn instance stopped at 12:59:40 and was evicted at 13:02:26, roughly 2 minutes 46 seconds later, with self-preservation already disabled. Registration, renewal, the server's response cache and the client's own poll compose into a window nobody configured directly.

Blast radiusEvery caller keeps sending to the dead address for the length of that window, which is where retry budgets get consumed.

FixNone in the thread. In practice the mitigation is not faster eviction but making callers tolerate a dead address: fail fast, retry another instance, and treat the registry as advisory.

Design ruleIn a lease-based registry, measure the end-to-end removal window rather than adding up the configured intervals. Then size retries and connection timeouts against the measured number.

Sourcespring-cloud-netflix #3652, 2020-04-05

Operator report

Class 2: layered timeouts, retries and pools compose badly

AssumptionA gateway, a load balancer and a circuit breaker, each with sensible defaults, add up to sensible behaviour.

What happenedConnections accumulated in CLOSE_WAIT behind the gateway's routing filter, whose default connection time-to-live is "-1, i.e. infinite", and were handed back out of the pool; the pool saturated and requests stopped. The same repository carries nine separate issues about the interaction of the circuit-breaker timeout, the load balancer read timeout and retries.

Blast radiusGateway-wide. Reported 16 March 2017 and recurring in issues through June 2021.

FixBound the connection lifetime, and set the outer timeout from the inner one rather than independently.

Design ruleAny two layers that both time out and both retry multiply. Write the budget down once, derive each layer's value from it, and give every pooled resource a finite maximum age.

Sourcespring-cloud-netflix #1785, and the timeout and retry issue set

Operator report

Class 3: the dependency freezes while you are still running it

AssumptionA widely adopted library from a large engineering organisation will keep receiving fixes, and a bug you report will be triaged.

What happenedA fix for a real defect on Java 11 sat unmerged from November 2022 until August 2026, when it closed because the author deleted their fork. Two gateway regressions reported after an upgrade in January 2026, with Netty buffer leak traces attached, were closed as not planned. Hystrix currently shows 52 open pull requests against 6 open issues.

Blast radiusDeferred rather than immediate. It arrives as an unpatchable dependency during an incident, or as a version you cannot upgrade past.

FixThe published fix is social, not technical: the status notice names a successor and invites the community to take ownership by email.

Design ruleBefore adopting a vendor's open-source infrastructure, check the ratio of open pull requests to closed ones and the age of the oldest open one. That ratio is the real support commitment; the README is marketing for it.

SourceHystrix PR #2033, Zuul issue #2021

Advisory record

Class 4: abandonment moves the security burden, it does not remove it

AssumptionWhen the originating company steps back, an actively used project simply continues under a fork.

What happenedNetflix discontinued Conductor maintenance on 13 December 2023. Eighteen months later a critical remote command execution, CVSS 9.8, was published against conductor-core, fixed in a version numbered by the fork rather than by Netflix.

Blast radiusEveryone still running the archived code, who must now track a different coordinate system for versions and advisories.

FixThe fork shipped the patch. That is the good outcome; the bad outcome is the same event with no fork.

Design ruleAn archived repository is not a frozen risk, it is a moving one. On the day a dependency is archived, re-point your scanners at whatever fork carries the advisories, or plan the removal.

SourceGHSA-8gqp-hr9g-pg62, 2025-06-30, conductor-oss

Reconstruction

Class 5: the fan-out, which is the failure that has no incident report

AssumptionA shared library is the cheap way to apply one rule everywhere.

What happenedIt is cheap to apply and expensive to change. The evidence is indirect and consistent: a sidecar built in 2014 purely to reach non-JVM applications, a stated move to a shared RPC runtime for "multi-language support", and a downstream framework that needed one release to delete fourteen modules and thirteen months of notice to do it.

Blast radiusMeasured in quarters. Ribbon was frozen in April 2016 and removed downstream in January 2021, four years and nine months later.

FixMove the rule to a plane whose update unit is not the application deployment.

Design ruleBefore you put a rule in a library, write down the number of services multiplied by the number of runtimes, and ask whether you would accept that as the cost of changing one line. That number is the real design constraint.

SourcePrana, Ribbon status, Spring Cloud 2020.0

Figure 5 · The states a vendor-run dependency passes through, and where you have to act

Every transition here is documented in this corpus for at least one project, and the only branch you control is the last one. Sources: Hystrix status, Conductor discontinuation, the fork and the advisory that followed it.

Diagram source

Numbers you can plan against

Dates behave like measurements in this material. The interval between two commits is the most reliable quantity a repository gives you.

Quantity	Value	Where	Context	As of	Source
Semaphore-isolated calls per day	hundreds of billions	Netflix	Scale the library-era isolation ran at	wiki edited 2017	Hystrix wiki
Thread-isolated calls per day	tens of billions	Netflix	Same source, the more expensive isolation mode	wiki edited 2017	Hystrix wiki
Load balancer frozen	2016-04-18	Ribbon	Commit ba0d2a1, "Add maintenance project status to README"	2026-09-09	commit history
Circuit breaker frozen	2018-11-19	Hystrix	Status section added; final release 1.5.18, matching internal 1.5.11	2026-09-09	commit a7df971
Registry rewrite abandoned	wiki edit 2018-07-13	Eureka 2.0	"discontinued", artefacts "use at your own risk"	2026-09-09	Eureka wiki
Configured lease expiry	90 s	Eureka	Documented removal time after renewals stop	wiki, checked 2026-09-09	Eureka at a glance
Measured eviction delay	2 min 46 s	Operator report	Self-preservation disabled, eviction timer 9 s, expiry 90 s	2020-04-05	issue #3652
Registry refresh interval	30 s	Eureka	Client poll for the registry	wiki, checked 2026-09-09	Eureka at a glance
Downstream modules removed	14	Spring Cloud 2020.0.0	General availability 27 January 2021, announced 23 December 2019	2021-01-27	release notes
Freeze to downstream removal, load balancer	4 yr 9 mo	Derived	2016-04-18 to 2021-01-27, arithmetic on the two dates above	derived 2026-09-09	inputs
Freeze to downstream removal, circuit breaker	2 yr 2 mo	Derived	2018-11-19 to 2021-01-27	derived 2026-09-09	inputs
Open pull requests, frozen circuit breaker	52	Hystrix	Against 6 open issues and 24.5k stars	2026-09-09	repository
Open pull requests, frozen load balancer	40	Ribbon	Oldest visible from 7 February 2018, despite the README offering review	2026-09-09	pull request queue
Oldest open question about the replacement API	2015-09-02	Ribbon #243	"replacement API is unclear", still open eleven years later	2026-09-09	issue #243
Orchestrator maintenance ends	2023-12-13	Conductor	Repository archived, org listing shows last update 22 December 2023	2026-09-09	README
Critical advisory against the fork	CVSS 9.8	conductor-core	Command injection, fixed in 3.21.13, 18 months after the handover	2025-06-30	GHSA-8gqp-hr9g-pg62
Successor orchestrator scale	100,000s of workflows per day	Maestro	"millions of jobs every day", publisher's own figure, not independently measured	checked 2026-09-09	README
Gateway drops the reactive API	2024-07-08	Zuul 4.0.0	RxJava Observable replaced by CompletableFuture	2024-07-08	release notes
Promised open-source RPC replacement	none published	Netflix org	Promised "not before Q3 of 2016"; searches for grpc and rpc return no such repository	2026-09-09	org search

Read these carefully

The two call-volume figures are the only throughput numbers in this corpus, they are the publisher's own, and the wiki page carrying them was last edited in 2017. Treat them as an order of magnitude for what a library-era stack was carrying, not as a current statement about Netflix. The Maestro figure is likewise self-reported.

Repository "last updated" dates are the weakest quantity here and are used only where a date is corroborated by a commit, a release or a status notice. A dependency bump moves that date without anyone maintaining anything, which is exactly how a frozen project can look alive: Hystrix shows activity in December 2025 and has not been developed since 2018.

The evidence wall

Every artefact behind this page, graded. The full ledger, with the quote supporting each claim, ships beside this file as sources.md.

What this evidence base is missing, and why

The research environment could reach two hosts: github.com and raw.githubusercontent.com. Every other host was refused by the network policy, so there is no engineering blog, no conference talk and no paper in this guide. That removes the sanctioned narrative layer and leaves the artefacts nobody writes for an audience. It also means any claim that would normally rest on a blog post is either absent here or marked as a reconstruction. Read the page as a repository archaeology, which is what it is.

Decision recordNetflix2016-04-18

Ribbon README, "Project Status: On Maintenance"

Lists each module with its internal status, four of them marked not used, and states that the production ones are wrapped in an internal client with no new functionality. Names gRPC as the direction, for "multi-language support and better extensibility".

Carry forwardThe published library is a snapshot of an internal need. Read the component-by-component status before adopting the whole.

https://github.com/Netflix/ribbon

SourceNetflixchecked 2026-09-09

Ribbon open pull request queue

40 open pull requests, the oldest visible from 7 February 2018, against a README that says complete pull requests would be reviewed and accepted.

Carry forwardThe queue is the support commitment. Compare it with the stated one before you depend on either.

https://github.com/Netflix/ribbon/pulls?q=is:pr+is:open

SourceNetflix2015-09-02

Ribbon issue #243, replacement API unclear

An issue reporting that a deprecated client is still used throughout the documentation and that the replacement is unclear. Open at the check date, eleven years later.

Carry forwardDeprecation without a documented migration path is a decision to leave users where they are.

https://github.com/Netflix/ribbon/issues/243

Decision recordNetflix2018-11-19

Hystrix status notice

Declares maintenance mode, states that issues will not be reviewed nor pull requests merged, pins the final release to the last internally used version, points new projects at resilience4j and invites the community to take ownership.

Carry forwardA good freeze notice names a successor and a version boundary. Demand both from your own deprecations.

https://github.com/Netflix/Hystrix

Case studyNetflixwiki, edited 2017

Hystrix wiki, production volume

"Today tens of billions of thread-isolated, and hundreds of billions of semaphore-isolated calls are executed via Hystrix every day at Netflix." The ratio is the interesting part: the cheap isolation mode carries an order of magnitude more traffic.

Carry forwardThread-per-call isolation is a premium you pay on a minority of calls. Decide which calls deserve it before you standardise.

https://github.com/Netflix/Hystrix/wiki

SourceNetflix2022-11 to 2026-08

Hystrix pull request #2033

A fix for an exception thrown when resizing thread pools on Java 11. Opened November 2022, never merged, closed in August 2026 when the contributor deleted their fork.

Carry forwardIn a frozen project, your fix is your fork. Budget for carrying patches, or leave.

https://github.com/Netflix/Hystrix/pull/2033

Decision recordNetflixwiki edit 2018-07-13

Eureka wiki, the 2.0 discontinuation

"The existing open source work on eureka 2.0 is discontinued. The code base and artefacts that were released as part of the existing repository of work on the 2.x branch is considered use at your own risk."

Carry forwardA rewrite that is abandoned in public is more informative than one that quietly stalls. Ask vendors directly about the state of any announced next generation.

https://github.com/Netflix/eureka/wiki

SourceNetflixchecked 2026-09-09

Eureka at a glance

Documents the mechanism that decides how stale a caller's view can be: a 30 second registry poll, renewals, and removal "in about 90 seconds" when renewals stop. Also states that clients keep working when every registry server is down.

Carry forwardAvailability under registry failure and speed of removing dead instances are the same dial. Pick which one you are buying.

https://github.com/Netflix/eureka/wiki/Eureka-at-a-glance

Operator reportOperator, via Netflix issue tracker2023-08-07

Eureka issue #1510, refresh task rejected

A saturated executor rejects the cache refresh; the registry view freezes; a redeployed dependency becomes unreachable for that pod, with "No route to host" on every call.

Carry forwardExport the age of every cached view and fail readiness on it. A frozen refresher is invisible to liveness checks.

https://github.com/Netflix/eureka/issues/1510

Operator reportSpring Cloud users2020-04-05

spring-cloud-netflix #3652, eviction measured

Timestamps in the thread show 2 minutes 46 seconds from process stop to eviction with a 90 second lease and a 9 second eviction timer.

Carry forwardMeasure the removal window end to end in your own environment; the configured intervals are inputs to it, not the answer.

https://github.com/spring-cloud/spring-cloud-netflix/issues/3652

Operator reportSpring Cloud users2017-03-16

spring-cloud-netflix #1785, CLOSE_WAIT connections reused

Connections stuck in CLOSE_WAIT are handed back out of the pool behind a routing filter whose default connection lifetime is infinite, until the pool blocks requests.

Carry forwardGive every pooled connection a maximum age. An infinite time-to-live turns a transient peer failure into a permanent pool defect.

https://github.com/spring-cloud/spring-cloud-netflix/issues/1785

SourceSpring Cloud users2018 to 2021

The timeout, retry and circuit-breaker issue set

Nine issues in one repository about the same interaction: an outer circuit-breaker timeout, an inner read timeout, and retries between them producing duplicate or abandoned work.

Carry forwardLayered resilience needs one budget, allocated downward. Independent defaults at each layer is the recipe for the whole class.

https://github.com/spring-cloud/spring-cloud-netflix/issues

Decision recordNetflixchecked 2026-09-09

concurrency-limits, the case against tuned constants

States that stress-test-derived request ceilings go stale under autoscaling and that the service then "falls over by becoming non-responsive", and derives a limit at runtime by treating concurrency as a TCP congestion window, with Little's Law as the model.

Carry forwardIf a number in your configuration has to be revisited whenever capacity changes, it is a control loop with a human in it. Automate the loop or accept the staleness.

https://github.com/Netflix/concurrency-limits

SourceNetflixchecked 2026-09-09

Prana, the 2014 sidecar

Exposes the Java client libraries for discovery, load balancing and configuration over HTTP so non-JVM applications can participate, and records that the implementation is not used internally.

Carry forwardWhen a second runtime appears, the library becomes a protocol problem. Design the out-of-process path before you need it, not after.

https://github.com/Netflix/Prana

SourceNetflixchecked 2026-09-09

Zuul, how filters are loaded

Filter source is written to directories that are polled for changes, then compiled into the running server and applied to subsequent requests.

Carry forwardThe gateway earns its place when policy can be pushed to it faster than a deployment. Without hot loading it is one more service to deploy.

https://github.com/Netflix/zuul/wiki/How-It-Works

Decision recordNetflix2020-04-11

Zuul issue #771, "Zuul 3 Improvements", closed as not planned

Proposes dropping the hard dependencies on Guice and Groovy, removing RxJava from the API, and making the name resolver and load balancer pluggable so Eureka and Ribbon are no longer required. Closed without being done; one item shipped four years later.

Carry forwardDependency removal lands when it is packaged as one change per release, not as a clean-up epic.

https://github.com/Netflix/zuul/issues/771

SourceNetflix2024-07-08

Zuul 4.0.0 release notes

Replaces RxJava Observable in the async filter API with CompletableFuture, removes the debug routing infrastructure, and changes connection draining to an event on the pipeline.

Carry forwardEven the vendor eventually leaves its own abstraction. Prefer the platform primitive when the library adds no leverage.

https://github.com/Netflix/zuul/releases/tag/v4.0.0

Operator reportOperator, via Netflix issue tracker2026-01-14

Zuul issue #2021, buffer leaks after upgrade, closed as not planned

Netty leak detection fires after upgrading the gateway, with traces from the message buffering path. Closed without a fix, alongside a companion report on the following version.

Carry forwardAdopting a company's internal gateway means adopting its upgrade priorities. Pin versions and own the debugging.

https://github.com/Netflix/zuul/issues/2021

Decision recordSpring Cloud (VMware)2021-01-27

Spring Cloud 2020.0 release notes

Removes fourteen Netflix modules from the release train in one version, including the circuit breaker, the load balancer, the gateway, the configuration library and the sidecar, having announced the plan thirteen months earlier.

Carry forwardYour framework's removal date, not the upstream freeze date, is the deadline you actually have. Track both.

https://github.com/spring-cloud/spring-cloud-release/wiki/Spring-Cloud-2020.0-Release-Notes

Decision recordgRPC2020-03-18

Proposal A27, xDS-based global load balancing

Moves gRPC from its own balancing protocol to the xDS configuration bus, on the reasoning that xDS is becoming the standard for configuring data plane software generally, and wires it into both the resolver and the balancing policy.

Carry forwardThe durable choice is the configuration bus, not the balancing algorithm. Pick the client that can subscribe to yours.

https://github.com/grpc/proposal/blob/master/A27-xds-global-load-balancing.md

Decision recordNetflix2023-12-13

Conductor discontinuation notice

States the date maintenance ends, points to community forks, and leaves the repository readable. The organisation listing shows the archive with its last update on 22 December 2023.

Carry forwardAsk any vendor-run open-source dependency one question: what happens on the day it stops being used internally?

https://github.com/Netflix/conductor

SourceGitHub Advisory Database2025-06-30

GHSA-8gqp-hr9g-pg62, critical injection in the forked orchestrator

CVSS 9.8 command injection in conductor-core below 3.21.13, eighteen months after the original project was discontinued, patched by the fork.

Carry forwardAfter an archive, your advisory feed must follow the fork's coordinates or you stop receiving notifications for code you still run.

https://github.com/advisories/GHSA-8gqp-hr9g-pg62

Case studyNetflixchecked 2026-09-09

Maestro, the successor orchestrator

Published separately from the discontinued project and described as a fully managed workflow service scheduling "hundreds of thousands of workflows, millions of jobs every day" under a strict service level objective.

Carry forwardWhen a vendor archives one project and publishes another in the same problem space, the archive is a migration signal, not an abandonment of the problem.

https://github.com/Netflix/maestro

SourceNetflixchecked 2026-09-09

Organisation repository listing and searches

Shows what is still being touched: the gateway in September 2026, discovery in August 2026, configuration in July 2026, adaptive limits in January 2026. Searches for grpc and rpc return no successor to the load balancer, ten years after one was promised.

Carry forwardAn organisation's repository list, sorted by last update, is the fastest map of which of its published systems still matter to it.

https://github.com/orgs/Netflix/repositories?sort=updated

Sourceresilience4jchecked 2026-09-09

resilience4j, the named successor

A fault tolerance library built as composable decorators, where "you have the choice to select the decorators you need and nothing else", contrasting with a framework that owns thread pools and a dashboard.

Carry forwardThe successor to a heavy framework is usually a smaller thing you assemble. Check what runtime you are inheriting, not just what features.

https://github.com/resilience4j/resilience4j

SourceNetflixchecked 2026-09-09

Archaius and Servo, the configuration and telemetry planes

Archaius exists so that values change without a restart and is still maintained; Servo states that it receives minimal maintenance and points to Spectator, which is the most recently updated repository in the organisation.

Carry forwardThe planes that survive a decade are the ones whose whole purpose is changing something at runtime.

https://github.com/Netflix/archaius

Build a miniature, then productionise it

Seven rungs. The first three are an afternoon each; the crossing into production shape happens at rung four, where a change stops requiring a deployment.

Count your fan-out before writing any code

Pick one cross-cutting rule you currently ship in a shared library, for example the default socket timeout. Count the services that depend on it, the runtimes involved, and the median time from a library release to full adoption in your estate.

Done when: you can state the cost of changing that one value as a number of deployments and a number of weeks. Teaches: the constraint that drove every decision in this guide.

Two services, a registry, and a client-side cache

Run a registry, register two instances, and have the caller poll and cache. Kill an instance without deregistering it and measure the time until callers stop choosing it.

Done when: you have measured the removal window rather than read it from configuration. Teaches: why the configured lease is a lower bound.

Break the refresher, not the registry

Leave the registry healthy and stop the caller's refresh task, the way a saturated executor does. Watch the caller keep routing to addresses that no longer exist while every health check stays green.

Done when: you have added a cache-age metric and a readiness check that fails on it. Teaches: silent detector failure, the most expensive class in section 4.

Static limit against autoscaling

Put a fixed concurrency limit in front of a dependency, tune it under load, then double the instance count and repeat the load test with the same limit. Record where it stops protecting and starts throttling.

Done when: you can show the limit is wrong in both directions after a capacity change. Teaches: why the successor library derives limits rather than accepting them.

Replace it with a control loop

Swap the constant for an adaptive limiter driven by measured latency, in the congestion control style, and rerun both load tests. Instrument the limit itself as a time series.

Done when: the limit tracks a capacity change with no human input, and you can explain a limit drop from the latency series. Teaches: that an adaptive limit needs its own observability or it is unauditable.

Move one rule off the deployment path

Take a routing or retry rule out of the service and put it where it can change without a rebuild: a dynamic property, a gateway filter loaded at runtime, or a control plane the client subscribes to. Time a change from edit to effect.

Done when: the change takes effect in seconds or minutes and no service was redeployed. Teaches: the actual value of the gateway and the control plane, stated in units of time.

Run a freeze drill on your riskiest dependency

Pick the vendor-run open-source component you would least like to lose. Record the date of its last release, the count and age of open pull requests, whether its README names a successor, and where its advisories would appear if the project were archived tomorrow.

Done when: the answers are written into your architecture decision record along with the exit trigger. Teaches: that adoption is an ongoing bet on someone else's internal roadmap.

Keep hunting

These are the queries and URL patterns that produced this page. They work on any organisation, and they are the whole method when the blogs are unreachable.

Dating a change of direction

github.com/<org>/<repo>/commits/master/README.md
github.com/<org>/<repo>/commit/<sha>
github.com/orgs/<org>/repositories?sort=updated
github.com/<org>/<repo>/releases

Finding the argument, not the announcement

github.com/<org>/<repo>/pulls?q=is:pr+is:closed+is:unmerged
github.com/<org>/<repo>/issues?q=is:issue+"outage"
github.com/<org>/<repo>/issues?q=is:issue+is:closed+reason:"not planned"
github.com/<org>/<repo>/wiki

Testing what a project promises

github.com/<org>/<repo>/pulls?q=is:pr+is:open+sort:created-asc
github.com/orgs/<org>/repositories?q=<promised successor>
github.com/advisories?query=<org or package>
raw.githubusercontent.com/<org>/<repo>/<branch>/README.md

Following the change downstream

github.com/<framework>/<release repo>/wiki release notes removed modules
github.com/<downstream>/issues?q=is:issue+<upstream library>+timeout+retry
github.com/grpc/proposal design proposals for routing and load balancing
github.com/<fork org>/<repo> README continuation of the original project

One habit is worth more than any single query. When a repository states a plan with a date in it, write the date down and come back to it. The Ribbon notice promising an open-source gRPC replacement "not before Q3 of 2016" is the most informative sentence in this corpus precisely because ten years later there is nothing in the organisation's public repositories that fulfils it.

References

Netflix, Ribbon, "Project Status: On Maintenance" Repository README, notice added 18 April 2016. Checked 2026-09-09.
Netflix, Ribbon README commit history Commit ba0d2a1, 18 April 2016. Checked 2026-09-09.
Netflix, Ribbon open pull requests 40 open at check date. Checked 2026-09-09.
Netflix, Ribbon issue #243 Opened 2 September 2015, open at check date. Checked 2026-09-09.
Netflix, Hystrix, status notice Repository README, notice added 19 November 2018. Checked 2026-09-09.
Netflix, commit a7df971, "Update README.md" Adds the Hystrix status section, 19 November 2018. Checked 2026-09-09.
Netflix, Hystrix wiki Production call volumes, page last edited 2017. Checked 2026-09-09.
Netflix, Hystrix pull request #2033 Opened 12 November 2022, closed unmerged 11 August 2026. Checked 2026-09-09.
Netflix, Eureka wiki home, 2.0 discontinuation notice Page last edited 13 July 2018. Checked 2026-09-09.
Netflix, "Eureka at a glance" Wiki page, undated. Checked 2026-09-09.
Netflix, Eureka issue #1510 Filed 7 August 2023. Checked 2026-09-09.
Netflix, Eureka issue #1073 Filed 30 May 2019. Checked 2026-09-09.
Netflix, Eureka releases Latest visible 6 August 2024. Checked 2026-09-09.
Netflix, Zuul, "How it works" Wiki page, undated. Checked 2026-09-09.
Netflix, "How we use Zuul at Netflix" Wiki page, undated. Checked 2026-09-09.
Netflix, Zuul issue #771, "Zuul 3 Improvements" Opened 11 April 2020, closed as not planned. Checked 2026-09-09.
Netflix, Zuul 4.0.0 release notes 8 July 2024. Checked 2026-09-09.
Netflix, Zuul issue #2021 Filed 14 January 2026, closed as not planned. Checked 2026-09-09.
Netflix, concurrency-limits Repository README. Checked 2026-09-09.
Netflix, Prana Repository README. Checked 2026-09-09.
Netflix, Archaius Repository README, 2.x branch. Checked 2026-09-09.
Netflix, Servo Repository README. Checked 2026-09-09.
Netflix, Conductor, discontinuation notice Effective 13 December 2023; archive last updated 22 December 2023. Checked 2026-09-09.
Conductor OSS, community continuation Repository README. Checked 2026-09-09.
GitHub Advisory Database, GHSA-8gqp-hr9g-pg62 Published 30 June 2025, CVSS 9.8. Checked 2026-09-09.
Netflix, Maestro Repository README. Checked 2026-09-09.
Netflix, DGS framework Repository README, support horizons for versions 5 through 11. Checked 2026-09-09.
Netflix, Falcor Repository page. Checked 2026-09-09.
Netflix, RxNetty Repository README, 0.5.x branch status. Checked 2026-09-09.
Netflix, organisation repository listing Sorted by last update. Checked 2026-09-09.
Netflix, organisation search for "grpc" Returns only the archived Conductor. Checked 2026-09-09.
Spring Cloud, 2020.0 release notes General availability 27 January 2021. Checked 2026-09-09.
Spring Cloud Netflix, issue #3652 Filed 5 April 2020. Checked 2026-09-09.
Spring Cloud Netflix, issue #1785 Filed 16 March 2017. Checked 2026-09-09.
Spring Cloud Netflix, timeout and retry issue set Nine issues between 2018 and 2021. Checked 2026-09-09.
gRPC, proposal A27, "xDS-Based Global Load Balancing" Mark D. Roth, last updated 18 March 2020. Checked 2026-09-09.
resilience4j Repository page. Checked 2026-09-09.