Deciding who may do what: a field guide to production authorization

The territory

One question on every request, three places to answer it, and a failure record that clusters at the two extremes.

<10ms

95th-percentile check latency Google's Zanzibar held over three years, at millions of checks per second

Zanzibar paper, 2019

885M

Documents exposed 2014–2019 by one missing object-level check in First American's EaglePro

NYDFS consent order, 2023

7h 27m

Global Google Cloud outage when Service Control, the check on every API call, crashed in every region at once

GCP incident report, June 2025

5½ yr

Age of the still-open feature request for the Zanzibar paper's consistency tokens in Ory Keto

keto#517, opened April 2021

State the problem without naming a product and it sounds almost too small to have an architecture: given a caller, an action and an object, return yes or no. What makes it a systems problem is where the question sits. It is on the hot path of every request, so the answer has to arrive in single-digit milliseconds. It depends on data that changes constantly, memberships, roles, shares, ownership transfers, so the answer has to track a moving target. And unlike most subsystems, both wrong answers are expensive in different currencies: a wrong yes is a breach, a wrong no is an outage.

The organisations that have written seriously about solving it fall into three camps. Google published Zanzibar in 2019, a dedicated global service storing trillions of access-control entries as relationship tuples and answering millions of checks per second for Drive, Calendar, YouTube and Cloud [1]. Airbnb, Carta and Gojek each built or adopted a Zanzibar-shaped service and published numbers [5][6][9]. Netflix runs a central service, PACS, in the critical path of a multi-million-requests-per-second product [10]. And a third group looked at all of that and declined: Figma evaluated Open Policy Agent, Zanzibar-style systems and Oso, rejected all three, and built a permissions DSL inside the application instead [4].

The finding that reframes the topic

Read the incident record next to the architecture record and they barely overlap. The published breaches, Optus, First American, the whole OWASP category the industry calls broken object level authorization, are checks that were never made: nothing to do with policy models or consistency. The published outages are the opposite pole: the check existed, was centralised, and its failure took everything with it, most vividly Google Cloud in June 2025. The sophisticated architecture and the actual loss events solve for different failure modes, and a design review should ask about both.

Figure 1 · The landscape: one enforcement point, three places the decision can run

Every deployment has an enforcement point; the camps differ on where the decision engine and its data live. Reconstructed from Figma, Google and Netflix accounts.

Diagram source

Scope, stated up front. This guide covers the decision and enforcement layer: where the check runs, how it stays fast, how fresh its data is, and how it fails. It does not cover authentication or session management, secrets handling (a separate guide in this series covers the build pipeline's credentials), database row-level security, or the organisational work of designing role hierarchies. Those share a boundary with this material but a different failure record.

How it is actually built

Underneath the product names, every published fast-check system is the same machine: a log of relationship facts, a denormalized index over them, and a cache with an invalidation story it is honest about to varying degrees.

Start with the data model, because every camp converged on it independently. Zanzibar stores permissions as tuples of the form object#relation@user [1]. Himeji's basic unit is a tuple written entity #relation @principal [5]. Carta's AuthZ builds a graph out of RelationTuples [7]. Slack's role management, though framed as classic RBAC, reduces to the same shape: a role is a set of permissions, granted to a user within an org or a workspace, which is a tuple with the container as the object [8]. The convergence matters to an architect because it means the storage problem is settled; the open problems are all downstream of it.

The second shared component is the one the marketing rarely leads with: a purpose-built cache, because graph traversal at request time cannot hit the latency target. The divergence is only in when the traversal happens. Airbnb fans out at write time: when a resource is mutated, Himeji writes the expanded permission set, so reads are lookups; Alan Yao's post is explicit that this is the read-heavy trade, fanning out "on the writes instead of the reads", and that the sharded per-availability-zone cache in front of it targets roughly a 98% hit rate [5]. Google keeps traversal at read time but flattens the expensive part, nested group membership, into the Leopard index, which precomputes transitive membership so a twelve-hop chain becomes one lookup [1]. These are the same decision made in two directions, and both accounts say why: the check is only fast because most of the answer was computed before the question arrived.

Third, the consistency machinery, which is where the camps genuinely part company. A cache of permission data can serve an answer that was true a moment ago and is false now. The Zanzibar paper names the resulting exploit the new-enemy problem and spends much of its design budget on it: clients store an opaque token (a zookie) with each content version, obtained via a content-change check at save time, and later checks are evaluated at or after that snapshot, so ACL changes and content changes can never be observed out of order [1]. SpiceDB carries the mechanism forward as ZedTokens with per-request consistency settings [13]. Everyone else largely runs without it; section 3 takes up what that costs and section 4 what the record says about it.

Figure 2 · Reference architecture of a relationship-based check service

The read path is a cache in front of a graph evaluator; the write path feeds the denormalizer and mints the consistency token. Reconstructed from the Zanzibar paper and Airbnb's Himeji post; the token path is omitted by most reimplementations.

Diagram source

The tuple log

Facts, not rules: who relates to what. Rules live in a schema (Zanzibar's namespace config, SpiceDB's schema, OpenFGA's model) that says which relations imply which permissions. Facts change constantly; rules change at deploy cadence. Splitting them is what makes the service generic.

Runs this way at Google, Airbnb, Carta, Gojek

The precomputed answer

Leopard flattens group nesting at Google; Himeji expands permissions at write time at Airbnb. Different mechanics, same claim: request-time traversal is the thing you must design out, because it is the difference between a lookup and a graph walk whose depth your product's sharing model controls.

Reported by Google and Airbnb

The enforcement point

Separate from the decision, and separately configurable: Envoy's ext_authz filter calls an external checker and, by default, rejects with 403 when that checker errors; Kubernetes' authorization chain lets each webhook declare Deny or NoOpinion on failure. Where Airbnb calls Himeji from the data layer, most adopters enforce in middleware.

Configuration surface: Envoy proto, KEP-3221

The in-application camp deserves its own paragraph, because it is not a lesser version of the service camp; it is a bet on a different constraint. Figma's post is the clearest statement: their permission rules were entangled with the product's data model, the pain was correctness drift (bugs, support tickets and delayed projects traced to rule complexity, starting in early 2021), and their fix was a DSL whose policies reference domain objects while a backend engine decides how to load the data, in what order, from which replicas, with what caching [4]. That is Zanzibar's separation of rules from data reproduced inside one codebase, minus the network hop and minus the obligation to model everything as tuples. Gojek's account is the counter-experience: their in-app version decayed into what they describe as a four-hundred-line check_permission function of nested conditionals before they rebuilt on SpiceDB [9]. The difference between those two outcomes is discipline about the rules/data split, not the camp chosen.

The decisions that matter

Four forks, each with a published rejection on the record, and the condition that flips each one.

Decision 1: does the check run in your application, or in a service?

Chosen

Central service: Google, Airbnb, Carta, Netflix, Gojek. One place to audit, one team owning latency, every service gets the same answer.
In-app DSL: Figma, after evaluating OPA, Zanzibar-style systems and Oso.

Rejected

Figma rejected the service camp because their rules needed the product's own data model; a generic tuple vocabulary did not reach the core of the problem [4].
Gojek rejected in-app checks after living with the nested-conditional version [9].

Flips when

Count the services that enforce rules over the same objects. At one, the DSL camp wins on data locality. The service camp wins roughly when that count passes a handful, because N copies of the rules is N chances to drift, which is the exact failure class in section 4.

Decision 2: strict consistency tokens, or eventual consistency with a TTL?

Chosen

Google: zookies, causal ordering between ACL and content updates, freedom elsewhere [1].
SpiceDB: ZedTokens, consistency chosen per request [13].

Postponed

Ory Keto: snapshot tokens proposed April 2021, still an open issue in September 2026 [15].
Most self-built systems: neither Himeji's nor Carta's public account describes a client-visible consistency token.

Flips when

Ask what revocation must mean. If removing access must bind to specific content versions (the fired employee must not see the doc as it exists after the firing), you need the token machinery. If seconds of staleness are acceptable, a short TTL is enormously cheaper. There is no middle setting.

Figure 3 · The new-enemy sequence: two writes, observed out of order

The exploit needs no attacker skill, only replica lag between an ACL write and a content write. The zookie pins the later check at or after the ACL change. From the Zanzibar paper and AuthZed's worked example.

Diagram source

Decision 3: when the checker is unreachable, fail open or fail closed?

Defaults on the record

Envoy ext_authz: closed. On checker error or 5xx, reject with 403; opening requires setting failure_mode_allow [17].
Kubernetes: per-webhook choice, Deny or NoOpinion, in the structured authorization config [18].

Revised after impact

Google Cloud ran Service Control's checks fail-closed and inline; after June 2025 the published remediation is to isolate them so a failing check fails open and API requests still serve [20][21].

Flips when

Split by what the request does. Reads of low-sensitivity data through a checker that is down: open is defensible and Google now agrees. Writes, money movement, or anything the CISO enumerated: closed, and the 403s are the pager. A single global setting is the wrong shape for this decision.

Decision 4: point checks in a loop, or a first-class list operation?

Chosen

OpenFGA: a dedicated ListObjects API, deliberately bounded by ListObjectsMaxResults and a deadline, with no pagination because paginating a graph traversal would serialize it [19].
AuthZed: BulkCheck, adopted by the customer in the Spanner incident below.

Rejected

A generic multi-relation list API, rejected in the same RFC: "the query cost of supporting a generic API endpoint like this could be quite expensive" [19].
Cedar rejected a batched is_authorized outright in 2023: batching forces a choice of what stays constant across the batch, and the real cost was in entity loading, not call overhead [16].

Flips when

The moment a UI filters a collection by permission. A product page that lists fifty items issues fifty checks unless the API is list-shaped; AuthZed's case file shows a client emitting 600+ identical checks in 250 ms doing exactly this [14]. Design the list operation before the first list screen ships, or cap it the way OpenFGA did.

Figure 4 · A decision tree that ends in actions, not products

The first question is not scale; it is how many services enforce rules over the same objects, because rule drift across copies is the documented failure class. Synthesized from Figma, Gojek and the OpenFGA RFC.

Diagram source

Decision	Chosen	Rejected	Because	Evidence
Check placement	Central service (5 orgs) or in-app DSL (Figma)	The other camp, respectively	Rule drift across services vs. data locality inside one	Figma 2024, Gojek 2026
Freshness	Zookies (Google, SpiceDB)	Postponed by Keto since 2021; absent from most self-builds	Correctness of revocation vs. write latency and client complexity	keto#517
Checker failure	Closed by default (Envoy); per-authorizer (K8s)	Global fail-closed, revised by Google after June 2025	Blast radius of the checker vs. sensitivity of the data	ext_authz, Register 2025
List filtering	Bounded ListObjects / BulkCheck	Generic multi-relation lists; Cedar's batched check	Graph-traversal cost; batching misplaces the optimisation	OpenFGA RFC 2022, cedar rfcs#14
Fan-out timing	Write-time (Airbnb); read-time + Leopard index (Google)	Naive read-time traversal	Read-heavy ratio makes writes the cheap place to pay	Airbnb 2021

What broke in production

Two failure classes dominate the public record: the check that was never made, and the checker whose failure was the outage. The third class everyone designs against has no published incident at all.

Postmortem

Optus: the check was removed by accident, on a domain nobody remembered

AssumptionThe API had access controls, so the API was controlled, on every domain where it was exposed.

What happenedPer the ACMA's Federal Court filing, a September 2018 coding error left the API's access controls ineffective on both its domains. The error was noticed in August 2021 and fixed on the main domain only; a dormant, internet-facing target domain kept the broken control until attackers used it over three days in September 2022, reaching records of 9.5 million people.

Blast radius9.5M current and former customers; a regulator lawsuit; four years of latent exposure.

FixLitigation is the published record; the structural lesson is inventory, not patching: the fixed and unfixed copies of the same control diverged.

Design ruleAn access control you cannot enumerate is one you cannot claim to have. Every enforcement point must appear in an inventory that decommissioning checks against; a dormant domain is an enforcement point.

SourceACMA statement, 2024-05-22; iTnews coverage of the filing

Postmortem

First American: the missing check was found internally, then not fixed for five months

AssumptionA document link implies an authorised requester; and separately, that a pen-test finding marked urgent would be routed to someone who could act.

What happenedEaglePro served title documents by direct object reference with no per-object check. Documents with nonpublic personal data were reachable from October 2014 to May 2019, roughly 885 million of them. The company's own December 2018 penetration test found the flaw and said to address it "as soon as possible"; it was fixed only after a journalist published in May 2019.

Blast radius885M documents over 4.5 years; the first NYDFS cybersecurity enforcement action; a $1M NYDFS penalty (2023) and an earlier SEC settlement near $500K (2021).

FixThe consent order requires access controls and identity management as governed processes, with classification and risk assessment feeding them.

Design ruleObject-level checks are not a code review item; they are a control with an owner. If a finding says "missing authorization check", it is a sev-class incident that has not been exploited yet, and it needs an incident's routing, not a backlog ticket's.

SourceNYDFS consent order, 2023-11-27; Krebs, 2021

Postmortem

Facebook View As: three correct features composed into a token for the wrong user

AssumptionEach feature's authorization was reviewed, therefore their composition was authorized.

What happenedPer Facebook's 2018 disclosures: the View As preview wrongly showed a video uploader; that uploader (shipped July 2017) wrongly minted an access token with the mobile app's permissions; and inside View As, the token was minted for the user being impersonated, not the viewer. Attackers chained the three from at least September 14, 2018 until detection on September 25.

Blast radiusTokens for about 50M accounts reset, plus 40M precautionary; the October update measured 30M tokens actually stolen.

FixView As switched off pending review; token minting paths audited; the feature interaction reviewed as its own surface.

Design ruleCredentials are authorization state. Any code path that mints a token is an authorization decision and needs the same review gate as a permission check; "who is this token for" is the question the third bug failed.

SourceSecurity Update, 2018-09-28; October update

Postmortem

Google Cloud: the check on every API call crashed in every region at once

AssumptionService Control, the binary doing authorization and quota checks on Google Cloud API calls, was protected by the usual rollout discipline.

What happenedA quota-policy code path added on May 29, 2025 shipped, per the incident report as quoted by The Register, without appropriate error handling and without a feature flag. On June 12 a policy row with unintended blank fields entered the regional Spanner tables and replicated globally within seconds; the untested path dereferenced null and Service Control crash-looped in every region. Checks failed closed, so the failure surfaced as global API errors across dozens of services.

Blast radius10:51 to 18:18 US/Pacific, June 12, 2025; 50+ Google Cloud and Workspace services; downstream outages at customers worldwide.

FixPublished remediations: feature-flag protection enforced for critical binaries, audit of systems consuming globally replicated data, and modularising Service Control so a failing check fails open and requests still serve.

Design ruleCentralising the check creates a component whose failure mode is "everything." Its policy data is production input and needs validation at ingestion; and the fail-open/fail-closed choice must be made per check class before the incident, because Google made it after.

SourceGCP incident report; The Register, 2025-06-16

Case study

AuthZed: the client turned every list page into a check storm

AssumptionThe checker's capacity budget was set by request volume, not by the shape of individual client calls.

What happenedAuthZed's case file describes a customer intermittently issuing up to 800 CheckPermission calls within milliseconds, including one user producing over 600 requests in 250 ms for the same permission, ACL-filtering a list one object at a time. p95 check latency reached a full second and Spanner capacity saturated.

Blast radiusRecurring latency spikes over weeks; degraded client product; emergency capacity spend as a stopgap.

FixThe client moved from fan-out point checks to BulkCheck; observability and optimisations from the investigation landed in SpiceDB v1.28.

Design ruleA check API without a bulk and list form will be used as one anyway, N times per page. Ship the bounded list operation with the point check, and alert on identical-check fan-out per caller.

SourceZed File #Z-4902 (vendor account, undated; retrieved 2026-09)

Figure 5 · June 12, 2025: one blank field reaches every region's checker

The propagation is the design: policy metadata replicated globally in seconds, so the crash was global in seconds. Sequence per the incident report as corroborated by The Register.

Diagram source

The class with no incident on record

Nothing in this corpus, and no public postmortem this hunt could find, attributes a breach to the third failure class: a stale cached authorization decision, the new-enemy window that zookies exist to close. Read that either way. Staleness exploits may be rare enough not to matter for most products, or they may be invisible, because a leak through a seconds-wide replica window produces no error, no crash and no log line that looks wrong. Both readings argue for the same modest step: know your staleness window's width, write it down, and make revocation of the highest-tier resources bypass the cache. Treat anything stronger as unproven either way; this is the open edge of the public record.

Numbers you can plan against

The latency envelope is remarkably consistent across a decade of accounts: a production check costs single-digit milliseconds at p95, and everyone pays for it with a cache.

Metric	Value	At	Context	As of	Source
Check latency, p95	<10 ms	Google	Zanzibar, measured over 3 years in production	2019	paper
Check volume	millions/s	Google	Trillions of ACL entries, hundreds of client services	2019	paper
Availability	>99.999%	Google	Zanzibar, 3-year window	2019	paper
Check latency, p99	12 ms	Airbnb	Himeji, one year after launch	2021-03	blog
Throughput growth	0 → 850k entities/s	Airbnb	March 2020 to March 2021, at 99.999% availability	2021-03	blog
Cache hit target	~98%	Airbnb	Sharded cache, one instance per AZ per shard	2021	blog
Check latency	<10 ms	Carta	RelationTuple graph service	2021-09	blog
Check latency, p95 (claimed)	5 ms	AuthZed	Vendor claim, "millions of queries/s, billions of relationships"	2026-09	README
Degraded p95 under check storm	~1 s	AuthZed customer	600+ identical checks in 250 ms from one user	retrieved 2026-09	case file
Exposure from one missing check	885M documents	First American	October 2014 to May 2019	2023-11	consent order
Records reached via dormant endpoint	9.5M	Optus	Attack window 17-20 September 2022; error latent from 2018	2024-05	ACMA
Checker-outage duration	7 h 27 m	Google Cloud	Service Control crash-loop, all regions, fail-closed	2025-06	report
Tokens stolen via minting bug	30M	Facebook	50M reset plus 40M precautionary; window July 2017 to Sept 2018	2018-10	update

Read these carefully

Measured, from primary accounts: the Google, Airbnb, Carta and incident figures. Claimed, without independent measurement: SpiceDB's 5 ms p95 (vendor README) and everything in the AuthZed case file, which is a vendor telling a flattering war story, honestly labelled here as such. Derived: the 7 h 27 m duration is arithmetic on the report's own 10:51 and 18:18 timestamps. Unknown, because nobody publishes it: what a check costs in dollars per billion at any of these organisations, and any measured width of a staleness window in a system without consistency tokens. Cost figures for this subsystem are the biggest hole in the public record; plan with your own load tests, not with this table.

The evidence wall

Every source behind this page, graded. The ledger shipped beside this file records one claim per row with its supporting passage. This session ran behind a restricted network; GitHub sources were fetched directly and the rest were read through search-engine retrieval, noted in the ledger.

Postmortem Google Cloud2025-06

Incident report: multiple GCP products experiencing service issues

The authorization and quota gatekeeper for every GCP API call crashed globally on a policy row with blank fields, through a code path that shipped without error handling or a feature flag.

Carry forwardThe checker's policy data is production input; validate it at ingestion, and decide fail-open per check class before the incident.

status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW

Postmortem ACMA / Optus2024-05

ACMA statement and Federal Court filing on the 2022 Optus breach

A 2018 coding error disabled the API's access controls; the 2021 fix covered the main domain and missed a dormant one, which carried the breach in 2022.

Carry forwardEnforcement points need an inventory; a control fixed in one copy and not another is a control you no longer have.

acma.gov.au/acma-statement-2022-optus-data-breach

Postmortem NYDFS / First American2023-11

Consent order: First American Title Insurance

885M documents reachable by direct link for 4.5 years; the company's own pen test found the missing check five months before a journalist forced the fix.

Carry forwardA "missing authorization check" finding is an unexploited incident and needs incident routing, not a backlog ticket.

dfs.ny.gov/…/ea20231127_first_american

Postmortem Meta2018-09 / 2018-10

Security Update, and the October follow-up

Three individually reviewed features composed into a path that minted access tokens for the wrong user; 30M tokens confirmed stolen of ~90M reset.

Carry forwardToken minting is an authorization decision; review feature compositions that touch it as their own attack surface.

about.fb.com/news/2018/09/security-update

Postmortem The Register2025-06-16

Google Cloud caused outage by ignoring its usual code quality protections

Secondary account quoting the incident report's key admissions: no error handling, no feature flag, and remediation that includes making the check fail open.

Carry forwardEven the check's author revised fail-closed after seeing its global blast radius; your default deserves the same scrutiny.

theregister.com/2025/06/16/google_cloud_outage_incident_report

Source Ory Keto2021-04 → open

Issue #517: Provide consistency guarantees using snapshot tokens

The zookie mechanism, proposed for Keto in April 2021 with candidate designs, remains an open feature request five and a half years later.

Carry forwardThe paper's consistency machinery is the part reimplementations postpone; if you adopt a clone, ask specifically what happened to the zookies.

github.com/ory/keto/issues/517

Source Cedar (AWS)2023-08

rfcs#14: batch is_authorized, closed unmerged, labelled rejected

The rejection argument: a batch API hard-codes which request parts vary, and the real cost sat in entity loading, addressed instead by preprocessing and opaque handles.

Carry forwardBefore adding a batch endpoint, find where the per-call cost actually is; batching the wrong layer freezes your API for nothing.

github.com/cedar-policy/rfcs/pull/14

Source Envoychecked 2026-09

ext_authz filter: failure_mode_allow

The most widely deployed enforcement point rejects with 403 when the authorization service errors or returns 5xx, unless explicitly configured to allow.

Carry forwardKnow your proxy's default before the checker's first bad day; fail-closed at the edge means the checker's availability is your product's.

github.com/envoyproxy/envoy/…/ext_authz.proto

Source AuthZedchecked 2026-09

SpiceDB repository

The most direct open Zanzibar descendant: schema-defined relations, per-request consistency, ZedTokens. README claims 5 ms p95 at millions of queries per second.

Carry forwardPer-request consistency is the pragmatic middle: pay for freshness only on the checks where staleness is an exploit.

github.com/authzed/spicedb

ADR Kubernetes2023-06

KEP-3221: Structured authorization configuration

The apiserver's authorization chain became an ordered, file-configured list where each webhook declares its own failure policy: Deny, or NoOpinion and continue.

Carry forwardFail-open versus fail-closed is per-authorizer configuration, not doctrine; the chain shape lets different sensitivities coexist.

github.com/kubernetes/enhancements/…/3221…/README.md

ADR OpenFGA2022-07

RFC: ListObjects API

The design record for permission-filtered lists: bounded results, a deadline, no pagination because paginated graph traversal defeats parallel evaluation, and an explicitly rejected generic variant on query-cost grounds.

Carry forwardList filtering is a different operation from a point check, with its own cost model; price it separately in your API and your capacity plan.

github.com/openfga/rfcs/blob/main/20220714-listObjects-api.md

Case study AuthZedretrieved 2026-09

Zed File #Z-4902: Spanner Spikes

A vendor incident narrative: client-side ACL filtering emitted hundreds of identical point checks per page, p95 hit a second, and the durable fix was BulkCheck plus SpiceDB v1.28 optimisations. Vendor-told, and reads like it.

Carry forwardAlert on identical-check fan-out per caller; it is the signature of a list screen using a point-check API.

authzed.com/blog/zed-file-z-4902-spanner-spikes

Blog Figma2024-03

How we built a custom permissions DSL at Figma

The strongest published argument for the in-app camp: OPA, Zanzibar-style services and Oso all evaluated and declined; policies reference product objects while the engine owns loading, replicas and caching.

Carry forwardSeparating rules from data loading is the load-bearing idea; it is available inside your app without adopting anyone's service.

figma.com/blog/how-we-rolled-out-our-own-permissions-dsl-at-figma

Blog Airbnb2021-05

Himeji: a scalable centralized system for authorization

The most numerically transparent adoption account: tuple model, write-time fan-out, per-AZ sharded cache with a ~98% hit target, 850k entities/s and p99 of 12 ms one year in.

Carry forwardIn a read-heavy product, pay the fan-out on writes; the read path then degrades like a cache, which you know how to operate.

medium.com/airbnb-engineering/himeji…

Blog Carta2021-06 / 2021-09

AuthZ: Carta's permissions system; and: user authorization in under 10 ms

Two posts: the starting condition (five conflicting legacy permission systems) and the destination (a RelationTuple graph service answering in under 10 ms).

Carry forwardThe migration driver is usually consolidation of contradictory systems, not greenfield elegance; audit how many permission systems you already run.

medium.com/building-carta/user-authorization-in-less-than-10-milliseconds…

Blog Slack2021-05

Role management at Slack

The RBAC end of the spectrum, done carefully: permissions as actions, roles as sets, delegated at org and workspace scope for Enterprise Grid.

Carry forwardRole systems and relationship systems answer the same check; choose by how much your permissions follow containment structure.

slack.engineering/role-management-at-slack

Blog Gojek2026-06

We built a Zanzibar-style IAM system from scratch

A 2026 adoption account on SpiceDB, Go and PostgreSQL, written from the far side of the in-app approach's decay into unmaintainable conditionals.

Carry forwardThe in-app camp without a rules/data split trends toward the 400-line check function; the split is mandatory in either camp.

medium.com/gojekengineering/we-built-a-zanzibar-style-iam-system…

Blog Oso2021-09

Why authorization is hard

The problem taxonomy from the library camp's vendor: enforcement placement, the "which actions can this user take" inversion, and decisions needing data from more than one service.

Carry forwardThe yes/no check is the easy 20%; list filtering and cross-service data are where the architecture is actually decided.

osohq.com/post/why-authorization-is-hard

Blog AuthZedretrieved 2026-09

New enemies: enforcing causal ordering in permissions checking

The clearest worked example of the new-enemy problem outside the paper, and candid that closing it costs consistency machinery most deployments skip.

Carry forwardIf you run eventually consistent, the staleness window is a security parameter; measure it and put it in the threat model.

authzed.com/blog/new-enemies

Paper Google / USENIX ATC2019-07

Zanzibar: Google's consistent, global authorization system

The reference document for the field: tuples, zookies, Leopard, and the measured envelope (trillions of ACLs, millions of checks/s, p95 under 10 ms, five nines over three years).

Carry forwardThe paper's core is not the graph model; it is that consistency, latency and availability were engineered as one budget.

usenix.org/system/files/atc19-pang.pdf

Paper AWS2024-04

Cedar: a new language for expressive, fast, safe, and analyzable authorization

The policy-language camp's strongest artifact: semantics modelled in Lean with proved properties, implemented in Rust, benchmarked by its authors against OpenFGA and Rego.

Carry forwardIf auditors must analyse policies statically, language design beats graph reach; that is the axis Cedar optimises that tuples do not.

arxiv.org/abs/2403.04651

Talk Netflix / QCon Plus2021-11

Authorization at Netflix scale (Travis Nelson)

Centralizing the check in the critical path of a multi-million-RPS service, made survivable by a lightweight gRPC client with custom caching filters and isolated failure domains. Timestamped claims could not be verified this session; cited to the talk page.

Carry forwardAt extreme read volume, the checker's client library is the real system; the service behind it is a backing store for the client's cache.

infoq.com/presentations/authorization-scalability

Talk USENIX ATC '192019-07

Zanzibar conference presentation (Ruoming Pang et al.)

The recorded presentation of [1]; the page carries the abstract's scale figures. The video host was unreachable from this research session, so no timestamped claims are cited from the recording itself.

Carry forwardWatch alongside the paper for the operational Q&A the text omits.

usenix.org/conference/atc19/presentation/pang

Vendor CISA / ACSC / NSA2023-07

AA23-208A: Preventing web application access control abuse

Three national agencies issuing design guidance because one bug class, the absent object-level check, kept producing national-scale breaches.

Carry forward"Checks on every request that touches sensitive data" is now written government guidance; secure-by-default framework behaviour is the recommended mechanism.

cisa.gov/news-events/cybersecurity-advisories/aa23-208a

Vendor OWASP2023

API1:2023 Broken object level authorization

Top of the API security list for two consecutive editions; the canonical statement that every endpoint receiving an object ID must check the caller against that object.

Carry forwardBOLA holding the #1 spot across editions is the empirical ranking of failure classes; weight your review effort accordingly.

owasp.org/API-Security/editions/2023/en/0xa1…

Blog Krebs on Security2021-06

First American Financial pays farcical $500K fine

The reporter who disclosed the EaglePro exposure, on the SEC settlement that preceded the NYDFS order; useful as the independent narrative of the discovery.

Carry forwardThe external-researcher path found in hours what internal routing sat on for months; make the report-to-fix path shorter than the journalist's.

krebsonsecurity.com/2021/06/first-american-financial…

Build a miniature, then productionise it

Six rungs from a toy graph to the operational surface. The crossing from toy to real happens at rung three, where staleness becomes observable.

A tuple store and a check

Implement object#relation@user tuples in any relational table plus a recursive check: direct tuple, or membership in a group that has the relation. Model one real feature of a product you know (folder sharing, org roles).

Done when: checks answer correctly for nested groups three levels deep. Teaches: why facts and rules separate, and where recursion cost lives.

An oracle and property tests

Write the same rules a second time as naive per-object code, then property-test the tuple engine against the oracle on random object graphs. This is the parity harness any real migration runs, at toy scale.

Done when: 10,000 random cases agree, and one seeded rule bug is caught. Teaches: how permission migrations are verified without betting the product.

A cache with a measured staleness window

Put a TTL cache in front of the check, then script the new-enemy attack from figure 3: revoke, immediately write, read as the revoked user. Log the window during which the stale allow serves.

Done when: you can state the window in milliseconds for three TTL settings. Teaches: that the staleness window is a measurable security parameter, not a vibe.

A consistency token

Mint a monotonic token on every ACL write; store it with content writes; make checks carry the token and bypass any cache entry older than it. Re-run the rung-three attack.

Done when: the attack window is zero and you have measured the added check latency. Teaches: what zookies cost, and why reimplementations postpone them.

An enforcement point with a failure policy

Front a demo service with Envoy's ext_authz pointed at your checker. Kill the checker under load with failure_mode_allow false, then true, and watch what the client sees each way.

Done when: you can show 403s in one mode and silent allows in the other, with graphs. Teaches: the fail-open/fail-closed decision as an observable, per-route choice.

The list endpoint, then the storm

Build a page listing 1,000 objects filtered by permission, first as N point checks, then as a bounded list operation with a result cap and deadline. Load-test both against the same checker.

Done when: the point-check version measurably degrades checker p95 and the list version does not. Teaches: why AuthZed's incident happened and why OpenFGA bounded ListObjects.

Keep hunting

The queries that found this material, grouped by the layer they surface. The vocabulary is the tool: zookie, new enemy, BOLA and failure_mode_allow each unlock a literature the generic terms never reach.

Production experience

authorization service "we built" engineering blog permissions check latency
Zanzibar "we built" OR "we migrated" IAM lessons learned
"authorization at scale" p99 OR p95 latency cache "we"
permissions DSL "we rolled out" OR "we replaced" migration parity

The failure record

IDOR "broken object level authorization" incident postmortem breach report
ACMA Optus "coding error" access controls federal court
NYDFS consent order access controls "penetration test" vulnerability
incident report authorization OR quota "fail open" OR "fail closed" global outage

The design record

zookie OR ZedToken consistency "new enemy" site:github.com issues
repo:cedar-policy/rfcs is:pr is:closed is:unmerged
"ListObjects" authorization RFC graph traversal "query cost"
failure_mode_allow ext_authz default deny 403

Papers to production

Zanzibar paper "Leopard" index zookie "content-change check"
Cedar authorization language Lean verified arXiv
"inspired by Zanzibar" implementation postgres OR spanner OR cockroach
OWASP API1 2023 broken object level authorization endpoint checks

References

Pang, Caceres, Burrows et al., Zanzibar: Google's Consistent, Global Authorization System USENIX ATC '19, July 2019. Checked 2026-09-13.
USENIX ATC '19 presentation page for Zanzibar (talk video) USENIX, July 2019. Checked 2026-09-13.
Cutler et al., Cedar: A New Language for Expressive, Fast, Safe, and Analyzable Authorization arXiv 2403.04651 / OOPSLA, April 2024. Checked 2026-09-13.
Jorge Silva, How We Built a Custom Permissions DSL at Figma Figma blog, 2024-03-13. Checked 2026-09-13.
Alan Yao, Himeji: A Scalable Centralized System for Authorization at Airbnb Airbnb Tech Blog, May 2021. Checked 2026-09-13.
AuthZ: Carta's highly scalable permissions system Building Carta, June 2021. Checked 2026-09-13.
Aaron Tainter, User authorization in less than 10 milliseconds Building Carta, 2021-09-03. Checked 2026-09-13.
Role Management at Slack Slack Engineering, May 2021. Checked 2026-09-13.
Armaan Jain, We Built a Zanzibar-Style IAM System From Scratch Gojek Product + Tech, 2026-06-04. Checked 2026-09-13.
Travis Nelson, Authorization at Netflix Scale QCon Plus November 2021, via InfoQ. Checked 2026-09-13.
Sam Scott, Why Authorization is Hard Oso, 2021-09-15. Checked 2026-09-13.
AuthZed, Enforcing Causal Ordering in Distributed Systems: The Importance of Permissions Checking AuthZed blog, undated in retrieval. Checked 2026-09-13.
SpiceDB repository AuthZed / GitHub. Checked 2026-09-13.
AuthZed, Zed File #Z-4902: Spanner Spikes AuthZed blog, undated. Checked 2026-09-13.
ory/keto issue #517: Provide Consistency Guarantees using Snapshot Tokens GitHub, opened 2021-04-01; open as of 2026-09-13.
cedar-policy/rfcs pull #14: create a batch version of is_authorized (rejected) GitHub, closed unmerged 2023-08-22. Checked 2026-09-13.
Envoy ext_authz filter proto: failure_mode_allow, status_on_error GitHub, main branch. Checked 2026-09-13.
KEP-3221: Structured Authorization Configuration Kubernetes enhancements, approved 2023-06-15; alpha in v1.29. Checked 2026-09-13.
OpenFGA RFC: ListObjects API (jon-whit) GitHub, 2022-07-14. Checked 2026-09-13.
Google Cloud incident report: Multiple GCP products experiencing Service issues Google Cloud Service Health, June 2025. Checked 2026-09-13 (host unreachable from the research network; content corroborated via [21]).
The Register: Google Cloud caused outage by ignoring its usual code quality protections 2025-06-16. Checked 2026-09-13.
ACMA statement on the 2022 Optus data breach ACMA, 2024-05-22. Checked 2026-09-13.
iTnews: Optus breach allegedly enabled by access control coding error June 2024. Checked 2026-09-13.
NYDFS consent order: First American Title Insurance Company New York Department of Financial Services, 2023-11-27. Checked 2026-09-13.
Krebs on Security: First American Financial Pays Farcical $500K Fine June 2021. Checked 2026-09-13.
Facebook: Security Update 2018-09-28. Checked 2026-09-13.
Facebook: An Update on the Security Issue 2018-10-12. Checked 2026-09-13.
CISA/ACSC/NSA advisory AA23-208A: Preventing Web Application Access Control Abuse 2023-07-27. Checked 2026-09-13.
OWASP API Security Top 10 2023: API1 Broken Object Level Authorization OWASP, 2023. Checked 2026-09-13.