LLM Rate Limiting & Traffic Management Service  ·  View 11 of 24  ·  Runtime

Hierarchical Limit Evaluation

Five scopes that must all pass, evaluated in a fixed order so the rejection reason is deterministic.

Editable source SVG draw.io All views
Resolve
Resolve
Request Context
from JWT claims only
Request Context...
Scope Chain
5 keys, ordered
Scope Chain...
Organisation
Organisation
Org RPM / TPM
500 rpm · 2M tpm
Org RPM / TPM...
Org Budget
monthly USD cap
Org Budget...
Team
Team
Team RPM / TPM
share of org
Team RPM / TPM...
Principal
Principal
User RPM
10 req/s
User RPM...
User Concurrency
max in flight
User Concurrency...
Model
Model
Model RPM / TPM
per model policy
Model RPM / TPM...
Model Concurrency
20 in flight
Model Concurrency...
Provider
Provider
Provider Global RPM
protects upstream
Provider Global RPM...
Provider Health
breaker state
Provider Health...
Outcome
Outcome
ALLOW + reserve
request_id issued
ALLOW + reserve...
REJECT 429
reason + retry_after
REJECT 429...
BUDGET_EXCEEDED
BUDGET_EXCEEDED
PROVIDER_SATURATED
PROVIDER_SATURATED
Hierarchical Limit Evaluation — every scope must pass
Hierarchical Limit Evaluation — every scope must pass
Application we own
Application we own
Decision point
Decision point
Risk / gap
Risk / gap
failure / alternate
failure / alternate
Scopes are evaluated in a fixed order so the reason code is deterministic. The first failing scope names the rejection; retry_after comes from that scope's window.
Scopes are evaluated in a fixed order so the reason code is deterministic. The first failing scope names the rejection; retry_after comes from that scope's window.
v 1.0 · owner Data & AI Global Practice
v 1.0 · owner Data & AI Global Practice
Text is not SVG - cannot display

Decisions

  • Scope order is fixed — organisation, team, principal, model, provider — so two identical requests always receive the same reason code. An order that varied would make client retry logic untestable.
  • The first failing scope names the rejection and supplies retry_after from its own window. A user-level limit gives a sub-second retry; a monthly budget gives a much longer one, and the client can tell them apart.
  • Guardrails at policy publish time enforce that a child scope's quota cannot exceed its parent's. Without that, a team limit larger than the org limit is silently meaningless.

Reason codes

  • RATE_LIMIT_EXCEEDED · TOKEN_QUOTA_EXCEEDED · CONCURRENCY_LIMIT_EXCEEDED · BUDGET_EXCEEDED · PROVIDER_SATURATED · PROVIDER_UNAVAILABLE (FR11).
  • Every 429 response carries reason, retry_after in seconds, and the limiting scope so a tenant can see whether it is their own limit or a shared one.
  • PROVIDER_UNAVAILABLE returns 503 rather than 429, because retrying against the same provider will not help.

Omissions

  • The mechanics of making these checks atomic are not on this diagram — view 12 carries them.
  • Priority handling sits between the provider check and the outcome and is drawn in view 15.
  • Concurrency at the org and team level exists but is left off to keep the cascade readable; it behaves exactly like the user and model cases shown.