LLM Rate Limiting & Traffic Management Service  ·  View 19 of 24  ·  Operations

Delivery Pipeline

Four things ship on independent cadences — service code, Lua scripts, policy and infrastructure — and each gets its own gate.

Editable source SVG draw.io All views
Commit
Commit
Build
Build
Verify
Verify
Canary
Canary
Production
Production
limiterd service
limiterd service
PR + review
PR + review
Go build, SBOM, cosign
Go build, SBOM, cosign
Race + fuzz + 10k rps soak
Race + fuzz + 10k rps soak
Argo Rollouts 5%
SLO analysis gate
Argo Rollouts 5%...
Full rollout
auto-rollback on p99
Full rollout...
Lua limit scripts
Lua limit scripts
Script in repo
Script in repo
SHA pinned to release
SHA pinned to release
Golden-file limit tests
Golden-file limit tests
Shadow dual-run compare
Shadow dual-run compare
SCRIPT LOAD on all shards
SCRIPT LOAD on all shards
Policies & config
Policies & config
Policy-as-code PR
Policy-as-code PR
Schema + guardrail lint
Schema + guardrail lint
Dry-run on traffic replay
Dry-run on traffic replay
One canary tenant
One canary tenant
Published to policy.v1
Published to policy.v1
Infrastructure
Infrastructure
OpenTofu PR
OpenTofu PR
Plan + cost diff
Plan + cost diff
OPA policy checks
OPA policy checks
Apply to staging
Apply to staging
Apply via Argo CD
Apply via Argo CD
Delivery Pipeline — code, scripts, policy and infrastructure
Delivery Pipeline — code, scripts, policy and infrastructure
Lua scripts are versioned and rolled out separately from the binary, because a change to the limit algorithm is a behaviour change even when no Go code moved.
Lua scripts are versioned and rolled out separately from the binary, because a change to the limit algorithm is a behaviour change even when no Go code moved.
v 1.0 · owner Data & AI Global Practice
v 1.0 · owner Data & AI Global Practice
Text is not SVG - cannot display

Decisions

  • Lua limit scripts are versioned and released separately from the Go binary. A change to the limit algorithm is a behaviour change even when no Go code moved, and treating it as a config file is how a quota bug reaches production unreviewed.
  • Script changes go through a shadow dual-run: both versions execute against live traffic, only the current one is authoritative, and the verdicts are compared. A discrepancy blocks promotion.
  • Canary analysis gates on the SLO directly — decision p99 and reject-rate delta — not on pod health. A pod can be perfectly healthy while rejecting twice as much traffic as it should.

Gates

  • Race detector and fuzz tests on the decision path; a 10k rps soak in staging before any canary.
  • Golden-file tests for the limit scripts: a fixed sequence of requests must produce an identical verdict sequence across versions.
  • Argo Rollouts at 5% for 15 minutes with automatic rollback if decision p99 exceeds 10 ms or the reject rate moves more than 10% against baseline.

Assumptions

  • Images are signed with cosign and an SBOM is produced per build; admission control rejects unsigned images in production.
  • OpenTofu is used rather than Terraform for licence reasons; the module structure is otherwise conventional.
  • Staging carries a synthetic tenant set derived from production shape, not production data.