LLM Rate Limiting & Traffic Management Service  ·  View 18 of 24  ·  Operations

Policy Control Plane

Design question 5 on propagation, answered: from an administrator's edit to a live decision, and how the delay is measured rather than assumed.

Editable source SVG draw.io All views
Author
Author
Platform Admin
Platform Admin
Admin Console
React
Admin Console...
Policy-as-code Repo
YAML, reviewed
Policy-as-code Repo...
Validate
Validate
policy-api
Go · REST
policy-api...
Guardrail Check
child ≤ parent quota
Guardrail Check...
Dry-run on Replay
24 h of traffic
Dry-run on Replay...
Persist
Persist
Policy Store
PostgreSQL · versioned
Policy Store...
Transactional Outbox
same commit
Transactional Outbox...
Publish
Publish
Outbox Relay
at-least-once
Outbox Relay...
policy.v1
Kafka · log-compacted
policy.v1...
Propagate
Propagate
limiterd Policy Cache
p99 < 2 s
limiterd Policy Cache...
Gateway Route Cache
same topic
Gateway Route Cache...
Verify
Verify
policy_version Gauge
per pod
policy_version Gauge...
Staleness Alert
fires above 60 s
Staleness Alert...
min version lag
min version lag
Policy Control Plane — from admin edit to live decision
Policy Control Plane — from admin edit to live decision
Person or role
Person or role
Application we own
Application we own
Decision point
Decision point
Data store
Data store
Queue / topic
Queue / topic
Security / platform
Security / platform
Risk / gap
Risk / gap
synchronous
synchronous
Answer to design question 5 on propagation. A compacted topic means a pod that restarts replays only the current policy set, so cold start does not depend on PostgreSQL being reachable.
Answer to design question 5 on propagation. A compacted topic means a pod that restarts replays only the current policy set, so cold start does not depend on PostgreSQL being reachable.
v 1.0 · owner Data & AI Global Practice
v 1.0 · owner Data & AI Global Practice
Text is not SVG - cannot display

Decisions

  • A transactional outbox written in the same commit as the policy row. Publishing to Kafka from application code after the commit is the standard way to lose a policy change, and this design refuses it.
  • The topic is log-compacted, so a restarting pod replays only the current policy set rather than the whole history. Cold start reaches steady state in about 1.2 s and does not depend on PostgreSQL being reachable (FR8).
  • Every pod exports the policy version it is running as a gauge. Propagation delay is then a query — the spread between the published version and the fleet minimum — rather than a claim in a document (NFR9).

Guardrails before publish

  • A child scope's quota may not exceed its parent's; the API rejects the write rather than accepting a limit that can never bind.
  • A dry run replays the last 24 hours of decision metadata against the proposed policy and reports how many requests would have been rejected. Large policy cuts are caught before they reach a tenant.
  • Propagation target is p99 under 2 s; the staleness alert fires when any pod is more than 60 s behind.

Risks

  • A policy change is a behaviour change with no deployment. It goes through the same review as code (view 19) and is applied to one canary tenant before the fleet.
  • The outbox relay is at-least-once, so a pod can see the same policy version twice. Policy application is idempotent by version, so duplicates are inert.
  • A partitioned pod continues on its last-known-good policy indefinitely rather than failing. That is the correct choice for availability, and it is why the staleness alert exists.