LLM Rate Limiting & Traffic Management Service  ·  View 06 of 24  ·  Structure

Integration Modes

Three ways to adopt the platform, and which of them can guarantee that reconciliation actually happens.

Editable source SVG draw.io All views
Request
Request
Authorize
Authorize
Execute
Execute
Observe actuals
Observe actuals
Reconcile
Reconcile
A · Inline proxy (default)
A · Inline proxy (default)
Envoy ext_authz
zero app change
Envoy ext_authz...
Check RPC
p99 < 10 ms
Check RPC...
Gateway calls provider
Gateway calls provider
Gateway parses usage block
Gateway parses usage block
Automatic commit
client cannot skip it
Automatic commit...
B · SDK / API
B · SDK / API
App calls /v1/authorize
App calls /v1/authorize
HTTP decision + request_id
HTTP decision + request_id
App calls provider directly
App calls provider directly
App reads provider usage
App reads provider usage
App calls /v1/commit
reaper covers crashes
App calls /v1/commit...
C · Queued bulk
C · Queued bulk
Job enqueued
priority BULK
Job enqueued...
Admission at dequeue
spare capacity only
Admission at dequeue...
Worker pool executes
Worker pool executes
Worker reports actuals
Worker reports actuals
Credit returned to pool
Credit returned to pool
Integration Modes — who closes the reconciliation loop
Integration Modes — who closes the reconciliation loop
Mode A is the default because the gateway sees the provider's own usage block, so reconciliation cannot be skipped by a misbehaving client.
Mode A is the default because the gateway sees the provider's own usage block, so reconciliation cannot be skipped by a misbehaving client.
v 1.0 · owner Data & AI Global Practice
v 1.0 · owner Data & AI Global Practice
Text is not SVG - cannot display

Why Mode A is the default

  • In Mode A the gateway sees the provider's own usage block, so the commit is issued by infrastructure the tenant does not control. A misbehaving or crashing client cannot skip it.
  • In Mode B the client owns the commit. The reaper bounds the damage — an uncommitted reservation expires after 120 s — but a tenant that never commits will systematically over-hold quota and is detected by a per-tenant commit-ratio alert.
  • Mode C exists because batch work should consume leftover capacity rather than compete for it. Admission is evaluated at dequeue time, not at submit time, so a queued job never holds a reservation while it waits.

Numbers

  • Reservation TTL 120 s, chosen as roughly 4× the p99 completion latency of the slowest supported model.
  • Commit-ratio alert fires below 0.98 over a 15-minute window for any tenant on Mode B.
  • Mode C admission runs at 1 Hz per worker pool and consumes only capacity unused by interactive tiers.

Assumptions

  • Streaming responses are supported in all three modes; the commit is issued when the stream terminates, not when it starts.
  • Clients on Mode B are expected to send a commit even on failure, with a status field. A failed call still consumed input tokens at most providers.