An AI inference platform handles authentication, quota accounting, prompt assembly, safety filtering, model invocation, token counting, billing emission and response streaming in one request handler. Latency is currently fine. Argue for or against separating these concerns, and identify which separations pay for themselves first.

The honest case for leaving it alone

It works, latency is fine, and every separation adds a hop, a failure mode and a deployment. On a latency-sensitive path, "one process does the whole thing" is a legitimate architecture, and teams that split reflexively usually regret it. Do not separate for tidiness.

The case for separating - and it is specific

These concerns differ along three axes that matter more than tidiness.

Change cadence. Safety filtering and prompt assembly change weekly or faster. Token counting and billing change rarely and must be correct. Coupling a fast-moving concern to a correctness-critical one makes every safety tweak a billing risk and every billing freeze a blocker on safety work.

Failure semantics. If billing emission fails, the request should still succeed and the event should be recovered later. If safety filtering fails, the request must not succeed. Those are opposite failure policies sharing one try/except.

Blast radius under load. Quota accounting is shared state and a contention point. Model invocation is a long, expensive, GPU-bound operation. Holding a quota lock or a database connection across a multi-second generation is how a platform discovers that its connection pool, not its GPUs, is the capacity limit.

Which separations pay first

1. Billing emission off the request path. Write the usage record transactionally with the request's own state and publish asynchronously through an outbox. This removes a failure mode that can either drop revenue or fail a paid request, and it is the cheapest change on the list.
2. Admission control ahead of the expensive work. Authentication, quota and rate limiting belong at the edge, before a GPU is reserved. Rejecting at the gateway costs microseconds; rejecting after model start costs seconds of the scarcest resource you own.
3. Safety filtering as an explicit stage with its own timeout budget, defined fail-closed behaviour and independent versioning — not necessarily a separate service, but a separate contract.

What to leave together

Prompt assembly, model invocation and streaming stay in one component. They share state, share a deadline, and separating them buys nothing but serialisation overhead on the hottest path.

The judgement

Separation of concerns is not "one thing per file". It is one failure policy and one change cadence per boundary. Draw the lines where those differ; leave them alone where they agree.