A workspace product adds AI features over user content. Which architectural decisions dominate, and which are commonly deferred at cost?

The decisions that dominate

1. Permission enforcement at retrieval. The model must only see content the requesting user may see. This is the single most consequential decision, because a retrieval layer that filters after ranking, or that reads from an index built before a permission change, leaks content across users — and that is a security incident, not a quality issue.

The correct pattern is authorisation applied against current truth at retrieval time, with the index over-retrieving and then filtering. Grants may be eventually consistent; revocations may not.

2. Latency budget and where the model sits. Model calls are slow relative to everything else. The architecture must decide which interactions are synchronous, which stream, and which are asynchronous jobs with a result delivered later. Getting this wrong produces a product that feels broken rather than slow.

3. Grounding and retrieval quality. The dominant determinant of output quality is what content reaches the model, not the model. That makes retrieval — chunking, indexing, ranking, freshness — the main engineering surface.

4. Cost per interaction, attributed. Costs vary by orders of magnitude between interactions. Without attribution by user, workspace and feature, the product cannot be priced and unprofitable usage patterns are invisible.

What is commonly deferred at cost

Evaluation. Teams ship on impressions and discover regressions only through user complaints. Without a held-out evaluation set and a comparison harness, every prompt or model change is an unmeasured risk, and the team cannot tell improvement from noise.

Prompt and configuration versioning. Prompts change frequently and behave like code. Unversioned prompts mean an output cannot be reproduced or explained, and a regression cannot be bisected.

Output handling as untrusted input. Model output rendered into a page, executed as a query, or passed to a tool is a serious vulnerability class — and it is not addressed by anything in the model layer.

Caching strategy. Repeated retrieval and repeated generations for identical inputs are frequently a large fraction of cost, and retrofitting a cache after the interaction design has settled is much harder.

The framing

The model is a dependency with unusual properties — slow, expensive, non-deterministic and occasionally wrong — and the architecture is mostly about the system around it. Teams that treat the model as the architecture ship prototypes; teams that treat retrieval, permissions, evaluation and cost as the architecture ship products.