Human in the Loop
Gating by reversibility and blast radius, and avoiding approval fatigue.
4 to work through
-
intermediate
A platform uses AI to propose substitutions when items are unavailable. Where should the human sit in the loop, and what determines it?
2 min answer -
advanced
A pipeline combines automated processing with human review at scale. How should the boundary between them be designed?
2 min answer -
advanced Multiple choice
An AI system routes decisions to human review. The override rate is 0.3%. Is the oversight working?
2 min answer -
advanced
Design the moderation path for user-generated content at high volume, where both false positives and false negatives are costly.
2 min answer
3 terms in this topic
Automation Ratchet
The effect where improving automation makes the cases reaching humans systematically harder, so reviewer throughput falls and error rates rise even a…
practiceEscalation Threshold
The rule determining when a model's output is acted on automatically and when it is routed to a person.
patternHuman-in-the-Loop Design
Placing human review at the points where model error is consequential, designed so the review is genuinely effective rather than nominal.
Neighbouring topics
AI-Era Architecture
General material on architecting systems that include models.
LLM Application Architecture
The shape of a production system with a model in the request path.
RAG Architecture
Retrieval, grounding, citation and the permissions RAG can enforce.
Vector Databases
Approximate nearest-neighbour search, filtering and re-indexing.
Embeddings
Dense representations, model coupling and the migration they imply.
Chunking & Retrieval
Structure-aware splitting, hybrid search and why chunking dominates quality.
Reranking
Cross-encoders improving precision more than a bigger embedding model.
Model Selection
Capability, latency, cost and the evaluation that decides between them.
AI Gateways
Centralised routing, keys, quotas, caching, logging and safety policy.
Prompt & Version Management
Prompts as reviewed, versioned, evaluated production configuration.
Agent Architectures
Loops, planning, memory and the boundaries an agent must not cross.
Tool Calling
Typed tool interfaces, narrow parameters and per-tool authorisation.
Multi-Agent Systems
Coordination, hand-off and whether more agents actually help.
LLM Evaluation
Held-out sets, rubric judging, CI gates and production sampling.
AI Observability
Logging prompts, versions, retrieved context and cost per request.
Guardrails
Deterministic checks on input and output that fail closed.
Prompt Injection Defence
Breaking the private-data, untrusted-input, outbound-channel combination.
AI Cost Management
Token accounting, routing, caching and the context-window budget.
ML Platform
Feature stores, training pipelines, registries and deployment.