Search the practice set
126 questions, 454 terms and 400 topics in 20 areas.
60 results for “AI Observability”
AI Gateway
A shared proxy in front of model providers that centralises routing, keys, quotas, caching, logging and safety policy.
Observability
The property of being able to answer new questions about a system's internal state from its external outputs, without shipping new code.
Alert Fatigue
The desensitisation that follows from alerts that are frequent, non-actionable, or not tied to user impact — after which real alerts are missed too.
Application Performance Monitoring
Instrumentation inside the application that attributes latency and errors to specific code paths, queries and dependencies.
Cardinality
The number of distinct time series produced by a metric, which is the product of the distinct values of all its labels — and the main driver of monitoring cost.
Context Window
The maximum number of tokens a model can attend to in one request, holding the system prompt, history, retrieved context, tools and the answer.
Correlation ID
A single identifier attached to one logical operation and included in every log line it produces, anywhere in the system.
Distributed Tracing
Following one logical request across every service it touches by propagating a shared trace identifier and recording timed spans.
Embedding
A dense numeric vector representing a piece of content, positioned so that semantically similar content sits nearby.
Golden Signals
The four measurements that cover most of what matters for a request-driven service: latency, traffic, errors and saturation.
Guardrail
A deterministic check applied to a model's input or output, enforcing rules that cannot be left to the model itself.
Health Check
An endpoint the platform polls to decide whether an instance should be restarted or should receive traffic — two different questions needing two different checks.
Human in the Loop
Requiring human review or approval at a defined point in an automated flow, chosen by the reversibility and cost of the action.
LLM Evaluation
A repeatable measurement of whether an AI system's outputs are good enough, on cases that reflect the actual task.
Model Context Protocol
An open protocol that standardises how AI applications connect to external tools, data sources and prompts.
Model Router
Directing each request to a model chosen by the task's difficulty, cost and latency budget, rather than sending everything to the largest model available.
Pipeline Orchestration
Coordinating the execution of data tasks by dependency rather than by clock, with retries, backfill and observability built in.
Prompt Injection
An attack in which text from an untrusted source is interpreted by the model as instructions rather than as data.
Prompt Registry
A versioned store of production prompts with their model bindings, parameters and evaluation results, so a prompt change is a reviewable, traceable, reversible deployment.
Prompt Versioning
Treating prompts as versioned, reviewed, tested artefacts rather than as strings edited in place.
You are asked to give an internal AI agent access to the customer database, the ticketing system and outbound email so it can resolve support tickets. What is your response?
What the interviewer is testing Whether you recognise a specific and well documented security pattern, and whether you can propose a workable design instead of
Your observability bill is now 40% of your compute bill. Leadership wants it cut in half without going blind. What do you cut?
What the interviewer is testing Whether you understand what each telemetry type is for , so you can cut the redundant parts rather than cutting uniformly — whic
A client wants an assistant that answers questions from 50,000 internal documents which change weekly. RAG or fine-tuning? What actually determines the quality?
What the interviewer is testing Whether you understand what each technique actually does, and whether you know that RAG quality is a retrieval problem. Why RAG
An LLM feature that worked last week now gives worse answers. Nothing was deployed. How do you find out what changed, and what should have been in place?
What the interviewer is testing Whether you treat an AI feature as a system with configuration and dependencies, or as a black box that mysteriously drifts. Wha
AI Observability
Logging prompts, versions, retrieved context and cost per request.
AI Cost Management
Token accounting, routing, caching and the context-window budget.
AI Gateways
Centralised routing, keys, quotas, caching, logging and safety policy.
AI-Era Architecture
General material on architecting systems that include models.
Observability
General material on understanding a system from its outputs.
Observability Cost
Telemetry bills, cardinality control and retention tiering.
Agent Architectures
Loops, planning, memory and the boundaries an agent must not cross.
Alert Fatigue
How noise makes the real page invisible, and the structural fix.
Alerting
Symptom-based, actionable, user-impacting — and linked to a runbook.
Application Performance Monitoring
Attributing latency to code paths, queries and dependencies.
Business Metrics
Orders per minute alongside error rate, because healthy is not enough.
Cardinality
The label that multiplies series count and the bill with it.
Chunking & Retrieval
Structure-aware splitting, hybrid search and why chunking dominates quality.
Correlation IDs
One identifier propagated through every hop and every log line.
Dashboards
Answering 'is it us' in under a minute, for someone who was asleep.
Debugging Distributed Systems
Localising a regression when every service reports healthy.
Distributed Tracing
Reconstructing one request's path across every service it touched.
Embeddings
Dense representations, model coupling and the migration they imply.
Guardrails
Deterministic checks on input and output that fail closed.
Health Checks
Liveness versus readiness, and the check that causes the outage.
Human in the Loop
Gating by reversibility and blast radius, and avoiding approval fatigue.
LLM Application Architecture
The shape of a production system with a model in the request path.
LLM Evaluation
Held-out sets, rubric judging, CI gates and production sampling.
Log Management
Aggregation, retention tiering, search and the cost of keeping everything.
Logging
What to log, at what level, and what must never appear in a log.
ML Platform
Feature stores, training pipelines, registries and deployment.
Metrics
Counters, gauges and histograms, and percentiles rather than means.
Model Selection
Capability, latency, cost and the evaluation that decides between them.
Multi-Agent Systems
Coordination, hand-off and whether more agents actually help.
OpenTelemetry
Instrumenting once against an open standard rather than a vendor agent.
Profiling
Continuous CPU and memory attribution in production.
Prompt & Version Management
Prompts as reviewed, versioned, evaluated production configuration.
Prompt Injection Defence
Breaking the private-data, untrusted-input, outbound-channel combination.
RAG Architecture
Retrieval, grounding, citation and the permissions RAG can enforce.