Search the practice set
275 questions, 991 terms and 600 topics in 30 areas.
60 results for “Model Evaluation & Red-Teaming”
Adversarial Evaluation
Deliberately attempting to make a model behave badly, because a probabilistic system with no fixed expected output cannot be verified by conventional testing.
Inference Telemetry
Recording the full context of each model interaction — inputs, outputs, tokens, latency, model version and evaluation scores — so quality and cost can be investigated.
LLM-as-Judge
Using a model to score another model's output against a rubric, providing scalable evaluation where exact-match assertions do not apply.
Prompt Registry
A versioned store of production prompts with their model bindings, parameters and evaluation results, so a prompt change is a reviewable, traceable, reversible deployment.
Advisory Review Model
Running architecture review as a consulting service that improves designs rather than as an approval gate that permits them.
Architecture Cost Model
A calculation, made during design, of what an architecture will cost to run at expected and at peak volume.
Arrival Rate Model
Driving a load test by requests arriving per second regardless of how the system responds, rather than by a fixed number of virtual users.
C4 Model
A set of four nested diagram levels — context, container, component, code — that keeps each diagram at one consistent level of abstraction.
Capability Model
A structured map of what a business does, independent of how it is organised or which systems support it, used to align technology investment with function.
Cost Accountability Model
The arrangement determining whether teams merely see their costs or are financially charged for them, and the behaviour each produces.
Cost Model Sensitivity
Identifying which assumptions in a cost projection dominate the outcome, so effort goes into the estimates that actually matter.
Embedding Model Migration
The process of moving a corpus to a new embedding model, which requires re-embedding everything because vectors from different models are not comparable.
Execution Model Fit
Matching a workload's traffic shape, duration and state requirements to the execution model that suits it, rather than choosing one model for everything.
Flag Evaluation Latency
The delay between page load and the client knowing which variant to show, which produces a visible flicker unless the flag is resolved before render.
Governance Operating Model
The arrangement of decision rights, review points and automated controls through which architectural intent is maintained across an organisation.
LLM Evaluation
A repeatable measurement of whether an AI system's outputs are good enough, on cases that reflect the actual task.
Model Card
A structured record of what a model is for, how it was built and where it should not be used — written for the people who will deploy or be affected by it.
Model Context Protocol
An open protocol that standardises how AI applications connect to external tools, data sources and prompts.
Model Inventory
A complete register of models in use with their purpose, owner, risk tier and validation status — the artifact everything else in model governance depends on.
Model Router
Directing each request to a model chosen by the task's difficulty, cost and latency budget, rather than sending everything to the largest model available.
Model Routing
Directing each request to the cheapest model capable of handling it, rather than sending all traffic to the most capable one.
Model Translation Boundary
The layer at which an external system's model is converted into your own, preventing its concepts and accidents from spreading into your domain.
Operating Model
How an organisation arranges people, process, technology and governance to deliver its capabilities, which constrains architecture as strongly as any technical factor.
Platform Team Product Model
Running an internal platform team with product management disciplines — users, roadmap, support, metrics — rather than as an infrastructure function.
Process Model
A representation of how work actually flows through an organisation, distinguished from a capability by describing sequence, actors and handoffs.
RED Method
A minimal per-service dashboard: Rate, Errors, Duration — the request-centric view of whether users are being served.
Read Model
A data structure shaped for a specific query rather than for the domain, maintained separately from the write model.
Read Model Projection
A denormalised store built and maintained from write-side events specifically to serve one query shape efficiently.
SQL Transformation Model
A named, versioned, tested SELECT statement that declares its own dependencies, turning transformation logic into reviewable software.
Shared Responsibility Model
The division of duties between provider and customer, which shifts with the service model and is routinely misunderstood in the customer's disfavour.
Spotify's Squad Model and Its Retrospective
The widely-copied Spotify model of squads, tribes, chapters and guilds was a snapshot that did not work as documented even at Spotify — a caution about importing organisational design.
Technology Evaluation Criteria
A standing set of dimensions for assessing a candidate technology, ensuring that operability and longevity are weighed alongside capability.
Workload Model
A description of the traffic mix, arrival pattern and data distribution a load test reproduces, which determines whether the test's results mean anything.
An internal AI assistant gives confidently wrong answers. The team wants to upgrade to a better model. What do you check first?
Establish whether the model ever saw the right content Log the retrieved chunks alongside each answer, then take the wrong answers and check: was the correct so
A business unit wants an assistant answering questions from 200,000 internal documents. They ask whether to fine-tune a model or use retrieval. How do you decide?
Retrieval, for this requirement, and the reasoning is not about quality Four properties decide it: Freshness. Documents change. Retrieval reflects a change as s
A deployed model performed well in validation and its business metric has declined over four months. Nothing has been deployed. What do you investigate?
The model did not change; its world did Three distinct causes, and they need different responses: Data drift. The input distribution has moved — a new customer
A document collaboration product needs sharing with individuals, teams, and inherited folder permissions. Which authorization model?
The requirement is relationship shaped The questions this product must answer are: is this user a member of a team that has access to a folder that contains thi
A vendor SaaS product embeds a model that scores customers, and its output drives an automated decision in your process. Your model governance framework covers models you build. What do you do?
The obligation does not transfer with the outsourcing You are accountable for the decision. That the scoring is performed by a vendor changes who operates the m
Run a threat model on a new payment integration: our service calls a third-party payment provider and receives webhooks. Where are the interesting threats?
Draw the boundaries first Three trust boundaries, and nearly every interesting threat lives on one of them: 1. User → our service (untrusted input, authenticate
The business wants to deploy a model that ranks loan applications, with a credit officer making the final decision. What must the architecture provide, and what will you insist on before go-live?
Classify first, because it determines everything else This decides access to credit for individuals. Under any consequence based classification it is high risk:
A CDO proposes moving to a data mesh because the central data team is a bottleneck with a nine-month backlog. How do you assess the proposal?
Agree with the diagnosis, examine the prescription The bottleneck is real and it is structural rather than a matter of capacity. A central team receives data fr
A design review presents a new event-driven platform. What cost questions do you ask before approving it?
What the interviewer is testing Whether cost is part of your architecture review or an afterthought handled by finance later, and whether you know the specific
A regulator asks whether customer data is encrypted. The team says yes, disks are encrypted. Is that a sufficient answer?
What disk encryption actually protects against Someone obtaining the physical medium or a raw storage snapshot. In a cloud context that means a provider employe
Model Evaluation & Red-Teaming
Adversarial testing of a probabilistic system with no fixed expected output.
Model Selection
Capability, latency, cost and the evaluation that decides between them.
Assurance, Audit & Model Risk
General material on assurance, architectural governance and risk oversight.
C4 Model
Context, container, component and code as four separate diagrams.
LLM Evaluation
Held-out sets, rubric judging, CI gates and production sampling.
Model Documentation
Model cards, intended use, limitations, and the record a regulator will ask for.
Model Risk Management
Inventory, validation, monitoring and challenge for models that make consequential decisions.
Research & Evaluation
Assessing a technology quickly without adopting it by accident.
Three Lines Model
Ownership, oversight and independent assurance, and where architecture sits in it.