Search the practice set
275 questions, 991 terms and 600 topics in 30 areas.
60 results for “Flaky Test Management”
Test Quarantine
Moving an intermittently failing test out of the blocking suite into a tracked, owned backlog, so a red build keeps meaning something.
Test Data Provisioning
Getting each test the data it needs, in a state it can rely on, without copying production personal data into a weaker environment.
Prompt Regression Suite
A set of test cases with expected properties, run against a prompt on every change, to detect quality regressions before deployment.
Application Portfolio Management
Maintaining an inventory of every application with its owner, cost, business value and technical health, and using it to decide what to invest in, replace or retire.
Boundary Volatility Test
Evaluating a proposed service boundary by asking whether the things on either side change for different reasons and at different rates.
Consent Management
Capturing, storing, honouring and evidencing a data subject's permissions for specific processing purposes, including withdrawal.
Control Test Automation
Executing a control's test continuously against the whole population rather than sampling it annually, which changes both the detection latency and the strength of the evidence.
Differentiation Test
The question of whether a capability is a source of competitive advantage, used as the primary filter in build-versus-buy decisions.
End-to-End Test Cost Curve
How the cost of a broad end-to-end suite grows with its size while its marginal value falls, and where the two cross.
Focus Management
Deliberately controlling where keyboard focus sits after an interface change, which is what makes a dynamic application usable without a mouse.
Global Traffic Management
The layer that decides which region a given user reaches, using DNS, anycast or an edge network, and that performs regional failover.
Identity and Access Management
The system of record for principals, credentials and permissions, and the policy engine that decides what each principal may do.
Offset Management
How a consumer records its position in a stream, and the decision that determines whether processing is at-least-once or at-most-once.
Refactoring Under Test
Changing internal structure without changing behaviour, with tests as the mechanism that makes the claim verifiable.
Secrets Management
Storing, distributing, rotating and auditing credentials so that they never live in code, images or configuration files.
Soak Test
Running sustained realistic load for hours or days to expose defects that accumulate over time rather than appearing under peak load.
Stress Test
Driving load beyond expected capacity to observe how the system behaves at and past its breaking point.
Telemetry Cost Management
Controlling observability spend through sampling, retention tiering and cardinality limits without losing diagnostic capability.
Test Double Boundary
The line inside a test between what is real and what is substituted, which determines exactly what the test can and cannot prove.
Test Pyramid
A distribution of tests weighted towards many fast unit tests, fewer integration tests, and very few slow end-to-end tests.
Test Strategy Altitude
Deciding which risks are verified at which level, so that each layer tests something the layers below it structurally cannot.
Your test quarantine has grown to 140 tests over a year. What has gone wrong and how do you recover?
What went wrong Quarantine without a cap and without a deadline becomes a graveyard. Each individual decision was reasonable — move the flaky test aside, raise
A team's CI suite fails roughly one run in three for reasons unrelated to the change. Everyone reruns until green. How do you recover the situation?
Recognise what has actually been lost The suite is no longer a gate. Once the team's reflex on red is "rerun", that reflex is applied to genuine failures too, a
End-to-end tests fail intermittently and nobody owns them. QA says the developers broke them; developers say the tests are flaky. How do you resolve this?
The ownership gap is the actual problem A test suite owned by nobody is maintained by nobody, and each failure becomes a negotiation rather than a fix. That is
A DR test fails: the secondary region cannot launch enough instances. What happened, and what standing checks prevent it?
What happened Service quotas in the secondary region are far lower than in the primary , because nothing has ever run there at scale. Quotas are per account and
A front-end team wants to replace their global state library because "state management is unmanageable". How do you evaluate the request?
Ask what is actually in the store The complaint almost always resolves to one distinction not having been made: server state versus UI state . Server state is d
Netflix built its own CDN; Dropbox moved storage off S3. Both are usually wrong. What conditions made them right, and how do you test for those conditions?
What the interviewer is testing Whether you can extract the conditions from a famous decision rather than the decision itself. These two cases are the most comm
Six teams share one integration test environment. Bookings are made a week ahead and releases slip when someone overruns. How do you fix it?
Name the cost first The queue is not an inconvenience; it is lead time. Measure it: for the last twenty changes, how many days elapsed between "ready to test" a
When should a service call another synchronously, and when should it publish an event instead? Give me the deciding test, not a preference.
The deciding test Does this user action succeed or fail based on this callee's response? If yes, the call is synchronous, because you need the answer to decide.
A deployed model performed well in validation and its business metric has declined over four months. Nothing has been deployed. What do you investigate?
The model did not change; its world did Three distinct causes, and they need different responses: Data drift. The input distribution has moved — a new customer
A vendor SaaS product embeds a model that scores customers, and its output drives an automated decision in your process. Your model governance framework covers models you build. What do you do?
The obligation does not transfer with the outsourcing You are accountable for the decision. That the scoring is performed by a vendor changes who operates the m
An AI feature launched two months ago now costs more per month than the rest of the platform. What do you investigate?
Get cost per request, decomposed Token cost splits into input and output, and they price differently. Break the bill down by feature, by user, and by input vers
An LLM feature that worked last week now gives worse answers. Nothing was deployed. How do you find out what changed, and what should have been in place?
What the interviewer is testing Whether you treat an AI feature as a system with configuration and dependencies, or as a black box that mysteriously drifts. Wha
An end-to-end suite of 340 tests takes four hours and fails spuriously about half the time. The team wants to parallelise it. Is that the right move?
Parallelising treats the symptom It might halve the runtime. It will not touch the flakiness — in fact parallelisation often worsens it, by exposing shared stat
An estate has database passwords in environment variables across 200 services. Design the migration to a secrets manager.
Sequence it by risk, not by convenience Phase 0 — stop the bleeding. Secret scanning in CI and on the existing repositories, blocking new commits containing cre
Flaky Test Management
Quarantine, detection, and the trust a suite loses once red stops meaning broken.
Test Data Management
Realistic data without copying production personal data into a weaker environment.
End-to-End Test Economics
Why broad end-to-end suites get slow, flaky and abandoned, and what to keep.
AI Cost Management
Token accounting, routing, caching and the context-window budget.
Application Portfolio Management
Inventory, ownership, cost and health for every application.
Artifact Management
Immutable versioned outputs, promotion between repositories, and retention policy.
Change Management vs CD
Reconciling CAB-era controls with continuous delivery without pretending either away.
Chaos as a Test
Fault injection with a hypothesis, a blast radius and an abort condition.
Exception & Waiver Management
Time-boxed, owned deviations with a remediation date, rather than permanent silence.
Fleet Management
Inventory, health, configuration and grouping across devices you will never see.
Identity & Access Management
Workload identity, roles, permission boundaries and usage-based review.
Incident Management
Command roles, severity levels and mitigation before diagnosis.
Integration Test Boundaries
What sits inside a test's boundary, what is faked, and the confidence that follows.
Key Management
Rotation, separation of duty, envelope encryption and crypto-shredding.
Log Management
Aggregation, retention tiering, search and the cost of keeping everything.
Master Data Management
One authoritative record for a customer or product across systems that each have their own.
Model Risk Management
Inventory, validation, monitoring and challenge for models that make consequential decisions.
Non-Functional Test Strategy
Testing availability, latency, security and recovery rather than only behaviour.
Performance Test Design
Workload models, warm-up, think time, and the distribution the average hides.
Prompt & Version Management
Prompts as reviewed, versioned, evaluated production configuration.
Secrets Management
Runtime injection, dynamic credentials and rotation applications survive.
Test Architecture Strategy
Choosing what to verify where, given the failure modes that actually occur.
Test Pyramid Shapes
Pyramid, trophy and honeycomb, and the system properties that justify each shape.
Accessibility Architecture
Semantics, focus management and announcements designed in rather than audited in.