Testing in Production
Synthetic transactions, dark launches and shadow traffic, done deliberately and safely.
6 to work through
-
advanced
"Testing in production" is either professional practice or negligence depending on what is meant. What are the legitimate forms, and what must be true for each?
2 min answer -
advanced
A canary takes 5% of traffic on a new build and is compared with the baseline on error rate and p99. What happens if the load balancer sends only new connections to the canary?
3 min answer -
advanced
A team wants to test in production. What does that legitimately mean, and what does it require first?
2 min answer -
advanced
On 1 August 2012 Knight Capital lost roughly $460M in about 45 minutes. The SEC's 2013 order records that new order-routing code had been rolled out to eight servers over several days and a technician did not copy it to one of them, where a flag reused by the new code reactivated dormant functionality. Every unit and integration test had passed. Which variable was never under test, and what check would you add?
3 min answer -
advanced
What does testing in production genuinely mean, and what must exist before it is responsible?
2 min answer -
advanced
You propose shadow traffic and canary analysis. A risk officer says "you are testing on customers". How do you respond?
2 min answer
5 terms in this topic
Deployed-Artefact Assertion
A check run after every deployment that compares the build identifier reported by each running host against the intended release, so a partially appl…
practiceSide-Effect Isolation
Ensuring that test, synthetic or shadowed traffic in production cannot produce real-world effects - the control that separates professional productio…
practiceSynthetic Monitoring
Continuously executing a real user journey against production from outside it, so a broken journey is detected before a user reports it.
practiceTesting in Production
Deliberately validating behaviour against real traffic, data and scale, using controls that bound the blast radius.
conceptTraffic-Mix Skew
A canary or experiment arm receiving a population that differs systematically from the baseline - so the comparison measures who was routed rather th…
Neighbouring topics
Testing & Quality Architecture
General material on designing a testing strategy as an architectural concern.
Test Architecture Strategy
Choosing what to verify where, given the failure modes that actually occur.
Test Pyramid Shapes
Pyramid, trophy and honeycomb, and the system properties that justify each shape.
Integration Test Boundaries
What sits inside a test's boundary, what is faked, and the confidence that follows.
Contract Testing at Scale
Keeping dozens of services compatible without an environment that runs all of them.
Consumer-Driven Contracts
Consumers declaring what they rely on, and providers verifying against those declarations.
Test Data Management
Realistic data without copying production personal data into a weaker environment.
Synthetic Data
Generating data with the shape and edge cases of the real thing, and where it misleads.
Environment Parity
The differences between staging and production that decide which bugs survive to release.
Service Virtualisation
Standing in for a dependency you cannot call, and keeping the stand-in honest.
End-to-End Test Economics
Why broad end-to-end suites get slow, flaky and abandoned, and what to keep.
Non-Functional Test Strategy
Testing availability, latency, security and recovery rather than only behaviour.
Performance Test Design
Workload models, warm-up, think time, and the distribution the average hides.
Chaos as a Test
Fault injection with a hypothesis, a blast radius and an abort condition.
Security Testing in the Pipeline
SAST, DAST, dependency and secret scanning, and what to do with the findings.
Accessibility Testing
Automated checks, their ceiling, and the manual testing that has to sit above it.
Mutation Testing
Measuring whether tests would actually notice a defect, not just cover a line.
Flaky Test Management
Quarantine, detection, and the trust a suite loses once red stops meaning broken.
Quality Gates
Thresholds that block a release, who may override them, and how they decay.