Non-Functional Test Strategy
Testing availability, latency, security and recovery rather than only behaviour.
5 to work through
-
beginner
A requirements document says "the system must be highly available and performant". How do you turn that into something testable?
2 min answer -
advanced
A trading platform's functional tests all pass and production fails under load. What should the non-functional test strategy contain?
2 min answer -
advanced
ByteDance's Monolith paper (RecSys 2022) describes a recommendation system with minute-level parameter synchronisation whose training parameter servers are snapshotted only once a day, so a server failure loses up to a day of updates. The authors say they expected to need more frequent snapshots. What did they do instead of arguing about it, and what does that give a non-functional test strategy?
3 min answer -
advanced
On 31 January 2017 a GitLab engineer removed the wrong PostgreSQL data directory, and the recovery mechanisms they believed they had turned out not to work. Roughly six hours of data was lost. Which property had never been tested, and what does that say about non-functional testing generally?
2 min answer -
advanced
Which non-functional properties can be tested pre-release, and which can only be verified in production?
2 min answer
2 terms in this topic
Non-Functional Acceptance
Treating quality-attribute targets as acceptance criteria with automated verification, so a release can fail on latency the way it fails on a broken …
practiceNon-Functional Test Strategy
The plan for how each quality attribute claim will be demonstrated, tying every NFR to a test that could falsify it.
Neighbouring topics
Testing & Quality Architecture
General material on designing a testing strategy as an architectural concern.
Test Architecture Strategy
Choosing what to verify where, given the failure modes that actually occur.
Test Pyramid Shapes
Pyramid, trophy and honeycomb, and the system properties that justify each shape.
Integration Test Boundaries
What sits inside a test's boundary, what is faked, and the confidence that follows.
Contract Testing at Scale
Keeping dozens of services compatible without an environment that runs all of them.
Consumer-Driven Contracts
Consumers declaring what they rely on, and providers verifying against those declarations.
Test Data Management
Realistic data without copying production personal data into a weaker environment.
Synthetic Data
Generating data with the shape and edge cases of the real thing, and where it misleads.
Environment Parity
The differences between staging and production that decide which bugs survive to release.
Service Virtualisation
Standing in for a dependency you cannot call, and keeping the stand-in honest.
End-to-End Test Economics
Why broad end-to-end suites get slow, flaky and abandoned, and what to keep.
Performance Test Design
Workload models, warm-up, think time, and the distribution the average hides.
Chaos as a Test
Fault injection with a hypothesis, a blast radius and an abort condition.
Security Testing in the Pipeline
SAST, DAST, dependency and secret scanning, and what to do with the findings.
Accessibility Testing
Automated checks, their ceiling, and the manual testing that has to sit above it.
Mutation Testing
Measuring whether tests would actually notice a defect, not just cover a line.
Flaky Test Management
Quarantine, detection, and the trust a suite loses once red stops meaning broken.
Testing in Production
Synthetic transactions, dark launches and shadow traffic, done deliberately and safely.
Quality Gates
Thresholds that block a release, who may override them, and how they decay.