Data-Dependent Tests and Behavioural Suites
Why aggregate metrics are a poor gate, how curated behavioural cases catch regressions that averages hide, and the discipline of turning every production failure into a permanent test.
A model improves from 91.2 to 91.6 percent accuracy and ships. It also stopped handling negation correctly, which affects 0.3 percent of traffic and is invisible in the aggregate. Behavioural testing exists because an average over a test set is a weak instrument for detecting whether specific, important capabilities survived a change.
Testing capabilities rather than accuracy
The CheckList framing (Ribeiro et al., 2020, ACL) organises this into three test types, and the vocabulary is useful well beyond NLP.
Minimum functionality tests are small curated sets probing one capability. Negation handling, numeric comparison, handling of a specific entity type. Each is a set of examples with known correct answers, and each fails loudly when that capability regresses, naming the capability rather than reporting a decimal shift.
Invariance tests assert that a change to the input which should not affect the output does not. Substituting a name, changing an irrelevant date, paraphrasing without changing meaning. Violations reveal spurious sensitivity, which is often the mechanism behind a fairness problem or a robustness failure.
Directional expectation tests assert that a change should move the prediction a particular way without specifying how much. Adding a strongly negative phrase should not increase a positive sentiment score. These are weaker assertions and are robust across model versions, which makes them good long-lived tests.
The regression suite as institutional memory
The discipline that compounds: every production failure becomes a test case before the fix is deployed. Over a year this accumulates into a suite that encodes exactly what has gone wrong before, which is far better targeted than any test set drawn at random.
This suite has a different purpose from the evaluation set and should be kept separate. Evaluation estimates performance on the distribution; the regression suite asserts that specific known failures stay fixed. Mixing them corrupts the evaluation estimate, because the regression cases are deliberately unrepresentative.
Sliced evaluation
Between the aggregate and the individual case sits slicing: report metrics per segment, by language, device, customer size, input length, or any axis where behaviour plausibly differs. A change that improves the average while degrading a slice is common, and slicing is what turns it from an unpleasant surprise into a visible tradeoff at review time.
The slices worth tracking are the ones that matter commercially or ethically, plus the ones where sample sizes are small enough that aggregate metrics cannot see them.
When it breaks
Behavioural tests are curated, so they are biased. They test what someone thought to test, which is not the same as what matters. They complement distributional evaluation and cannot replace it, and a suite that has grown large can create a misleading sense of coverage.
Tests written by the model's authors share the authors' blind spots. The failures that survive to production are the ones nobody imagined, which is an argument for sourcing cases from support tickets, user reports and adversarial review rather than from the team.
Generated test cases inherit the generator's limitations. Using a model to produce test cases is efficient and produces cases within that model's conception of the problem, systematically missing what it also does not know.
Passing every behavioural test is not a quality claim. The suite asserts that known capabilities work; it says nothing about the rest of the distribution. A model can pass 400 behavioural tests and be worse overall, which is why both gates are needed and why neither can be dropped for the other.
12 flashcards for this concept
Click a card to reveal the answer.