Design the moderation path for user-generated content at high volume, where both false positives and false negatives are costly.

What the interviewer is testing

Whether you design a tiered system by consequence rather than proposing a single classifier.

The shape of the problem

Volume makes full human review impossible. Model-only decisions are unacceptable because both error types are costly: a false positive removes legitimate speech, a false negative leaves harmful content live. And the two have asymmetric costs that differ by category — the cost of missing child safety content is not comparable to the cost of missing spam.

So a single threshold is wrong by construction.

The design

Tier by category and confidence.

High-confidence, high-severity: block immediately and automatically, with an appeal path. The cost of a false positive is accepted because the cost of a false negative is unacceptable.

High-confidence, low-severity: automated action with lighter review.

Uncertain: route to human review, with a queue prioritised by potential harm and reach — content being seen by many people is more urgent than the same content seen by nobody.

Low confidence, low severity: no action, sampled for measurement.

Reactive path: user reports feed the same queue with their own priority, because they catch what models miss.

Making the human review actually work

Route only cases needing judgement, not a stream of obvious ones. A reviewer seeing mostly easy cases stops looking, and automation bias does the rest.

Show the evidence and the specific policy in question, so the reviewer forms a judgement rather than confirming one.

Track the override rate. Near zero means the review is not functioning; very high means the model should not be gating that category.

Manage throughput and welfare, which for this domain is a genuine operational requirement rather than a nicety.

What a strong answer adds

The feedback loop: reviewer decisions are labelled training data, and closing that loop is what improves the models. But it biases the training set towards what the model already routes for review, so a random sample must also be reviewed to measure true performance on the population.

And the appeal path as an architectural requirement — decision logging sufficient to explain an individual outcome, which under emerging AI regulation is an obligation for higher-risk systems rather than a courtesy.

Common weak answers

One classifier with one threshold. Human review of everything, which does not scale.