Inference & Serving
27 min
The Mode Is Not the Answer: How Decoding Stopped Being a Search Problem
Run exact search over a trained translation model and, for more than half of a standard test set, the highest-scoring output is the empty string. Beam search works only because it fails to find what the model actually prefers. Th…