Fill-in-the-Middle and Completion Models

A developer's cursor is almost never at the end of a file. It is inside a function with code above and below, and the completion has to fit both. A model trained purely left to right has never been asked to condition on a suffix, and it will happily generate a plausible continuation that duplicates or contradicts what follows.

The transformation

Fill-in-the-middle training solves this without changing the architecture (Bavarian et al., 2022, arXiv:2207.14255). During pretraining, a fraction of documents are split into prefix, middle and suffix, and reordered with sentinel tokens so that the model sees the prefix and suffix and predicts the middle. The objective remains next-token prediction; only the ordering changes.

The result is a model that can be prompted at inference with the code before and after the cursor and asked for what goes between. The paper's central finding is that this capability comes essentially free: models trained with a moderate FIM fraction lose nothing on ordinary left-to-right generation while gaining infilling.

The two arrangements, prefix-suffix-middle and suffix-prefix-middle, differ in whether the suffix or the prefix is adjacent to the generated region, and implementations pick one and must use it consistently between training and inference. A mismatch produces output that is coherent and wrong in a way that looks like a quality problem.

Latency governs the design

Inline completion appears as the developer types, which places it in a budget of a few hundred milliseconds. That constraint dictates almost every other choice.

The model is small, because time to first token dominates and it scales with model size. The context is truncated aggressively, because prefill scales with prompt length, so the retrieval that would improve quality has to be cheap or precomputed. Requests are cancelled continuously as the developer keeps typing, so most generations are discarded and cost is spent on completions nobody sees. Caching on the prefix hash catches the common case of a developer pausing and resuming at the same point.

The acceptance rate, the share of shown suggestions that are kept, is the metric that matters, and it is a joint property of quality and of the timing and length of what was shown.

When it breaks

Long suggestions have low acceptance. A multi-line completion is more likely to contain something wrong, and a developer rejects the whole thing for one bad line. Shorter, high-confidence completions are accepted far more often, so aggressiveness is a tuning parameter with a real optimum.

Suggesting into an incomplete thought is disruptive. A completion appearing while the developer is mid-decision interrupts rather than assists, which is why timing heuristics, suppressing suggestions immediately after a keystroke or inside certain syntactic positions, matter as much as model quality.

Accepted is not correct. Acceptance measures that the suggestion looked right at a glance, and studies of accepted completions find defects at a rate the acceptance metric does not reflect. It is a usage metric, not a quality one.

Repository context has to be precomputed. There is no time at completion for retrieval that involves embedding a query and searching an index, so useful cross-file context must come from cheap signals: open files, recent edits, and imports already parsed.