A source-code platform must make code across millions of repositories searchable, where repositories differ enormously in size and activity. Which indexing decisions dominate, and what makes this different from indexing documents?

Why code is not documents

Standard text indexing tokenises on word boundaries and applies stemming and stop-word removal. Every one of those assumptions is wrong for code:

• Developers search for substrings and symbols — getUserById, ->, SELECT *, a regular expression. Word tokenisation cannot answer those.
• Punctuation is meaningful. Stripping it destroys the query.
• Case matters sometimes and not others.
• The same token appears in enormous quantities of near-identical code (vendored dependencies, generated files, forks), so ordinary term-frequency ranking produces useless results.

The consequence is a different index structure — typically n-gram or trigram indexes that support substring matching, which are far larger than word indexes and demand a different cost model.

The decisions that dominate

1. What not to index. The highest-leverage decision. Forks that are byte-identical to their parent, vendored dependencies, generated files, binaries and very large files can be excluded or deduplicated by content hash. Content-addressed deduplication across repositories is enormous leverage, because the same open-source file exists in a vast number of copies.

2. Index sharding by repository, not by term. Keeps a repository's data local, makes permission filtering tractable, and lets an unavailable shard degrade a subset of results rather than all of them.

3. Permissions at query time, not index time. Access changes constantly; a user's visible set is computed per query and intersected with results. Baking permissions into the index means every access change triggers reindexing, which is untenable. The design consequence is that the index must over-retrieve and then filter, and the ranking must survive that filtering.

4. Incremental indexing driven by push events, with an explicit freshness budget. Full reindexing at this corpus size is a multi-day operation, so it must never be the routine path — only a disaster recovery path.

5. Tiered freshness. Default branches of active repositories index within seconds; inactive repositories can lag by hours. Uniform freshness across an enormously skewed corpus means paying the active-repository cost for the whole estate.

The scale asymmetry

Repository sizes span many orders of magnitude. A design that treats them uniformly fails at both ends: per-repository overhead dominates for millions of tiny repositories, while a handful of enormous monorepos exceed any per-shard limit. The answer is explicit size classes with different handling — small repositories batched many-to-a-shard, huge ones split across shards.

What breaks in production

• Reindex storms after an index format change, competing with live indexing for the same capacity.
• Permission leakage through result counts, timing or ranking signals, even when documents are correctly filtered — a real and subtle class of bug.
• Query cost unbounded by regular expressions, requiring a cost limiter and a timeout rather than trusting the query.
• Index and repository drifting apart when indexing jobs fail silently, which needs a reconciliation process rather than trust.