Text Analysis: Stemming, Lemmatisation and Stopwords

A support engineer searches a ticket index for "not working". The index was built with Lucene's EnglishAnalyzer, whose default stop set of 33 words includes "no" and "not" (Apache Lucene, EnglishAnalyzer source). The query becomes a single stemmed term, work, and matches every ticket that says the fix is working. Nothing crashed and no log line was written. The search engine did exactly what its text analysis told it to.

Every lexical index, from an inverted index scored by BM25 to a query-likelihood model, sees only the terms the analyzer emits. The analyzer decides what counts as the same word, and that decision determines recall and precision before any ranking function runs.

The analyzer is a contract between indexing and querying

An analyzer is a pipeline. Lucene's EnglishAnalyzer runs a standard tokenizer, then a possessive filter that strips a trailing "'s", lowercasing, stopword removal, an optional keyword marker that protects listed words from stemming, and finally the Porter stemmer. The same chain must run on documents at index time and on queries at search time. If the two drift apart, for example a query path that lowercases but does not stem, the query term running looks for a postings list that does not exist, because the index only ever stored run.

The contract has a second consequence: analysis is baked into the postings. Changing the stemmer or the stop list does not affect documents already indexed, so any change to analysis means a full reindex.

Stemming: rules that ignore meaning

Porter's algorithm strips suffixes in a sequence of steps, each allowed only when the remaining stem is long enough by a crude syllable measure. In the original paper "generalizations" becomes "generalization", then "generalize", then "general", then "gener", and on a 10,000-word vocabulary the algorithm produced 6,370 distinct stems, about a one-third reduction (Porter, 1980, An Algorithm for Suffix Stripping, Program 14(3)).

The rules know nothing about meaning, and the errors run in both directions. Running the author's reference implementation shows the over-stemming: "university" and "universe" both become univers, "organization" becomes organ, "generous" and "general" share gener, "experiment" and "experience" share experi, and "news" becomes new. It also shows under-stemming: "absorb" stays absorb while "absorption" becomes absorpt, and "european" never meets "europe". The standard textbook example is that Porter conflates operate, operating, operates, operation, operative, operatives and operational to oper, which hurts a query about operational research (Manning, Raghavan & Schütze, 2008, Introduction to Information Retrieval, §2.2.4).

Lemmatisation: dictionaries and part of speech

A lemmatiser maps a word to its dictionary form using a vocabulary and morphological analysis, often with part-of-speech tagging, so "saw" becomes "see" as a verb and stays "saw" as a noun. Krovetz's stemmer is a middle path: suffix rules checked against a dictionary so that outputs are real words. Krovetz reported that grouping morphological variants, inflectional and derivational, significantly improved retrieval (Krovetz, 1993, Viewing Morphology as an Inference Process, SIGIR).

The literature does not agree on how much any of this helps English. Harman tested three suffixing algorithms on the Cranfield, Medlars and CACM collections and found no significant improvement for any of them (Harman, 1991, How Effective Is Suffixing?, JASIS 42(1), 7-15). The textbook consensus that followed is that stemming raises recall while harming precision, that neither stemming nor full morphological analysis improves English retrieval much in aggregate, and that the picture is different for morphologically rich languages such as Spanish, German and Finnish, where CLEF evaluations repeatedly showed large gains.

Stopwords: cheaper than they look, costlier than they seem

Word frequencies are heavily skewed. In the Reuters-RCV1 collection, the 30 most common words account for about 30 percent of tokens, and dropping the 150 most common words cuts 25 to 30 percent of non-positional postings (Manning, Raghavan & Schütze, 2008, §5.1). Stemming after case folding shrinks the RCV1 dictionary by about 17 percent, from 391,523 to 322,383 terms.

Those savings once justified stop lists of two to three hundred words. The trend since has been towards small lists or none: gap-encoded postings compress common terms very well, IDF already weights "the" near zero, and dynamic pruning skips long postings lists that cannot change the top \(k\). Web search engines generally do not use stop lists, because removal breaks queries such as "flights to London" and "President of the United States".

When it breaks

Negation and names disappear. A stop list that includes "not" turns "not working" into "working", and any of the longer lists that also include "who" turns a search for the band The Who into an empty query. Stop lists written for recall on news prose are dangerous on support tickets, product search and titles.

Over-conflation costs precision where it is most visible. A stemmer that merges "university" with "universe" adds matches deep in the list and, occasionally, at rank one. Web-style search, judged on the first page, feels precision loss more than it benefits from recall gains.

Asymmetric pipelines fail silently. Index-time and query-time analyzers diverge after a configuration change, a synonym filter added on one side, or a client that pre-processes queries. Recall drops for a class of queries and no error is raised.

Every change is a reindex. Trying a different stemmer on a production index means rebuilding it, so analysis choices are made early, with the least data, and rarely revisited.

Hybrid pipelines must keep two views of the text. Stemmed, stopped terms suit a BM25 first stage. A neural reranker or dense encoder expects natural text with its own subword tokenizer, and feeding it analyzed tokens degrades it. Store the original text alongside the analyzed index.