Full-text search with relevance ranking
Finding the documents that best match a few words, with stemming and typo tolerance, and returning the best ones first.
What it asks of a system
Full-text search is not LIKE '%word%'. The text is split into tokens and normalized (lower-cased, stemmed so “satisfies” matches “satisfy”, stop words dropped), and an inverted index maps each term to the documents that contain it. At query time the engine scores every match, by default with BM25, which weighs how often a term appears in a document against how common it is across all documents, so the most relevant results come first rather than in table order.
To tell whether a system really handles it, check which languages its analyzers support, whether it ranks results or only filters them, whether it handles phrases, typos, synonyms and facets, and how quickly a new or changed document becomes searchable. At scale, look at how the index is sharded and replicated and whether you can combine text relevance with filters and other signals such as recency.
Good at it · 7
Compare side by sideSearch indexes
Vector databases
PineconeCommercialDocument indexes rank `string` fields with BM25 and Lucene query syntax, in the same index as the vectors.A managed, serverless vector database that keeps its indexes in object storage and serves dense-vector, sparse-vector and BM25 full-text search from the same index.
turbopufferCommercialBM25 inverted indexes over one or more fields, combinable with vector ranking and attribute filters in one query.A managed vector and full-text search engine that keeps every index in object storage and caches the active parts on NVMe and in memory, trading occasional slow cold queries for a much lower price per GB.Sources
- PostgreSQL docs: Full text search, introduction (why LIKE falls short, tokens, lexemes, ranking)
- Elasticsearch docs: Similarity settings (BM25 is the default)
- Elastic blog: Practical BM25, part 2, the BM25 algorithm and its variables (April 2018)
Why each technology is listed comes from the sources on its own page.