Full-Text Search

The Problem

Users expect instant, relevant search results across all content — documents, products, messages, user profiles. SQL LIKE queries are too slow and lack relevance ranking. Full-text search requires specialized data structures (inverted indexes), text analysis (tokenization, stemming, synonyms), relevance scoring (TF-IDF, BM25), and query features like typo tolerance, faceted filtering, and autocomplete. The search index must stay in sync with the primary database, handle schema evolution, and scale to billions of documents with sub-100ms query latency.

Architectural Approaches

Architecture

Loading diagram...

Key Components

Data Model

Loading diagram...

Inverted Index Internals

An inverted index maps terms to the documents that contain them — the opposite of a forward index (document → terms).

How it works:

1. Analysis: Text is processed through an analyzer chain:
   • Tokenizer: “The quick brown fox” → [“The”, “quick”, “brown”, “fox”]
   • Lowercase filter: → [“the”, “quick”, “brown”, “fox”]
   • Stop word removal: → [“quick”, “brown”, “fox”]
   • Stemmer: → [“quick”, “brown”, “fox”] (no change here, but “running” → “run”)
2. Indexing: Each term maps to a posting list of document IDs:
   • “quick” → [doc1, doc5, doc12]
   • “brown” → [doc1, doc7]
   • “fox” → [doc1, doc3, doc7]
3. Querying: For “quick fox”, intersect the posting lists:
   • “quick” ∩ “fox” → [doc1] (AND query)
   • “quick” ∪ “fox” → [doc1, doc3, doc5, doc7, doc12] (OR query)

BM25 scoring: The standard relevance algorithm considers term frequency (how often the term appears in the document), inverse document frequency (rarer terms are more important), and document length normalization (longer documents don’t unfairly dominate).

Keeping the Index in Sync

The search index is a derived data store — the primary database is the source of truth. Keeping them in sync is the hardest operational challenge.

Approaches:

Dual-write: Application writes to both the database and the search index in the same request. Simple but fragile — if one write fails, they’re out of sync. No atomicity guarantee.

Change Data Capture (CDC): Use Debezium to stream database changes (INSERT, UPDATE, DELETE) from the WAL/binlog to the indexing pipeline. Near real-time (1-5 second lag), reliable, and decoupled from application code. The gold standard.

Event-driven: Application publishes domain events (OrderCreated, ProductUpdated) to a message bus. The indexing service consumes events and updates the index. Requires events to contain enough data to build the search document.

Periodic re-index: Full re-index on a schedule (hourly, daily). Simple but creates a staleness window. Use as a safety net alongside CDC — periodically verify the index matches the database and repair divergences.

Handling deletes: Soft-deletes in the database should trigger index removal. Hard-deletes require CDC or event-driven approaches — the record is gone from the database, so you can’t query it for index updates.

Search-as-You-Type and Autocomplete

Instant search results as the user types requires special optimization:

Prefix matching: Index terms as edge n-grams. “elasticsearch” → [“e”, “el”, “ela”, “elas”, …]. Allows matching partial terms during typing. Increases index size but enables instant prefix search.

Completion suggester: Elasticsearch’s completion suggester uses a finite-state transducer (FST) stored entirely in memory. Extremely fast (<1ms) but only for exact prefix matches. Ideal for search box autocomplete.

Debouncing: Don’t send a query on every keystroke. Wait 150-300ms after the user stops typing before sending the query. Reduces server load by 70-80%.

Result caching: Cache autocomplete results aggressively. “iph” → cached results for “iphone” suggestions. Short TTL (60-120 seconds) is fine since autocomplete results change infrequently.

Typo tolerance: Typesense and Meilisearch handle this natively. For Elasticsearch, use fuzzy queries with an edit distance of 1-2. “iphne” matches “iphone” with edit distance 1.

Highlighting: Return matching fragments with the search terms highlighted. Use Elasticsearch’s highlight API or post-process with regex. Show highlighted snippets in the search dropdown for better UX.

Real-World Examples

Architectural Tradeoffs

# Learn the System Design of: Full-Text Search

You are an expert software architect and teacher. I want to understand full-text search as a practical system design capability. (Inverted indexes, relevance tuning, and search-as-you-type for any content type).

Use the following architectural guide as a learning reference. Avoid one-size-fits-all recommendations. Keep assumptions, costs, and tradeoffs explicit.

---

import { Mermaid, ComponentList, ComponentCard, ApproachList, Approach, Example, Tradeoff } from '@/components/mdx';

## The Problem

Users expect instant, relevant search results across all content — documents, products, messages, user profiles. SQL LIKE queries are too slow and lack relevance ranking. Full-text search requires specialized data structures (inverted indexes), text analysis (tokenization, stemming, synonyms), relevance scoring (TF-IDF, BM25), and query features like typo tolerance, faceted filtering, and autocomplete. The search index must stay in sync with the primary database, handle schema evolution, and scale to billions of documents with sub-100ms query latency.

## Architectural Approaches

<ApproachList>
<Approach
  name="Elasticsearch / OpenSearch"
  pros={[
    'Extremely mature — battle-tested at massive scale',
    'Rich query DSL: fuzzy matching, boosting, aggregations, geo',
    'Distributed architecture — scales horizontally by adding nodes',
    'Huge ecosystem: Kibana, Logstash, client libraries for every language',
  ]}
  cons={[
    'Operationally complex — JVM tuning, shard management, cluster health',
    'High memory requirements (heap + OS cache)',
    'Eventual consistency between primary DB and search index',
    'Can be expensive to run (compute + storage)',
  ]}
>
Deploy a dedicated search cluster using **Elasticsearch** or its fork **OpenSearch**. Index documents from your primary database. Query using the rich DSL for full-text search, aggregations, geospatial queries, and more. The industry standard for large-...

[Full prompt copied to clipboard]