In brief: Search is a derived read model; its design must explain freshness, deletion and how truth is rebuilt.
The problem
Direct synchronous indexing slows writes and still loses updates on failure. Shared indexes risk tenant leakage, relevance changes are subjective, and schema evolution can require rebuilding billions of documents without taking search offline.
Why the simple answer is not enough
A search engine trades transactional consistency for specialised retrieval. The product must state how long results may lag and which operations fall back to the database.
Tenant filters must be injected by a trusted query wrapper, not supplied optionally by callers.
System flow
authoritative change→Outbox or CDC
durable indexing event→Indexer
versioned transform→Search index
tenant-filtered read model
Architecture decisions
Use versioned documents
Database sequence or aggregate version prevents delayed events from overwriting newer indexed state.
Separate schema from alias
Create immutable index versions and point a stable read alias to the active version.
Measure relevance offline and online
Use labelled queries for precision and recall, then confirm clicks or successful outcomes without blindly optimising engagement.
Going deeper
Deletion is a first-class flow
Privacy deletion must remove primary data, indexes, caches and backups according to their policies, with durable evidence.
Rebuild capacity is resilience
If a full index cannot be rebuilt within the recovery objective, the derived system has an untested dependency on its current state.
Implementation path
- Capture committed changes through outbox or CDC.
- Index with document version and server-owned tenant identity.
- Build a new index, validate it, then switch an alias atomically.
Technical example: Alias-based reindex
create index products_v3(mapping_v3)
bulk index snapshot with sourceVersion
consume changes after snapshot watermark
compare counts, samples and query set
atomic alias products_read: v2 -> v3Keep the previous index for a bounded rollback window and continue feeding both until the cutover is trusted.
Failure modes to design for
- A delayed update resurrects deleted data.
- A missing tenant filter returns another customer's document.
- Reindex overloads the source database.
What to observe
| Signal | Why it matters |
|---|---|
| Indexing lag by event type | Measures search freshness and deletion risk. |
| Rejected stale versions | Shows out-of-order delivery is contained. |
| Zero-result and successful-search rate | Tracks relevance from a user outcome. |
How to validate the design
- Delay and reorder index events around updates and deletes.
- Run cross-tenant negative queries at the lowest search client layer.
- Reindex at production-like volume while writes continue.
Safe rollout plan
Introduce durable events and version checks on the current index first. Build the next mapping in parallel, compare a representative query corpus, then move a small tenant cohort through a routing alias before global cutover.
Production checklist
- Search never becomes the source of truth for transactions.
- Delete and privacy events have measured propagation objectives.
- Relevance changes are evaluated with a fixed query set and business metrics.
Takeaway
Reliable search combines rebuildable data flow, explicit freshness and measurable relevance.