Vector Search in Production: Semantic Retrieval from Principles to Deployment

Summary: Vector search maps queries and documents into one semantic space and replaces brute-force comparison with approximate nearest neighbor search, making it the key technical path to millisecond semantic retrieval and better on-site search relevance.

When an enterprise knowledge base, product catalog or ticket store grows to million-scale, keyword matching alone increasingly misses what users actually mean. Questions that are semantically close but worded differently are exactly where traditional retrieval fails. Vector search uses an embedding model to turn text into vectors, then applies approximate nearest neighbor (ANN) algorithms to find the closest results in semantic space, so that "phrased differently, still found" becomes real. This article moves from principles to deployment and gives an engineering path you can follow directly.

Key Takeaways

The essence of vector search is using an embedding model to map text into vectors, then recalling via approximate nearest neighbor (ANN) in semantic space, not character-by-character matching.
HNSW is the default index for million-scale corpora; IVF+PQ suits billion-scale, memory-constrained scenarios; the choice depends on scale and budget.
Production systems should almost always use hybrid retrieval of "dense vectors + keywords (BM25)", then rerank with a cross-encoder to raise precision.
Recall quality must be measured with recall@k on a real query set, with SLOs defined on p95 latency; average latency alone is not enough.
Metadata filtering must be executed natively inside the ANN query; post-hoc filtering breaks graph index assumptions and sharply lowers recall.
The zero-result rate and the similarity-score distribution are the two signals most worth watching after launch; anomalies usually surface before user complaints do.
Quantization (for example int8) cuts memory footprint to about 1/4, a common trade that exchanges single-digit recall loss for controllable cost.

1. Semantic retrieval maps text into a comparable vector space

The core idea of vector search is this: an embedding model converts both the query and every document into high-dimensional vectors so that semantically similar content sits closer together in vector space, and an approximate nearest neighbor (ANN) algorithm then finds the nearest results. Compared with exact nearest neighbor brute-force comparison over all vectors, ANN trades single-digit recall loss for a 10x to 1,000x speedup, which is precisely why it is indispensable in production. Mainstream embedding dimensions run between 384 and 1536, and the distance metric must match the model: after L2 normalization, cosine and inner-product ranking give identical results, while a mismatched metric silently degrades relevance. The embedding model itself sets the ceiling more than the index does: a stronger encoder usually beats endlessly tuning an index built on weak embeddings.

2. Index selection trades off recall, latency and memory

At different scales, the trade-offs across the index family diverge sharply. HNSW (Hierarchical Navigable Small World) offers an excellent recall-latency balance on million-scale corpora and is the default choice for text retrieval, though its memory footprint is high; IVF (inverted file) restricts the search scope to a few buckets through clustering, suiting very large scale but depending on good clustering and probe-count configuration; PQ and its variants (such as OPQ) compress vector storage, cutting memory several times over at the cost of slightly degraded similarity precision. In practice, an HNSW efSearch between 64 and 128 typically yields 95%–98% recall on a million-scale index while holding latency to 30–60 ms. Memory planning is equally direct: a million 768-dimension float32 vectors take about 3GB raw, with HNSW graph overhead usually adding another 20%–50%; switching to int8 quantization brings that down to about 800MB.

3. A production retrieval pipeline has four stages: embedding, hybrid search, reranking and fallback

Vector search on its own is rarely robust enough: enterprise queries often contain exact terms such as product codes or policy IDs, which dense vectors tend to lose, so production pipelines should almost always adopt hybrid retrieval, using dense ANN for semantic matching and sparse retrieval such as BM25 to catch exact terms, merging via reciprocal rank fusion or weighted scoring, and finally reranking candidates with a cross-encoder. A common pattern is to retrieve wide first (top-200) and then rerank precisely (top-20); the two-stage design saves about forty percent of latency compared with single-stage exact retrieval. For zero results, set a similarity-threshold fallback together with "did you mean" rewrite suggestions, and when necessary turn high-frequency failed queries into synonyms or new documents. Reranking improves precision but adds latency, so trigger it only when retrieval scores are ambiguous rather than paying 50–300 ms every time.

4. Deployment and measurement determine whether vector search stays stable

Launching vector search is only the beginning; the real engineering difficulty lies in measurement and operations. Offline, run recall@k, precision@k and MRR/NDCG against a "golden query set"; online, watch p50/p95/p99 latency, the similarity-score distribution and the zero-result rate, and alert when recall drops more than 2%. An embedding model version change requires a full index rebuild, while a single-document edit can use incremental upsert; the two must not be confused. On cost, HNSW memory is the main expense; IVF-PQ can cut memory 4–10x, and a semantic cache (for example a 0.90 similarity threshold) can reduce repeated-query database access by 60%–75%. Watch out for the scale wall: as data objects approach thirty million and vectors approach four billion, average latency can jump from about 300 ms to 1100 ms and single-node memory from 18GB to 48GB, at which point sharding and self-tuning strategies must already be in place.

FAQ

Q: What is the difference between vector search and traditional keyword-based search?

A: Keyword search matches terms literally and tends to miss recall on synonyms, rewrites or colloquial phrasing; vector search converts text into semantic vectors and finds nearest results by semantic distance, so it understands "phrased differently". Production systems usually combine both through hybrid retrieval rather than choosing one.

Q: HNSW or IVF, which one should I pick?

A: At million-scale with sufficient memory, prefer HNSW, which is the most balanced on recall and latency; at billion-scale or under memory constraints, choose IVF+PQ, trading a small recall loss for controllable memory. A two-stage setup also works: IVF for coarse ranking, HNSW for fine ranking.

Q: Why do my vector search results look irrelevant?

A: The most common cause is a mismatch between the embedding model and your business corpus, or a distance metric not aligned with the model; the next is an efSearch set too low, leaving too few recall candidates. First benchmark exact nearest neighbor to measure ANN recall loss, then tune.

Q: Why does metadata filtering make recall collapse?

A: If filtering runs after the HNSW search completes (post-hoc), candidates have already been truncated by the graph structure and little may remain. Choose an engine that executes filtering natively during the ANN query so that filtering participates in retrieval.

Q: What level should the zero-result rate be held to?

A: There is no universal threshold, but it is one of the signals most worth watching after launch. Start by treating the "top 10 zero-result queries" as a weekly content-gap list, and bring the similarity-score distribution into monitoring; anomalies usually surface before complaints do.

Q: Do I need to rebuild the index after upgrading the embedding model?

A: Yes. A change in embedding dimension or model version changes the vector space, so the old index and new queries no longer share one space and a full rebuild is required; single-document additions and deletions can use incremental upsert. The two must not be mixed.

Q: How can I cut vector search cost without adding more machines?

A: Three tactics are most common: use int8 quantization to cut memory to about 1/4, use a semantic cache to intercept repeated queries, and tune efSearch down to the lowest value that still meets the SLO. Memory is the main cost of HNSW, so quantization pays off most directly.

About Tengence

Vector search is the technical foundation of enterprise on-site semantic retrieval, and complements the query understanding and result ranking of Tengence Search naturally. Tengence (TENGENCE) is the enterprise growth engine of the AI era, focused on public-traffic acquisition and private-domain conversion; its core products are Tengence GEO (dual-engine GEO + SEO) and Tengence Search.

Data Sources

ScaleMind: Vector Databases for RAG in Production: presents the ANN trade-offs among recall, latency, filtering and cost, and gives the production architecture of the HNSW/IVF index families, hybrid retrieval and reranking.
Machine Learning at Scale: HNSW and ANN Search: notes that ANN trades single-digit recall loss for a 100–1000x speedup, and that RAG can often tolerate 90%–95% recall while fraud scenarios may need 99.9%.
Vector Databases Part 5: Advanced Optimization and Reranking: gives measured results for efSearch 64–128 delivering 95%–98% recall on a million-scale index, and two-stage retrieval cutting latency by about forty percent.
PulseRevOps: Low-Latency RAG Pipeline Architecture: estimates 2–10 ms HNSW retrieval for a million 768-dimension vectors, with int8 quantization cutting memory from about 3GB to about 800MB.