On-Premise AI Search in 2026: Architecture and Capacity Planning
Summary: The most common failure in on-premise AI search is not an insufficient model but wrong capacity math, an absent retrieval layer, and governance added at the end. This article splits the problem into four layers, each with a basis you can quantify.
Key Takeaways
- Retrieval first, model second. Without a reliable retrieval layer, a larger model only fabricates answers more fluently.
- Hybrid retrieval is the default, not an option. Keyword recall handles exact matching, vector recall handles semantic proximity; missing either collapses on the long tail.
- Size capacity by concurrent questions, not by document count. Document volume drives indexing cost; concurrency and context length drive inference cost.
- The real constraint is usually data residency and audit, not performance. Design the architecture around those two, not the reverse.
- Five evaluations must precede any launch commitment. Skipping any one costs far more than the evaluation itself.
1. What each of the four layers solves
| Layer | Responsibility | Key metric | Common gap |
|---|---|---|---|
| Retrieval | Recall candidate passages | Recall rate, latency, index freshness | Vector only, fails on rare exact terms |
| Model serving | Generation and reranking | Time to first token, throughput, VRAM | Sizing by documents, not concurrency |
| Orchestration | Decompose questions, call tools, assemble context | Calls per question, retry rate | No timeout or degradation strategy |
| Governance | Permissions, audit, traceability, redaction | Audit coverage, block rate | Built last, forcing rework |
A rule of thumb on build order: retrieval and governance first, model serving and orchestration after. Retrieval sets the ceiling and governance decides whether you can launch at all; models can be swapped, while those two layers are the most expensive to change later.
Orchestration is the most under-estimated. Real questions need multiple steps — what is this, what to do about it, what the specific process is. Products that answer single-turn questions well collapse on multi-turn ones, and the fault usually sits in orchestration rather than the model.
Want to know how much of the ChatGPT and Perplexity answers your brand owns?
Explore TENGENCE GEO2. Retrieval: hybrid by default
Vector-only retrieval visibly fails on two query classes: exact identifiers (order numbers, model codes, clause numbers) and rare proper nouns. Keyword-only retrieval fails on colloquial phrasing and synonyms.
So the default is hybrid: two recall paths, then fusion.
Three common fusion methods. Weighted score fusion is simple but needs tuning and is unstable across queries. Reciprocal rank fusion (RRF) looks only at rank, not absolute scores, has few parameters and is robust — good for a starting point. Learning to rank has the highest ceiling but requires labels and continuous maintenance.
Practical advice: ship RRF first, introduce learning-to-rank once you have data. RRF's only real parameter is the smoothing constant, which is cheap to tune and easy to explain.
Index freshness must also be quantified. The delay between a document update and its being searchable determines whether "what happened today" can be answered. In on-premise deployments the usual pattern is incremental indexing plus scheduled full rebuilds; both periods belong in the architecture document, not in an operator's head.
3. Model serving: how to size capacity
Capacity estimation starts from three quantities: concurrent questions, average context length, output length.
Step one: estimate VRAM. Model weights take one part; KV cache takes another. Longer context and higher concurrency grow the cache. Many plans size VRAM from weights alone and discover after launch that concurrency does not hold.
Step two: estimate throughput. Concurrency per GPU and time to first token are in tension: batching raises throughput and increases first-token wait. What the product demands of "how fast the first token appears" usually matters more to experience than tokens per second, and that must be explicit in the plan.
Step three: estimate cost. On-premise cost is not only GPUs; it includes power, data-centre space, operations staff and model upgrades. A common error prices hardware only and ignores three years of operating cost.
One more: the reranker. Reranking before generation strongly affects answer quality, and it consumes both VRAM and latency budget. Put reranking and generation on the same resource sheet, or you end up with "generation is fast but the whole thing is slow".
4. Hybrid deployment and policy boundaries
On-premise does not mean entirely local. The usual shape is hybrid: sensitive data and retrieval on-premises, general capabilities calling external services, or the reverse.
Before designing a hybrid, answer three questions. Which data absolutely cannot leave the jurisdiction? Which queries may be sent out after redaction? What granularity must the audit retain?
These three decide the split point. The split point is usually not "where the model runs" but "at which step data gets aggregated". Once aggregation happens externally, the risk exists even if the raw text stayed local.
Audit granularity must be decided early. Recording only "who queried what" is usually insufficient; you also need "which passages were returned and which documents they came from". Those two are worth more in a dispute and cost more in storage, so retention periods belong in the design phase.
5. Five evaluations before any launch commitment
Recall evaluation. Build a test set from real questions and measure recall and ranking quality. Without this, generation quality cannot be judged.
Hallucination evaluation. Construct questions with no answer in the corpus and see whether the system admits it does not know. This exposes orchestration problems best.
Permission evaluation. Construct out-of-scope queries, verify the block fires, and check audit log completeness.
Capacity evaluation. Load test at 1.5× peak concurrency and watch where first-token latency and error rates bend.
Degradation evaluation. Simulate model unavailability, retrieval timeout and external dependency failure, and verify graceful degradation rather than total outage.
A last practical note on sequencing: buy hardware and licences against a written capacity model, not against a vendor demo. Demos run on clean corpora with friendly queries; production runs on duplicated documents, stale permissions and multi-language content. Every layer sized separately, then a pilot on a representative sample, is what separates a launch date that holds from one that slips twice.
FAQ
Q: Do we need GPUs on premises, or can we use hosted inference?
Depends on the residency constraint, not on performance. If sensitive passages cannot leave, inference must be local; otherwise a hybrid with redaction plus external inference is usually cheaper.
Q: How many documents can one GPU handle?
The wrong question. Document count sets index size; concurrency and context length set inference load. Size from peak concurrent questions.
Q: Is RRF good enough, or do we need a learned ranker?
RRF is good enough to launch and to explain. Move to a learned ranker when you have click or judgement data and a way to measure lifts.
Q: How do we keep two indexes consistent?
Single write path, versioned documents, and a monitor for the window where one path lags. Short windows are acceptable; unmonitored ones become "random failures".
Q: What is the most under-budgeted item?
Operations and model refresh. Three-year operating cost typically rivals the hardware line.
Related Reading
- Hybrid search architecture: BM25 plus vector with RRF
- Enterprise search API integration guide
- GEO diagnostic tools compared
Data Sources
- What is Azure AI Search — Microsoft Learn — hybrid retrieval, semantic ranking and index freshness in a managed service.
- k-nearest neighbor search — Elastic documentation — dense vector retrieval mechanics and index configuration.
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020 — the retriever-plus-generator architecture on-premise deployments must reproduce.