Agentic Search Architecture: Five Patterns That Survive Production

Summary: The reasoning loop is the cheap part. What separates a working agentic search system from a demo is the control loop around it: typed tools, context isolation, budgets, and an evaluator that can send work back.

Key Takeaways

The gain is real and measured. Agentic retrieval improves nDCG at 10 by 8.7 points using the same embedding model as standard retrieval.
The cost is real too. The same study reports 107.4 seconds per query versus 0.67 seconds for a single retrieval pass, and 764.1K input tokens per query.
Plan-and-Execute beats ReAct when sub-queries are independent. Latency lands near twice standard RAG, versus four times for ReAct on parallelisable questions.
Context isolation is the pattern that prevents error propagation. Hierarchical host-manager-worker designs keep a bad sub-query from poisoning the final answer.
Most production failures are budget failures. A loop with no token cap and no wall-clock cap is not an architecture, it is an incident waiting for a query.

I. What "agentic" actually adds

Classic retrieval is one pass: embed the query, fetch top-k, rerank, answer. Agentic search adds a decision layer — the system decides what to search, how many times, and whether the evidence it got is good enough to answer.

That decision layer buys generalisation. On out-of-domain tasks, fixed retrieval pipelines degrade while the same agentic loop stays competitive across two different leaderboards. It also buys a specific, measurable quality gain: the 8.7-point nDCG at 10 improvement holds with the embedding model held constant, which means the gain comes from query formulation and iteration, not from a better encoder.

The price is visible in the same measurement: 107.4 seconds per query against 0.67 seconds, and 764.1K input tokens with 5.8K output tokens. Two orders of magnitude in latency and a five-figure token bill per question is not a rounding error — it changes what the system can be used for, and it is the reason architecture matters more than the model choice.

II. Five patterns, and when each one is wrong

Pattern	Shape	Best for	Fails when
Plan-and-Execute	One planning step produces typed sub-queries, then parallel execution	Multi-part questions, dashboards, compliance checks	Sub-queries depend on each other's results
ReAct loop	Think-act-observe repeated until the answer is supported	Exploratory research, unknown corpora	Latency budget under five seconds
Hierarchical workers	Host decomposes, managers aggregate, workers search in parallel with isolated context	Large corpora, long sessions	Orchestration overhead exceeds retrieval time
Retrieval-as-tool	The model calls search, filters, and SQL as typed functions	Mixed structured and unstructured data	Tool schemas drift from the index schema
Evaluator-gated	A checker scores evidence, rejects and re-plans	High-stakes answers, regulated domains	The evaluator shares the planner's blind spots

The first pattern is the workhorse. When a question decomposes into independent parts, planning once and executing in parallel keeps latency near a small multiple of standard retrieval instead of letting it grow with the number of reasoning steps.

ReAct is better when you genuinely do not know what to look for. It is also the pattern most likely to blow a latency budget, because each observation costs a model call.

Hierarchical designs exist for a boring reason: context saturation. If every sub-query's results land in one window, the later reasoning degrades. Isolating worker context and aggregating at a manager layer keeps the final synthesis clean, and parallelism absorbs part of the added cost.

Retrieval-as-tool is what most teams actually ship, because it lets the same agent answer "what is the policy" and "how many tickets breached SLA last month" without two pipelines. The failure mode is schema drift: the tool description says one thing, the index says another, and the agent silently searches the wrong field.

Evaluator-gated designs add a second model that scores whether the retrieved evidence supports an answer. Useful in regulated settings, but only if the evaluator is trained or prompted on a different failure distribution than the planner — otherwise both miss the same thing.

One more pattern deserves a name because it is what actually keeps cost sane: caching at the sub-query level rather than the answer level. Two questions that share a party name and a date range can reuse the same retrieval result if the cache key is the normalised sub-query plus the index version. Answer-level caching fails because the final answer differs on every phrasing; sub-query-level caching hits often enough to matter, and it is invalidated cleanly when the index changes.

The counter-argument is that cached retrieval makes the loop lazy. It does, in the sense that the planner reuses a prior result instead of reformulating — which is the behaviour you want, because reformulating a working sub-query burns tokens for no gain.

III. The control loop we build

State outside the model. Keep the plan, the executed sub-queries, and the evidence set in a store the orchestrator owns. Models rewrite their own scratchpads; the orchestrator must not depend on that.
Typed tools with validated schemas. Every tool call is validated against a schema before execution, and the schema is generated from the index definition so it cannot drift.
Two hard budgets. A token cap and a wall-clock cap, both enforced by the orchestrator, not by the prompt. When either is hit, return the best-supported partial answer with a confidence note.
Evidence ledger. Every retrieved passage gets an ID, a source, a timestamp and a score. The final answer cites IDs, not prose. This is what makes the answer auditable.
Evaluator with a re-plan path. If evidence fails the check, the loop returns to planning with the failure reason as input, once. A second failure returns a refusal with what was missing.
Observability at step granularity. Log per-step latency, token spend and retrieved counts. Aggregate traces hide the one step that costs 80% of the budget.

Point three is where most teams under-invest, and it is cheap. A prompt that says "be efficient" is not a budget; a counter that terminates the loop is.

IV. What we changed in a live contract-search deployment

A representative case: an enterprise contract repository, 1,400,000 documents, average query a legal question with two or three conditions. The first agentic version used a ReAct loop and hit a 9-second p95 with a token spend that made the finance team ask questions.

We replaced it with Plan-and-Execute. One planning call produced three to five typed sub-queries (party, clause type, date range, jurisdiction), executed in parallel against the same hybrid index, then a single synthesis step. p95 fell to 3.4 seconds, and nDCG at 10 improved because the sub-queries split the conditions instead of hoping one embedding captured all of them.

Two things we did not expect. First, the planner needed a hard cap of five sub-queries — beyond that, recall stopped improving and latency grew linearly. Second, the biggest win came from the evidence ledger, not from the loop: once answers cited passage IDs, the legal reviewers' correction rate dropped, because they could check the exact clause instead of re-reading the answer.

A note on tool schemas, because this is the failure that hides longest. When a tool is described in prose ("search contracts by counterparty"), the model will pass a free-text counterparty and the index will match it against the wrong field — a party name against a document title, say. Define the tool with an enum of index fields and a validated type per field, and let the model choose among them. Prose tools produce plausible queries against the wrong fields, which is worse than a failed call, because the evidence looks relevant.

V. Boundaries and the mistakes that look like sophistication

More agents is not more capability. Beyond three to four workers, orchestration overhead dominates and the answer quality plateaus.
Parallelism helps only independent work. Fan-out over dependent sub-queries produces plausible-looking answers built on the wrong evidence.
The 107-second figure is a research setting, not a service level. Production deployments hit it only on unbounded loops; the fix is budgets, not a faster model.
Streaming hides latency but does not remove it. A user watching tokens arrive still abandoned the query at 30 seconds in our logs.
Evaluators inherit planner bias. Use a different model family, or a rule-based checker, for the gate.
Cost per query is the real constraint. At 764.1K input tokens per query, a 10,000-queries-a-month deployment is a capacity planning problem before it is a quality problem.

A last structural point about evaluation: score the loop on task success and evidence coverage, not only on ranking metrics. A system that returns a beautifully ranked set of passages which do not jointly answer the question has improved nDCG and failed the task. Track both, and treat a rise in ranking quality with a flat task success rate as a sign that the planner is producing sub-queries the index can rank but cannot satisfy.

FAQ

Q: Do we need an agent at all?
A: No, if one retrieval pass answers the question. Measure the share of queries that need a second sub-query; below 15%, a better reranker is cheaper.

Q: Which pattern first?
A: Plan-and-Execute with typed tools. It is the one that scales across query types with predictable latency.

Q: How do we cap cost?
A: Hard token and time budgets in the orchestrator, plus a per-tenant monthly ceiling with a graceful degradation path.

Q: Is context isolation worth the complexity?
A: Yes above roughly 20 retrieved passages per session; below that, a single window with a good summariser is fine.

Q: How do we evaluate the loop?
A: Fixed query set with graded relevance, nDCG at 10 for ranking quality, plus task success and per-step latency for the loop.

Q: What about security?
A: Tool-level authorisation, not prompt-level. The agent should be unable to call a tool the user's role cannot use.

About Tengence

Tengence Search ships agentic retrieval as a governed capability: typed tools, hard budgets, an evidence ledger and per-step observability, so the loop's quality gain survives production latency targets.

Data Sources

Beyond semantic similarity: performance and costs of agentic retrieval — nDCG at 10 plus 8.7 points on the same embedding model; 107.4 seconds versus 0.67 seconds per query; 764.1K input and 5.8K output tokens.
Agentic RAG for enterprise production — Plan-and-Execute latency near twice standard RAG versus four times for ReAct on parallelisable queries.
InfoSeeker: hierarchical parallel agent framework — host-manager-worker decomposition with strict context isolation.
Production agentic search architecture — control loop, evaluator, runtime budgets and observability as the production layer.