The problem
Retrieval that works on a demonstration corpus often breaks at enterprise scale. A few hundred clean documents forgive a great deal. Millions of documents, many separate client deployments and heavy request volume expose every shortcut in the design.
Three failures are especially common. Flat chunking, which splits every document into equal pieces of text, discards structure that retrieval depends on. Text-only search misses information that exists only in figures, diagrams and scanned pages. And a single slow component becomes a bottleneck for every user at once.
What we built
A production-scale retrieval platform handling millions of documents across 160+ client deployments, designed to preserve document structure, search across modalities and hold up under heavy load. The work covered the system architecture, optimization of the language-model layer, and deployment.
How it works
Hierarchical ingestion
Documents are ingested with their structure intact: sections, subsections and the relationships between them. This matters because the right unit of retrieval varies. Some questions are answered by a single paragraph; others need a whole section, or need to know which section a passage belongs to. A hierarchy lets retrieval work at the right level of a document instead of on arbitrary fragments that may have been cut mid-thought.
Multimodal search
Much of the information in enterprise documents is not in running text. It lives in charts, tables, diagrams and scanned images. Image analysis is part of search, so content that exists only in a figure can still be found and used in an answer.
Retrieval at scale
A vector database serves retrieval. Documents and questions are represented as vectors, numerical representations of meaning, so the system can find passages that are relevant even when they do not share the question’s exact words.
A mobile chatbot
An integrated mobile chatbot brings the platform to users on their phones, which is where many people actually need an answer.
Designing for throughput
The platform is optimized for 100K+ requests per second. Throughput at that level has to be designed in from the beginning, because it shapes the architecture: how work is distributed, which results can be cached, and where the expensive steps sit in the request path.
The language model is usually the most expensive step, which is why optimizing that layer was part of the work. Retrieval narrows each question to a small set of relevant material before the model sees it, so the model does less work per request, and the system stays responsive under load.
Why many deployments is a different problem
Running across more than 160 client deployments is not the same as running one very large system. Each client’s documents must stay separate, each deployment must behave predictably, and a change made for one must not break another. A platform at this scale is as much an exercise in isolation and consistency as in raw performance.
What the results mean
The platform runs across 160+ client deployments and is optimized for 100K+ requests per second. The first measures breadth: the same design working across many organisations and document collections. The second measures headroom: the capacity to serve heavy traffic without degrading.
Where it can go wrong
- Retrieval quality at scale. The more documents there are, the more plausible-but-wrong passages compete with the right one. Relevance needs continuous measurement, not a one-off test.
- Stale indexes. Documents change. An index that falls behind returns confident answers from outdated material.
- Isolation errors. In a multi-client platform, leaking one client’s content into another’s results is the most serious possible failure, and it has to be designed out rather than tested for afterwards.
When this approach fits
This architecture is justified by scale: very large document collections, many separate deployments, or heavy traffic. For a single team with a modest document set, a far simpler retrieval setup is usually faster to build and easier to maintain.
Questions to ask before building one
If you are considering a system like this, these are the questions we would want answered first.
- Is your scale genuinely large? This architecture is justified by volume. A modest collection is better served by something simpler.
- How much of your knowledge is in images and tables? If a lot, multimodal search is essential rather than optional.
- How will you measure retrieval quality? Build a set of real questions with known answers before you launch, and keep using it.
- How must data be separated? If you serve multiple clients or departments, design isolation in from the start.