In high-complexity institutions, the knowledge is already documented. The problem is that nobody knows where it lives, or when it expires.
The quality department as an accidental help desk
A large public health institution, with hundreds of employees spread across clinical, administrative and technical departments, maintains a living corpus of internal documents. Standard Operating Procedures, institutional policies, contingency plans, technical manuals, protocols, work instructions. The whole set existed. It was written down, went through periodic review, had an owner and an expiration date. None of that was new.
What was new was the volume of requests hitting the quality department every week: "where is this document?", "what's the current version of the X protocol?", "is the patient identification SOP still valid?". Questions whose answers were sitting in Google Drive, sometimes in a folder two clicks away. By osmosis, the quality team had become the human search interface for a collection nobody knew how to navigate.
That is the invisible cost of a knowledge base without a retrieval engine. The knowledge isn't wrong. It's inaccessible at operational scale.
The client framed the problem in plain terms: employees need to be able to ask the system and get an answer. Without depending on one key person. Without opening Drive folders and hoping for luck.
The temptation of textbook RAG, and why we paused
The canonical RAG (Retrieval-Augmented Generation) architecture for this scenario would be obvious: index the PDFs in a vector database, generate embeddings for each chunk, search by semantic similarity to the query, pass the relevant chunks to the LLM, return an answer. It's the flow most tutorials show. It's also the flow that, in this specific case, would have created two serious problems right out of the gate.
First crack: embeddings don't answer governance questions
Semantic search is great for "what does this document say about patient identification." It's useless for "which documents expire in the next 30 days" or "which procedures fall under the quality manager's responsibility."
Those last two aren't content questions. They're metadata questions. The answer isn't in the body of any document. It's in the next_review_date field in the header and the responsible field in the footer. A vector database doesn't index those fields in a queryable way. For this class of query, embeddings simply aren't the right mechanism.
And this class of query matters a great deal for compliance management. Knowing what's expiring, knowing what's out of date, knowing who owns what: these are exactly the questions an auditor asks. If the system can't answer them, half the business value disappears.
The same question against two indexes: one semantic, one structured
Second crack: continuous reindexing as an operational tax
Active corpora change. SOPs get revised. New protocols come in. Old versions get archived. In a vector database, any change to a document requires rechunking and re-embedding the entire file and, depending on the implementation, eventually reindexing the whole ANN (Approximate Nearest Neighbor) index.
For a corpus of 600 documents with frequent updates, that cycle becomes significant operational overhead. It isn't intractable, but it is real technical debt: API costs for embeddings, sync latency, the risk of a stale index during the reindexing window.
Every applied AI project, in practice, requires a layer of observing the process as it exists today. Before writing a line of code, look at what's already there.
What observation revealed: the titles are already the index
Before choosing the architecture, we spent time looking at the corpus as it was. Not as we wished it were. As it was.
The discovery was straightforward: this institution's documents had been written to be found. Titles were descriptive by institutional convention. "SOP 01 · Patient Identification Policy". "Work Instruction 14 · Managing Companions During MRI Procedures". Each title worked as a summary of its content.
If an employee asks about patient identification, the title already points to the relevant document. We don't need a 1,536-dimension vector to figure that out. We need decent lexical search over the title, reinforced with a summary generated once, at indexing time.
The logic of the decision became clear: the investment worth making was in the quality of the catalog, not in embedding infrastructure.
The titles are already the index: BM25 over the catalog makes embeddings unnecessary
The architectural decision: the catalog as a filterable dimension, BM25 as the search engine
We dropped the vector database. In its place, we built a structured catalog in JSONL (one entry per document) with the following fields extracted at indexing time:
title,doc_type,department: identity and taxonomysummary,tags,entities: generated by an LLM (GPT-4o-mini, because it's 10× cheaper than Sonnet for batch indexing tasks)next_review_date,last_review_date,issue_date,version: extracted via regex from the headerresponsible,approver: extracted via regex from the footer, with an LLM fallback when the regex fails
Each document is read exactly once during indexing. The extracted text is cached locally. Reindexing an updated document means reprocessing one JSONL entry, not recomputing an embedding and updating an ANN index.
Search uses rank-bm25 over a BM25 text corpus built from title + summary + tags. BM25 is deterministic: the same query returns the same ranking every time. It's debuggable. And for a corpus with controlled vocabulary (consistent clinical and institutional terminology), it performs well.
The taxonomy that can't hallucinate
The doc_type field (whether a document is an SOP, a policy, a managed protocol, a work instruction) comes from the folder where the file lives in Drive. Not from the LLM.
TAXONOMY_PREFIXES = {
"pop": ["POP"],
"politica": ["POLÍTICA INTERNA"],
"plano_contingencia": ["PLANO DE CONTINGÊNCIA"],
"instrucao_trabalho": ["INSTRUÇÃO DE TRABALHO"],
"protocolo_gerenciado": ["PROTOCOLO GERENCIADO"],
"protocolo_tecnico": ["PROTOCOLO TÉCNICO"],
"manual": ["MANUAL"],
"tutorial": ["TUTORIAL"],
"programa_interno": ["PROGRAMAS"],
"cartilha": ["CARTILHAS INFORMATIVAS"],
"documento_corporativo": ["DOCUMENTOS CORPORATIVOS"],
}
Order matters: protocolo_gerenciado (managed protocol) has to be tested before protocolo, or the generic prefix swallows everything. Matching is by prefix, with case and accent normalization. PDFs in folders outside the taxonomy are silently discarded, which is intentional behavior to keep the index clean.
The result: the system cannot classify an SOP as a manual. The type comes from the folder structure, which the quality managers already maintain. There's no room for hallucination in a field derived deterministically from the filesystem.
The taxonomy that can't hallucinate: doc_type derives from the folder structure
The agentic MapReduce architecture
The context window problem in RAG is real. If we retrieve 10 full documents and send them to an LLM, we're talking about ~50,000 tokens. Answer quality drops. Cost goes up. And the orchestrator loses reasoning capacity for synthesis because the context is clogged with raw text.
The pattern we adopted mirrors the logic of systems like Perplexity and OpenAI's Deep Research: an orchestrator that never reads full documents, plus N disposable sub-agents that each read one document and return only the relevant extraction.
Agentic MapReduce pattern: orchestrator + parallel sub-agents
The orchestrator receives the user's question, decides which documents deserve a full read, and hands each one to a separate sub-agent. Each sub-agent has a clean context: one document, one question. It returns only the relevant passage, not the whole PDF. The orchestrator combines the extractions and synthesizes the answer.
The practical result: the orchestrator processes ~25× fewer tokens than it would if it received the full documents. The context window stays free for reasoning instead of clogged with raw text.
The 6-node pipeline
LangGraph pipeline: 6 nodes with second pass and fast path
The pipeline has six nodes. The sequence matters as much as the nodes themselves.
analyze_query is where Claude Sonnet receives the raw question and decides what to do with it. It classifies the query into one of five categories (simple, aggregation, comparison, exploratory, temporal) and extracts the structured filters implicit in the natural language. "Documents owned by the quality manager expiring within 90 days" becomes { temporal_filter: "expiring_soon", responsible_filter: "quality manager", doc_type_filter: null }.
search_catalog applies those filters to the JSONL catalog and runs BM25 over the resulting set. No LLM call here. It's deterministic code. A relevant edge case: when the query is purely about metadata (no content keywords), BM25 with an empty query returns arbitrary scores. The node detects this and does a full scan with the filters applied and a uniform score. Correct and deterministic.
triage_candidates is the branching node. The LLM evaluates the candidates returned by BM25 and decides: is it worth reading the full documents, or does the catalog already have enough information to answer? For temporal queries and aggregations over more than eight documents, the fast path kicks in: we skip document reading and answer straight from the metadata.
dispatch_sub_agents dispatches the sub-agents in parallel. Each one receives the text of one document plus the original question plus the metadata context. It returns a compressed extraction with a relevance rating. The orchestrator never sees the full document.
check_second_pass checks whether the retrieved documents cross-reference other documents that weren't in the initial result. If they do, the pipeline loops back to node 4 for a second round. It's a conditional loop in the LangGraph graph, which is why the pipeline is a graph and not a linear chain.
synthesize_answer combines the extractions, filters out the low-relevance ones, and builds the final answer with citations and direct links to the PDFs in Drive. One implementation detail worth mentioning: when the set of extractions comes back empty or with zero relevance, the pipeline short-circuits without calling the LLM. The "not found" message is built in code, not generated by a model. Cheaper, more consistent.
The metadata layer: where the real value lives
Extracting responsible and approver deserves its own paragraph, because it was the point of greatest technical resistance.
The canonical footer on the institution's documents is a three-column table: "ELABORADO POR", "VERIFICADO POR", "PUBLICADO POR" (prepared by, reviewed by, published by). Each column holds the name and department of the person responsible. The problem is that once extracted from the PDF with PyMuPDF, that text becomes a linear sequence with no column structure, and adjacent columns bleed into each other.
The strategy: regex with negative lookahead on the neighboring labels to prevent bleed between columns, applied to the first and last 3,000 characters of the text (the footer can appear on any page, but it's typically the last one). If the regex fails on any field, we call GPT-4o-mini with a short prompt asking only for the two names in JSON. The result carries a source: "regex"|"llm"|"mixed"|"none" field for telemetry.
In the initial sample: ~50% regex hit rate. The remaining 50% were typically short forms with no standard footer, where None is the correct result, not an error.
The practical consequence is that queries like "documents owned by the quality department manager" work. The responsible_filter is applied to the catalog as a substring match, and the relevant set of documents comes back without the LLM having to infer authorship from the body text.
The bug we learned not to make again
During development, we ran into a defect pattern worth documenting, because it's subtle and recurring in RAG systems with metadata filtering.
The filter worked. The system found the right documents. But the synthesized answer didn't mention the owner, even when the user had explicitly asked about it. The query correctly narrowed the set down to a single document owned by a specific person, but the synthesis prompt assembled the catalog_text without the responsible field. The synthesis LLM could see the document, but it had no way to state "this document is owned by so-and-so," because that information wasn't in the context it received.
Every piece of metadata used as a filter criterion also has to be present in the synthesis context. Filtering and presentation are separate layers, and they have to be aligned explicitly.
The fix touched three places: dispatch_sub_agents for the temporal path, synthesize_from_catalog, and the regular sub-agent prompt. It's a defect that doesn't show up in ordinary content tests, only in governance tests. Exactly the queries that matter most to the client.
What we'd do differently
RBAC in the pipeline. The current system has no per-user access control. Any authenticated employee can ask about any document. For HR documents or sensitive data from specific departments, that would need to change: an identity filter node in the OrchestratorState before dispatch. Leaving out RBAC was a conscious decision about initial scope, not an oversight.
Domain-agnostic vocabulary in analyze_query. The current prompt has heuristics for synonyms from clinical and quality vocabulary. If the system is extended to other departments with very different vocabulary (HR, legal, finance), those prompts will need revision or per-domain variants. That's not trivial work.
Incremental sync limitation. Syncing with Drive uses the Changes API, which detects new, modified or deleted files. It can't re-derive the doc_type from the folder structure, because the API only returns file metadata. If a document is moved from a type A folder to a type B folder, the sync doesn't detect the type change. A full index rebuild is required in those cases. It's a known limitation worth monitoring in corpora that get reorganized often.
How to replicate this for your own corpus
If you have a corpus of internal documents in an organization with the same characteristics (descriptive titles, a folder structure with conventions, metadata in headers and footers, continuous updates), this checklist covers the steps:
1. Audit the corpus before writing any code. Look at 50 random documents. Do the titles already summarize the content? Do the headers have expiration dates? Do the footers name owners? If so, a structured catalog will outperform embeddings for most queries.
2. Map the taxonomy before touching indexing.
Walk the folder structure and list every naming variant per document type (typos included: every organization has them). Define TAXONOMY_PREFIXES before writing a single line of indexing code. Order specific before generic.
3. Use a cheap model for indexing, a good model for answers. Indexing is a batch process that tolerates the occasional error. GPT-4o-mini costs ~10× less than Sonnet to generate summaries, tags and entities. The orchestrator and the sub-agents, which interact directly with the user, deserve the better model.
4. Implement filters as a mirror of the catalog fields.
For every filterable catalog field (doc_type, responsible, temporal), add the corresponding filter to the orchestrator state. And make sure the field arrives serialized in the synthesis prompts, not just in the search filter. This is the most common mistake.
5. Explicitly test governance queries, not just content queries. "What does procedure X say" is the obvious query. But "which documents expire in 60 days" and "everything owned by such-and-such manager" are the queries worth the most to the client. Cover both axes in your tests.
6. Build the fast path from day one. For temporal queries and aggregations over many documents, answering from metadata alone is faster, cheaper and often more accurate than sending sub-agents to read documents. The routing is simple, and the investment pays off.
7. Short-circuit "not found" in code. When the set of extractions is empty or below the relevance threshold, build the message in code. Don't spend an LLM call generating "sorry, I couldn't find anything relevant." It's more consistent and cheaper.
The principle that generalizes
This system isn't about RAG. It's about recognizing that, for most real-world enterprise corpora, the value isn't in more sophisticated embeddings. It's in well-modeled metadata and a layer of deterministic filters ahead of semantic retrieval.
Before choosing the tool, observe the process. The corpus you're handed already has structure. Your job is to expose that structure, not to try to replace it with inference.
Embeddings are useful when content is the only index available. When the organization already maintains a taxonomy, owners, dates and document types (and any organization that has survived more than five years has that metadata), the right investment is in retrieving it reliably and making it queryable in natural language.
The practical result: a system that answers both "what does the protocol for managing companions during MRI say" and "which of the quality manager's documents expire in the next 90 days" with the same reliability, the same response time, and a direct citation to the original file.
What's the difference between RAG with a vector store and the BM25 + structured catalog approach?
BM25 is a lexical ranking function: it scores documents based on the frequency and distribution of the query terms. It's deterministic, has no API cost, and is debuggable. Vector stores use cosine similarity between embeddings, which captures semantics but not structured filters.
For corpora with descriptive titles and rich metadata (dates, owners, types), BM25 over a catalog plus deterministic filters performs as well as or better than embeddings on most practical queries. The exception is when the query and the document use completely different vocabulary. In those cases, embeddings have the edge. The choice depends on the corpus.
Why use two LLM providers (Anthropic and OpenAI) instead of one?
The decision is about cost and quality at the same time. Indexing is a structured batch task: generating a summary, extracting tags, identifying entities. GPT-4o-mini costs ~10× less than Claude Sonnet for this class of task, and the quality difference is irrelevant when the output is a JSON field with 3 tags.
Orchestration and synthesis are different. Query analysis has to classify the type, extract implicit filters, and handle ambiguity in Brazilian Portuguese. The final synthesis has to be faithful to the text of the documents and coherent. That's where Sonnet justifies its cost. The general rule: a cheap model where the task is structured and batch, a good model where the user will read the output.
Can the system answer questions that require reading multiple documents at once?
Yes, through the MapReduce pattern. Sub-agents read documents in parallel and return compressed extractions. The orchestrator receives only the relevant passages from each document, not the full documents, and synthesizes the answer from those extractions.
This solves two problems: the context window limit (you can't send 10 full PDFs to an LLM and keep quality) and cost (sub-agents with clean contexts process ~25× fewer tokens than an orchestrator that reads everything). The query "compare protocol A with protocol B" works exactly this way.
How does automatic sync with Drive work?
The Google Drive Changes API returns a list of files created, modified or deleted since the last sync, identified by a token persisted locally. The indexer processes only the delta. It doesn't rebuild the whole catalog on every sync.
The known limitation is that the Changes API returns only file metadata, not the file's position in the folder tree. If a document is moved between folders of different types, the sync doesn't detect the doc_type change. In those cases, a full rebuild is required. It's an acceptable trade-off for corpora that are rarely reorganized.
What is the "catalog-only" fast path, and when does it kick in?
For certain classes of query, the metadata catalog already contains the complete answer, with no need to read the documents' content. Temporal queries ("which documents expire in 30 days"), aggregations over many documents ("list all the manuals from department X"), and queries purely about ownership are the typical cases.
In those cases, the triage_candidates node triggers the fast path: the pipeline skips dispatch_sub_agents and synthesizes the answer directly from the catalog's summaries and metadata. It's faster (no PDF read I/O), cheaper (no sub-agent calls), and often more accurate (the catalog has exactly the fields the query needs, with no content noise).