Every quarter a small business or a city department asks us to build them “a chatbot that answers from our documents.” Most of them have somewhere between 200 and 1,000 files. And about half the time, our honest answer is: not yet, and maybe not ever the way you’re picturing it. RAG is a real pattern that solves a real problem, but it is also the most over-scoped request we get from smaller orgs. Here is how we decide — the same way we decide when we scope the work.

What RAG actually does

Retrieval-augmented generation is a pattern, not a product. When someone asks a question, the system searches your documents for the most relevant passages, then hands those passages to a language model and tells it to answer using only what it was given. The model writes the answer; your documents supply the facts.

The point is grounding. A raw language model answers from its training data and will invent a plausible-sounding policy number if you let it. A RAG system answers from your policy document and cites the passage it used. For an SMB, the value is rarely “chat with our files” — it’s “stop losing twenty minutes a day to questions already answered in a folder nobody can find.”

That’s the upside. The catch is that retrieval has a floor of complexity. You’re now running an embedding step, a vector index, a chunking strategy, a reranker, and an eval set — five moving parts that all have to stay healthy. Under a certain corpus size, that machinery costs more to own than the problem costs to suffer.

The small-corpus question

Here’s the part vendors skip. Below roughly 1,000 documents, RAG is often the wrong tool, and two simpler approaches usually win.

Just put the documents in the prompt. Modern long-context models read a lot of text per call. If your entire relevant corpus is a few hundred pages, you can paste it straight into the system prompt and skip retrieval entirely. No vector store, no chunking, no reranker — nothing to maintain. Anthropic’s prompt-engineering guidance calls this out directly: when the data fits, stuffing context is simpler and more accurate than building a pipeline. It costs more per query, but for low query volume that math favors you for a long time.

Buy good search instead. A lot of “we need RAG” requests are really “our search is broken.” If your documents already live in SharePoint or Google Drive and people just can’t find them, the fix may be tagging, a cleaner folder structure, or turning on the search you already pay for — not a new AI system. We’ve closed more than one engagement by fixing a SharePoint taxonomy and walking away.

So when does RAG win inside that 100-to-1,000 band? When two things are true at once: the same questions get asked over and over, and the corpus is too big or changes too often to keep re-pasting by hand. A 600-document HR policy library that 300 employees query daily is a fair RAG candidate. The same 600 documents read once a quarter by one compliance officer is not — just give that person better search.

When RAG beats fine-tuning

People conflate these constantly, so it’s worth being blunt. RAG and fine-tuning solve different problems, and for almost every SMB the answer is RAG.

Fine-tuning changes how a model behaves — its tone, its format, its default reasoning style. It does not reliably teach a model new facts, and it bakes whatever you trained on into a frozen snapshot. RAG changes what a model knows at answer time by feeding it fresh, specific passages on every query.

	RAG	Fine-tuning
Best at	Injecting current, specific facts	Shaping tone, format, behavior
When a doc changes	Reindex one file in minutes	Retrain the whole model
Source of an answer	A citable passage you can point to	Opaque weights, no citation
Cost to start	Low to moderate	Higher; needs labeled data + GPUs
Wrong-answer risk	Bounded by what you retrieved	Confident hallucination, no source

If your problem is “the model needs to know our current refund policy,” that’s RAG — and reindexing a changed policy is a five-minute job, not a retraining run. Fine-tuning earns its cost when you need consistent output structure at scale, which is rare for an org under 1,000 documents.

The cleanup-first rule

This is the part that decides whether a RAG project succeeds before a single line of code gets written, so we made it a rule: clean the corpus first, or don’t build RAG at all.

RAG retrieves whatever you point it at. Point it at a SharePoint with five conflicting versions of the same SOP, three of them outdated, and you don’t get a smart assistant — you get a confident liar. The retrieval step will dutifully surface the wrong 2019 policy, and the model will cite it with total composure. Garbage in, confidently-wrong out.

For Bay Area public-sector teams the stakes go past embarrassment. A city department under California records-retention rules can’t have an assistant quoting a superseded ordinance to a resident. Compliance-aware retrieval means the index knows which document is current and which is archived — and that only works if a human did the work of marking them. We require at least one round of document cleanup in every RAG engagement, and we won’t ship for an org that won’t fund it.

A quick way to gauge readiness — call it the is-your-corpus-RAG-ready checklist:

1. Single source of truth per topic. One current version of each policy, not five drafts.
2. Recency is knowable. Every document has a clear “last reviewed” date.
3. Machine-readable text. No scanned PDFs that are really just images.
4. Access boundaries are clear. You know which docs which roles may see.
5. Someone owns it. A named human keeps the corpus current after launch.

Three or fewer of these, and your money is better spent on cleanup than on a pipeline.

The pipeline, briefly

If you clear the checklist, here’s what the build actually involves — four stages, none optional.

• Chunking. Documents get split into passages. Too big and retrieval gets noisy; too small and you sever the context that makes a passage make sense. Chunking is where most public RAG implementations are quietly broken, and it’s tuned to your document shape, not copied from a tutorial.
• Retrieval. Each chunk is embedded into a vector and stored in an index — Postgres with pgvector is plenty for this scale. A query gets embedded the same way; the index returns the nearest passages.
• Reranking. The first-pass results get re-scored by a model that reads query and passage together, so the genuinely best passages rise to the top. This single step moves answer quality more than swapping the underlying model.
• Citation. Every answer points to the source passage it used. No source, no answer. This is non-negotiable in our systems and the first thing a compliance officer checks.

How to know it works

You cannot eyeball a RAG system into correctness, and “it seemed fine in the demo” is how teams ship a liar. You measure it.

Build a set of 50 to 100 real questions from the people who’ll actually use the system, with the correct source document noted for each. Then measure two numbers. Recall asks: of the questions that have an answer in the corpus, how often did retrieval surface the right passage? Precision asks: of the passages it surfaced, how many were actually relevant? Low recall means people get “I don’t know” when the answer exists. Low precision means the model is reading junk and may answer from it.

We build that eval set with your team, run the pipeline against it, and share the eval itself — not just a score. If recall sits at 60% on launch day, you’ll see it, and you’ll know exactly which questions are failing before a single employee gets a wrong answer. This is the same eval-first discipline we apply to the agents we build: measure first, ship second.

We run this pattern on ourselves

We’re not theorizing. AVA, the assistant our own MSP runs daily, retrieves over years of historical support tickets — a RAG pattern over a structured data source — to draft notes grounded in what we actually did last time, with the source ticket cited. We cleaned that corpus, built the eval, and watched the precision numbers before we trusted it. That’s why this guide is opinionated instead of generic: every rule here is one we paid for first.

Common questions

Is RAG worth it for a small business with only a few hundred documents? Often no — not as a built pipeline. Under about 1,000 documents you can frequently paste the relevant text straight into the prompt, or just fix the search you already have. RAG earns its keep when the same questions repeat at volume and the corpus is too large or too fast-changing to handle by hand.

What’s the difference between RAG and fine-tuning for my use case? RAG feeds the model fresh facts at answer time and lets you cite the source; fine-tuning reshapes the model’s tone and behavior but won’t reliably teach it new facts. For nearly every org under 1,000 documents, “the model needs to know our current information” means RAG, not fine-tuning.

Why does everyone say RAG hallucinates if it’s supposed to use my documents? Because it retrieves whatever you point it at. A messy corpus with outdated, conflicting files gets surfaced and quoted with full confidence. That’s the cleanup-first rule: a RAG system is only as honest as the documents behind it.

How do I know if my documents are ready for RAG? Run the is-your-corpus-RAG-ready checklist above: one current version per topic, knowable recency, machine-readable text, clear access boundaries, and a named owner. Score three or fewer and your budget belongs in cleanup, not a pipeline. The AI Readiness Assessment walks through this for your specific situation.

Can RAG respect records-retention and access rules for a public-sector team? Yes, if the index knows which documents are current versus archived and which roles may see what — which means a human has to mark that up first. For Bay Area public-sector teams, compliance-aware retrieval is exactly the part that can’t be skipped, and it’s the first thing we scope.

Most orgs under 1,000 documents don’t have a RAG problem; they have a corpus problem wearing a RAG costume. Fix the documents, try stuffing the context, fix the search — and if the questions still pile up faster than people can answer them, then you have earned the pipeline. Knowing which of those four situations you’re actually in is the entire job, and it’s a far cheaper conversation to have before the build than after.