RAG vs Fine-Tuning: Choose the Right System for Work

The model has answered 38 support questions correctly, then invents a refund policy on question 39, which is where a prototype becomes engineering work because RAG, fine-tuning, and a bigger model create different systems to maintain.

I have no case-study evidence of shipping RAG or a fine-tuned model, so this is a reasoned guide informed by related work using constrained structured extraction, where I had to define exactly what a model could claim, while RAG changes the evidence a model sees at request time, fine-tuning changes behaviour through examples, and prompting plus structured extraction changes neither but narrows the job until a general model can do it safely enough.

Start with the failure, not the fashionable component

“RAG vs fine-tuning” describes implementation choices before the failure, so capture five real bad requests - user input, sources available then, wanted and actual answers, and why the result failed - because “the model hallucinated” hides the distinction.

Was information absent, buried in irrelevant text, or retrieved from the wrong document; did retrieval cut the qualifying sentence; did the model ignore an instruction or return valid-looking JSON with a nonsensical quantity; or was the request inherently uncertain, with no honest product response?

For a false claim about a current internal policy, inspect the context because you cannot fine-tune every paragraph revision: retrieve the current policy, show the passage, and refuse when the source is missing; if the model sees the right policy but produces prose instead of a discriminated JSON object, the output contract is failing, while “Chicken wrap, chips, maybe a sauce” requires an uncertain calorie estimate in a known shape, not a private document corpus.

Delay architecture until a small failure set reveals a pattern, because twenty uncomfortable examples - stale information, near-duplicate documents, hostile instructions in a source file, short questions with ambiguous referents, and questions whose answer is “I cannot establish that from these sources” - are enough to stop a loud demo deciding the system design.

Retrieval is for evidence that changes and needs a trail

RAG grounds responses in material too large, private, changeable, or specific for every prompt: an employee handbook changed after legal review, current billing rules, a contract clause, product documentation, incident runbooks, internal process notes, inventory records, or account-specific facts.

It gives the model a limited evidence packet and lets the user inspect its source, and freshness is why it is not a fine-tune: when a pricing rule changes on Tuesday, ingest the new source and remove or mark the old version, because a fine-tune holds statistical associations rather than a readable policy with a revision date, so it cannot identify an old example or cite a paragraph it did not see at runtime.

RAG is not a “vector database” checkbox, so define the source boundary - allowed repositories, folders, pages, database records, and attachments; their owners; and authority when versions conflict - because semantic similarity cannot resolve a decommissioned handbook against a current one; index status, date, owner, audience, and permission scope.

Extraction can corrupt everything invisibly because PDFs have duplicated headers, misordered columns, detached qualifying footnotes, and scanned pages requiring OCR, while chunking has the same consequence: a 150-word chunk can omit exceptions two paragraphs later and a 2,000-word chunk adds unrelated context and wrong-sentence opportunities, so prefer document structure such as a policy section, FAQ and answer, or procedure step with preconditions, then test questions requiring the qualifier; overlap can preserve a boundary or return four near-identical chunks, hide the useful one, and increase request cost.

Evaluate retrieval separately: for each question, label source passages supporting a correct answer and measure recall at $k$ by asking whether the right evidence appears in the top $k$, while inspecting precision too, or how much irrelevant material reached the model; then ask whether the answer is supported by the retrieved text, whether its citation supports its claim, and whether it states uncertainty for incomplete sources, keeping retrieval and generation failures in separate columns so prompt rewriting cannot conceal an index serving the wrong document version.

An index is a production dependency, not a one-time import

The first ingestion run is easy, but the next day somebody creates or edits a support article or policy, deletes a page, or moves a folder, and without propagation the index makes old information look authoritative.

A small source can be crawled on a schedule with a content hash per document, reliable webhooks can enqueue extraction and embedding after every change, and a database-backed source can emit an outbox event in the transaction changing its record; the contract matters - allowed staleness, deletion propagation, and proof that a chunk belongs to the expected version.

Retain source identifiers, source version or content hash, retrieval timestamp, chunking version, embedding model version, and indexing access rules because a disputed answer must be reconstructable; “the embedding search returned it” explains nothing.

Permissions are costly because an answer generator must not retrieve a human-resources document for someone who cannot open it; filtering after retrieval is too late if private text has reached a prompt or log, so apply the predicate in the retrieval query or an index partition, which can make a broad internal assistant far costlier than its prototype.

Plan for weak search because similarity scores are not calibrated truth meters: “I found no source I can rely on” should show search terms, offer nearest documents where appropriate, route to a human, or ask a narrow follow-up rather than become a polished guess.

RAG's costs extend beyond tokens - extraction failures, source ownership, re-indexing after chunking changes, embedding-model migrations, access-control changes, storage, observability, evaluation maintenance, and answer investigation - so it earns them only when current, traceable evidence is part of the promise.

Fine-tuning is for repeated behaviour you can show in examples

Fine-tuning helps when the base model has the right context but fails a stable behavioural task you can express in many good examples: classification, controlled house style, a fixed taxonomy, predictable tool order, or compact output too inconsistent or verbose for prompting.

It is not an answer to “our knowledge base changes every week”, because training on last week's documents creates a lagging source of truth, loses direct citations, and makes correction harder; retrieve current documents or pass the relevant record directly, since learned weights are not content management.

Before training, collect actual-task examples, define the input boundary and acceptance rule, remove duplicates, and split training, validation, and held-out tests by failure mode rather than a random shuffle that leaks near-identical rows, then version the data and record each example's reason for inclusion.

Compare a baseline prompt on the same held-out cases, and if it meets the requirement do not take on a model version; fine-tuning must improve a product metric - exact schema validity, classification error, human correction time, or cost at required volume.

Retraining means collecting and reviewing examples, filtering bad labels and sensitive data, preparing data, paying training compute, registering the model, running and comparing evaluations, changing deployment configuration, monitoring, and keeping a regression route back; training may be quick, but safe deployment work is not.

Categories are renamed, products added, writing policy changed, and user phrasing shifts, so when the task is unstable a fine-tune is a permanent training pipeline that can pay at volume, but the API does not make it free.

Do not fine-tune merely for valid JSON: use provider-supported structured output or a schema-constrained tool interface, validate in the application, and reject failed domain rules, because JSON Schema can permit a quantity no person accepts; syntax is a contract, not truth.

A narrow prompt and structured extraction often win

The closest first-hand parallel is Food & Fit, a nutrition-tracking product whose model work is neither RAG nor fine-tuning, because its text path sends a meal description to an OpenAI-compatible chat-completions endpoint with gpt-4o-mini, requesting calories, protein, carbohydrates, fat, and an itemised list through strict JSON Schema with no additional properties at temperature 0.3, while the iOS client displays the parsed estimate and saves it only after the user chooses to save it.

“Two eggs, toast and butter” needs a bounded, honest estimate rather than an internal-corpus search, while a barcode reads published nutrition fields from Open Food Facts - deterministic lookup, not a model guess - and the Worker times out after 25 seconds, limits attested callers to 100 daily AI requests and other callers or IP fallbacks to 10, and compresses photos toward 1 MiB before base64 submission, which are availability and cost controls around a metered request rather than model-quality controls.

The provider schema enforces response shape, but the Worker has no independent nutritional plausibility bounds, no reconciliation of item and meal totals, no committed model-quality evaluation set, and no token or cost telemetry evidenced in reviewed code, so a valid object can still be a bad food estimate; improve this prompt and extraction contract before retrieval or training by defining units and missing portions, returning confidence or uncertainty only when calibrated and useful, checking impossible values and item sums in the application, and maintaining a reviewed set of real meal descriptions and images, because the human review before save is safety, not decorative UI.

Invoice fields in a supplier email, a lead form to CRM fields, a maintenance note to a work order, and a support request routed to one of five queues are similarly bounded extraction, so if data arrives in the request and the result has a known shape, start there; add an index only for external changing evidence, and train only when examples prove prompt and constraints cannot hold the behaviour.

Combining them only makes sense after each part earns its place

A service assistant may retrieve a customer's current contract and turn supported facts into a tightly controlled case summary, while a compliance tool may retrieve an active rule and classify a submission into a fixed review queue, and combining them retains both obligations: measure source freshness, extraction quality, chunk boundaries, retrieval recall, citation support, output validity, domain correctness, latency, and cost because a retrieval version and model version can each change a result, doubling investigation surface.

Build in this order:

1. State the user decision or action the system supports.
2. Collect a small evaluation set, including declines.
3. Pass known facts directly and use a prompt plus structured output for bounded extraction.
4. Add retrieval when current or private evidence outside the request is necessary, evaluating it separately.
5. Fine-tune only after a prompt baseline fails a stable, measurable behavioural requirement and good examples exist.

The sequence is deliberately boring: expensive machinery follows evidence.

For delivery mechanics around a model API, see my OpenAI API integration checklist, and I have also written about tests as contracts for AI agents, because the useful question is what a failing output is allowed to do next.

I build the surrounding system as carefully as the model call - data boundaries, validation, monitoring, and an escape route when the answer is not good enough - and my CV and rates are here.