Before any Python, any GPU rental, any training run, this lesson asks the question that actually matters: should you be doing this at all. Course 10 built a real retrieval pipeline that hands a model your actual site content at answer time. That solves a specific, common failure: a model that doesn’t know something. Fine-tuning and custom embeddings solve a different, much rarer failure: a model that knows the facts but represents or expresses them wrong. Confusing the two problems is the single most common reason fine-tuning projects fail to justify their cost, so this lesson draws the line precisely before the rest of the course builds on top of it.
Course 10, “Build a RAG Knowledge Base in WordPress,” and Course 24, “Private & Self-Hosted AI for WordPress,” completed, not skimmed. This lesson assumes you already have a working retrieval pipeline and already understand what self-hosting a model involves, and it’s written for readers deciding whether to go further than that, not readers deciding whether to start.
Step 1: Separate “doesn’t know” from “doesn’t behave right”
Every complaint about a WordPress AI feature’s output quality reduces to one of these two categories, and they have almost nothing in common:
Knowledge problems. The model answers a question about your return policy with a generic, wrong-for-you answer, or invents a product you don’t sell. This is a retrieval failure or a missing-retrieval failure, the model was never given your actual policy text or product catalog at answer time. No amount of fine-tuning fixes this reliably, because fine-tuning bakes patterns into weights at training time, and your return policy will change again next quarter. Retrieval, done well, fixes this permanently, since it hands the model current, correct information every single time it’s asked.
Behavior problems. The model has the right facts in front of it, you handed it the correct product description in the RAG prompt, and it still writes marketing copy that doesn’t sound like your brand, or a support answer that ignores your team’s tone guidelines, or output that’s supposed to be a specific JSON shape and drifts from it under load. This is what fine-tuning is actually for: changing how the model behaves given information, not what information it has.
Step 2: Where RAG wins outright, no contest
If your problem looks like any of these, stop here, go implement or improve Course 10’s pipeline instead of continuing this course:
- The model doesn’t know about a recent post, product, or policy change.
- Content changes often enough that baking it into weights would go stale within weeks.
- The fix is “give the model the right passage of text,” not “change how it writes.”
- You haven’t yet tuned chunking, retrieval scoring, or prompt construction in Course 10, since a badly-tuned RAG pipeline can look like a “the model doesn’t understand my domain” problem when it’s actually a “the model never received the right chunk” problem.
That last point is worth sitting with. A huge share of what looks like a case for fine-tuning, in practice, turns out to be retrieval that was never tuned properly. Confirm your RAG pipeline is actually working well, real top-N results that are genuinely relevant, before concluding it can’t solve your problem.
Step 3: The genuine remaining cases for fine-tuning
After RAG is working and tuned, a real, narrower set of problems remains that fine-tuning can address and prompting usually can’t fully close:
- House style RAG keeps failing to reproduce. You’ve handed the model your best writing samples in the prompt itself, and it still doesn’t consistently match a specific, distinctive editorial voice across long-form content.
- Domain vocabulary and phrasing. A model trained mostly on general web text systematically misuses or avoids the specific terminology your industry expects, medical, legal, or technical shorthand a general-purpose model treats as unusual.
- Structured output reliability under load. A prompt-only approach mostly produces the JSON shape or format your integration needs, but fails often enough at scale that the failure rate itself is the problem, and fine-tuning on hundreds of correct examples measurably tightens that reliability.
Step 4: The genuine remaining case for custom embeddings
Separately from behavior, a generic embedding model can be the actual bottleneck in an otherwise well-built RAG pipeline: it ranks chunks by general-purpose semantic similarity, and your domain’s notion of “similar” doesn’t match that. A support knowledge base full of dense technical shorthand, or a content library where two pieces that read as unrelated to a generic model are actually close in your domain, are real cases where the embedding model itself, not the retrieval logic around it, is the constraint. Lesson 5 covers this specifically.
You do not need both. A site can have a text-generation behavior problem with no embedding problem, or a retrieval-ranking problem with a language model that behaves fine. Diagnose each separately rather than assuming one implies the other.
Step 5: A tool the WordPress ecosystem is building for a different problem entirely
Worth naming explicitly, since it’s easy to conflate: WordPress’s own Abilities API lets a site expose real WordPress actions, creating a post, updating an order, to an AI agent as discoverable, callable functions. That solves “the model needs to actually do something in WordPress,” through tool-calling, at request time, with no training involved at all. It’s a useful contrast: plenty of “the model doesn’t know how to use my site” problems are solved by giving the model a real tool to call, not by fine-tuning it on your site’s procedures. Keep tool-calling and fine-tuning as separate answers to separate questions.
Test it: five questions before you commit
Answer these honestly before continuing to Lesson 2:
- Is Course 10’s RAG pipeline actually built, tuned, and returning genuinely relevant results for real user queries?
- Is the problem you’re solving demonstrably about behavior or ranking, not missing information a better retrieval prompt could supply?
- Do you have, or can you realistically produce, enough real training examples, typically hundreds at minimum, to fine-tune on?
- Do you have access to a GPU, your own or rented, and the time to run and iterate on a training job outside WordPress entirely?
- Do you have a way to measure “better” objectively, a held-out test set, before you start, so you’re not guessing at the end (Lesson 7)?
If any answer is no, that’s not a failure, it’s the honest signal that RAG, better prompting, or more Course 10 tuning is still the higher-leverage next step.
A fine-tuned model is a snapshot of a training run. Your site’s content, tone, and product line will keep changing after that snapshot is taken, and a fine-tuned model doesn’t update itself the way a RAG pipeline’s retrieved content does automatically. Committing to fine-tuning means committing to re-running this process periodically, not doing it once and forgetting about it.
Recap
RAG and fine-tuning solve different problems: RAG answers “does the model know this,” fine-tuning answers “does the model behave or write this way.” Most WordPress AI quality problems are knowledge problems, and Course 10’s pipeline, tuned properly, closes them. The genuine remaining cases, house style RAG can’t reproduce, domain vocabulary, output reliability under load, or an embedding model misranking domain-specific content, are real and worth solving, but they’re a genuine minority. The rest of this course assumes you’ve honestly answered Step 5’s five questions and the answer is still yes.