Every lesson up to this point produced something that runs. None of that proves it’s better than what you started with. This lesson is the one that turns “I fine-tuned a model” into “I can show you it actually improved,” using a real, standard machine learning practice: a held-out test set, data the model never saw during training, and a direct before/after comparison against the exact same set for both the base model and the adapted one. This is general ML methodology, not a WordPress feature, and treating it with the same rigor a serious ML team would is the difference between a genuine improvement and a model you merely spent GPU hours changing.
Lesson 2’s held-out slice, set aside before training and never used in Lesson 3 or Lesson 5. If you didn’t set one aside, the honest option is to go back and fine-tune again on a smaller training set with a genuine held-out portion, evaluating against data the model trained on tells you nothing trustworthy.
Like Lessons 3 and 5, the scripts here run in Python, on the same machine used for training. This is standard machine learning evaluation practice applied to your models, not a WordPress-specific tool, and it would look identical if the model being evaluated had nothing to do with WordPress at all.
Step 1: Confirm the held-out set is genuinely untouched
Before running anything, verify the held-out examples were excluded from both Lesson 3’s training data and Lesson 5’s contrastive pairs, not merely a random sample pulled after the fact from the same pool the model trained on:
# File: verify_holdout.py
import json
with open("training-data.jsonl") as f:
train_prompts = {json.loads(line)["prompt"] for line in f}
with open("holdout-data.jsonl") as f:
holdout_prompts = {json.loads(line)["prompt"] for line in f}
overlap = train_prompts & holdout_prompts
print(f"Overlap between train and holdout: {len(overlap)}")Any nonzero overlap here invalidates the evaluation that follows, since the model has already seen those exact examples during training, and it will predictably perform well on them regardless of whether it generalized to anything else.
Step 2: Evaluate the fine-tuned generation model, base vs tuned
Run the identical held-out prompts through both the original base model and Lesson 3’s fine-tuned model, and compare the completions directly:
# File: evaluate_generation.py
from transformers import pipeline
import json
base_pipe = pipeline("text-generation", model="meta-llama/Llama-3.1-8B-Instruct", device_map="auto")
tuned_pipe = pipeline("text-generation", model="./site-model-merged", device_map="auto")
with open("holdout-data.jsonl") as f:
holdout = [json.loads(line) for line in f]
results = []
for example in holdout:
prompt = f"### Instruction:\n{example['prompt']}\n\n### Response:\n"
base_output = base_pipe(prompt, max_new_tokens=200)[0]["generated_text"]
tuned_output = tuned_pipe(prompt, max_new_tokens=200)[0]["generated_text"]
results.append({
"prompt": example["prompt"],
"reference": example["completion"],
"base_output": base_output,
"tuned_output": tuned_output,
})With results in hand, two real, complementary ways to score it:
- Automated text-similarity metrics, ROUGE or BLEU between each model’s output and
the real reference completion, using Hugging Face’s
evaluatelibrary, a quick, repeatable signal for whether tuned output is objectively closer to your actual published style than base output is. - A structured rubric scored by a separate, stronger model or a human reviewer, rating each pair on specific dimensions, tone match, vocabulary accuracy, adherence to format, blind to which output came from which model, so the comparison isn’t biased by knowing which is “supposed” to win.
# File: score_with_metrics.py
import evaluate
rouge = evaluate.load("rouge")
base_scores = rouge.compute(
predictions=[r["base_output"] for r in results],
references=[r["reference"] for r in results],
)
tuned_scores = rouge.compute(
predictions=[r["tuned_output"] for r in results],
references=[r["reference"] for r in results],
)
print("Base model ROUGE:", base_scores)
print("Tuned model ROUGE:", tuned_scores)A genuine improvement shows up as a consistent, meaningful gap in the tuned model’s favor across the held-out set, not a single cherry-picked example that happened to look better.
Step 3: Evaluate the custom embedding model with Recall@k
For an embedding model, the equivalent held-out set is pairs of a real query and the chunk that genuinely answers it, pulled from real support tickets or search logs the way Lesson 5 described, kept separate from the pairs actually used in training:
# File: evaluate_embeddings.py
from sentence_transformers import SentenceTransformer, util
import json
base_model = SentenceTransformer("BAAI/bge-base-en-v1.5")
tuned_model = SentenceTransformer("./site-embeddings-model")
with open("holdout-query-pairs.jsonl") as f:
holdout_pairs = [json.loads(line) for line in f]
def recall_at_k(model, pairs, k=5):
hits = 0
all_chunks = [p["relevant_chunk"] for p in pairs]
chunk_embeddings = model.encode(all_chunks)
for i, pair in enumerate(pairs):
query_embedding = model.encode(pair["query"])
scores = util.cos_sim(query_embedding, chunk_embeddings)[0]
top_k_indices = scores.argsort(descending=True)[:k]
if i in top_k_indices:
hits += 1
return hits / len(pairs)
print("Base model Recall@5:", recall_at_k(base_model, holdout_pairs))
print("Tuned model Recall@5:", recall_at_k(tuned_model, holdout_pairs))Recall@k answers a direct, honest question: for a real held-out query, was the actually- relevant chunk among the top k results the embedding model would have surfaced. This is the same kind of metric the Massive Text Embedding Benchmark uses to compare embedding models generally, applied here to your own domain-specific data instead of a public benchmark.
Step 4: Structure this as a repeatable test, not a one-time check
Run once, this evaluation tells you about today’s model. Run as a repeatable, automated comparison, checked back in alongside your training scripts and re-run whenever you retrain, it tells you whether each new version is actually still an improvement over the last one you shipped. If you’ve built the testing and evaluation practices from Course 27, structure this comparison the same way that course treats any other repeatable quality check: a script with a clear pass/fail threshold, run against a fixed held-out set, checked before a new model version replaces the one currently serving production traffic.
Test it: run the full comparison and read the verdict honestly
python evaluate_generation.py && python score_with_metrics.py
python evaluate_embeddings.pyA genuine improvement looks like a consistent gap in the tuned model’s favor across both the automated metric and, ideally, the rubric-scored comparison, not a narrow win on one metric and a loss on another. A result that’s roughly a wash, or worse on any metric, is a real, valid outcome, it means the training data, the hyperparameters, or the underlying decision from Lesson 1 needs revisiting, not that the evaluation script is wrong.
ROUGE scores and Recall@k are proxies, not the actual thing you care about, whether real users get better answers. Before fully committing Lesson 4’s or Lesson 6’s swap in production, run the adapted model or embeddings alongside the original for a genuine sample of real traffic, not just the offline held-out set, and confirm the improvement holds up outside the exact conditions the held-out set was built under.
Recap
A held-out test set, genuinely excluded from training, and a direct before/after comparison against the same base model or embedding model you started with, are the real, standard way to know whether fine-tuning or custom embeddings actually worked. ROUGE or a rubric score for generation, Recall@k for embeddings, run identically against both versions, turn “I trained something” into an honest answer about whether it’s better. Structuring this as a repeatable, automated check rather than a one-time manual comparison is what keeps every future retraining run honest too.