Fine-Tuning Strategy#

Fine-tuning is not a button that makes an AI “know your business.” It is a costly way to change a model’s behaviour. Before you train, prove that prompting, retrieval, or tools cannot solve the real problem more safely and cheaply.

⏱ ~12 min read · ~30 min hands-on 🔗 needs: Prompt Engineering · RAGAS Evaluation · MLflow

What fine-tuning changes#

A pretrained model has learned broad patterns from its original training. Fine-tuning continues training it on examples for a narrower task. In supervised fine-tuning (SFT), each example demonstrates the input and the desired output.

base model
   + high-quality examples of one repeated task
   → adapted model that is more likely to respond in that pattern

Fine-tuning is often useful for stable, repeated behaviour:

• a support assistant that always produces a particular schema and tone,
• classifying a fixed set of document types,
• extracting fields from invoices in a known format,
• generating code in an internal style, or
• teaching a model how to use a specialised vocabulary or response convention.

It is a poor first answer for changing facts, a one-off task, or an unreliable workflow. A model can memorise bad examples and become more confidently wrong.

Pick the smallest thing that solves the problem#

Many useful systems combine these: a fine-tuned model follows a reliable format, RAG supplies current policy text, and tools fetch account-specific facts. Do not choose one technique for ideological reasons—use an evaluation to choose.

Start with an evaluation, not a dataset#

Write down what “better” means before collecting examples. An eval set is a private, held-out collection of realistic tasks and scoring rules. Never tune on it.

real task → baseline model + prompt → held-out eval → find repeated failure
                                                ↓
                              make only examples that target that failure
                                                ↓
                              fine-tune → same held-out eval → compare

For a meeting-notes assistant, “sounds professional” is vague. A useful rubric might be:

Score the baseline first. If a better prompt or structured-output schema already passes, stop: there is no fine-tuning problem to solve.

Training data is a product, not a dump#

High-quality, diverse, representative examples matter more than a large pile of scraped text. Each example teaches both what to do and what “good” looks like.

A good example is#

• Representative: resembles real inputs, including short, messy, ambiguous, and edge cases.
• Correct: reviewed against a written rubric, not merely generated by another model.
• Consistent: uses one chosen response format and policy.
• Complete: contains enough context for the target output to be justified.
• Legal and safe: you have permission to use it; secrets, personal data, and copyrighted/private material are removed or properly authorised.

A bad example is#

• duplicated dozens of times to make the dataset “bigger”,
• a prompt where the answer depends on information not provided,
• internally inconsistent (“always refuse” in one example, “answer anyway” in another),
• an answer copied from a production customer conversation without consent, or
• an answer that rewards fluent fabrication rather than a transparent “I don’t know”.

⚖️ Fine-tuning data is an upload, not a thought experiment. Read the provider’s data-retention, access-control, and deletion policy before using customer data. Keep an internal record of source, licence/consent, reviewer, and removal process for every dataset version.

The three-way split#

Keep these sets separate from the beginning:

For small projects, split by customer, document, or time period, not randomly by individual sentence. Otherwise a nearly identical version of the same document can land in train and test, giving you a fake score.

Try it — make a tiny, reviewable dataset and eval#

Do not start a paid training job yet. This exercise creates the assets that make a training job meaningful. It uses a fictional meeting-notes task; replace it only with material you are allowed to use.

1. Create a dataset layout#

mkdir -p finetune-notes/{data,evals}
cd finetune-notes

Save this as data/train.jsonl. JSONL means one valid JSON object per line. Most chat fine-tuning APIs accept a message-based structure, though exact supported roles and formats vary by provider/model—check its current guide before upload.

{"messages":[{"role":"system","content":"Turn the transcript into concise meeting notes. Use exactly these headings: Decisions, Actions, Risks. Do not invent people, dates, or facts. Write 'Not stated' where needed."},{"role":"user","content":"Transcript: Priya: We will launch the beta on 15 September. Omar will prepare the deployment checklist by Friday. The payment-provider approval is still pending."},{"role":"assistant","content":"## Decisions\n- Launch the beta on 15 September.\n\n## Actions\n- Omar: prepare the deployment checklist by Friday.\n\n## Risks\n- Payment-provider approval is still pending."}]}
{"messages":[{"role":"system","content":"Turn the transcript into concise meeting notes. Use exactly these headings: Decisions, Actions, Risks. Do not invent people, dates, or facts. Write 'Not stated' where needed."},{"role":"user","content":"Transcript: The team discussed the mobile redesign. No decision was made. Mei will compare two accessibility libraries. No risks were mentioned."},{"role":"assistant","content":"## Decisions\n- Not stated.\n\n## Actions\n- Mei: compare two accessibility libraries.\n\n## Risks\n- Not stated."}]}

These two rows are format examples, not enough data to train a useful model. A real dataset needs many reviewed examples, including failures: vague owners, conflicting statements, missing dates, and requests that the assistant must decline or escalate.

Save a different, unseen case as evals/final.jsonl:

{"input":"Transcript: Carlos decided to pause the migration until the security review. Aisha will schedule that review next week. The old database reaches end-of-support in October.","expected":{"headings":["Decisions","Actions","Risks"],"must_contain":["pause the migration","Aisha","end-of-support in October"],"must_not_contain":["a date for the security review","an invented owner"]}}

2. Validate the JSONL before it becomes expensive#

Save this as check_dataset.py and run uv run check_dataset.py data/train.jsonl. It catches malformed JSON and basic message-shape errors; human review still decides whether an answer is good.

import json
import sys
from pathlib import Path

path = Path(sys.argv[1])
errors = []

for line_number, line in enumerate(path.read_text().splitlines(), start=1):
    try:
        item = json.loads(line)
        messages = item["messages"]
        roles = [message["role"] for message in messages]
        if roles != ["system", "user", "assistant"]:
            errors.append(f"line {line_number}: expected system, user, assistant")
        if not all(isinstance(message.get("content"), str) for message in messages):
            errors.append(f"line {line_number}: every content value must be text")
    except (json.JSONDecodeError, KeyError, TypeError) as error:
        errors.append(f"line {line_number}: {error}")

if errors:
    print("Dataset has errors:")
    print("\n".join(errors))
    raise SystemExit(1)

print(f"OK: {path} contains JSONL examples with the expected message shape.")

3. Establish a baseline and score it#

Run the final.jsonl input through your chosen base model and the same system prompt. Save the output and score it against the four checks above. Only after that should you train and compare the tuned model on the untouched final eval.

Track both runs in MLflow (or a simple CSV at first): base-model version, prompt version, data version, evaluator/rubric, cost, latency, and score. This is how you discover whether training actually helped.

Choose a tuning method later#

This lesson is about the decision and data. The next lessons cover the mechanics, but you should recognise the trade-off:

Do not choose LoRA, QLoRA, epochs, or a large base model until the baseline and data quality justify the work. You can use an efficient method to produce a perfectly efficient bad model.

When it fails#

Your turn (≈30 min)#

1. Choose one repeated task, and write its input, desired output, and one non-goal.
2. Make a five-item held-out eval with a pass/fail rubric before collecting training data.
3. Test your strongest base-model prompt on that eval and record quality, cost, and latency.
4. Write 10–20 original, reviewed candidate training examples, including ambiguous and missing-information cases. Keep them separate from the eval.
5. Run a JSONL/schema check and ask a classmate or AI to find contradictions, unsupported answers, and accidental personal data.
6. Make a one-sentence decision: tune now, improve prompting/RAG/tools, or collect better data—and state the evidence.

Checklist#

• I can explain why tuning differs from RAG and tool calling.
• I have a baseline and an untouched evaluation set before training.
• Each example is representative, correct, reviewed, and authorised for use.
• Train, validation, and final test data cannot leak into each other.
• I will compare quality, cost, latency, and safety—not only a single score.
• I know a fine-tuned model can still hallucinate and needs system-level safeguards.

Go deeper#

• Hugging Face: fine-tune a pretrained model — practical dataset/training workflow.
• Hugging Face PEFT — adapters and parameter-efficient methods.
• MLflow Tracking — record your baseline, data version, and tuned comparison.
• Agent evaluation and benchmarking — build a useful evaluation before optimising a system.