Fine-Tuning Explained

When fine-tuning is the right call

Fine-tuning is a behavior tool, not a knowledge tool. Reach for it when you want the model to:

• Write in a specific voice
• Output a specific format reliably
• Handle a narrow task with high quality at a small-model price
• Internalize a long system prompt so you can drop it from per-call cost

It is not the right tool for teaching the model new facts. For that, use RAG.

The data is the project

A fine-tuning project is 80% data work. You need:

• 100+ examples at minimum, ideally 300–1000.
• Pristine labels — every example you train on, the model will mimic. Garbage in, more confidently garbage out.
• A holdout of 30–50 examples the model never sees during training.
• Coverage of the edge cases you care about, not just the easy ones.

If your data is shaky, fine-tuning won't fix it. It will amplify it.

LoRA / QLoRA — the modern default

For open-weights models, LoRA (Low-Rank Adaptation) is the standard approach in 2026. Instead of updating every parameter in a 70B model, you train a small adapter (typically <1% of the parameter count) that sits on top.

Benefits:

• Cheap — fine-tune on a single GPU.
• Fast — minutes to hours, not days.
• Reversible — toggle the adapter on/off; serve multiple adapters from one base model.
• Storable — adapters are tens of MB, not hundreds of GB.

QLoRA goes further by quantizing the base model to 4-bit during training. Trains on consumer hardware.

The training loop, simplified

1. Build dataset (300–1000 input/output pairs)
2. Split: 90% train, 10% eval
3. Format as instruction-following pairs
4. Run LoRA training (e.g. axolotl, unsloth, OpenAI fine-tune API)
5. Evaluate on eval set + holdout
6. Iterate on data, not hyperparameters

Hosted vs open-weights

Hosted (OpenAI, Anthropic via partners, Google):

• Pro: Zero ops. Run a job, get a model ID.
• Con: Limited control. Higher per-token cost than base. You're locked in.

Open-weights (Llama, Qwen, DeepSeek):

• Pro: Full control, full ownership, no vendor lock-in. Can self-host or use Together / Fireworks for inference.
• Con: Requires GPU access for training; ops overhead for serving.

For most production teams: hosted for first iterations, open-weights when economics demand it.

Common mistakes

• Training on too few examples — 30 examples teaches the model to mimic 30 specific outputs, not your underlying intent.
• Skipping eval — without evals, you have no idea if the model improved.
• Inconsistent formatting — every example must follow the same input/output schema, exactly.
• Overfitting — too many epochs on too little data → the model parrots training inputs.
• Fine-tuning when prompting would work — the most common mistake. Always try harder prompts first.

When to walk away

If after one round of fine-tuning your eval scores haven't moved meaningfully, stop. The problem is almost always:

1. Your prompt was actually fine — the task isn't the bottleneck.
2. Your data is too noisy / inconsistent.
3. The base model can't do this task at any prompt level — switch models, not techniques.

Fine-tuning isn't a magic upgrade. It's a precision tool for narrow problems with clear examples.