When Fine-Tuning Is the Wrong Answer

Fine-tuning is the wrong answer whenever the real problem is missing facts, a vague task, bad evaluation, or a prompt you have not properly tried. It is a powerful tool, but it is slower to iterate on than almost any alternative, so it should earn its place.

1. You have no evaluation

If you cannot measure whether outputs are good, you cannot tell whether a tuned model improved. Teams often fine-tune, eyeball three outputs, and declare success. Build a test set of at least fifty representative inputs with expected results or a scoring rubric before you train anything.

2. The model lacks facts

Say you want answers about your company's refund policy. Training on the policy text can make the model sound familiar with it, but recall is unreliable, and the model may blend your policy with generic policies from pretraining. Retrieval puts the actual text in front of the model, which is more accurate and can be updated by editing a document. Fine-tuning for facts is the single most common misuse.

3. The task is not defined

If two humans on your team would disagree about the right answer, a model trained on their labels will learn the disagreement. Write labelling guidelines, measure agreement between annotators, and settle ambiguity first. Noisy labels set a ceiling on quality that no hyperparameter can lift.

4. You have not tried a strong prompt

Many failures disappear with clear instructions, a few well-chosen examples, a defined output schema and a stronger base model. A prompt change takes minutes. A fine-tune takes days once you include data work. Always record a prompt-only baseline, as post 16 explains.

5. Your requirements change often

Every change in policy, label set or output format means new data and a new training run. If the specification moves weekly, keep the behaviour in the prompt or in configuration, where a change is a text edit.

6. You have too little or poor data

A handful of examples can teach a format, but not a hard skill. If you have twenty examples, use them as few-shot prompts instead. If your data is scraped, duplicated or inconsistent, cleaning it will help more than training on it. Posts 21 to 29 cover data work.

7. You need guarantees

No fine-tune guarantees a behaviour. If you need strict rules, such as never outputting a certain field or always returning valid JSON, use constrained decoding, schema validation and retries around the model. Treat the tuned model as a probabilistic component and wrap it.

A quick self-test

checks = {
    "have_eval_set": True,
    "prompt_baseline_measured": True,
    "failures_are_behavioural": False,   # failures are really missing facts
    "have_good_examples": True,
    "spec_is_stable": True,
}

blockers = [name for name, ok in checks.items() if not ok]
if blockers:
    print("Do not fine-tune yet. Fix:", ", ".join(blockers))
else:
    print("Fine-tuning is a reasonable next step.")

This prints Do not fine-tune yet. Fix: failures_are_behavioural, which would send you to retrieval. The point is not the code but the habit of writing down your reasons.

Cheaper fixes to try first

• Rewrite the prompt with explicit rules and two or three examples.
• Switch to a stronger or more recent base model.
• Use structured output or JSON schema features of your inference stack.
• Add retrieval for knowledge gaps.
• Split one hard task into several simple steps.
• Add validation and automatic retry on failure.

When it is still the right call

Fine-tune when the failures are behavioural, the spec is stable, you hold a few hundred or more clean examples, and either quality, latency or cost cannot be reached by prompting. That is the case for classification at scale, strict extraction formats, consistent voice, and small on-device models. Post 2 covers the full comparison.