The problem
General-purpose language models are very good at general-purpose text. Domain-specific generation is harder. When terminology, format and tone have to be right every time, prompting alone often is not enough: the model keeps drifting back toward how it writes in general.
Fine-tuning closes that gap by adapting the model itself. But fine-tuning is frequently done as a series of one-off experiments. Each run uses slightly different data and settings, results are hard to reproduce, and six months later nobody can say exactly how a given model was produced or whether a new one is actually better.
What we built
A repeatable framework for fine-tuning language models on domain-specific generation, used across multiple industries. It turns fine-tuning from an experiment into a process that a team can run again, compare and improve. The work covered model optimization and custom training pipelines.
How it works
The models
The framework has been used to fine-tune GPT-3.5, Llama 2 and flan-t5. Supporting several model families matters, because the right base model depends on the task, the deployment constraints and the cost of running it.
Parameter-efficient tuning: LoRA and Q-LoRA
Fully retraining a large model means updating billions of parameters, which is expensive in compute and memory. LoRA (low-rank adaptation) takes a different approach: it leaves the original model’s weights unchanged and trains a small number of additional parameters that adjust its behaviour. The result captures most of the benefit of fine-tuning at a fraction of the cost.
Q-LoRA goes further by loading the base model in a compressed, quantized form while training the adapters. That reduces memory requirements enough to fine-tune larger models on far more modest hardware.
Few-shot learning
Where labelled data is limited, few-shot learning lets a model learn a task from a small number of examples. It is useful early on, before enough data exists for a full fine-tuning run.
Built on PyTorch
The training pipelines are built on PyTorch.
Why a framework rather than a script
The value of a framework is repeatability. When the data, settings and evaluation are captured consistently, two things become possible that one-off experiments do not allow.
- Honest comparison. A new model can be compared fairly against the current one, on the same evaluation, so improvement is measured rather than assumed.
- Reproducibility. Any model can be traced back to exactly how it was produced, which matters for debugging and for anyone who inherits the system.
When fine-tuning is the right call
Fine-tuning is not our default. We reach for it only after cheaper approaches have been tried. In rough order:
- Better prompting. Clear instructions and good examples solve more problems than people expect.
- Retrieval. If the model lacks knowledge, give it the right documents at question time. Retrieval is usually better than fine-tuning for knowledge, because knowledge changes and retrieval stays current.
- Fine-tuning. When the model has the knowledge but needs to behave differently, in format, tone or terminology, consistently, fine-tuning is the right tool.
Treating it as a repeatable framework is what makes that decision cheap to revisit as base models improve.
Where it can go wrong
- Poor training data. A fine-tuned model learns whatever is in its data, including its mistakes. Data quality matters more than quantity.
- Overfitting. A model tuned too narrowly can lose general ability it still needs.
- Evaluating on the wrong thing. Improvement on a convenient test set is not improvement on the real task. Evaluation has to reflect real use.
Questions to ask before building one
If you are considering a system like this, these are the questions we would want answered first.
- Have you exhausted prompting and retrieval? Fine-tuning should be the answer to a problem those could not solve.
- Is your need knowledge or behaviour? Retrieval is usually better for knowledge; fine-tuning is better for consistent behaviour.
- Do you have good training examples? A few hundred excellent examples beat thousands of mediocre ones.
- How will you know the new model is better? Define the evaluation before the first training run.