Fine-Tuning in Practice: LoRA, PEFT, and When Not to Fine-Tune

What fine-tuning actually changes

The most expensive misunderstanding in model adaptation is treating fine-tuning as a way to teach the model new facts. Teams collect their product documentation, tune a model on it, and then watch it hallucinate the same policies it was supposedly trained to know. The failure is predictable, because gradient updates on a few thousand examples do not reliably install retrievable knowledge in a network with billions of parameters. What they do reliably install is behavior: output format, tone, terminology, the reflex to ask a clarifying question before quoting a price, the habit of emitting valid JSON on the first try.

Think of it as the difference between the how and the what. Fine-tuning shapes the how: a claims-triage model that always produces the four-field summary your downstream system parses, a support model that sounds like your brand instead of a generic assistant, a code model that follows your internal conventions. The what, meaning current facts, entitlement-scoped documents, anything that changes weekly, belongs in retrieval, where it can be updated without a training run and audited per query. The comparison is worked through in RAG vs Fine-Tuning.

This split also explains where fine-tuning shines. Tasks with a stable, well-defined shape and a fuzzy specification that resists prompting (extract these fields, classify with this taxonomy, rewrite in this voice) respond dramatically to a few hundred good examples. Tasks that hinge on fresh or proprietary information do not, no matter how large the training set.

The methods landscape: LoRA, QLoRA, full FT

Full fine-tuning updates every weight in the model. It offers maximum plasticity, and for enterprise use it is almost never justified: it needs the most hardware, it is the most prone to catastrophic forgetting, and it produces a full-size artifact per task that you must store, version, and serve separately. It survives mainly in labs and in heavy distillation pipelines, covered in Distillation.

LoRA (2021) is the workhorse. It freezes the base model and trains small low-rank matrices injected alongside the existing weight matrices, typically far below 1 percent of total parameters. The result, an adapter, is a file on the order of tens to hundreds of megabytes rather than a multi-gigabyte model copy. Quality on narrow tasks is generally competitive with full tuning, and the frozen base sharply limits how much general capability the tune can destroy. QLoRA (2023) goes further for training economics: it quantizes the frozen base to 4-bit (NF4) and backpropagates through it, cutting training memory enough that mid-size models tune on a single commodity GPU.

DPO-style preference tuning deserves a separate mention because its input is different: pairs of responses where one is preferred. That makes it the right tool when the goal is judgment or tone (concise over verbose, cautious over confident) rather than a demonstrable input-to-output mapping. In practice, supervised LoRA teaches the task and a light preference pass polishes the manner.

The dataset is the product

Once LoRA collapses the compute cost, the training data becomes the entire project. This is where fine-tuning efforts succeed or fail, and it is the part organizations most consistently underinvest in. The uncomfortable finding, replicated across teams: hundreds to low thousands of genuinely excellent examples routinely beat tens of thousands of mediocre ones. Every sloppy example is an instruction the model will follow. If 5 percent of your training set has the wrong label or a lazy answer, you have trained a model that is wrong or lazy 5 percent of the time, on purpose.

Treat the dataset with the same rigor as production code, which means a pipeline, not a spreadsheet someone exported once:

• Deduplicate aggressively, including near-duplicates. Repeated examples silently overweight one behavior and inflate apparent training progress.
• Carve out a held-out eval split before training starts, and treat any leakage between train and eval as a build-breaking defect. A tune evaluated on its own training data always looks brilliant.
• Decontaminate against your benchmarks: if eval cases resemble training cases, your scores measure memorization, not capability.
• Use synthetic augmentation, filtered hard. A strong model can draft variations and edge cases cheaply, but only examples that survive automated checks and human review get in. Generation patterns are covered in Synthetic Data.
• Check provenance and licensing per example. Customer transcripts need privacy review; scraped or vendor-derived content needs a license that permits training. Retroactively removing tainted data from a shipped model is somewhere between expensive and impossible.

The strategic upside of doing this well: a clean, labeled, versioned dataset outlives any base model. Models will be swapped; the dataset, like a good eval suite, compounds.

Running the tune: what actually matters

The training run itself is the least mysterious part of the project, and the hyperparameter folklore around it is mostly noise. A few settings carry nearly all the weight. Epochs: few, often one to three. Behavioral tuning converges fast, and every additional pass over a small dataset is an invitation to memorize it. Learning rate: the single most damage-capable knob. Too high and the model degrades broadly within minutes of training; the discipline is to start from the community's established defaults for your base model and method, change one thing at a time, and resist the urge to sweep ten parameters on a dataset of eight hundred examples.

What separates professional runs from hobby runs is not the config, it is the instrumentation. Checkpoint frequently and run evals during training, not after it. The held-out split from the previous section earns its keep here: eval loss and task-level scores per checkpoint turn "did it work?" from a post-hoc guess into a curve you can read. Teams that only evaluate the final artifact routinely ship a checkpoint that was already past its best.

For forgetting specifically, keep a small general-task probe set (reasoning, instruction following, safety refusals) in the eval loop alongside your task metrics. LoRA's frozen base limits the blast radius, but adapters can still degrade behaviors adjacent to the tuned task, and adjacent is exactly where nobody thinks to look. The harness pattern is the same one described in the Eval Harness: versioned cases, thresholds, and a gate the tune must pass before anyone calls it done.

Serving and versioning tuned models

Adapters change the serving economics as much as the training economics. Because a LoRA adapter is megabytes rather than gigabytes, it can be loaded, swapped, and versioned like a configuration artifact instead of a model deployment. Modern inference stacks exploit this with multi-adapter serving: one copy of the base model resident on the GPU, many adapters attached to it, selected per request. Ten tuned "models" for ten tasks or tenants no longer means ten times the hardware.

 request: {task: "triage"}      request: {task: "sql"}
        |                              |
        v                              v
 +---------------------------------------------+
 |          SHARED BASE MODEL (frozen)         |
 |            one copy in GPU memory           |
 +---------------------------------------------+
     |            |            |
 [adapter A]  [adapter B]  [adapter C]
  triage       sql-gen      brand-voice
  v3 (live)    v7 (live)    v2 (canary)
            

Version adapters the way you version prompts and indexes: immutable, tagged with the exact base model, dataset revision, and eval scores that produced them, and routed through the same gateway that handles the rest of your model traffic. An adapter without its lineage is undeployable the first time you need to reproduce or roll back a behavior.

Then there is the cost nobody budgets for: the re-tune treadmill. An adapter is welded to the exact base checkpoint it was trained on. When the base model is upgraded, and as of mid-2026 the useful ones are upgraded often, your adapter does not transfer; it must be retrained and re-evaluated on the new base. Every fine-tune is therefore a subscription, not a purchase: recurring dataset refreshes, re-training runs, and full regression evals, owned by a named team. Price that in before the first run, because it is the line item that quietly decides the ROI section below.

The ROI math

Strip away the enthusiasm and fine-tuning is a capacity investment with a break-even point. You spend a fixed cost up front (dataset construction, which dominates; training compute, which as of mid-2026 is often trivial by comparison, on the order of tens to a few hundred dollars for a LoRA run; plus evaluation and integration engineering) to lower a variable cost: a tuned small model typically undercuts a prompted frontier model by an order of magnitude or more per token, with lower latency and, often, a shorter prompt because the instructions moved into the weights.

An illustrative shape, not a quote: suppose the all-in fixed cost of a tune, dominated by the people building the dataset and evals, lands somewhere in the tens of thousands of dollars. Suppose the tuned model saves on the order of half a cent per request against the frontier alternative. Break-even sits in the millions of requests. A high-volume workload (document triage, ticket classification, structured extraction at hundreds of thousands of calls per day) crosses that line in weeks. A workload running a few thousand calls a day may never cross it before the re-tune treadmill resets the clock.

The math therefore favors fine-tuning under three simultaneous conditions: the task is narrow (a small model can master it), stable (the spec will not shift under you within months), and high-volume (per-request savings actually accumulate). Remove any one and the prompted frontier model wins, for an unglamorous reason: a prompt change ships in minutes and costs nothing, while a behavior change to a tuned model costs a dataset revision, a training run, and a regression cycle. For volatile products, that iteration speed is worth far more than per-token savings. Where fine-tuning sits relative to prompting, RAG, and distillation is mapped in The Adaptation Ladder.

The go/no-go checklist

Everything above compresses into five questions. They are sequenced deliberately: each one is cheaper to answer than the one after it, and a single honest "no" ends the conversation for now.

1. Have you exhausted prompting and RAG? A serious prompt engineering pass plus retrieval closes the gap more often than not, at a fraction of the cost and with none of the maintenance. Fine-tuning is what you do after these plateau, with evidence that they plateaued.
2. Will the task be stable for months? If the output spec, taxonomy, or tone requirements are still being debated, every debate outcome is a retraining cycle. Tune settled tasks, prompt moving ones.
3. Do you have enough quality examples? Hundreds of clean, deduplicated, rights-cleared examples with a held-out split. If assembling them sounds hard, notice that this is the actual project, and it does not get easier after kickoff.
4. Is an eval suite in place before training? Task metrics plus general-capability probes, versioned and gated. Without it you cannot detect overfitting, forgetting, or regressions on the next base model, which means you cannot detect whether the tune worked.
5. Does the re-tune treadmill have a named owner? Someone must own dataset refreshes, re-tuning on base model upgrades, and regression sign-off, indefinitely. An adapter without an owner is abandonware with a serving bill.

If all five are yes, fine-tune with confidence: the conditions under which it pays are real, and when they hold, a tuned small model is one of the best cost and latency levers in the stack. If any answer is no, do not fine-tune yet. That is not a failure of ambition. It is the same judgment that keeps you from building a data center to host a spreadsheet: the boring option is winning on merit, and it leaves you free to tune later, when the task has earned it.