Full Fine-Tuning vs PEFT

Introduction

Last lesson cut memory with fewer bits per weight (quantization). This one cuts it with fewer trainable weights — the second great lever of fine-tuning mechanics, and the one that put fine-tuning in everyone's hands: PEFT (Parameter-Efficient Fine-Tuning).

The question is simple: when you adapt a model, do you update all its weights (full fine-tuning) or just a tiny sliver (PEFT)? The answer, for ~80% of real work, is the sliver — and not because it's a cheap compromise. PEFT trains <1% of the parameters yet reaches 95–99% of full fine-tuning's quality, while also being modular, forget-proof, and small enough to fine-tune a 70B on a single GPU. This lesson is why, and when full fine-tuning still earns its cost.

An infographic titled 'Full Fine-Tuning vs PEFT', the fourth lesson of section two on fine-tuning mechanics, comparing updating every weight against parameter-efficient fine-tuning. Full fine-tuning updates all of a model's weights, giving maximum flexibility but paying the full cost: about sixteen bytes per parameter in memory, catastrophic forgetting of general abilities, and a complete model checkpoint per task, for example forty gigabytes for each downstream dataset. PEFT, parameter-efficient fine-tuning, freezes the pre-trained base and trains only a tiny set of new or selected parameters, typically under one percent of the weights, such as a few million trainable parameters on a 7-billion model. This yields four wins. First, memory: only the small adapter needs gradients and optimizer states, cutting requirements roughly ten to twenty times, which is the primary reason serious fine-tuning now runs on a single consumer GPU. Second, modularity: because the base is unchanged, you store one shared base plus a tiny few-megabyte adapter per task and hot-swap adapters at inference, instead of N full copies, so a forty-gigabyte-per-task explosion becomes a few megabytes per task. Third, no catastrophic forgetting, because the frozen base retains its general knowledge. Fourth, quality parity: LoRA typically reaches ninety-five to ninety-nine percent of full fine-tuning quality while training under one percent of parameters. The PEFT family includes LoRA, the dominant low-rank adapter method, classic bottleneck adapters, prefix and prompt tuning that learn soft virtual tokens, and BitFit that trains only biases. PEFT wins about eighty percent of production fine-tuning workloads; full fine-tuning still owns the other twenty percent, and the deciding factor is not cost but the depth of behavioral change needed: when you must infuse substantial new domain knowledge or fundamentally alter core capabilities, and you have the data and compute, full fine-tuning can be more effective. The takeaway is to default to PEFT, usually LoRA, and reach for full fine-tuning only for deep changes.

Full Fine-Tuning: All Weights, Maximum Cost

Full fine-tuning does exactly what L189's loop describes, on every weight: a gradient and an optimizer state for all of them. It gives the model maximum flexibility to change — and pays for it three times over:

Memory — the full ~16 bytes/parameter from L190 (a 7B needs ~112 GB to train).
Catastrophic forgetting — moving every weight toward your task moves them away from everything else (the L186 trap; full FT forgets ~20%).
A full copy per task — each fine-tune is a complete new model. Need 5 specialized models? That's 5 full checkpoints (e.g. 5 × 40 GB).

Full FT is the sledgehammer: powerful, but expensive, forgetful, and unwieldy if you need more than one specialized model.

PEFT: Freeze the Base, Train a Sliver

PEFT flips the default: freeze the entire pre-trained base and train only a small set of new or selected parameters — typically under 1%. On a 7B model that's a few million trainable parameters instead of seven billion.

The magic is in what frozen buys you. A frozen weight needs no gradient and no optimizer state (recall L190 — those were 5/6 of the memory), and it can't be forgotten because it never moves. The few trainable parameters live in a tiny adapter that you can save separately and snap onto the shared base at run time. From this one idea — train little, freeze the rest — flow four wins.

The Four Wins of PEFT

1) Memory — 10–20× less. Only the tiny adapter carries gradients and optimizer state, so a fine-tune that needed a cluster now fits a single consumer GPU. (This, plus QLoRA's 4-bit base, is why a 7B trains in ~6 GB.)

2) Modularity — one base, many tasks. Because the base is untouched, each task's adapter is an independent, few-MB file. You store one base + N tiny adapters and hot-swap the right one per request — instead of loading a different full model. The classic example: a model whose full checkpoint is 40 GB would cost 40 GB per task with full FT, but a few MB per task with PEFT.

3) No catastrophic forgetting. The frozen base keeps all its general knowledge and safety — you're adding a skill, not overwriting the model (the L186 fix, by construction).

4) Quality parity. For most tasks, a well-configured PEFT method reaches 95–99% of full fine-tuning's score while training <1% of the weights. The gap is usually too small to matter.

The PEFT Family

PEFT is a family of methods that differ in where they put the trainable parameters:

LoRA (Low-Rank Adaptation) — injects tiny low-rank matrices beside the frozen weights. The dominant method (next lesson, in depth).
Adapters — small bottleneck layers inserted between the model's layers (the original PEFT, ~0.5% params).
Prefix / Prompt / P-tuning — learn a handful of soft "virtual token" vectors that condition the frozen model (you tune the input, not the weights).
BitFit — train only the bias terms (a fraction of a percent).
(IA)³ — learn to scale activations.

In practice the library does the wiring — wrapping a model in PEFT is a few lines, and it tells you exactly how little you're training:

from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM

base = AutoModelForCausalLM.from_pretrained('meta-llama/Llama-3.2-3B')   # the FROZEN base
config = LoraConfig(r=16, lora_alpha=32, target_modules='all-linear')    # a tiny low-rank adapter
model = get_peft_model(base, config)

model.print_trainable_parameters()
# trainable params: 12,156,928 || all params: 3,224,906,752 || trainable%: 0.38

# You train 0.38% of the weights. The other 99.62% are FROZEN -> no gradients,
# no optimizer state, no catastrophic forgetting. The trained adapter saves as a ~24 MB
# file you hot-swap onto the shared base at serve time (model.save_pretrained('support-adapter')).

When Full Fine-Tuning Still Wins

PEFT isn't always the answer — it wins about 80% of production fine-tuning; full FT owns the other 20%. And the deciding line isn't cost — it's the depth of behavioral change you need:

Infusing substantial new behavior/capability — if you need to fundamentally change how the model reasons or operates (not just add a format/skill), updating all the weights has more capacity to do it. PEFT is constrained by the frozen base.
Some simpler classification tasks — where more trainable parameters and longer training reliably help, full FT can edge out PEFT.
You have the data and compute — and the marginal last few % of quality is worth the full bill.

But notice the default has flipped: the question used to be "can we afford to fine-tune?"; now it's "is this change deep enough to justify full fine-tuning over a LoRA?" For most teams, the honest answer is no — use PEFT.

See It: The PEFT vs Full Fine-Tuning Lab

Watch the two diverge. Pick a base model, choose PEFT or full FT, then drag the number of tasks you need. The pipeline shows PEFT as one frozen base + swappable adapters and full FT as N separate copies — and the storage bar makes the modularity win unmissable: full FT explodes by a whole model per task, while PEFT grows by a few MB.

Interactive: the PEFT vs Full Fine-Tuning Lab. The user builds a multi-task deployment and watches the two approaches diverge. They pick a base model size (7B, 13B, 70B), drag the number of tasks they need from one to eight, and toggle the method between full fine-tune and LoRA/PEFT. A pipeline shows the structural difference with arrow connectors: PEFT is one frozen shared base fanning out to tiny swappable adapters, one per task, while full fine-tuning is N independent full-model copies with nothing shared. Live stats compare trainable parameters, one hundred percent versus under one percent, training VRAM at about sixteen bytes per parameter versus ten to twenty times less, catastrophic forgetting present versus none because the base is frozen, and quality at one hundred percent versus about ninety-five to ninety-nine. The centerpiece is a storage bar that makes the modularity win visceral: full fine-tuning stores N full checkpoints, N times the model, while PEFT stores one base plus N few-megabyte adapters, so as the task count grows full fine-tuning explodes and PEFT barely moves, and a verdict explains that PEFT wins about eighty percent of workloads while full fine-tuning earns its cost only for deep behavioral change.

Crank the task count to 8 and the picture is decisive: full fine-tuning ships eight full models; PEFT ships one base + eight tiny adapters — same quality, a fraction of the footprint, and you can hot-swap between them on a single server.

Why This Matters

PEFT is the reason a solo developer can fine-tune a frontier-class open model on a rented GPU, and the reason a company can serve dozens of customer-specific models from one base without a storage bill that scales with customers. Knowing the full-vs-PEFT trade — and that the line is depth of change, not money — means you default to the cheap, modular, forget-proof option and only spend on full fine-tuning when the task genuinely demands it. It's the single highest-leverage architectural decision in a fine-tuning project, and it leads directly into the method you'll actually use: LoRA.

🧪 Try It Yourself

Make the storage case. Imagine your team needs 6 task-specific variants of an 8B model. Estimate, for each approach: (1) training memory (full ≈ 8B × 16 vs PEFT ≈ 8B × 2.4), (2) storage to deploy all 6 (full = 6 × the ~16 GB checkpoint vs PEFT = one base + 6 tiny adapters), and (3) whether you'd suffer catastrophic forgetting. Then open the Lab, set 8B + 6 tasks, and check your storage numbers. Finally, name one scenario from your own work where you'd actually choose full FT — and justify it by depth of change, not cost.

Mental-Model Corrections

"PEFT is a cheaper, lower-quality compromise." It trains <1% of weights yet hits 95–99% of full quality — for most tasks the difference is noise.
"PEFT makes the model smaller." No — the base is the same size (and still loaded). PEFT trains a tiny adapter on top of a frozen base.
"Each PEFT task still needs its own full copy." The opposite — that's PEFT's superpower: one shared base + tiny swappable adapters. Full FT is the one that needs N copies.
"Full fine-tuning is always more thorough, so it's safer." It also forgets more and costs ~20× the memory/storage. Use it only for deep behavioral change.
"LoRA is PEFT." LoRA is the most popular PEFT method; PEFT also includes adapters, prefix/prompt tuning, BitFit, and more.

Key Takeaways

Full fine-tuning updates all weights → max flexibility, but ~16 B/param memory, catastrophic forgetting, and a full copy per task.
PEFT freezes the base and trains <1% of the weights (a tiny adapter) → 10–20× less memory, no forgetting, and 95–99% of full-FT quality.
Modularity is the killer win: one shared base + few-MB swappable adapters (one model serves many tasks) instead of N full checkpoints.
The family: LoRA (dominant), adapters, prefix/prompt tuning, BitFit, (IA)³ — they differ in where the trainable parameters go.
Default to PEFT (~80% of cases). Reach for full FT only for deep behavioral change (lots of new capability), where the line is depth, not cost.

Next: L193 — LoRA & QLoRA Explained — exactly how the dominant PEFT method works (low-rank adapters), and how QLoRA's 4-bit base brings it all together.