LoRA Fine-Tuning Run

In L11 (Project Brief & Data Curation) you engineered a clean, leakage-safe dataset — three JSONL files in chat format. Now you run the training and turn that data into a model. Here's the good news: thanks to LoRA and QLoRA, the part that used to need a cluster now fits in a single afternoon on one affordable GPU — even a free Colab T4. The data was the hard part; the run is the fun part.

The old way — full fine-tuning — updates every weight in the model: billions of parameters, hundreds of gigabytes of optimizer state, multiple high-end GPUs, and a fresh full-size copy of the model for every task. LoRA (Low-Rank Adaptation) flips that: freeze the entire base model and train a tiny pair of low-rank matrices bolted onto each layer — under 1% of the parameters — getting ~99% of the quality at a fraction of the memory and cost. QLoRA goes further: load the frozen base in 4-bit, so even an 8-billion-parameter model trains on a 16 GB card.

You'll learn to:

• Configure LoRA — set rank, alpha, and (most importantly) target modules.
• Set up QLoRA — a 4-bit base + the VRAM math that decides whether your run fits or OOMs.
• Pick the recipe — learning rate, epochs, batch size — the handful of hyperparameters that matter.
• Run it with TRL's SFTTrainer and read the loss curves to tell converging from overfitting from diverging.

Why this is a senior skill. The training script is short, but the judgment — what rank, what learning rate, will it fit, is that loss curve healthy or memorizing — is what separates a model that ships from a wasted GPU day. We use the course stack: Hugging Face TRL + PEFT + bitsandbytes, on a small open student (e.g. Llama-3.2-3B).

Scope: this lesson owns the training run. Judging the result against the baseline → L13 (Evaluation vs Baseline).

Full fine-tuning updates all of the model's weights. That means, in memory, you carry the weights, their gradients, and the optimizer's moment estimates (AdamW keeps two per parameter) — roughly 16 bytes per parameter, so an 8B model needs ~130 GB just for training state, before activations. And every task you fine-tune produces a full-size copy of the model to store and serve.

PEFT (Parameter-Efficient Fine-Tuning) — of which LoRA is the dominant form — changes the economics entirely:

• Trainable parameters: <1%. You freeze the base and train tiny adapters, so gradients and optimizer state shrink ~100× — the memory that dominated full fine-tuning nearly vanishes.
• No catastrophic forgetting (mostly). The base is frozen, so its general capabilities are preserved; you're adding a small skill, not overwriting the model.
• Tiny, swappable artifacts. A LoRA adapter is a few to a few hundred MB, not tens of GB. One frozen base can serve many adapters — swap the task by swapping a small file.

For ~80% of fine-tuning jobs — including ours — PEFT is the right default. Full fine-tuning earns its cost only for deep, broad behavioral change (a new language, a foundational capability). For teaching a small model to emit triage JSON, LoRA is plenty — and it's why this whole project fits on one GPU.

The insight behind LoRA: when you fine-tune, the change to a weight matrix has low intrinsic rank — it can be approximated by the product of two much smaller matrices. So instead of learning the full update ΔW (a d × d matrix), LoRA learns ΔW = B · A, where A is r × d and B is d × r, with the rank r tiny (8, 16, 32) compared to d (thousands).

At each adapted layer, the forward pass runs two paths in parallel and sums them:

h = W·x + (α / r)·(B·A·x)

• W is the frozen original weight — it never changes (no gradients, no optimizer state).
• B·A·x is the trainable low-rank update — the only thing that learns.
• α / r is a scaling factor that controls how much the adapter contributes.

Because A and B together have only r·(d + d) parameters versus d² for the full matrix, you train a sliver of the weights — for a 4096-dim layer at r=16, that's about 0.4%. At inference you can even merge B·A back into W so there's zero added latency — the adapter literally folds into the original weights. That's the magic: a tiny pair of thin matrices stands in for a giant weight update.

In code, LoRA is a LoraConfig with three knobs that matter:

What each does:

• r (rank) — the adapter's capacity. Too low under-fits (can't capture the task); too high over-fits and eats your parameter savings. 8–32 covers most tasks; 16 is a great default.
• lora_alpha — the scaling (α / r). The common heuristic is α = 2r (so r=16 → α=32), which roughly doubles the adapter's effective learning rate.
• target_modules — which linear layers get adapters, and this is the most impactful choice. The QLoRA paper found that adapting all linear layers ("all-linear" — attention and the MLP projections) matters more than rank: above r=8, coverage beats rank. Adapting only the attention q/v is the old default and leaves quality on the table.

Rule of thumb: start at r=16, α=32, target_modules="all-linear", dropout=0.05 and only tune from there if the eval (L13) tells you to. More rank is rarely the fix; better data and good coverage usually are.

QLoRA is the trick that put fine-tuning on consumer hardware: quantize the frozen base model to 4-bit (the NF4 format, tuned for neural-net weight distributions), keep the LoRA adapters in higher precision (bf16), and train. The frozen base contributes no gradients or optimizer state anyway, so storing it in 4-bit cuts the weights portion of VRAM by ~4× with minimal quality loss. Load it with bitsandbytes:

Now the VRAM math that decides if your run even starts. Training memory is four parts: weights (4× smaller in 4-bit), LoRA + optimizer (tiny — only the adapters train), activations (scale with batch × sequence length; gradient checkpointing trades compute to shrink them ~3×), and overhead. The payoff is dramatic: a run that would OOM in 16-bit fits comfortably in 4-bit. When you do hit an out-of-memory error, the fixes in order are: stay 4-bit, lower the per-device batch (use gradient accumulation to keep the effective batch), shorten max_seq_length, and keep gradient checkpointing on. The interactive at the end lets you feel exactly where the cliff is.

Fine-tuning has dozens of knobs, but a handful decide the outcome. The 2026 gold default for a LoRA SFT run:

Two failure modes live at the extremes of the learning rate: too high and the loss explodes into NaNs (diverges); too low and the model plateaus, underfit and generic. And the danger of epochs on a small dataset is overfitting — each extra pass over a few hundred examples teaches memorization, not generalization. The whole recipe is just "learn the task without memorizing it," and LoRA is sensitive to LR — when in doubt, 2e-4 and watch the curves.

TRL's SFTTrainer ties it together: it takes your chat-messages JSONL from L11, applies the model's chat template, masks the prompt so loss is computed only on the assistant turn, and runs the loop. You hand it the model, the LoraConfig, your datasets, and an SFTConfig of hyperparameters:

Because your data is in the messages format, SFTTrainer handles the chat template + completion-only loss masking automatically — no manual prompt formatting. Note load_best_model_at_end with eval_loss: it evaluates on the validation set as it trains and keeps the best checkpoint, not the last — your safety net against overfitting. When it finishes, save_model writes the adapter (a tiny adapter_model.safetensors), not a full model copy.

While it trains, the train and validation loss curves tell you everything. Watching the gap between them is the skill — there are four shapes:

• ✅ Converging — both train and val loss descend together and flatten. Healthy. The model is learning the task and it generalizes. Ship the best-val checkpoint.
• ⚠ Overfitting — train loss keeps falling while val loss turns up. The gap is the tell: the model is memorizing the training set. Fix with fewer epochs / early stopping, more (or more diverse) data, or a touch more dropout.
• ⚠ Underfitting — both stay high / flat. It hasn't learned. Fix with a higher learning rate (toward 2e-4) and/or more epochs; check that the data and target modules are right.
• ❌ Diverging — loss oscillates wildly or explodes to NaN. The learning rate is too high. Lower it. This is the #1 way to wreck a LoRA run.

Early stopping formalizes the first two: monitor val loss and stop when it stops improving (a patience of a couple of evals), keeping the checkpoint at the val minimum. One caveat from the research — for generation tasks, val loss can keep looking fine while the model over-memorizes, so don't trust the curve alone: the real verdict comes from task accuracy on the held-out test set, which is exactly what L13 measures.

The whole run, assembled — load the 4-bit base, attach LoRA, train on your L11 data with the recipe, save the adapter:

That's a complete, runnable QLoRA fine-tune. It produces a small adapter — your trained skill, bolted onto a frozen open base. But converging is not the same as good. A loss curve that flattens tells you training worked; it does not tell you the model is actually better than the Claude baseline at triage. That verdict — does the tuned model win, and is it worth shipping? — is L13 (Evaluation vs Baseline).

Before L13, your training run should be done and defensible:

• A LoraConfig with justified r / alpha / target_modules (start 16 / 32 / all-linear).
• A QLoRA setup (4-bit NF4 base via bitsandbytes) that fits your GPU — you can explain the VRAM breakdown and the OOM fixes.
• The recipe applied — lr 2e-4, 2–3 epochs, cosine + warmup, effective batch via grad-accum, sensible max_seq_length.
• Trained with TRL SFTTrainer on your L11 messages JSONL, with validation during training and load_best_model_at_end.
• You can read the loss curves — and your run converged (not overfit/underfit/diverged).
• A saved adapter (adapter_model.safetensors, tens of MB) — and the best checkpoint, not just the last.

If you have a converged adapter and can defend every hyperparameter, you're ready to find out whether it's actually good — in L13.

This lab is the whole run made playable. Configure a real QLoRA job — base model, GPU, 4-bit vs 16-bit, LoRA rank, batch, sequence length, learning rate, epochs, dataset size — and watch the pipeline play out the two gates and the payoff: does it fit (VRAM), does it train (the outcome), and is it worth shipping (eval delta + artifact).

• Load an 8B in 16-bit on a 12 GB card → OOM before training starts. Flip to 4-bit QLoRA and watch it fit.
• Drag the learning rate up past ~1e-3 → diverged. Bring it back to 2e-4 → converged.
• Pile on epochs over a tiny dataset → overfit: task accuracy up, but general capability down (forgetting).
• Land a clean run and read the artifact: a tiny LoRA adapter + a quantized GGUF ready to serve.

Notice the two gates. One — memory: QLoRA's 4-bit base and gradient checkpointing are what turn an OOM into a run that fits a free T4. Two — the recipe: the learning rate decides converge-vs-diverge, and epochs-vs-dataset-size decides converge-vs-overfit. A green run here is a small adapter that beat the base — which L13 will measure for real.

Reason these out, then check against the lab and the code.

1. Full fine-tuning an 8B model needs ~130 GB of training memory, but a QLoRA run fits in ~6 GB. Name the two changes that close that 20× gap.

2. Your loss curve shows training loss steadily falling while validation loss bottoms out and starts rising. What's happening, and what are two fixes?

3. A teammate sets learning_rate=2e-3 'to train faster.' What will the loss curve do, and what's the safe value?

4. Between raising the rank from 16 to 64 and adapting all linear layers instead of just attention, which usually improves quality more — and what does that tell you?

5. Your run converged — train and val loss both flattened nicely. Are you done? Why or why not?

Answers.

1. (a) LoRA freezes the base, so you no longer store gradients and optimizer state for billions of params — only for the tiny adapters (~100× less of the part that dominated). (b) QLoRA loads the frozen base in 4-bit, cutting the weights memory ~4×. Frozen + 4-bit = the run fits one small GPU.

2. Overfitting — the model is memorizing the training set; the rising val loss (and the gap) is the tell. Fixes: fewer epochs / early-stop at the val minimum, more or more diverse data, or a bit more LoRA dropout. Keep the best-val checkpoint, not the last.

3. It will diverge — the loss oscillates or explodes to NaN. A too-high LR is the #1 cause of a wrecked LoRA run. Use 2e-4 (LoRA already wants ~10× a full-FT LR; 2e-3 is another 10× too far).

4. Adapting all linear layers (target_modules="all-linear") usually helps more. The QLoRA paper showed that above r=8, coverage of target modules matters more than rank — so reach for broader coverage before bigger rank (which also costs more memory and risks overfitting).

5. No — converging only proves training worked, not that the model is good. A flat loss curve can hide over-memorization, and it says nothing about whether the tuned model actually beats the Claude baseline on real triage or holds its general capability. The real verdict is task accuracy on the held-out test set — that's L13 (Evaluation vs Baseline).

• "Fine-tuning needs a cluster." → With QLoRA, an 8B model fine-tunes on a single 16 GB GPU — even a free Colab T4. Frozen 4-bit base + tiny adapters changed the economics.
• "LoRA trains the whole model." → It freezes the base and trains a low-rank adapter (<1% of params). The base never moves — which is also why it doesn't forget.
• "Higher rank = better." → Above r=8, target-module coverage beats rank. Adapt all linear layers before you raise rank; more rank mostly adds memory and overfitting risk.
• "Use the same learning rate as full fine-tuning." → LoRA wants a higher LR (~2e-4, ~10×). Too low → underfit; too high → diverge (NaNs). It's the most important knob.
• "More epochs = better." → On a small dataset, extra epochs memorize. Use 2–3 (1–2 under 500 examples) and early-stop at the val minimum.
• "QLoRA is just LoRA but worse." → QLoRA adds a 4-bit base for big memory savings at minimal quality loss. 16-bit LoRA is slightly sharper/faster if it fits; QLoRA is how you fit when it doesn't.
• "A flat loss curve means it's good." → A flat curve means training converged, not that the model is better than the baseline. Converging can even hide over-memorization. Judge it on the held-out test (L13).
• "Save the last checkpoint." → Save the best-validation checkpoint (load_best_model_at_end). The last step is often past the overfitting point.

• LoRA freezes the base and trains a tiny low-rank adapter (h = Wx + (α/r)·BAx) — <1% of the params, ~99% of the quality, no catastrophic forgetting, and a few-MB swappable artifact.
• Config that matters: r (capacity, 16), alpha (scaling, ~2r), and — most important — target_modules="all-linear" (coverage beats rank above r=8).
• QLoRA = a 4-bit NF4 base + bf16 adapters → cuts weight memory ~4× so an 8B fits a 16 GB GPU. Know the VRAM parts (weights / LoRA+opt / activations / overhead) and the OOM fixes (4-bit, smaller batch, shorter seq, grad-checkpoint).
• The recipe: lr 2e-4 (the key knob), 2–3 epochs, cosine + warmup, effective batch via grad-accum. Too-high LR → diverge; too many epochs on small data → overfit.
• Run it with TRL SFTTrainer on your L11 messages JSONL (auto chat-template + completion-masking), validate during training, and keep the best-val checkpoint — a small saved adapter.
• Read the curves: converge (both descend), overfit (val turns up — the gap), underfit (both high), diverge (NaN). But the curve isn't the verdict.
• Next — L13 (Evaluation vs Baseline): converging proves training worked; now prove the tuned model actually beats the Claude baseline on the sacred held-out test set — and is worth shipping.