The Memory Bottleneck & Memory Math

Introduction

Last lesson — L189 (How Training Actually Works) — ended with a number that should bother you: training needs roughly ~4× the model's size just to hold the loop. This lesson makes that exact, because it's the single biggest practical constraint in fine-tuning: can I even fit this on my GPU?

Here's the surprise that trips up everyone new to this: a 7-billion-parameter model needs far more than 7B-worth of memory to train. In fact, full fine-tuning a 7B needs ~112 GB — more than a single 80 GB A100. The model that happily runs on a 16 GB laptop GPU can't be trained on a data-center card. Why? Because running a model and training it are completely different memory problems. Let's do the math — and then see exactly how LoRA and QLoRA collapse it.

An infographic titled 'The Memory Bottleneck & Memory Math', the second lesson of section two on fine-tuning mechanics, explaining exactly why a model needs far more GPU memory to train than its parameter count suggests. During training, four things sit in VRAM: the model weights, a gradient for every weight, the optimizer states, and the activations from the forward pass that the backward pass needs. The math is bytes-per-parameter. In mixed precision with AdamW, full fine-tuning pays about sixteen bytes per parameter: weights at two bytes in fp16, gradients at two bytes, and the AdamW optimizer at eight bytes because it keeps a momentum and a variance per weight in fp32, plus a fp32 master copy of the weights. So a 70-billion parameter model needs roughly 140 gigabytes for weights, 140 for gradients, and 560 for the optimizer, about 840 gigabytes to full-fine-tune, which is around eleven 80-gigabyte GPUs, and even a 7-billion model needs about 112 gigabytes, more than a single 80-gigabyte card. Training needs three to five times more memory than inference, which only holds the weights plus a small key-value cache. Activations are the variable line, scaling with batch size and sequence length, and gradient checkpointing trades compute to shrink them. The collapse comes from PEFT. LoRA freezes the base model, so 99.5 percent of the weights need no gradients and no optimizer states, only a tiny adapter does, cutting a 70B run to about 140 gigabytes. QLoRA additionally quantizes the frozen base to 4-bit NormalFloat, a seventy-five percent cut on the weights, bringing a 70B down to about 36 gigabytes so it trains on a single 80-gigabyte GPU, and a 7B fits a 16-gigabyte card, at the cost of twenty to forty percent slower throughput from dequantization. The takeaway is that memory is the real bottleneck of fine-tuning, the optimizer state is the biggest line, and freezing the base plus 4-bit quantization is what makes fine-tuning fit on affordable hardware.

The Four Things Living in GPU Memory

During a training step, four things must sit in VRAM at once (all four from the loop you just learned):

Weights — the model's parameters themselves.
Gradients — backprop produces one gradient per weight (same size as the model).
Optimizer states — AdamW keeps a momentum and a variance per weight (plus a master copy). This is usually the biggest line.
Activations — the intermediate outputs of every layer from the forward pass, kept around because the backward pass needs them. Scales with batch size and sequence length.

Inference only needs #1 (weights) + a small KV cache. That's the whole reason training needs 3–5× more memory than inference — and why a model you can run on a small GPU you often can't train on a big one.

The Bytes-per-Parameter Math

Memory is just bytes-per-parameter × parameters. First, how many bytes a number takes, by precision:

Precision	Bytes/param
fp32 (full)	4
fp16 / bf16 (half)	2
int8	1
int4 / NF4	0.5

Now add up the four components for full fine-tuning in the standard mixed-precision + AdamW setup — the famous ~16 bytes/param:

weights (fp16): 2 · gradients (fp16): 2 · AdamW optimizer (momentum + variance, fp32): 8 · (+ a fp32 master copy of weights: ~4) → ~16 bytes/param.

Notice the optimizer is half the bill. Here's the math as code:

# Training VRAM = weights + gradients + optimizer + activations. Compute the bottleneck.
P = 70e9                      # parameters (70B)
gb = lambda nbytes: nbytes / 1e9

# FULL fine-tuning, mixed precision + AdamW  (the classic ~16 bytes/param):
weights   = gb(P * 2)         # fp16:              2 bytes/param  ->  140 GB
gradients = gb(P * 2)         # fp16:              2 bytes/param  ->  140 GB
optimizer = gb(P * 8)         # AdamW (m, v fp32): 8 bytes/param  ->  560 GB   <- the killer
print('full FT 70B:', round(weights + gradients + optimizer), 'GB')   # ~840 GB  ->  ~11x H100

# QLoRA: 4-bit FROZEN base (no grads/optimizer for it) + a tiny ~0.5% adapter:
base    = gb(P * 0.5)                 # 4-bit NF4:  0.5 bytes/param  ->  35 GB
adapter = gb(P * 0.005 * 10)          # adapter weights+grads+optimizer (tiny)  ->  ~3.5 GB
print('QLoRA 70B:', round(base + adapter), 'GB')   # ~38 GB  ->  fits ONE 80GB GPU
# Same model. 840 GB vs 38 GB. That ~22x gap is the whole point of this lesson.

The 70B Example: ~840 GB to Train, ~140 GB to Run

Put real numbers on a 70B model and the bottleneck is undeniable:

	Full fine-tune	Inference
weights	140 GB	140 GB
gradients	140 GB	—
optimizer (AdamW)	560 GB	—
activations	10–100+ GB	small KV cache
total	~840 GB	~140 GB

~840 GB to full-fine-tune means ~11× 80 GB GPUs — a cluster. The same model infers on ~140 GB (two cards). And scale down: even a 7B is 7 × 16 = ~112 GB to full-fine-tune — still more than one 80 GB A100. This is why "just fine-tune it" is a hardware project, not a script — unless you change the math.

Activations — the Variable Line

Three of the four components scale with the model (fixed once you pick the model + method). The fourth — activations — scales with your batch size × sequence length × number of layers, so it's the line you control at run time. Long sequences and big batches can push activations from a few GB to 100+ GB.

The standard fix is gradient (activation) checkpointing: instead of storing every layer's activations, store only a few and recompute the rest during the backward pass — trading ~30% more compute for a ~5–10× cut in activation memory. It's the first lever you reach for when you're close to fitting. (Drag the batch / sequence sliders in the calculator below to feel this line move.)

How LoRA & QLoRA Collapse the Bottleneck

The whole reason PEFT (parameter-efficient fine-tuning) exists is to attack this memory math directly — and it's startlingly effective:

LoRA freezes the base model and trains only a tiny adapter (~0.5% of the weights). Frozen weights need no gradients and no optimizer states — so #2 and #3 vanish for 99.5% of the model. A 70B drops from ~840 GB to ~140 GB (now dominated by the still-fp16 base weights).
QLoRA goes further: it quantizes the frozen base to 4-bit (NF4) — a 75% cut on the weight line — plus double quantization and paged optimizers. A 70B drops to ~36 GB, so it trains on a single 80 GB GPU; a 7B QLoRA fits a 16 GB card (a free Colab T4).

That's ~22× less memory than full fine-tuning — the difference between a GPU cluster and a laptop. The trade-off is ~20–40% slower training (the 4-bit base must be de-quantized on every forward pass). You'll learn quantization in L191 (Numerical Precision & Quantization) and LoRA/QLoRA in depth in L193 (LoRA & QLoRA Explained) — this lesson is why they matter.

See It: The Training VRAM Calculator

Do the memory math by dragging. Pick a model and method, set the batch and sequence length, and watch the four bars. The move that makes it click: take a 70B, start on full FT (≈840 GB, ~11 GPUs), then flip → LoRA (gradients + optimizer collapse) and → QLoRA (the weights bar shrinks 4×) until it fits a single 80 GB GPU.

Interactive: the Training VRAM Calculator. The user configures a training run with five inputs, base model size (1B, 8B, 13B, 70B), method (full fine-tune, LoRA, QLoRA), optimizer (AdamW or 8-bit Adam), and batch size and sequence length on sliders, and the calculator breaks the GPU memory into its four components live, weights, gradients, optimizer states, and activations, shown as a stacked bar with gigabytes on each, a total, the cheapest GPU or GPUs that fit, and a training-versus-inference comparison with the multiple. The teaching is visceral: flipping from full fine-tune to LoRA collapses the gradients and optimizer bars because the base is frozen and only a tiny adapter needs them, and flipping to QLoRA shrinks the weights bar fourfold from 4-bit quantization, turning an 840-gigabyte eleven-GPU full fine-tune of a 70B into a 36-gigabyte run on a single 80-gigabyte GPU, while the batch and sequence sliders grow the activations.

Same model, same task — the only thing that changed is which parameters carry gradients and optimizer state, and how many bits each weight takes. That's the entire game of fine-tuning mechanics.

Why This Matters

Every "can we fine-tune this?" decision is, underneath, a memory decision — and the engineers who can do this math in their head don't waste a week discovering a run won't fit, or burn money renting an 8-GPU node for a job a single card could do with QLoRA. It also demystifies the entire toolbox: quantization, LoRA, QLoRA, gradient checkpointing, 8-bit optimizers aren't a grab-bag of tricks — they're each an attack on one of the four lines of this equation. Know the equation and the rest of the section is just which line am I cutting?

🧪 Try It Yourself

Do the math, then check it. Without the calculator, estimate the VRAM to full-fine-tune an 8B model with AdamW: weights (8B × 2) + gradients (× 2) + optimizer (× 8) ≈ ? GB. Will it fit one 80 GB A100? Now open the VRAM Calculator, set 8B + full FT, and check your number. Then flip to QLoRA — how many GB, and which single GPU does it fit on now? Finally, drag the sequence length to 8,192 and watch which bar grows — and name the technique that would shrink it.

Mental-Model Corrections

"A 7B model needs ~7 GB to fine-tune." It needs ~112 GB for full fine-tuning (16 bytes/param). ~14 GB is the inference number; training is ~5× that.
"The weights are the big memory cost." Usually the optimizer is — AdamW's momentum + variance is 8 bytes/param, more than weights + gradients combined.
"LoRA saves memory by making the model smaller." No — the base is the same size (and still loaded). LoRA saves by freezing it, so 99.5% of the weights need no gradients or optimizer state.
"QLoRA just means LoRA on a quantized model." Right idea — and that 4-bit base is exactly what cuts the weight line 4×, on top of LoRA's gradient/optimizer savings.
"If it runs, I can train it." Running needs weights only; training needs weights + gradients + optimizer + activations — 3–5× more.

Key Takeaways

Training VRAM = weights + gradients + optimizer states + activations. Inference is just weights (+ KV), so training needs 3–5× more.
Full fine-tuning ≈ 16 bytes/parameter (fp16 weights 2 + gradients 2 + AdamW optimizer 8 + master 4) — the optimizer is the biggest line. A 70B ≈ 840 GB; even a 7B ≈ 112 GB (> one 80 GB GPU).
Activations are the variable line (batch × seq × layers); gradient checkpointing trades ~30% compute for a ~5–10× cut.
LoRA freezes the base → no gradients/optimizer for 99.5% of weights (70B → ~140 GB). QLoRA also 4-bits the base → 70B on one 80 GB GPU, 7B on a 16 GB card — ~22× less than full FT, for ~20–40% slower training.
Every memory technique — quantization, LoRA/QLoRA, checkpointing, 8-bit optimizers — is an attack on one of the four lines.

Next: L191 — Numerical Precision & Quantization (GGUF/llama.cpp, GPTQ, AWQ) — exactly how those fewer-bit number formats work, the lever behind the weight line.