Estimating Cost & Effort

Introduction

Section 1 has taught you when to fine-tune (L185 (Good Reasons to Fine-Tune)), when not to (L186 (Bad Reasons to Fine-Tune)), and how to combine it with RAG (L187 (Fine-Tuning + RAG Together)). The last question before you commit is the one that kills or greenlights the project in a budget meeting: what does this actually cost — in money, time, and ongoing effort?

Here's the punchline up front, because it's the most common estimating mistake: the GPU bill everyone fixates on is the smallest line. With LoRA/QLoRA, training is often under $20. The real cost is everywhere else — the data, the serving, and the maintenance. This lesson gives you the four lines of a fine-tuning budget so you estimate the whole bill, not the part that's easy to Google.

An infographic titled 'Estimating Cost & Effort', the fifth lesson of the fine-tuning container and the close of section one, showing the four lines of a fine-tuning budget and the counter-intuitive truth that the GPU bill everyone fixates on is the smallest one. Line one, compute: with LoRA and QLoRA, fine-tuning a 7 to 8 billion parameter model takes two to four hours on a single A100 for under twenty dollars, often under ten, because parameter-efficient methods cut compute by more than ninety percent versus full fine-tuning, turning a run that needed an eight-GPU cluster and fifty thousand dollars in 2022 into a few hundred dollars on a single GPU, while full fine-tuning of even a 13B model on one GPU is essentially not done anymore for a one to two percent accuracy difference. Line two, data, is the real bill and about eighty percent of the work: human annotation costs roughly fifty cents to fifty dollars per example, while distillation, where a strong teacher model labels your inputs, costs under a third of a cent each, about twenty times cheaper, and the eighty-twenty rule means most of the gain comes from the first well-chosen twenty percent of examples, so quality beats quantity. Line three, serving, is recurring: a scale-to-zero endpoint for bursty traffic costs twenty to sixty dollars a month, but an always-on dedicated endpoint bills twenty-four seven for hundreds to thousands of dollars a month, often the dominant cost. Line four, maintenance, is the re-training treadmill as data drifts, specs change, and base models upgrade. A note on the landscape: OpenAI wound down its self-serve fine-tuning API in May 2026, pushing teams toward open-model LoRA on rented GPUs or toward smaller base models with prompt caching, so self-hosted LoRA is increasingly the path. The takeaway is to estimate the whole bill, not the GPU-hours: compute is a rounding error, and data plus serving plus maintenance are where the money goes.

The Four Lines of a Fine-Tuning Budget

Every fine-tuning project has the same four cost lines. Price all four or your estimate is fiction:

Compute — the GPU-hours to train. (The line everyone quotes — and the smallest.)
Data — sourcing, labeling, and cleaning your examples. (~80% of the work.)
Serving — running the fine-tuned model in production. (Recurring, often the biggest over a year.)
Maintenance — re-training as data drifts, specs change, and base models upgrade. (The treadmill from L186.)

Plus the one that doesn't show up on a cloud invoice: engineering time — and it's mostly spent on line 2. Let's size each.

Compute Is the Smallest Line (Thanks to LoRA/QLoRA)

The number that used to scare CFOs has collapsed. LoRA and QLoRA cut training compute by 90%+ versus full fine-tuning — a run that needed an 8-GPU cluster and $50,000 in 2022 now runs on a single GPU for a few hundred dollars, and for small models, much less:

Job	GPU & time	Compute $
QLoRA, 7–8B, ~5k examples	~2–4 h on one A100	< $20 (often < $10)
QLoRA, 70B	~8–12 h on one H100	~$10–16
Full FT, 8–13B	multi-GPU cluster	90×+ more — and rarely worth it

Full fine-tuning of even a 13B model on a single GPU is basically not done anymore — it's too slow and memory-hungry, and QLoRA gives up only ~1–2% accuracy (a rounding error). QLoRA's 4-bit quantization is what drops an 8B onto a free Colab T4. You'll learn the memory math behind this in L190 (The Memory Bottleneck & Memory Math) and the methods in L193 (LoRA & QLoRA Explained) — for now, just know: training compute is a sliver.

Data Is the Real Bill (And ~80% of the Effort)

If compute is the smallest line, data is usually the biggest — and almost always the most work. The cost swings wildly with how you get your labels:

You already have it (corrected production traces, logs): cheapest — but rare and still needs cleaning.
Distillation (a strong teacher model labels your inputs — the pattern from L185): < $0.003 per example — about 20× cheaper than humans. This is the default in 2026.
Human annotation: $0.50–$50 per example depending on expertise. It adds up fast — 600 high-quality RLHF annotations can cost $60,000 — and takes days to weeks for 5k–100k examples.

And the lever that saves the most: the 80/20 rule — ~80% of the performance comes from the first ~20% of well-chosen examples. Quality beats quantity. A few hundred clean, diverse, correct examples beat thousands of mediocre ones (and you'll size this properly in L195 (Data Quality, Coverage & Quantity)). Don't pay to label data you don't need.

Serving & Maintenance — The Recurring Lines

Training is a one-time cost; serving is forever. This is where a cheap-to-train model becomes expensive to run:

Scale-to-zero (autoscaling, bursty traffic — 100–1,000 req/day): ~$20–60/month. You pay only when it runs. Great for low/spiky volume.
Always-on dedicated endpoint: bills 24/7. A single GPU endpoint is hundreds/month; a 70B served always-on is $1,400–5,700/month depending on provider — often the single biggest line over a year, dwarfing the one-time build.

Then maintenance: every base-model upgrade, data drift, and spec change is a re-train (the treadmill from L186). Budget it as a recurring cost, not a one-off. The honest comparison isn't "$15 to train vs. a prompt" — it's "$15 + $X/mo serving + a re-train every quarter vs. a prompt you change for free."

Managed vs Self-Host (And a 2026 Shift)

Two ways to pay for all this:

Self-host: rent a GPU (A100 ~$1/hr, H100 ~$1.5/hr), train a LoRA (our stack — TRL/PEFT or Unsloth), and serve it yourself (or via a serverless endpoint). Cheapest and most flexible; you own the ops.
Managed: a platform hosts training + serving for a per-token premium.

A real 2026 landscape shift: OpenAI wound down its self-serve fine-tuning API in May 2026, recommending teams move to prompt caching + smaller base models (which now match fine-tuned economics for most workloads) — or to open-model LoRA. The practical effect: self-hosted LoRA on open weights (Llama/Qwen) is increasingly the path, which is exactly the stack this container teaches. Here's the four-line estimate as code:

# Back-of-envelope: the FOUR lines of a fine-tuning budget. Most teams price only line 1.
N = 5_000                       # training examples

# 1) COMPUTE - QLoRA on an 8B, ~3h on one A100 (~$1/hr). The line everyone fixates on:
compute      = 3 * 1.00         # ~ $3      (full FT would be ~90x more, on a multi-GPU cluster)

# 2) DATA - ~80% of the WORK. Distill from a teacher vs human-label:
data_distill = N * 0.005        # ~ $25     (a teacher labels each example for < $0.003)
data_human   = N * 1.50         # ~ $7,500  (~20x more - and days-to-weeks of effort)

# 3) SERVING - RECURRING. scale-to-zero (bursty) vs an always-on endpoint:
serve_scale  = 45               # ~ $45/mo
serve_always = 1.00 * 730       # ~ $730/mo  ->  $8,760/yr

# 4) MAINTENANCE - re-train on every drift / spec change / base upgrade (the treadmill).

print(f"compute ${compute}  |  data(distill) ${data_distill}  |  serving(always)/yr ${serve_always*12}")
# compute $3 is a ROUNDING ERROR next to data and a year of serving.
# Estimate the WHOLE bill - then ask if a prompt or smaller base model is cheaper end-to-end.

See It: The Cost & Effort Estimator

Price your own project. Set the model, method, dataset size, data source, and serving — and watch the "where the money goes" bar. Two things to try: switch QLoRA → full fine-tune and watch the VRAM/compute explode; and switch distill → human-label (or scale-to-zero → always-on) and watch the real bill appear while the GPU line barely moves.

Interactive: the Fine-Tuning Cost & Effort Estimator, a calculator. The user configures a project with five inputs: base model size (1B, 3B, 8B, 70B), method (full fine-tune, LoRA, QLoRA), dataset size on a slider, data source (you have it, distill from a teacher, or human-label), and serving (scale-to-zero or always-on). The estimator computes live: the VRAM needed and the cheapest GPU that fits (QLoRA dropping a model onto a single small GPU while full fine-tuning explodes to a multi-GPU cluster), training GPU-hours and dollar compute, data cost and person-days of effort, monthly serving cost, and a where-the-money-goes stacked bar for the first year split across compute, data, and serving. The teaching payload is visceral: the training compute everyone fixates on, with LoRA or QLoRA, is the smallest slice, while data, especially human-labeled, and an always-on serving endpoint dominate the real bill, and a verdict calls out which line is eating the budget and how to cut it.

The bar makes the lesson un-ignorable: the blue compute slice is a sliver next to data and a year of serving. Estimate the whole rectangle.

Why This Matters

Fine-tuning projects don't die because the GPU was too expensive — they die because someone estimated the GPU and forgot the other three lines, then got surprised by a $8k/year serving bill and a re-training treadmill. The engineer who can say "training is $15, but data is three weeks and $7k, serving is $700/month, and we re-train quarterly — versus a prompt that's free to change" is the one whose proposals get funded and whose systems don't blow the budget in month three. Estimating is a feature, not an afterthought.

🧪 Try It Yourself

Budget a real fine-tune. Take a project you'd consider (or use: "QLoRA an 8B to triage support tickets, 5,000 examples, ~2,000 requests/day"). Estimate all four lines: (1) compute (method + GPU + hours), (2) data (distill vs human × N — and how few examples you really need, per 80/20), (3) serving (scale-to-zero or always-on?), (4) maintenance (re-train how often?). Write the one-year total and the single biggest line. Then open the Estimator, reproduce it, and check: was your instinct about which line dominates correct? Finally — would a prompt or a smaller base model be cheaper end-to-end?

Mental-Model Corrections

"Fine-tuning is expensive because of GPUs." Not with LoRA/QLoRA — training is often < $20. The expense is data + serving + maintenance.
"More data is better, so budget for lots of labels." The 80/20 rule: most gain is in the first ~20% of well-chosen examples. Pay for quality, not volume — and distill instead of human-labeling where you can (~20× cheaper).
"It's a one-time cost." Serving is monthly and maintenance is a treadmill. A cheap-to-train model can be expensive to run.
"Full fine-tuning is more thorough, so it's worth the compute." It's ~90× the compute for ~1–2% accuracy — and often won't fit one GPU. QLoRA is the default.
"We'll just use a managed fine-tuning API." Check availability first — OpenAI wound down self-serve fine-tuning in 2026. Self-host LoRA on open weights, or compare against a smaller base + prompt caching.

Key Takeaways

A fine-tuning budget has four lines: compute, data, serving, maintenance (plus engineering time). Price all four.
Compute is the smallest line: LoRA/QLoRA cut it 90%+ — a 7–8B QLoRA is under $20, often under $10. Full FT is ~90× more for ~1–2% accuracy.
Data is the real bill (~80% of effort): human labeling is $0.50–$50/example; distillation is ~20× cheaper; and 80/20 means quality of the first ~20% of examples matters most.
Serving & maintenance are recurring: scale-to-zero (~$20–60/mo) vs always-on (hundreds–thousands/mo), plus a re-train treadmill — often the biggest cost over a year.
2026 shift: OpenAI deprecated self-serve fine-tuning → self-host LoRA on open models (this container's stack), or compare against a smaller base + prompt caching.
Estimate the whole bill, not the GPU-hours — and always check whether a prompt or a smaller base is cheaper end-to-end.

That closes Section 1 — When (and When Not) to Fine-Tune. Next, Section 2 — Fine-Tuning Mechanics opens with L189 — How Training Actually Works (Backprop Recap), then the memory math, quantization, and LoRA/QLoRA that make all these numbers real.