Serving the Model

You curated the data (L11), trained the adapter (L12), and proved it beats the baseline on a clean test (L13). All of that is worthless until the model is actually serving tickets in production — cheaply, fast, and reliably. This final lesson of the project takes your LoRA adapter and turns it into a live endpoint doing real work.

There's a path from "a file in out/triage-lora/" to "a model triaging thousands of tickets a day," and each step is a decision:

• The artifact — merge the adapter into the base for one simple model, or keep it separate to hot-swap many adapters on one GPU.
• Quantize for inference — shrink and speed it up with GGUF (local/Ollama) or AWQ (GPU/vLLM).
• Pick a runtime — vLLM for high-throughput production, Ollama for local/dev — behind an OpenAI-compatible endpoint so it's a drop-in for the model you're replacing.
• Serve the strategy — wire the cascade from the brief: the cheap tuned model takes every ticket as a first pass, and only the hard ones escalate to Claude. That's where the project's cost win is actually realized.

Why this is a senior skill. A trained model that sits in a bucket creates zero value. Knowing how to merge, quantize, serve, and route it — turning a fine-tune into a fast, cheap, monitored endpoint — is what makes the whole project pay off. We use the course stack: vLLM (production) / Ollama (local), an OpenAI-compatible API, and the cascade routing that lets a small model carry the load.

Scope: this lesson owns serving the model + the production routing. It also completes the project — built, trained, proven, shipped.

Your trained artifact is a LoRA adapter — a small file that only means something on top of its base model. You have two ways to serve it:

• Merge — fold the adapter's B·A back into the base weights W, producing one standalone model. Simplest to serve, and zero added inference latency (there's no separate adapter to apply). This is the right default for a single fine-tune like our triage model.
• Keep separate (multi-LoRA) — serve the frozen base once and swap small adapters per request. This lets you serve dozens of fine-tunes on one GPU (one per customer/task) — the adapters are tiny, so they hot-swap cheaply. vLLM supports this with --enable-lora --lora-modules. Brilliant when you have many tunes; overkill for one.

We'll merge — one task, one model, lowest latency:

Now out/triage-merged is a normal Hugging Face model directory — nothing about it says "fine-tune." It's ready to quantize and serve like any model. (Keep the adapter around too — it's your few-MB source of truth, and you may want multi-LoRA later.)

Serving a model in full bf16 is wasteful: it's large and slow. Post-training quantization shrinks the weights to 4-bit for inference, cutting memory ~4× and speeding up generation, with minimal quality loss. This is a different quantization than the QLoRA 4-bit you trained with — that was to fit training; this is to serve fast. Pick the format by where you serve:

• GGUF q4_k_m — for local / single-user serving with llama.cpp / Ollama. The community sweet spot: retains ~95% of full-precision quality at ~4× smaller. Dead simple to run.
• AWQ (or GPTQ) — for GPU / high-throughput serving with vLLM. AWQ gives the best quality-to-speed on a GPU and is what vLLM is happiest with under load.

For production (vLLM), quantize to AWQ:

Always re-check quality after quantizing on your held-out test set (L13) — 4-bit usually costs a point or two, occasionally more on a format-strict task like JSON. If the drop is acceptable, you've got a model that's a quarter the size and noticeably faster — for free.

The runtime is the engine that loads your model and turns requests into tokens — and the two you'll reach for solve different problems:

• vLLM — for production throughput. Its two innovations are PagedAttention (manages the KV cache like virtual-memory paging, so it packs far more concurrent requests into GPU memory) and continuous batching (it admits a queued request the moment another finishes, keeping the GPU full). Under 10+ concurrent requests, vLLM delivers roughly 3–8× the throughput of a serial runtime — exactly what a high-volume ticket firehose needs.
• Ollama / llama.cpp — for local & single-user. Trivial to run (ollama run triage), great on a laptop or a single dev box, excellent single-request latency. But its simple batching means throughput collapses under concurrency — wrong tool for a busy production endpoint.

Rule of thumb: Ollama to develop and demo, vLLM to serve at scale. Same model, same adapter — you just point a different engine at it. (Both can put your model behind the same OpenAI-compatible API, so the rest of your stack doesn't know or care which is running.)

Here's the move that makes your fine-tune a drop-in: serve it behind an OpenAI-compatible HTTP endpoint (/v1/chat/completions). Then any code that already calls an LLM — including the very Claude calls you're replacing — can point at your model by changing one base URL. vLLM ships this server built-in:

That's the integration payoff of the whole project: your specialized, ~20× cheaper model now answers at the same API shape as the frontier model it replaces. Swapping it in (or out) is a config change, not a rewrite — and it makes the cascade in a moment trivial to wire.

Serving at volume means thinking about two different speeds, not one:

• Latency (per request) — split into TTFT (time to first token, dominated by prefill of the prompt) and TPOT (time per output token, the decode speed). For triage — a short prompt and a short JSON output — both are small, so a tuned small model answers in tens of milliseconds vs a frontier model's 200–300 ms.
• Throughput (per GPU) — requests/second across all concurrent users. This is where vLLM's continuous batching earns its keep: instead of running requests one-by-one (slots sitting idle when short requests finish), it keeps the GPU saturated, multiplying requests/second.

The two trade off: cranking concurrency raises throughput but also each user's latency (their request shares the GPU). Tune batch size / max concurrency to your SLO — for a high-volume background triage pass you favor throughput (it's not user-facing); for an interactive feature you'd protect latency. Either way, measure cost per 1,000 tickets — the number that justified this whole project — and you'll find the small served model is a fraction of the frontier API's bill.

Now the payoff. The brief was never "replace Claude entirely" — it was "run a cheap model on the easy majority and save the frontier model for the hard minority." That's a cascade: try the cheap tuned model first, and escalate only when it's not confident.

• Every ticket → the tuned model first (your ~20× cheaper, fast endpoint). For the ~90% of routine tickets, it's done — cheaply.
• Escalate the uncertain ones → Claude. If the tuned model's confidence is low (or its output isn't valid schema), bump the ticket up to the frontier baseline. You spend frontier tokens only on the cases that provably needed them.

Done well, a cascade cuts cost 45–85% while keeping ~95% of quality — it can even beat a single frontier model on both. The one hard knob is the confidence threshold: set it too high and you escalate everything (paying more than just using Claude); too low and the cheap model answers things it can't, degrading quality silently. Tune it on your eval set.

The interactive below lets you feel this: send a batch of mixed tickets through all-frontier (overpay), all-cheap (the silent-failure trap), and cascade (the sweet spot), and watch cost and quality move. All-cheap is the trap — a cheap wrong answer is worse than an expensive right one, which is exactly why you keep the online eval from L13 running in production to catch silent degradation.

A served model is a production service, so the operational discipline from L10 (Ship It) applies here too — now for a model you host yourself:

• Package & place it — containerize the vLLM server, put it on a GPU instance, and autoscale on load (a triage firehose is bursty). A merged + AWQ 3B fits comfortably on a modest GPU.
• Guardrails — a timeout and a fallback to Claude if your endpoint is down (the cascade already gives you this for free), plus a sane max concurrency so you don't OOM the GPU.
• Monitor — track throughput, p95 latency, cost/1k, escalation rate, and the schema-valid % live. A rising escalation rate is your early warning that the world drifted from your training data.
• Keep evaluating online — sample live tickets and score them (the online eval from L13). The data flywheel closes here: today's hard, escalated tickets — now labeled by Claude — are tomorrow's training data for the next version of the tuned model.

That last point is the whole loop: serve → monitor → collect the escalations → retrain → re-evaluate → re-serve. Your fine-tune isn't a one-shot artifact; it's a product that improves as it runs.

The full serving path, assembled — merge, (quantize,) serve behind an OpenAI endpoint, and route through the cascade:

That's it — a specialized, cheap, fast model serving the easy majority, a frontier safety net for the hard minority, behind a standard API, with monitoring and an online-eval loop. The model you built, trained, and proved is now earning its keep in production. The project is complete.

Your model is served when:

• The adapter is merged into a standalone model (or kept separate for multi-LoRA, deliberately).
• It's quantized for inference (AWQ for vLLM / GGUF q4_k_m for Ollama) and you re-checked quality on the held-out test after quantizing.
• It's served behind an OpenAI-compatible endpoint (vLLM for production / Ollama for local) — a drop-in for the model it replaces.
• You can speak to throughput vs latency (continuous batching, TTFT/TPOT) and report cost per 1k tickets.
• The cascade is wired — tuned model first, escalate to Claude on low confidence — with a tuned threshold.
• It's deployed and monitored (autoscale, fallback, p95, escalation rate, schema-valid) with online eval running — closing the flywheel.

If a real ticket can hit your endpoint, get triaged by the cheap model (or escalated), and show up on a dashboard — you've shipped a fine-tuned model.

Serving isn't just running the model — it's routing to it wisely, and that's where your cheap fine-tune earns its keep. Read your project's two tiers into this lab: the cheap model is your tuned student, the frontier model is the Claude baseline. Flip between the four strategies and watch cost, quality, and savings move.

• All-frontier → every ticket to Claude: 100% quality, but you overpay on the easy majority.
• All-cheap → every ticket to the tuned model: cheapest, but the hard ones come back wrong — and silently. This is the trap.
• Router → a classifier sends each ticket to the cheapest capable model — great, if the classifier is good.
• Cascade → the brief's design: tuned model first, escalate the hard ones — ~frontier quality at a fraction of the cost, no classifier needed (the price: escalations pay twice and take ~2× latency).

Notice the trade. The tuned model swings cost dramatically, but all-cheap is worthless if it's wrong — a confident wrong triage fails silently and users churn. The cascade captures most of the savings without that risk, and the online eval from L13 is what keeps it honest in production.

Reason these out, then check against the lab and the code.

1. You're serving one triage model. Should you merge the adapter or serve it separately with multi-LoRA — and why?

2. Your endpoint is fast for one request but falls over at 50 concurrent tickets. What runtime feature fixes this, and which engine has it?

3. You quantized to 4-bit and shipped. What's the one check you must not skip before trusting it?

4. You set the cascade confidence threshold to 0.99 "to be safe." What goes wrong?

5. Why serve your fine-tune behind an OpenAI-compatible endpoint instead of a custom API?

Answers.

1. Merge it. For a single fine-tune, merge_and_unload() folds the adapter into the base → one standalone model, simplest to serve and zero added inference latency. Keep adapters separate (multi-LoRA) only when you need to serve many tunes on one GPU and hot-swap them per request.

2. Continuous batching (with PagedAttention for the KV cache) — it keeps the GPU full by admitting queued requests as others finish, giving 3–8× throughput under concurrency. vLLM has it; Ollama's simple batching is why it collapses under load. Serve production on vLLM.

3. Re-evaluate on the held-out test set (L13). 4-bit inference quantization usually costs a point or two of accuracy — sometimes more on a format-strict task like JSON. Confirm the quantized model still clears your bar before trusting it; never assume the merge/quantize was free.

4. You escalate almost everything to Claude — the cheap model is rarely 'sure enough,' so you pay frontier prices on nearly every ticket (often more than just using Claude directly, plus the extra first-pass call). Too low a threshold has the opposite failure: the cheap model answers things it can't, degrading quality silently. Tune the threshold on your eval set for the cost/quality point you want.

5. Because it makes your fine-tune a drop-in. Any code already calling an LLM (including the Claude calls you're replacing) can point at your model by changing one base URL — no rewrite. It also makes the cascade trivial (both tiers speak the same schema) and lets you swap models in/out as config, not code.

• "A trained model is the deliverable." → A model in a bucket creates zero value. The deliverable is a served endpoint doing real work — merged, quantized, routed, monitored.
• "Serving quantization is the same as the QLoRA 4-bit." → Different jobs. QLoRA's 4-bit was to fit training; AWQ / GGUF here are post-training quantization to serve fast. Re-check quality after.
• "Just run it in Ollama in production." → Ollama is great for local/single-user, but its throughput collapses under concurrency. Use vLLM (continuous batching + PagedAttention) to serve at scale.
• "Keep the adapter separate so it's flexible." → For one model, merge it (simpler, lower latency). Multi-LoRA separation is for serving many tunes on one GPU.
• "All-cheap — we trained it, so use it for everything." → All-cheap is the trap: the hard cases come back wrong, silently. Run a cascade — cheap first, escalate the uncertain ones to Claude.
• "A higher escalation threshold is safer." → Too high → you escalate everything and pay more than just using the frontier model. Too low → silent wrong answers. Tune the threshold for your cost/quality target.
• "Ship it and you're done." → A served model needs monitoring + online eval (L10/L13). Watch the escalation rate for drift, and feed escalated tickets back as next-version training data — the flywheel.
• "A cheap model that's sometimes wrong is fine because it's cheap." → A confident wrong answer fails silently and costs you users. Cheap is only valuable when it's right — which is why eval never stops.

That's the whole project — you took a model from a brief to a live, cheap endpoint. Across four lessons:

• L11 — Brief & Data Curation: chose a task fine-tuning actually wins (behavior, not knowledge) and engineered the dataset — sourced, distilled, cleaned, decontaminated, split, formatted.
• L12 — LoRA Fine-Tuning Run: trained a QLoRA adapter on a small open student — the dataset was the product, the run was the easy part — and read the loss curves.
• L13 — Evaluation vs Baseline: proved it on the sacred test — delta vs base + prompted Claude, a bootstrap CI (ship on the lower bound), a forgetting check, and the cost win.
• L14 — Serving the Model: merged, quantized, served it behind an OpenAI endpoint and routed the cascade — cheap model first, escalate to Claude.

The arc — data → train → evaluate → serve — is the full lifecycle of a custom model, and it's a genuinely senior capability: most engineers use models; you can now make one, prove it's worth it, and run it cheaply in production. You also saw the deepest lesson of fine-tuning: the dataset is the product, and the proof is the eval — the training run in the middle is almost an afterthought.

Where next. You've now shipped all three capstone projects — a RAG assistant, a multi-tool agent, and a fine-tuned model. That's a portfolio. The course closes with the Capstone & Career track: framing this work for interviews and system-design, building your portfolio, and your own open-ended capstone. You've done the hard part — you can build, measure, and ship AI. Now go show it. 🚀