Running Open Models Locally with Ollama

Last lesson, every call went to a cloud API — which means a key, a bill per token, and your data leaving your machine. But a huge and fast-improving class of models are open-weight: you can download them and run them on your own computer, for free, fully offline, with your data never leaving the building.

The tool that makes this almost effortless is Ollama. One command downloads a model and starts chatting. And the best part for you as a developer: Ollama speaks the same API dialect as OpenAI, so the code you just learned works against a local model with a one-line change.

You'll learn:

• Why you'd run a model locally (and the trade-offs)
• What Ollama is and its core commands
• Why a big model fits on your laptop: quantization
• How to pick a model for your hardware and task
• The OpenAI-compatible bridge — reuse your SDK against a local model

Cloud APIs give you the best models with zero setup — but local models win on a different set of axes:

• Free. After the (one-time) hardware, inference costs $0 per token. Great for high-volume experimentation that would rack up an API bill.
• Private. Your prompts and data never leave your machine — essential for sensitive, confidential, or regulated data.
• Offline. No internet, no API outage, no rate limits — develop on a plane.
• Full control. Pin exact versions, tweak settings, and never get surprised by a provider update.

The honest trade-offs: you're bounded by your hardware, so you'll run smaller models than the frontier; they're generally slower on consumer machines and not yet as capable as the biggest closed models (though the gap is closing fast). The sweet spot: local for development, privacy, and volume; cloud for maximum quality and scale.

Ollama is an open-source tool that downloads, manages, and runs open-weight LLMs locally — think "Docker for LLMs." It hides all the painful parts (model files, quantization, runtime, GPU/CPU setup) behind a dead-simple CLI and a local server.

Install it from ollama.com (a normal app install on macOS/Windows, or a one-line script on Linux). Once installed, a background server runs at http://localhost:11434 — remember that address; it's how your code will talk to local models.

One command downloads a model and drops you into a chat:

ollama run llama3.1:8b

The first run pulls the model (a few GB), then you're talking to it — entirely offline. The handful of commands you'll actually use:

ollama pull qwen2.5-coder:7b   # just download (no chat)
ollama run  llama3.1:8b        # download if needed, then chat
ollama list                    # what's installed
ollama ps                      # what's currently running (in memory)
ollama rm   llama3.1:8b        # delete a model to reclaim disk

Browse everything available at the model library (ollama.com/library) — it carries the major open families: Llama, Qwen, Gemma, Mistral, Phi, DeepSeek, gpt-oss, and more.

A 7-billion-parameter model in full precision is ~14 GB; a 70B is ~140 GB — far past a normal laptop. So how does Ollama run them? Quantization.

Quantization compresses the model's weights to lower precision — typically from 16-bit down to 4-bit — shrinking the file roughly 4× with only a modest quality loss. That's why Ollama's default is a 4-bit build (tagged like q4_K_M), and why a 7B model that should need 14 GB runs comfortably in ~4–5 GB.

A useful rule of thumb at 4-bit: ~0.5 GB of memory per billion parameters. So:

• 7–8B model → ~5 GB (fine on a 16 GB laptop)
• 70B model → ~40 GB (needs a workstation / big GPU)

Model tags encode this as name:size-quant, e.g. llama3.1:8b-q4_K_M. Higher-precision tags (q5_K_M, q8_0, fp16) are better quality but bigger and slower. 16 GB RAM is a practical starting point, and Apple Silicon Macs punch well above their weight thanks to unified memory.

Choose by task and what your hardware can hold. Some sensible starting points (all real, all one ollama pull away):

General rule: start with a 7–8B model, see if quality is enough, and only size up if your RAM/GPU allows. Bigger isn't worth it if a small model already does your task.

Here's the detail that makes Ollama a joy for AI engineers: it exposes an OpenAI-compatible API at http://localhost:11434/v1. So the exact OpenAI SDK code from last lesson works against a local model — you just change the base_url (and pass any throwaway api_key):

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",   # point at local Ollama
    api_key="ollama",                        # any non-empty string
)

resp = client.chat.completions.create(
    model="llama3.1:8b",                        # a model you've pulled
    messages=[{"role": "user", "content": "Say hi in one sentence."}],
)
print(resp.choices[0].message.content)

This is the payoff of last lesson's provider-agnostic advice: the same code talks to GPT, Claude (via compatible layers), or a local model — flip a base URL and you've swapped the engine. (Ollama also has its own native API and a pip install ollama library, but the OpenAI-compatible route maximizes code reuse.)

Neither replaces the other — they're tools for different jobs:

• Reach for local (Ollama) when: developing/prototyping, the data is sensitive or must stay private, you're offline, you're running high volume where API cost matters, or you want zero rate limits.
• Reach for a cloud API when: you need frontier-level quality, the largest models, painless scaling, or the newest capabilities — and you don't have (or don't want to manage) the hardware.

Many real systems use both: develop and test locally for free, then point the same code at a cloud model for production quality. (When local production serving needs more throughput than Ollama gives, teams graduate to servers like vLLM — a later topic.)

• Develop for free and offline. Prototype all day without burning API credits or needing a connection.
• Privacy by default. For confidential or regulated data, local inference keeps everything on your machine — sometimes the only acceptable option.
• Same SDK, swappable engine. Thanks to the OpenAI-compatible endpoint, local↔cloud is a base_url change — the provider-agnostic habit from last lesson pays off immediately.
• Quantization is a concept you'll reuse. It reappears in fine-tuning (QLoRA) and deployment — understanding "4-bit ≈ 4× smaller, modest quality loss" now will save you later.
• Right tool, right job. Knowing local exists keeps you from over-paying for cloud on tasks a small local model handles fine.

Your turn — go local. Install Ollama, then ollama run llama3.1:8b and chat with it offline. Now the payoff: point the OpenAI SDK at it — OpenAI(base_url="http://localhost:11434/v1", api_key="ollama") — and run last lesson's code unchanged. Same code, a free local model. Pull a coding model (qwen2.5-coder:7b) and feel the speed/quality trade vs a frontier API.

• "Local models are as good as GPT/Claude." Not the biggest ones — local is hardware-bounded, so smaller; great for many tasks, not (yet) for frontier-level work.
• "Ollama needs a fancy GPU." No — it runs on CPU and shines on Apple Silicon; a 7–8B model is fine on a 16 GB laptop. Bigger models want more memory/GPU.
• "Quantization barely changes anything / ruins the model." Neither — 4-bit cuts size ~4× with modest quality loss; it's the standard trade-off for local use.
• "I need to rewrite my code for Ollama." No — point the OpenAI SDK at localhost:11434/v1; same code.
• "Local means free with no downsides." Free per token, but you pay in hardware, speed, and a quality ceiling.

• Ollama runs open-weight models locally — free, private, offline — via a simple CLI; the server lives at http://localhost:11434.
• Core commands: ollama run (download + chat), pull, list, ps, rm. Browse models at ollama.com/library.
• Quantization (default ~4-bit) is why big models fit your machine: ~4× smaller, modest quality loss (~0.5 GB per billion params at 4-bit).
• Pick by task + hardware — start with a 7–8B model; size up only if RAM/GPU allows.
• The OpenAI-compatible API (/v1) lets you reuse your SDK against a local model by changing only base_url — local for dev/privacy/volume, cloud for frontier quality/scale.

Next: whether local or cloud, every model has a hard limit on how much text it can handle at once — we tackle tokens and context windows, and why they shape everything you build.