The AI Engineering Mindset (Build-vs-Buy, Eval-First, Iterate)

You now know what AI engineering is, who does it, and what you can build with it. This lesson is about how to think — the habits that separate someone who makes an impressive demo from someone who ships AI that actually works for real users.

The surprising part: the difference isn't talent or model access. It's mindset. Three habits — build-vs-buy, eval-first, and iterate — sit on top of one foundational shift: thinking probabilistically. Internalize these now and every later lesson will feel like technique in service of a mindset you already have.

You'll learn:

• The probabilistic mindset that rewires how you build
• Build vs. Buy — and why you should almost always buy first
• Eval-first — measuring quality instead of trusting vibes
• Iterate — starting simple and climbing the complexity ladder only as needed
• How a pro stitches all four together on a real feature

This is the foundation under everything else, and it's the hardest unlearning for traditional engineers.

Classic software is deterministic: the same input gives the same output, every time. You write an exact test (assert add(2, 2) == 4) and a green test means correct.

Foundation models are probabilistic: the same input can give different outputs (recall the sampling lesson — temperature, top-p), and the model can be confidently wrong. There is no single "correct" string to assert against.

That one fact rewires your whole approach:

Everything below — buy-vs-build, evals, iteration — is a response to living in a probabilistic world.

For nearly every decision — the model, the vector DB, observability — you'll choose between buy (use a hosted service / API) and build (self-host, fine-tune, or roll your own). The professional default, especially early, is buy.

• Buy = call a hosted frontier model (Claude, GPT, Gemini via API) and lean on managed services. You get frontier capability in minutes, for cents, with eval/tracing/guardrails often included.
• Build = self-host open models, fine-tune, or (almost never) pretrain. Powerful, but you take on serving, scaling, and a maintenance burden that's often ~5× the initial build.

The rule of thumb from 2026 practice: "Almost always start with buy, because speed matters more than optimization." With price-competitive models everywhere, self-hosting rarely wins on cost alone.

When build (self-host / fine-tune) is justified

Reach for build only when a concrete need forces it:

• Compliance / privacy — data legally cannot leave your infrastructure.
• Latency — you need tight, predictable latency, and network round-trips to a hosted API (plus cold starts) get in the way.
• Specialized behavior — a proprietary fine-tuned model that can't run on third-party infra.
• AI is your core differentiator (not just a feature) and you can staff a team for the long haul.

The mature pattern is hybrid "buy-and-build": buy the foundations, build only your differentiator. Apply it fractally — buy the model, buy the managed vector DB and observability, and build only the application logic that is your edge. Don't reinvent infrastructure that a vendor does better; spend your scarce effort where it's unique.

Because outputs are probabilistic, "it looks good when I try it" is not a quality signal — it's a vibe. Vibes don't catch the 1-in-20 confidently-wrong answer that embarrasses you in production, and they can't tell you whether your latest prompt tweak helped or hurt.

Eval-first means: before (and while) you build, you decide what "good" means and how you'll measure it.

• Collect a small set of real, representative examples (even 20–50 to start).
• Define success per example (exact answer? contains key facts? correct format? grounded in sources?).
• Look at your data — read actual outputs to find how it really fails.
• Let those evals drive your changes: every tweak is judged against the set, so you know if it improved.

This is the LLM analog of test-driven development, and it's the discipline taught inside frontier labs that most courses skip. We dedicate an entire container to it later — for now, just adopt the instinct: measure, don't assume.

Beginners over-engineer: they reach for a multi-agent system on day one. Professionals start with the simplest thing that could possibly work and add complexity only when evals prove it's needed.

Climb this ladder of complexity one rung at a time — each rung adds power and cost/latency/unpredictability:

1. A single prompt (zero-shot) — try this first.
2. A better prompt (few-shot, clearer instructions).
3. Add knowledge (RAG) — when it needs facts it doesn't have.
4. Add tools / a workflow — when it needs to act or chain steps.
5. An agent — only for genuinely open-ended tasks.
6. Fine-tune — only when prompting/RAG plateau and you have data.

The loop: ship the simplest version → measure with evals → find the single biggest failure → fix that one thing → repeat. This is the improvement flywheel, and it beats big-design-up-front every time, because real usage and real evals — not your guesses — tell you what to build next.

Watch all four — the probabilistic foundation plus the three habits — work as one. Say the ask is "auto-draft replies to customer support emails."

1. Is an LLM even right? (use-case lens + the when-not-to check from last lesson) — drafting language for a human to approve? Yes, great fit; keep a human in the loop (augment, not automate).
2. Buy, don't build. Start with a hosted model + a single prompt. No fine-tuning, no infra. Ship a thin internal tool this week.
3. Eval-first. Pull 30 real past emails, write down what a good draft looks like, and score the prompt's drafts against them. Now you have a number, not a vibe.
4. Iterate up the ladder. Biggest failure = it invents policy details → add RAG over the help-center docs. Re-measure. Next failure = tone is off → add few-shot examples of good replies. Re-measure. Only if it plateaus and you have lots of labeled data do you even consider fine-tuning.

Notice what you didn't do: build infrastructure, train a model, or design a 5-agent system on day one. You bought, measured, and climbed — and shipped value in week one.

• Vibe-driven development. Shipping because "it looked good in my five tests." Build a small eval set instead.
• Premature complexity. Reaching for agents/fine-tuning before a plain prompt has been tried and measured.
• Build-everything syndrome. Self-hosting models and rolling your own vector DB before you've validated the product. Buy first.
• Expecting determinism. Treating a probabilistic system like classic software — no guardrails, no fallback for the bad 5%.
• Optimizing before measuring. You can't improve what you don't evaluate; set up the eval before you tune.

Now run the mindset yourself on that exact feature. Ship the simplest version, read the eval, and fix the biggest failure — one rung at a time. Watch what happens if you jump straight to an agent or fine-tuning before the cheaper fix.

The eval — not your gut — told you what to build next. That is the whole mindset in motion: buy + simplest → measure → fix the biggest miss → repeat, climbing the ladder (prompt → RAG → few-shot → … → fine-tune) only as far as the data forces you.

Run the three checks. Take a feature — say 'auto-tag support tickets.' Apply the AI-engineering mindset:

1. Build vs buy: can a prompt + API do it, or do you really need a custom model? (Almost always: prompt + API.)
2. Eval-first: what's the one metric that says it works? (e.g. label accuracy on 50 real tickets.)
3. Smallest iteration: what's the tiniest version you can ship and measure this week?

If you can answer all three in two minutes, you're already thinking like an AI engineer.

• Think probabilistically: design for a distribution of outputs, set expectations, and measure — don't assume determinism.
• Build vs. Buy → default to buy: rent the model and managed services; build only your differentiator, and only when compliance, latency, or specialization forces it.
• Eval-first: decide what "good" means and measure it on real examples — the LLM version of test-driven development.
• Iterate: ship the simplest thing, measure, fix the biggest failure, repeat; climb the complexity ladder (prompt → RAG → tools → agent → fine-tune) only as evals demand.
• These four — the probabilistic foundation plus the three habits — are the throughline of the entire course.

Next: How This Course Works — prerequisites, optional primers, and how to get the most out of what's ahead. That wraps up our welcome to AI engineering.