Common AI Engineering Interview Questions

In L15 (The AI Engineering Interview Framework) you got the map of the five rounds; in L16 (System Design for AI Applications) you drilled the biggest one. This lesson is the question bank — the specific questions that come up again and again across the technical rounds, and a repeatable way to answer any of them.

Here's the mindset that makes this lesson work, and it's the opposite of how most people study: you are not here to memorize answers. Memorized answers collapse the moment an interviewer asks a follow-up. Instead, you'll learn one answer framework and apply it to questions across five clusters — and you'll notice something reassuring: you already know the answers, because you built the systems. The questions only feel intimidating until you realize they're asking about the RAG assistant, the agent, and the fine-tuned model you shipped.

The five clusters (the same ones from L15, which cover ~90% of technical questions):

• LLMs & transformers — attention, tokens, sampling, context windows.
• RAG & retrieval — chunking, hybrid search, reranking, RAG evaluation.
• Agents & tool use — ReAct, tools vs agents, memory, failure modes.
• Prompting & evaluation — production eval, prompt management, hallucination.
• Fine-tuning & serving — LoRA/QLoRA, catastrophic forgetting, cheap serving.

Plus a production/cost round that cuts across all of them, a rapid-fire set of curveballs, and the red-flag phrases that quietly sink otherwise-good answers. We'll start with the framework — the single most useful thing in this lesson — then walk the clusters question by question.

Before any specific question, internalize the four-beat framework (from L15). It works for every conceptual question an interviewer can ask, and it forces the depth that separates a hire from a rejection:

1. Define it crisply — one or two sentences. No waffle.
2. Why it exists — the problem it solves (the motivation).
3. The key trade-off — what it costs. There is always a cost.
4. When not to use it — the boundary. This is the beat juniors skip and seniors lead with.

Then add the two moves that signal senior every time: name the trade-off along quality · latency · cost · safety, and cite a number from a project you built.

Why this works: interviewers probe one level past the definition. Anyone can recite "RAG retrieves relevant documents." The framework forces you past that — to why retrieval beats parametric memory, the latency it costs, and when you'd reach for fine-tuning instead. That's the difference between sounding like you read about the technique and sounding like you shipped it.

The tell of a weak answer is a definition with no second sentence. The tell of a strong one is "…and the trade-off is X, so I wouldn't use it when Y — in my project, this cost us ~Z." Keep the four beats in your head and every question below becomes a fill-in-the-blanks.

The fundamentals round checks whether you understand what's happening under the API. The greatest hits:

Q: Walk me through self-attention — and why divide by √dₖ?

• Define: softmax(Q·Kᵀ / √dₖ)·V — every token scores its relevance to every other token and mixes their values, all in parallel.
• Why: it replaces recurrence, so the whole sequence is processed at once and each token can pull context from anywhere.
• Trade-off: it's O(n²) in sequence length — doubling the context roughly quadruples attention compute and memory.
• The √dₖ: without it, large dot products push softmax into saturation → vanishing gradients; scaling keeps training stable.

Q: Temperature vs top-p — which for a code generator?

• Temperature scales the logits before softmax (higher = more random); top-p (nucleus) samples from the smallest set of tokens whose probabilities sum to p. Both dial determinism ↔ creativity. For code/extraction/facts → low temperature (~0). Crank it only for creative work. Trap: thinking temperature adds knowledge — it doesn't.

Q: Is a bigger context window always better?

• No. Self-attention is O(n²) so cost and latency grow quadratically, and "lost in the middle" means models recall the start and end of a long context far better than the middle. Retrieve and compress instead of stuffing the window — more tokens ≠ more signal.

Bonus depth they love: the KV cache stores past keys/values to avoid recompute during decoding (memory grows with layers × heads × seq × batch — tens of GB at long context); GQA shrinks it ~4×; prompt caching reuses a shared prefix to cut input cost 80–90%. Drop one of these and you've shown you understand serving, not just modeling.

RAG is the most-asked architecture topic, because almost every product LLM system has retrieval in it.

Q: Why hybrid search instead of pure vector search?

• Define: hybrid fuses dense (embedding) retrieval with sparse keyword retrieval (BM25), usually via reciprocal-rank fusion. Why: dense captures semantics, BM25 captures exact lexical matches. Trade-off / the point: pure vector search misses exact keywords — product codes, names, error strings — that users actually type. When-not: a purely semantic corpus can survive on dense-only, but hybrid rarely hurts. Add a cross-encoder reranker on top: retrieve top 50–100 cheaply, rerank to the top 5–10.

Q: How do you evaluate a RAG system?

• Split the problem. Retrieval: Recall@k, Precision@k, MRR/NDCG. Generation — the RAG triad: faithfulness (answer supported by context), answer relevance, context relevance (were the chunks even useful — a retrieval failure, not a generation one). Run an offline golden set + LLM-as-judge (e.g. RAGAS) and an online thumbs/hallucination-rate signal. Never "we use BLEU" or "we eyeball it."

Q: A client wants the model to "know" their internal docs — fine-tune or RAG?

• RAG. Retrieval is the right tool for knowledge — fresh, attributable, no retraining when docs change. Fine-tuning changes behavior/format/style; it does not reliably inject facts. The classic trap the interviewer is fishing for is "fine-tune it on their docs to teach it the facts." The clean rule: fine-tune for form, retrieve for facts.

Other RAG questions worth a one-liner: chunking (small chunks = retrieval precision, large = context coherence; structure-aware for technical docs), lost-in-the-middle (reorder retrieved chunks by score, use fewer high-quality ones), and HyDE (embed a hypothetical answer instead of the query — costs one extra LLM call).

In 2026 the agent round is the hardest — companies running production agents want engineers who've seen what breaks. Three questions carry most of it.

Q: What's the difference between tool use and an agent?

• Tool use is a capability (the LLM can invoke a function); an agent is an architecture built on top of it — adding planning, memory, error recovery, and an autonomous loop (ReAct: Thought → Action → Observation). Trap: calling any function-calling chain "an agent."

Q: When would you NOT use an agent?

• When the task is stable, repetitive, or compliance-bound — then a workflow (you specify the steps) is cheaper, testable, and predictable. Agents compound errors, can loop, overflow context, and rack up unbounded cost. Start with a workflow; add agency only where the task genuinely needs dynamic reasoning — most production systems are a blend. This answer signals real production judgment.

Q: How do you make an agent production-safe?

• A step/turn budget + cost cap (so a loop can't run away); structured tool errors with retries + backoff (report failures, don't invent results); human-in-the-loop approval on irreversible side-effects; treat tool/retrieved content as data, not instructions (prompt-injection defense); and trajectory eval — golden trajectories, step-level accuracy, cost-per-task. The eval round filters most candidates here.

Round it out with the four memory types (in-context, episodic, semantic, procedural) and MCP vs function calling (MCP decouples tools via an open protocol — "LSP for agents"). Every one of these you built or hardened in Project 2.

"Evaluation is the hardest unsolved problem in LLM engineering" — and it's the round where strong candidates pull ahead, because most can't get past "we tested it."

Q: How do you evaluate an LLM feature in production?

• A multi-layered strategy, not one number. Offline: a golden set run as a regression gate before deploy, scored by an LLM-as-judge against a rubric. Online: A/B tests plus thumbs/implicit signals to catch drift and close the loop. The wrong answers — "we use BLEU," "we tested it once manually" — fail the round on their own.

Q: How do you manage prompts in production?

• Treat prompts as versioned code assets in a registry (LangSmith/Langfuse/a table). A prompt change is a behavior change — it deserves the same rigor as code: gate changes behind A/B + eval before promotion, and never hot-edit a live prompt with no version or rollback.

Q: How do you detect and reduce hallucination?

• Ground the answer with retrieval + citations so claims trace to a source; score faithfulness (every claim supported by context) and flag contradictions; design the system to abstain / say "I don't know" when retrieval or confidence is low. Trap: "just lower the temperature" — that is not a hallucination strategy; grounding + abstention is.

The through-thread: you can't improve what you don't measure. You built eval harnesses three times (RAG, agent trajectory, fine-tune) — when this round comes, describe a system you shipped, not a metric you read about.

The adaptation-and-serving cluster checks whether you can make a model yours and run it cheaply.

Q: Explain LoRA — the math and why it's efficient.

• Define: freeze the base weights W; learn a low-rank update ΔW = B·A (rank r). The layer becomes h = Wx + (α/r)·B·A·x. Why: task adaptation has low intrinsic rank, so r = 8–64 captures it with ~100× fewer trainable parameters (a 4096² layer at r=16 goes 16.7M → 131K). Trade-off: QLoRA adds a 4-bit NF4 frozen base to fit big models on one GPU at a tiny quality cost. Trap: "LoRA trains all the weights" — it trains the small B and A while the base stays frozen.

Q: How do you prevent catastrophic forgetting?

• Use PEFT (LoRA) so you only nudge a small update; mix 5–10% general instruction data into the fine-tuning set; lower the learning rate and stop early; and eval on held-out general tasks, not just your new one, to catch capability loss.

Q: How do you serve an LLM cheaply at scale?

• Throughput: vLLM PagedAttention + continuous batching → 2–4× (serving is memory-bound, so batching is the lever). Shrink: quantization (AWQ/GGUF q4) and speculative decoding (2–3× on predictable output). Route: a cheap model for the bulk, escalate hard cases → ~60–80% savings. Cache: semantic + prompt caching cut input cost ~80–90%. Trap: "run one big model on every request" — no cost story.

Every number here you can anchor to Project 3 and L14 (Serving the Model) — you fine-tuned and served a real model, so these aren't trivia, they're your résumé.

Beyond the five fundamentals clusters, senior loops add a "production realities" round. This is where cost reasoning is expected, and where you prove you've operated a system, not just built one.

Q: How would you cut the cost of an LLM feature by half?

• Two levers, two directions. Reduce tokens: semantic caching (80–90% input reduction on repeats), prompt compression, context pruning. Reduce price-per-token: model routing (cheap model for simple queries, escalate complex ones — 60–80% savings), prompt caching, quantization. Quantify it: "at 2B tokens/day on a frontier model that's ~$300K/month; routing + caching takes it well under$60K/month."

Q: How do you reason about LLM latency?

• Split it: time-to-first-token (TTFT) vs inter-token latency. Prefill (parallel) drives TTFT; decode (sequential) drives the rest. Stream the output to slash perceived latency for free; quote p95/p99, never the mean (LLM tails are long); watch out for cold starts (warm pools fix them).

Q: What does LLM observability require?

• Trace logging (prompt, response, tokens, cost, latency), quality metrics (LLM-judge on a sample), a latency breakdown (TTFT vs inter-token separately), error tracking, and drift detection — via LangSmith/Langfuse/Phoenix or Prometheus+Grafana with an LLM layer.

The meta-signal across this whole round: a number on everything. Tokens, dollars, p95, hit-rate. Candidates who quantify sound like they've run production; candidates who hand-wave sound like they haven't.

Loops sprinkle in quick "do you actually know this?" questions. Crisp one-liners (each still answerable with the framework if pressed):

• Mixture of Experts (MoE)? A router activates only a few expert FFNs per token, so total params scale for capacity while active params per token stay small (e.g. Mixtral 8×7B ≈ 47B total, ~13B active). Cost: load-balancing and multi-GPU communication.
• RLHF vs DPO? RLHF trains a reward model then optimizes with PPO + a KL penalty; DPO skips the RL loop and directly optimizes preferred-over-rejected — simpler, stabler, now the default.
• Speculative decoding? A small draft model proposes several tokens; the target model verifies them in one parallel pass → 2–3× speedup on predictable output, no quality loss.
• Prompt injection defense? Separate instructions from data, sanitize tool/retrieved content, privilege-separate, and monitor outputs — the root issue is that LLMs treat all tokens equally regardless of source.
• Embeddings — what are they? Dense vectors where semantic similarity ≈ geometric closeness; the bridge that makes retrieval semantic instead of keyword-only.
• Why does my agent call tools it doesn't need? Usually a prompt problem — the system prompt doesn't say when to use tools vs answer from memory; a fast classifier can route simple queries away from retrieval.

The pattern: even a one-liner ends with a trade-off or a cost. That single habit is what makes a quick answer sound senior instead of rehearsed.

Some answers fail not because they're wrong but because of a phrase that signals shallow experience. Train yourself to avoid these:

• "It just focuses on the important words." (attention with no mechanism) → give Q/K/V and the O(n²) cost.
• "Vector search is always best." → name dense-only's failure on exact keywords; reach for hybrid + rerank.
• "We'll fine-tune it on our docs so it knows the facts." → fine-tune for form, retrieve for facts.
• "We use an agent for everything." → start with a workflow; add agency only where needed.
• "We use BLEU" / "we tested it manually." → layered eval: golden-set regression gate + online A/B.
• "Just lower the temperature" (as a hallucination fix) → grounding + faithfulness checks + abstention.
• "Run the biggest model / biggest context window." → cost, latency, lost-in-the-middle; route and retrieve.
• "LoRA trains all the weights." → it trains the low-rank B·A; the base stays frozen.
• Quoting average latency. → quote p95/p99; the mean hides cold starts and slow decodes.
• No number, anywhere. → the fastest way to sound senior is to attach a metric to a claim.

Notice these are mostly absolutes — and the fix is always the same shape: name the trade-off and the boundary. Swap every absolute for a "…but the trade-off is X, so I wouldn't when Y."

Reading the question bank is passive; drilling it is how it sticks. This widget turns the bank into an active-recall loop.

• Pick a cluster and get a real question.
• Type your answer's key points first — active recall beats re-reading, and it shows you exactly where your answer is thin before you see the model one.
• Reveal the answer in the define → why → trade-off → when-not framework, plus the #1 red flag and which project you built it in.
• Self-rate (nailed / shaky / missed) to fill a per-cluster mastery bar and an overall readiness score.
• Work every cluster to 100% — and watch "study for the interview" become a measurable target instead of a vague dread.

The point in your hands: you're not memorizing fifteen answers — you're rehearsing one framework on fifteen questions, until reaching for define → why → trade-off → when-not is automatic. Do that, and an unseen question is just the framework with new nouns.

Answer these out loud in the four-beat framework — then check yourself.

1. "What's the difference between top-k and top-p sampling?" (15 seconds)

2. "Why would you add a reranker to a RAG pipeline, and what does it cost?"

3. "When would you choose fine-tuning over RAG?" Give a concrete example.

4. "Our LLM bill is $200K/month and the CFO wants it halved. What do you do?"

5. Spot the red flag: "We made the agent better by giving it access to every tool and letting it figure things out."

Answers.

1. Both truncate the sampling pool. Top-k keeps the k highest-probability tokens (fixed count); top-p / nucleus keeps the smallest set whose probabilities sum to p (adaptive count). Top-p adapts to how confident the model is, which is why it's usually preferred. Trade-off: both reduce diversity vs pure sampling — set them tight for facts, loose for creativity.

2. Define: a cross-encoder reranker rescores the top 50–100 cheap retrievals down to the best 5–10. Why: first-stage retrieval optimizes recall, not precision, so it returns semantically-near-but-irrelevant chunks. Trade-off: +latency (an extra model pass, ~150–250ms) and cost, so add it only if you're within the SLO. When-not: if top-1 recall is already ~95%, the rerank earns little.

3. Choose fine-tuning for behavior/form the prompt can't reliably get: a strict output format/schema, a consistent tone/style, a domain reasoning pattern, or to distill a big model's behavior into a cheaper one. Example: "enforce our exact JSON triage schema on every call" — fine-tune. "Answer questions about this week's pricing" — RAG, not fine-tune.

4. Quantify, then pull two levers. "$200K/month is a token-volume × price problem. First, semantic + prompt caching for repeats — often 40–70% fewer calls. Second, model routing — send the easy majority to a cheap model, reserve the frontier model for hard cases, ~60–80% on those. Together that comfortably halves it; I'd A/B-test quality at each step so we don't trade cost for a regression."

5. Red flags: no tool scoping (giving an agent every tool invites wrong-tool errors and prompt-injection surface), no eval ("better" with no metric), and no guardrails ("let it figure things out" = no step budget, no approval gate). Senior rewrite: "I gave it the minimal tool set, scoped by the task, with a step budget and trajectory eval — and measured task success before and after."

• "I need to memorize the answers." → You need the framework (define → why → trade-off → when-not). Memorized answers die on the first follow-up.
• "A good definition is a good answer." → Depth lives one level past the definition — the trade-off and the boundary. A definition with no second sentence is the tell of a weak answer.
• "Fine-tuning teaches the model new facts." → Fine-tuning changes behavior/form; RAG is for knowledge. Fine-tune for form, retrieve for facts.
• "Function calling = an agent." → Tool use is a capability; an agent is the loop (plan → act → observe) around it, with memory and recovery.
• "More context / a bigger model is better." → Both cost latency and money, and long context loses the middle. Route and retrieve, don't max everything.
• "Eval = a benchmark score." → Production eval is layered: a golden-set regression gate + LLM-judge + online A/B for drift. Not BLEU, not vibes.
• "Lower temperature fixes hallucination." → Grounding + faithfulness checks + abstention fix hallucination; temperature is a creativity dial.
• "I should sound impressive." → You should sound measured: name the trade-off, cite a number, own the decision. That is impressive.

• Don't memorize answers — practice one framework: define → why → trade-off → when-not, then name the trade-off (quality·latency·cost·safety) and cite a number from a project.
• LLMs & transformers: attention is softmax(QKᵀ/√dₖ)V, O(n²) in length; low temperature for facts; bigger context isn't free (cost + lost-in-the-middle).
• RAG: hybrid (BM25 + dense) + rerank beats pure vector; evaluate with the RAG triad (faithfulness · answer · context relevance); fine-tune for form, retrieve for facts.
• Agents: tool use is a capability, an agent is the loop; don't use an agent for stable tasks (use a workflow); make it production-safe with step budgets, approval gates, and trajectory eval.
• Evals: layered — golden-set regression gate + LLM-judge + online A/B; version prompts as code; fix hallucination with grounding + abstention, not temperature.
• Fine-tuning & serving: LoRA learns ΔW = B·A (r=8–64, ~100× fewer params), QLoRA adds 4-bit NF4; prevent forgetting by mixing general data; serve cheap with batching, routing, caching, quantization.
• Production: quantify everything — tokens, dollars, p95/p99, hit-rate — and avoid the red-flag absolutes ("vector is always best," "fine-tune for facts," "agent for everything," "we use BLEU").
• Next — L18 (Building Your AI Portfolio): make the three projects that answer all these questions visible — a clean README, a demo, and the metrics front-and-center — so your work speaks before you do.