The Open Research Problems

This whole course has been about what AI can do — and how to build with it. This lesson is the honest counterweight: what it still can't reliably do. After the hype of the last few sections (reasoning, computer use, coding agents), it would be easy to believe the remaining gaps are small engineering details. They're not. They are deep, open research problems — some of them decades old — and knowing them is what separates an engineer who ships responsibly from one who over-promises and gets burned.

Here's the framing that makes this useful rather than gloomy. Every open problem is also a design constraint and a signal of where the field is going. If you know hallucination is baked into how models are trained, you ground and verify instead of trusting. If you know reliability collapses over long horizons, you keep tasks short and a human in the loop. If you know benchmarks are saturating, you build your own evals. The open problems are a map of where to be careful and where the next breakthroughs must come from.

A crucial idea runs through all of them: which problem bites hardest depends on how autonomously you deploy. A human-reviewed assistant can tolerate a lot — a person catches the mistakes. A fully autonomous agent, acting unsupervised with tools, is exposed to every frontier at once. You'll feel this directly in the lab's autonomy slider.

In this lesson, the eight big frontiers:

• Hallucination & calibration — confidently wrong, by design.
• Long-horizon reliability — errors compound; agents drift and fail silently.
• Reasoning vs. pattern-matching — do they generalize, or interpolate?
• Continual learning & memory — models are frozen; they can't learn on the job.
• Alignment & interpretability — we can't fully read what's inside.
• The evaluation crisis — we're losing our ruler.
• Security & prompt injection — not solvable today.
• The data wall & sample efficiency — running low on fuel.

Scope: this is a perspective lesson that draws on the whole course. It's the penultimate lesson of The Frontier; the finale, staying current → L260, is about how to keep learning as these problems evolve.

The most famous failure, and a deeper problem than it looks. Hallucination is when a model states something plausible but false, confidently — and rarely says "I don't know." The naive view is that this is a bug we'll patch. The uncomfortable research finding is that it's partly baked into how models are built and graded.

• It's statistical, not accidental. A 2025 OpenAI analysis (Why Language Models Hallucinate) and earlier work (Calibrated Language Models Must Hallucinate) argue that calibration — and therefore some errors — is a natural consequence of the standard training objective (next-token prediction with cross-entropy). A model that never errs would have to be miscalibrated.
• Evaluation rewards guessing. Most benchmarks score accuracy, which rewards a confident guess over an honest "I'm not sure." So both training and the leaderboards we optimize for incentivize hallucination. It's a socio-technical problem — fixing it needs changes to how we evaluate, not just how we train.
• It's still large where it matters. Stanford's RegLab found LLMs hallucinate on 69–88% of specific legal queries; rates rise on recent events (static training data).

What you do about it (today): treat the model as a fluent reasoner, not a fact store. Ground answers in retrieved sources (RAG) and cite them; verify outputs against a source of truth; train or prompt for abstention ("say you don't know"); and surface uncertainty. These are engineering responses that shrink the blast radius — not a cure. Hallucination is, as of 2026, an open problem.

A single model call is usually pretty good. A hundred chained model calls is a different beast — and this is the wall every ambitious agent hits. Reliability is multiplicative. If each step is 95% reliable, a 20-step task is 0.95²⁰ ≈ 36% reliable; a 50-step task is near hopeless. Errors don't just add — they compound and cascade, because each mistaken step poisons the context for the next.

Worse, the failures are silent. As we saw with coding and computer-use agents, when an agent breaks you often get a clean, confident, completely wrong answer — no exception, no stack trace. Studies of deployed ML systems find the large majority degrade over time. The result: agents that demo beautifully on a 5-step task and fall apart on a 50-step one.

This is the gap between the agent demos you see and autonomous agents you can trust unsupervised. It connects to almost everything else: hallucination supplies the wrong steps, weak evaluation hides the failures, and alignment governs whether a confused agent does something harmful.

What you do about it: decompose long tasks into short, checkpointed sub-tasks; verify after each step (tests, L258); cap iterations and detect loops; and keep a human at the consequential moments. The honest status: robust long-horizon autonomy is unsolved — which is exactly why production agents today are kept on a short leash.

Here's the question that splits the field: when a model 'reasons,' is it actually reasoning — or interpolating patterns it saw in training? The evidence says: a lot of both, and we can't cleanly tell them apart.

The sharpest illustration is ARC-AGI, a benchmark of abstract visual puzzles built to resist memorization. The story in 2026 is striking:

• On ARC-AGI-1, top systems now match or beat the human baseline (a leading model hit ~87%). Looks solved.
• On ARC-AGI-2 — same idea, novel puzzles — the same model scores under 3%, and plain LLMs without reasoning scaffolds score ~0%. Performance falls off a cliff the moment the puzzle has no near-neighbor in training.

As researchers put it: the benchmark is solved, but the problem isn't. Models are extraordinary at interpolating within the vast space of patterns they've seen, and brittle at genuinely novel structure — they conflate similarity search with logical inference. This is the core of the "illusion of thinking" debate, and it's why reasoning models (L256) and test-time compute help (they search harder) but don't fully close the gap.

Why it matters to you: it predicts where models break — far from their training distribution, on truly new problems — and tempers claims of "general" intelligence. Robust out-of-distribution generalization is an open problem. Practically: don't assume a model that aces familiar tasks will transfer to a genuinely novel one; test on your actual edge cases.

Here's something that surprises newcomers: today's models can't learn on the job. A deployed LLM is frozen — its weights don't change as it works. Everything that looks like "learning" or "memory" in a product (a chatbot that remembers you, an agent that improves) is bolted on from the outside, not the model getting smarter.

The reason it's hard is a classic, unsolved problem: catastrophic forgetting. When you train a neural network on new information, the gradient updates overwrite the weights that held the old knowledge, and prior abilities degrade sharply. First characterized in the 1980s, it's still with us. It's a facet of the stability–plasticity dilemma: be plastic enough to learn the new, stable enough to keep the old — and we don't know how to get both at once.

So in 2026 "memory" is a stack of workarounds: the context window (short-term, wiped each session), RAG / vector stores (external long-term recall), and scratchpads / written notes (like Claude Code's CLAUDE.md). These are useful — but they're retrieval, not learning. The model that finishes a hard debugging session is exactly as capable at the start of the next one; it kept nothing.

Why it matters: it's why you can't just "let the agent learn" from production — improvement comes from you updating prompts, memory files, and fine-tunes, not the model adapting itself. True online / lifelong learning — a model that safely accumulates skill over time without forgetting — is a major open frontier.

This is the problem that underlies all the others, and the one with the highest stakes: making increasingly capable models reliably do what we intend — and being able to understand what they're actually doing inside.

Interpretability is the honest admission that we don't fully know how these models work. We trained them; we can't read them. A central obstacle is superposition: a single neuron participates in many unrelated concepts at once, so you can't just "look at neuron 4,217" and know what it means. The leading approach, mechanistic interpretability with sparse autoencoders (SAEs), decomposes those tangled activations into cleaner, human-meaningful features — and it's made real progress (Anthropic and others have extracted millions of interpretable features). But the field's own verdict in 2026 is blunt: the internals remain largely a mystery.

Alignment is the broader goal: as models get more capable and autonomous, how do we ensure they pursue our actual intent — not a misread proxy, not a reward-hacked shortcut (recall L258), not instructions injected by a malicious page (L257)? This includes honesty, refusing harm, and staying corrigible (lettable-to-be-corrected) even as capability grows.

Why it matters to an engineer: alignment and interpretability are why the rest of this section insists on human oversight, sandboxing, and verification. We deploy systems we can't fully inspect, so we wrap them in controls. It is the deepest active frontier — progress is real, a solution is not in hand.

Here's a quieter crisis that threatens the whole field's ability to make honest claims: we are losing our ruler. The benchmarks the industry relied on are failing in two ways at once, right as models get more capable.

• Saturation. Frontier models now cluster above 88–90% on the classic tests — GSM8K ~99%, MMLU ~93%, HellaSwag >95%, and even "designed-to-be-hard" BIG-Bench-Hard in the mid-90s. When everyone scores ~94%, score differences become statistical noise — the benchmark can no longer tell the models apart.
• Contamination. The tests are public, so they leak into training data. Models can reproduce benchmark answers from memory rather than ability — and audits found that even SWE-bench Verified lets frontier models reproduce gold patches or problem specifics verbatim. OpenAI deprecated SWE-bench in early 2026 over exactly this.

The result: leaderboard numbers overstate real capability, and it's genuinely hard to know which model is better, or whether "progress" is learning or memorizing. This connects straight to hallucination (accuracy-based scoring rewards guessing) and reasoning (memorization masquerading as inference).

What you do about it: trust memorization-resistant and private/dynamic benchmarks (ARC-AGI-2, FrontierMath, Humanity's Last Exam), discount any single leaderboard, and — most importantly — build your own evals on your own tasks (L246). Public benchmarks tell you about the field; only your eval tells you about your product. Trustworthy evaluation is itself an open problem.

We met this one in Computer Use & Browser Agents (L257), and it belongs on the list of genuinely unsolved problems because it is: as of 2026, we do not know how to prevent prompt injection 100% of the time.

The root issue is architectural. An LLM agent treats everything in its context as input — and it can't reliably tell trusted instructions (from you) apart from untrusted content (from a web page, email, or document it reads). So indirect prompt injection — malicious instructions hidden in content — can hijack an agent into ignoring you. It turns catastrophic via the lethal trifecta: private data + untrusted content + an exfiltration path = data theft, and a tool-using agent often has all three.

What makes it a frontier problem rather than a bug: capability and risk grow together. The very reach that makes an agent useful (read anything, act anywhere) is what an attacker turns against you, and there is no known complete defense — only mitigations.

What you do about it: defend in layers, assuming any one fails — sandbox the agent, withhold secrets, allowlist destinations, require human confirmation for consequential actions, and use injection classifiers. These shrink the blast radius; they don't close the hole. Security for autonomous agents is an open problem the whole field is actively working on.

The last frontier is about fuel: the recipe that got us here — more data + more compute + bigger models — is hitting a data wall.

Modern training runs are so large that they're running out of high-quality, information-dense human text. The web's good text can't keep pace with training budgets; sites increasingly block scrapers; and the most useful data has largely been consumed. This threatens the pre-training scaling laws that drove a decade of progress.

The leading workaround is synthetic data — models generating training data for the next models — but its science is immature: repeatedly training on synthetic data can give diminishing or even negative returns (a degradation sometimes called model collapse), and high-quality synthetic data is expensive and hard to verify. Critics call naive versions "data laundering."

Underneath sits a humbling fact: LLMs are wildly sample-inefficient. A model needs to read a meaningful fraction of the internet to learn what a child picks up from a tiny slice of experience. We don't know how to make machines learn as efficiently as people do.

Why it matters: it shapes where the next gains come from. With pre-training data scarce, the field is leaning on the second scaling law — test-time compute (L256) — plus RL / self-play (generate your own verifiable problems) and better data efficiency. Whether those carry progress as far as raw scale did is an open question.

Let's make the frontier concrete. The lab plots all eight open problems on a map: research progress toward solved (left → right) against how much it blocks real deployment (bottom → top). Notice immediately that everything sits on the unsolved (left) side — that's the honest picture.

Now grab the autonomy slider and move it from Assistant (a human reviews every output) toward Autonomous (no human in the loop). Watch the TOP RISKS list re-rank: with a human reviewing, hallucination is the top concern; as you remove the human, long-horizon reliability, alignment, and prompt injection take over and dominate — because those are the ones a human was silently absorbing. Which frontier matters depends on how you deploy.

Finally, click any problem to read what it is, why it's hard, and the best response — and watch the arrows light up its connections. Click Hallucination and see it feed the evaluation crisis and long-horizon reliability. That web is the point: these aren't eight separate bugs, they're a connected frontier, and progress on one ripples into the others.

Notice three things. One: none of the nodes are on the "solved" side — humility is the correct posture. Two: the same system is safe as an assistant and risky as an autonomous agent — the deployment choice is yours, and it sets which problems you own. Three: the connections mean there's no single fix; progress is a research program, and your job is to engineer around the gaps while it advances.

Predict first, then check in the lab (or reason it out).

1. A vendor claims their model "doesn't hallucinate anymore." Based on why hallucination happens, why should you be skeptical — and what would a more honest claim sound like?

2. A model scores 87% on ARC-AGI-1, beating the human baseline. A teammate says "it can reason as well as a human." What single follow-up number would you ask for, and why does it matter?

3. Your product manager wants to "let the agent learn from user corrections automatically so it gets smarter over time." What fundamental limitation makes this not work the way they imagine, and what actually improves over time instead?

4. Two teams deploy the same model. Team A ships a drafting assistant a human always reviews; Team B ships an unattended agent that books travel with a saved credit card. Which open problems are load-bearing for each, and why does Team B carry far more risk?

5. Your model scores 94% on three public benchmarks. Why is that weak evidence that it'll do well on your task, and what's the one evaluation you should trust instead?

Answers.

1. Hallucination is partly inherent to next-token training (calibration implies some error) and is rewarded by accuracy-based evals — so "never hallucinates" is implausible. An honest claim: "lower hallucination rate on benchmark X, and it abstains / cites sources more often" — a reduction with grounding and uncertainty, not elimination.

2. Ask for its ARC-AGI-2 score (or any novel, memorization-resistant benchmark). The same models that ace ARC-AGI-1 drop to <3% on ARC-AGI-2 — high familiar-task scores can be pattern interpolation, not transferable reasoning. The gap reveals generalization.

3. Deployed models are frozen and suffer catastrophic forgetting, so they can't safely learn online from corrections (and auto-training on user data risks drift/attacks). What improves is the scaffolding you control: prompts, memory files / RAG, curated examples, and deliberate fine-tunes — retrieval and human-driven updates, not the model teaching itself.

4. Team A (assistant): hallucination matters, but a human catches errors — most frontiers are absorbed by the reviewer. Team B (autonomous + tools + a credit card): long-horizon reliability (compounding errors with no human to catch them), prompt injection / the lethal trifecta (untrusted web content + a payment method + an exfil path), and alignment are all load-bearing at once. Removing the human exposes every gap simultaneously.

5. Public benchmarks saturate (everyone ~90%+, differences are noise) and are contaminated (answers leak into training), so they overstate ability and may reflect memorization. Trust an eval you build on your own real tasks (L246) — that's the only number that measures your product.

• "Hallucination is a bug we'll soon patch." → It's partly inherent to how models are trained and rewarded by accuracy-based evals — a socio-technical, open problem. Ground, verify, abstain; don't expect elimination.
• "If it works for 5 steps, it works for 50." → Reliability is multiplicative — errors compound and fail silently. Long-horizon autonomy is unsolved; keep tasks short and checkpointed.
• "It beat humans on the benchmark, so it reasons like one." → It may be interpolating patterns; the same model can crater on a novel variant (ARC-AGI-1 vs ARC-AGI-2 <3%). Generalization is the open question.
• "The agent learns and gets smarter as it's used." → Deployed models are frozen (catastrophic forgetting is unsolved). "Memory" is external retrieval, not learning; you improve the scaffolding.
• "We understand how the model works — we built it." → We can't fully read the internals (superposition); interpretability is an active frontier, not a solved science.
• "Higher benchmark scores = better model." → Benchmarks saturate and are contaminated; numbers overstate ability. Build your own evals.
• "Prompt injection just needs a better filter." → There is no complete defense — capability and risk grow together. Defend in layers and assume any one fails.
• "We'll just keep scaling data." → The data wall is real; synthetic data has diminishing returns. Gains now come from test-time compute, RL, and efficiency — an open bet.

• This is the honest lesson: today's AI has deep, open problems, and knowing them is an engineering skill — it tells you where to keep a human, what not to promise, and where breakthroughs must come from. None of these are solved.
• Reliability frontier: hallucination is partly inherent (calibration + accuracy-rewarding evals) — ground & verify, don't expect elimination; and long-horizon reliability collapses multiplicatively (errors compound, fail silently) — the gap between agent demos and trustworthy autonomy.
• Reasoning & learning frontier: models may interpolate, not reason (ARC-AGI-1 solved, same model <3% on ARC-AGI-2) — OOD generalization is open; and models are frozen with catastrophic forgetting, so continual / lifelong learning is unsolved ("memory" is external retrieval).
• Safety frontier: alignment & interpretability — we deploy systems we can't fully inspect (superposition; SAEs help but internals are a mystery), which is why we wrap them in oversight; and prompt injection has no complete defense (the lethal trifecta) — mitigate in layers.
• Measurement & fuel frontier: the evaluation crisis (benchmarks saturate + are contaminated — build your own evals, L246) and the data wall (high-quality text is running out; synthetic data has diminishing returns) — pushing gains toward test-time compute (L256), RL, and efficiency.
• The unifying idea: the problems interconnect, and which one bites depends on how autonomously you deploy — a human-reviewed assistant tolerates a lot; an unsupervised agent is exposed to every frontier at once. Honesty + verification + a human in the loop is the engineering posture.
• Next — L260 (Staying Current: How to Keep Learning): the finale — how to keep up as these frontiers move, so your knowledge doesn't go stale.