Skip to main content
CodePeetCodePeet
Capstone Projects & Career

Ship It

Introduction

This is the final step of the multi-tool agent project — and the moment everything pays off. You designed the agent (L6 — Project Brief & Design), gave it tools and memory (L7 — Tools + Memory), exposed and secured its tools over MCP (L8 — MCP Server Integration), and measured it with trajectory evaluation (L9 — Agent Evaluation). Now you ship it.

Here's the trap that catches most people: "it works on my machine, so deploy it." But your dev agent ran on in-memory state (lost on restart), with no guardrails (nothing stops a runaway loop or a runaway bill), and no monitoring (you'd find out it broke from an angry user, not a dashboard). Shipping an agent is not shipping a stateless API — an agent is long-running, stateful, takes real actions, can loop, and costs money per step. Production demands four things the demo didn't:

  • Durable state — Postgres-backed checkpoints and long-term memory, so runs survive restarts and redeploys.
  • Guardrails, designed in — loop and cost caps, a tool allowlist, human approval on writes, and fallbacks — set before deploy, not bolted on after an incident.
  • Observability + online evaluation — traces of every run plus the L9 evaluators running on live traffic, to catch the drift your offline gate never saw.
  • A safe rollout — a small canary and a fast rollback, so a bad change hits a handful of users, not all of them.

Why this is the senior skill. Agents fail in production overwhelmingly from missing guardrails, poor observability, and inadequate fallbacks — not from model quality. Getting an agent to work is a weekend; getting one you'd run in front of real users and real money is this lesson. We use the course stack: LangGraph with a Postgres checkpointer/store, deployed on LangGraph Server (LangSmith Deployment) or the Modal pattern from Project 1, observed with LangSmith / Phoenix.

Scope: this lesson owns taking the measured agent to production. It's the capstone of Project 2.

Hero infographic titled 'Ship It' for the fifth and final lesson of the multi-tool agent project, on a white background. The deck says: take the measured agent to production — durable state, guardrails, online eval, and a safe rollout; shipping an agent is not shipping a stateless API. The centre is a production launch diagram: a USER hits a small canary slice of traffic routed to the AGENT, which runs on three services — LangGraph API, Postgres (durable checkpoints + long-term store), and Redis (streaming) — wrapped by a ring of GUARDRAILS labelled loop limit, cost cap, tool allowlist, human approval on writes, and fallbacks. Live trajectories flow into an OBSERVABILITY panel running ONLINE EVAL (sampled), with monitors for success rate, tool errors, loop alerts, latency, and cost, feeding a PROMOTE-or-ROLLBACK decision gate. A small badge reads: a small canary limits the blast radius; online eval catches the drift the offline gate never saw. Three summary cards along the bottom read: 'Durable state — Postgres checkpointer + store, not InMemory'; 'Guardrails designed in — loop + cost caps, allowlist, human approval, fallbacks'; 'Online eval + canary — measure live traffic, roll out safely, roll back fast'. A family strip lists the five project lessons — Brief & Design, Tools + Memory, MCP Server Integration, Agent Evaluation, Ship It — with the fifth highlighted. Slate, sky, cyan, violet, amber, emerald accents; generous legible text.

From Dev to Prod — What Actually Changes

Your agent's logic doesn't change when you ship it — its operational substrate does. Three things that were fine in dev are liabilities in prod:

ConcernIn dev (L6–L9)In production
StateInMemorySaver / InMemoryStorePostgresSaver / PostgresStore — durable, resumable, shared
Limitsnone (you watch it run)loop + cost caps, tool allowlist, human approval on writes
Failureit crashes, you rerunfallbacks, retries, timeouts — degrade, don't crash
Qualityoffline eval on a golden setonline eval on live traffic + alerts
Releaseyou redeploy and hopecanary rollout + rollback

The good news: because you built on LangGraph with a clean design, most of this is configuration and wiring, not rewrites. Swapping InMemorySaver for PostgresSaver is a one-line change; the guardrails are compile-time and config flags; the online eval reuses your L9 evaluators. The rest of this lesson walks each row of that table, then hands you the launch console to feel the consequences.

Durable State in Production — Postgres, Not InMemory

In L7 (Tools + Memory) your agent used InMemorySaver (short-term thread state) and InMemoryStore (long-term cross-thread memory). Perfect for dev — and catastrophic in prod, because a restart, a deploy, or a crash wipes every in-flight conversation and all learned memory. Production state must be durable and resumable, which means a real database. Swap in the Postgres backends — same agent code, durable storage:

# src/agent/state.py — production memory: Postgres, not InMemory
from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.store.postgres import PostgresStore
from psycopg_pool import ConnectionPool

from agent.config import settings

# a pooled connection; autocommit + dict rows are required by the Postgres saver
pool = ConnectionPool(
    conninfo=settings().database_url,                 # from env, never hard-coded
    max_size=20,
    kwargs={"autocommit": True, "row_factory": "dict_row"},
)

checkpointer = PostgresSaver(pool)    # short-term: durable, resumable thread state
store = PostgresStore(pool)           # long-term: cross-thread memory (semantic index)

checkpointer.setup()                  # first run only: creates the checkpoint tables
store.setup()

That's the entire change from L7 — create_react_agent(llm, tools, checkpointer=checkpointer, store=store) now persists to Postgres. Now a thread can be resumed days later, a crashed run picks up from its last checkpoint, and memory outlives deploys. Two production notes: use a connection pool (max_size) so concurrent users don't exhaust connections, and set LANGGRAPH_STRICT_MSGPACK=true to restrict checkpoint deserialization to known-safe types (a security hardening for state you load back). Durable state is the foundation everything else sits on.

Serving the Agent — Deployment Options

How do you actually run the agent as a service? Two paths, both on the course stack:

Option A — LangGraph Server (managed). The Agent Server (LangSmith Deployment, formerly LangGraph Platform) runs your graph as a production API with streaming, persistence, and human-in-the-loop built in. You declare your graph in a langgraph.json and it provisions three services: the LangGraph API, Postgres (durable checkpoints/threads/runs), and Redis (streaming + run queue):

// langgraph.json — declares your agent for the Agent Server
{
  "dependencies": ["."],
  "graphs": { "agent": "./src/agent/graph.py:build_agent" },
  "env": ".env"
}

The server gives you assistants (versioned configurations of the graph), threads (durable conversations), streaming over /stream, and durability modes (async writes a checkpoint after each step; exit only at the end).

Option B — self-host (Modal). Reuse the Modal deployment pattern from Project 1's Deploy & Observe step: serve the compiled graph behind a FastAPI/ASGI app, with managed Postgres and Redis alongside. More control and a familiar pattern, at the cost of wiring the persistence and streaming yourself.

Which? Start with the managed server — it handles the stateful, long-running, human-in-the-loop machinery that's genuinely hard to get right. Reach for self-host when you need custom infrastructure or to colocate with the rest of your stack. Either way, the three-service shape — API + Postgres + Redis — is the production topology of a stateful agent.

Production Guardrails — Designed In, Not Bolted On

Guardrails are the difference between an agent that misbehaves and an agent that causes an incident. Set them at the infrastructure level, before deploy — soft limits to warn, hard limits to stop the worst case. Five you bake into the agent you built:

  1. Loop limit — a recursion_limit (from L6) caps how many steps a run may take, so a stuck agent can't loop forever burning tokens.
  2. Cost cap + token budget — a soft threshold logs a warning; a hard threshold kills the run. Runaway cost is the most common agent incident.
  3. Tool allowlist + least privilege — the agent may call only vetted tools, each scoped to minimum permissions (the discipline from L8 — MCP Server Integration).
  4. Human approval on writes — high-stakes actions (a refund, a deploy, an email) pause for a human (the interrupt from L7), now enforced at scale.
  5. Fallbacks, retries, timeouts — a failed tool call degrades gracefully instead of crashing the run.
# guardrails baked into the run
agent = create_react_agent(
    llm, tools, prompt=SYSTEM,
    checkpointer=checkpointer, store=store,
    interrupt_before=["tools"],          # pause before tool calls -> approve writes (L7/L8)
)

config = {
    "configurable": {"thread_id": thread_id},
    "recursion_limit": 25,                # loop guard (L6): stop runaway tool loops
}

SOFT_BUDGET, HARD_BUDGET = 40_000, 80_000   # tokens per run

def enforce_budget(usage):                  # cost guard: soft-warn + hard-stop
    if usage.total_tokens > SOFT_BUDGET:
        log.warning("run approaching token budget", extra={"tokens": usage.total_tokens})
    if usage.total_tokens > HARD_BUDGET:
        raise BudgetExceeded("hard token cap hit — killing the run")

Notice these aren't new code so much as production settings on the agent you already have: the recursion limit is the L6 loop guard, the write-approval interrupt is the L7/L8 human-in-the-loop, the allowlist is L8 least-privilege. The lesson is when: armed before the first request, not added after the first incident. The launch console below lets you switch each one off and watch exactly the failure it was preventing.

Observability & Online Evaluation

Offline evaluation (L9) proved the agent met the bar at release on your golden set. But production traffic is full of inputs your golden set never saw, and quality drifts — a model update, a changed prompt, a shifted user base. Online evaluation is how you catch that: you score live traffic continuously, not just once before launch.

First, trace everything — LangSmith (or Phoenix via OpenInference) captures each run as a tree of spans, so any problem links to the exact step. Then sample live runs and grade their trajectories with the same L9 evaluators, now online:

# online eval: sample live runs and grade their trajectories continuously
import random
from agentevals.trajectory.llm import create_trajectory_llm_as_judge
from langsmith import Client

judge = create_trajectory_llm_as_judge(model="anthropic:claude-sonnet-4-6")   # the L9 judge
ls = Client()

def after_run(run_id: str, messages: list):
    if random.random() < 0.10:                       # sample 10% of live traffic
        verdict = judge(outputs=messages)            # grade the real trajectory, online
        ls.create_feedback(run_id, key="trajectory_accuracy",
                           score=verdict["score"], comment=verdict["comment"])

Then monitor and alert on the signals that matter for an agent: task success rate, tool error rate, looping signals (excessive tool-call counts), latency, cost per run, and the online trajectory score. Wire automations so a drop in any of them pages you — drift, tool-failure spikes, latency, cost. And route low-scoring samples to human review; their labels recalibrate your evaluators over time. That review-and-improve loop is the data flywheel: production traffic becomes the dataset that makes the next version better. Online eval is the only signal that catches a silent quality regression — a hallucinated or mis-routed run still returns 200 OK.

Safe Rollout — Canary, Flags, and Rollback

Never big-bang a new agent (or prompt, or model) to 100% of traffic. Roll it out so that if it's bad, it's bad for a few users you can see, not everyone at once. The discipline:

  • Version everything together — prompt, graph, tools, and policies as one unit; pin the version in prod so nothing changes underneath you.
  • Gate on the offline suite — the L9 CI gate must be green before a change is even eligible to roll out.
  • Canary — send a small slice (1% → 10% → 50%) of traffic to the new version and watch the monitors and online eval. A feature flag lets you flip it instantly.
  • Promote or roll back — if the canary's metrics stay green, promote in stages; if anything goes red, roll back fast and investigate. A small canary means the blast radius of a bad change is tiny.

This is exactly the loop the launch console makes playable: arm your guardrails, pick a canary size, go live, read the online eval and incidents, and decide. A 1% canary that catches a regression and rolls back is a good day; a 100% deploy that ships the same regression to everyone is an outage. Same change — the rollout strategy is the difference.

Wiring It All Together — the Production Agent

Here's the whole thing assembled: the agent from L6–L9, now with durable state, guardrails, and a hook for online eval — ready for the Agent Server or Modal:

# src/agent/graph.py — production build
from langgraph.prebuilt import create_react_agent
from langgraph.checkpoint.postgres import PostgresSaver
from langgraph.store.postgres import PostgresStore
from psycopg_pool import ConnectionPool

from agent.config import settings, SYSTEM
from agent.tools import vetted_tools           # allowlist (L8: scoped, least-privilege)
from agent.llm import llm                       # claude-sonnet-4-6


def build_agent():
    pool = ConnectionPool(settings().database_url, max_size=20,
                          kwargs={"autocommit": True, "row_factory": "dict_row"})
    checkpointer = PostgresSaver(pool); checkpointer.setup()   # durable state
    store = PostgresStore(pool); store.setup()                 # long-term memory
    return create_react_agent(
        llm, vetted_tools, prompt=SYSTEM,
        checkpointer=checkpointer, store=store,
        interrupt_before=["tools"],            # human approval on writes (L7/L8)
    )
    # + recursion_limit + token budget per run (guardrails)
    # + tracing on (LangSmith/Phoenix) + after_run() online-eval sampling

Deploy it with langgraph.json (managed) or behind Modal (self-host), put it behind a canary, watch the online eval and monitors, and promote when green. That's a real, production-grade agent: durable, guarded, observed, and rolled out safely. Not a demo — a system you'd run in front of real users and real money.

✅ Definition of Done (this step)

Your agent is production-ready when:

  • Durable state — PostgresSaver + PostgresStore (pooled), .setup() run; threads and memory survive restarts/deploys; LANGGRAPH_STRICT_MSGPACK=true.
  • Served — deployed on LangGraph Server (langgraph.json: API + Postgres + Redis) or the Modal self-host pattern, with streaming.
  • Guardrails armed before deploy — recursion_limit (loop cap), token/cost budget (soft + hard), tool allowlist (least privilege), human approval on writes, fallbacks/retries/timeouts.
  • Observability — every run traced (LangSmith / Phoenix); dashboards for success rate, tool errors, loop signals, latency, cost/run.
  • Online eval — sampled live trajectories scored by the L9 evaluators, with alerts on drift/failures and a path to human review.
  • Safe rollout — versioned + pinned, the L9 gate green, a canary with a rollback plan and a runbook.

If you can deploy a change behind a 1% canary, watch its online eval, and roll it back in one click — you've shipped an agent like an engineer, not like a demo.

See It — The Agent Launch Console

This console is the whole lesson in one decision: go live, then promote or roll back. Arm the guardrails, pick a canary size, optionally inject drift, and click GO LIVE — then read the consequences.

  • Start all-armed at a 10% canary — online eval green, no incidents, verdict PROMOTE.
  • Switch off the loop limit and write-approval — watch the exact incidents appear (runaway loops burning money, unapproved writes auto-executing), cost/run spike, verdict flip to ROLL BACK.
  • Push the canary to 100% with a guardrail off — same bug, but now the blast radius is every user. Drop to 1% and it's contained to a handful.
  • Re-arm everything, then inject drift — every guardrail is on and the offline gate passed, yet online eval catches the live regression the golden set never saw.
The Agent Launch Console — ship the multi-tool agent to production, then decide promote or roll back, live. Arm the production guardrails (durable Postgres checkpointer + store, recursion/loop limit, cost cap + token budget, tool allowlist + human-approval on writes, fallbacks/retries), optionally inject post-deploy drift (a regression that passed the offline gate but degrades on live traffic), and pick a rollout — a 1% / 10% / 50% / 100% canary. Click GO LIVE: a batch of live trajectories streams in and the console shows the ONLINE-eval score (sampled), monitors (success rate, tool errors, $ per run vs cap), the incidents that escaped, the blast radius, and a verdict — PROMOTE or ROLL BACK. The lesson lands in your hands: every guardrail you switch off produces exactly the incident it would have prevented (runaway loops burning money, unapproved writes auto-executing, crashed runs), the canary size sets how many users a bad change reaches, and online eval is the only signal that catches the drift the offline L9 gate never saw. Deterministic teaching model, not a live deployment.

Notice three things. One: guardrails prevent specific incidents — turn one off and you see precisely what it was for. Two: the canary is your blast radius — the same bad change is a minor blip at 1% and an outage at 100%. Three: online eval is irreplaceable — it's the only thing that catches a regression that passed your offline gate and still returns a 200 OK.

🧪 Try It Yourself

Reason these out, then check against the console and the code.

1. Your dev agent uses InMemorySaver. You deploy two replicas behind a load balancer and restart one for a config change. What happens to the conversations on that replica — and what's the one-line fix?

2. An agent gets stuck calling the same tool over and over on one weird input. Which two guardrails stop the damage, and what does each one cap?

3. Your offline L9 eval gate is green, you ship to 100%, and a day later users complain the agent 'feels worse' — but every request still returns 200 OK. Why didn't the offline gate catch it, and what would have?

4. Why roll out a new agent version to a 1% canary first instead of 100%, when your eval suite already passed?

5. Your agent has a create_refund tool. What's the right production guardrail around it, and which earlier lesson does it come from?

Answers.

1. Every in-flight conversation on the restarted replica is lost (and any unsynced memory), because InMemory state lives in that process. Users mid-session get dropped. The fix: swap InMemorySaver / InMemoryStore for PostgresSaver / PostgresStore — durable, shared across replicas, and resumable after a restart.

2. The recursion/loop limit (recursion_limit, from L6) caps the number of steps so it can't loop forever; the cost cap / token budget caps the spend per run (soft-warn, then hard-stop) so a loop can't burn an unbounded bill. Together they bound both time and money for a stuck run.

3. The offline gate only tests your golden set at release — it can't see drift on live inputs it never contained, and a quality regression still returns 200 OK (it's not an error, it's a worse answer). Online evaluation — sampling live trajectories and scoring them with the L9 evaluators, with alerts — is what catches it. You need both: offline to gate releases, online to watch production.

4. Because passing the eval suite isn't proof of zero regressions on real traffic — the golden set is finite. A 1% canary limits the blast radius: if the new version is bad, it's bad for ~1% of users you're actively watching, and you roll back fast. At 100% the same bug is a full outage. Canary + rollback turns a potential incident into a non-event.

5. A human-approval interrupt on the write — create_refund is a high-stakes, hard-to-reverse action, so the agent should pause and require a human to sign off before executing (plus a tool allowlist and a scoped token). That human-in-the-loop pattern is from L7 (Tools + Memory) and the least-privilege/scoping from L8 (MCP Server Integration) — now enforced in production.

Mental-Model Corrections

  • "It works in dev, so it's ready to ship." → Dev ran on InMemory state, no guardrails, no monitoring. Production needs durable state, caps, online eval, and a rollback plan — shipping an agent is not shipping a stateless API.
  • "InMemory state is fine, I'll fix it later." → A single restart or deploy wipes every in-flight conversation and all memory. Use Postgres checkpointer + store from day one of prod.
  • "I'll add guardrails if something goes wrong." → Guardrails are infrastructure set before deploy (soft + hard limits), not a reaction to an incident. The first runaway loop shouldn't be how you learn you needed a cost cap.
  • "Offline eval (L9) is enough." → Offline gates the release; online eval watches live traffic for drift the golden set never saw. A regression that returns 200 OK is invisible without it. You need both.
  • "Ship it to everyone — the evals passed." → Passing a finite golden set isn't zero regressions on real traffic. Canary the change, watch it, and roll back fast. Small blast radius beats big-bang.
  • "The risk is model quality." → Agents fail in prod mostly from missing guardrails, poor observability, and inadequate fallbacks — operational gaps, not the model.
  • "Human approval slows everything down." → You gate only high-stakes writes, not every step. The cost of a confirmation click is nothing next to an unapproved refund or a wrong production deploy.
  • "Deploy and you're done." → Deploy is the start of operating the agent — monitor, evaluate online, feed the flywheel, and iterate. Production is a loop, not a finish line.

Key Takeaways

  • Shipping an agent ≠ shipping a stateless API. It's long-running, stateful, takes real actions, can loop, and costs money per step — production has to account for all five.
  • Durable state first. Swap InMemorySaver/InMemoryStore for PostgresSaver/PostgresStore (pooled, .setup()), so threads and memory survive restarts and deploys.
  • Serve it properly. LangGraph Server (langgraph.json → API + Postgres + Redis) or the Modal self-host pattern — the three-service shape is the production topology of a stateful agent.
  • Guardrails, designed in before deploy: loop limit (recursion cap), cost cap (soft + hard token budget), tool allowlist (least privilege), human approval on writes, fallbacks/retries. Each prevents a specific incident.
  • Observe + evaluate online. Trace every run (LangSmith / Phoenix); run the L9 evaluators on sampled live traffic; alert on drift, tool failures, latency, and cost; route low scores to human review (the flywheel). Online eval catches what the offline gate can't.
  • Roll out safely. Version + pin, gate on the L9 suite, canary (1% → 10% → 50%), and keep a fast rollback. A small blast radius turns a bad change into a non-event.
  • Agents fail in prod from missing guardrails, poor observability, and no fallback — not model quality. Engineering the operations is the job.

🎓 Project 2 Complete — What You Shipped

Take a moment — you built a production multi-tool agent end to end. Across five lessons you went from a blank page to a system you'd actually run:

  • L6 — Project Brief & Design: scoped the agent, chose the architecture (LangGraph create_react_agent), and wrote the brief.
  • L7 — Tools + Memory: gave it tools, short-term (checkpointer) and long-term (store) memory, and human-in-the-loop on writes.
  • L8 — MCP Server Integration: exposed tools as portable MCP servers, connected many at once, and secured the agent-tool boundary.
  • L9 — Agent Evaluation: measured the agent's trajectory — three levels, deterministic match + LLM judge, a CI gate.
  • L10 — Ship It: took the measured agent to production — durable state, guardrails, online eval, and a canary rollout.

That arc — design → build → integrate → evaluate → ship — is the full lifecycle of an AI product, and it's exactly what a senior AI engineer owns. You can now point at a real agent and speak to every layer: why it's architected this way, how its tools and memory work, how it's secured, how you know it's good, and how it runs safely in production.

Where next. The same lifecycle scales to harder agents and new modalities, and the course continues into fine-tuning and shipping your own models, then the capstone and career track. But the muscle you built here — measure it, guard it, ship it small, watch it live — is the one that separates engineers who demo agents from engineers who operate them. Ship it. 🚀