Anatomy of an LLM API Call (Messages, Roles, Params)

Introduction

Every AI feature you will ever build — a chatbot, a RAG system, an autonomous agent — is, underneath, one API call to a model. Master the anatomy of that single call and everything else in this course (prompting, retrieval, tools, agents) becomes just a smarter way of shaping the same request.

In this lesson we dissect what actually goes over the wire — the messages, the roles, the parameters, and the response — using both Claude (Anthropic) and OpenAI, since you'll meet both in the wild. (We install the SDKs in the next lesson; here we learn the shape.)

You'll learn:

The mental model: an LLM call is a stateless function
Messages & roles — system, user, assistant — and a key provider difference
The core parameters you'll set on every call
How to read the response (and why stop_reason and usage matter)
How multi-turn conversations actually work

The Mental Model: A Stateless Function Call

The single most important idea: an LLM API call is a stateless, pure-ish function.

output_text, metadata  =  model( instructions + conversation_so_far, parameters )

You send everything the model needs every time: the instructions, the entire conversation history, and the settings. The model remembers nothing between calls. There is no session on the server holding your chat — if you want the model to "remember" turn 1 at turn 5, you resend turns 1–4.

This one fact explains a huge amount of AI engineering: it's why the context window matters, why long conversations get slower and more expensive, and why "giving the model the right information" (context engineering) is a core skill. Hold onto it.

The Request, Part 1 — Messages & Roles

A request is mostly a list of messages, each with a role and content. There are three roles:

Role	Who it represents	Use it for
system	The developer (you)	Standing instructions: persona, rules, output format, context
user	The end user	The actual question or input
assistant	The model	Its previous replies (you include these to continue a conversation)

A conversation is simply an ordered list of user / assistant messages. To continue it, you append the model's last reply and the next user turn, then send the whole list again.

One provider difference worth memorizing

The system prompt is handled differently:

Anthropic (Claude): system is a top-level parameter, not a message. Putting {"role": "system"} in the messages array will error.
OpenAI: the system prompt is a message — the first item in messages, with "role": "system".

Same concept, two shapes. Get this wrong and your first Claude call fails with a confusing error — so it's worth knowing up front.

The Request, Part 2 — Core Parameters

Beyond messages, a handful of parameters shape every call. You met the sampling ones (temperature, top_p) in the decoding lesson — here's the working set:

Parameter	What it does	Note
`model`	Which model to call	e.g. `claude-sonnet-4-6`, `gpt-5.5`
`max_tokens`	Cap on the output length	Required by Anthropic; optional (but wise) on OpenAI
`temperature`	Randomness of sampling	0 = focused/deterministic-ish, higher = more varied
`top_p`	Nucleus sampling cutoff	Usually tune temperature or top_p, not both
`stop` / `stop_sequences`	Strings that end generation early	Great for structured output
`stream`	Stream tokens as they're generated	For responsive UIs (covered later)

Rule of thumb: set model, max_tokens, and temperature deliberately on every production call; leave the rest at defaults until you have a reason.

Anatomy in Code

Here is the same request to both providers, annotated. Notice the structural difference in where the system prompt lives.

# --- Claude (Anthropic) ---
from anthropic import Anthropic
client = Anthropic()

resp = client.messages.create(
    model="claude-sonnet-4-6",      # which model
    max_tokens=300,                  # REQUIRED: cap on output length
    temperature=0.7,                 # sampling randomness
    system="You are a concise travel assistant.",   # system = TOP-LEVEL param
    messages=[
        {"role": "user", "content": "Suggest 3 things to do in Kyoto in autumn."}
    ],
)

# --- OpenAI (same request) ---
from openai import OpenAI
client = OpenAI()

resp = client.chat.completions.create(
    model="gpt-5.5",
    max_tokens=300,
    temperature=0.7,
    messages=[
        {"role": "system", "content": "You are a concise travel assistant."},  # system = a MESSAGE
        {"role": "user",   "content": "Suggest 3 things to do in Kyoto in autumn."},
    ],
)

The Response — What Comes Back

The response is more than just text. Two fields you must always check: why generation stopped, and how many tokens it cost.

# --- Claude response ---
text   = resp.content[0].text          # the generated text
reason = resp.stop_reason              # 'end_turn' | 'max_tokens' | 'stop_sequence'
usage  = resp.usage                    # .input_tokens, .output_tokens

print(text)
if reason == "max_tokens":
    print("WARNING: output was truncated — raise max_tokens")
print(f"cost basis -> in: {usage.input_tokens}, out: {usage.output_tokens} tokens")

# --- OpenAI response (same ideas, different names) ---
text   = resp.choices[0].message.content
reason = resp.choices[0].finish_reason  # 'stop' | 'length' | 'content_filter'
usage  = resp.usage                     # .prompt_tokens, .completion_tokens

if reason == "length":
    print("WARNING: output was truncated — raise max_tokens")

Why these two fields matter so much:

Stop reason (stop_reason / finish_reason): if it's max_tokens / length, your answer was silently cut off mid-sentence. Beginners ship truncated output for weeks without noticing — always check it.
Usage (tokens in/out): this is your bill and your latency. Input tokens = everything you sent (system + full history + this message); output tokens = what the model generated. Watch both — they're the foundation of cost optimization later.

Multi-Turn — Building a Conversation

Because the API is stateless, a conversation is something you maintain: keep a list, append the assistant's reply and the next user message, and resend the whole thing.

messages = [{"role": "user", "content": "Suggest 3 things to do in Kyoto."}]
r1 = client.messages.create(model="claude-sonnet-4-6", max_tokens=300, messages=messages)

# append the model's reply, then the next user turn — and resend EVERYTHING
messages.append({"role": "assistant", "content": r1.content[0].text})
messages.append({"role": "user", "content": "Which of those is best in the rain?"})

r2 = client.messages.create(model="claude-sonnet-4-6", max_tokens=300, messages=messages)
# r2 can reference the earlier suggestions — because we re-sent them, not because the server remembered.

Every turn re-sends the growing history, so input tokens (and cost, and latency) climb with conversation length. That tension — give the model enough context, but not so much it's slow and expensive — is exactly what context engineering (a later section) is about.

Visualization

Diagram of an LLM API call. Left: a REQUEST box containing model and parameters (max_tokens, temperature), a system prompt, and a messages list (user / assistant turns). A center arrow labeled 'HTTPS POST (stateless)' points to a model. The model returns a RESPONSE box on the right containing the generated text (content), a stop_reason, and usage with input_tokens and output_tokens.

🧪 Try It Yourself

Write the call. Sketch the messages for a 2-turn chat (user asks, model answers, user follows up). Two checks:

Where does the system prompt go for Claude vs OpenAI? → top-level system= param vs the first message (role:'system').
Why must you include the assistant's previous reply? → the model is stateless; the only 'memory' is what you resend.

Build the API call — flip Claude vs OpenAI, add turns, and watch the payload shape and token bill change.

Common Pitfalls

Putting system in the messages array on Anthropic. It errors — system is a top-level parameter for Claude. (It is a message for OpenAI.)
Forgetting max_tokens. Required on Anthropic; on both, too low a value silently truncates output.
Assuming the model remembers. It doesn't — resend the history every turn.
Never checking the stop reason. max_tokens / length means your answer was cut off. Check it before trusting output.
Ignoring usage. Those token counts are your bill and your latency budget — instrument them from day one.

Key Takeaways

An LLM call is a stateless function: you send instructions + full conversation + parameters; the model returns text + metadata and remembers nothing.
The request is mostly a list of role-tagged messages (system, user, assistant) plus parameters (model, max_tokens, temperature, …).
Provider quirk: Anthropic's system is a top-level param; OpenAI's is a message. Anthropic requires max_tokens.
Always read stop_reason/finish_reason (truncation!) and usage (cost & latency).
Multi-turn = you resend the growing history, which is why context size drives cost — the seed of context engineering.

Next: we'll install the Anthropic and OpenAI SDKs and make these calls for real.