How LLMs Actually Run Things

The first time you call an LLM API on your own, it is almost anticlimactic. You send one block of text, and the model sends one back. That's it. You can't send another message unless you run your code again. It keeps no memory of what you sent, and it cannot take any next steps on its own. One input prompt, one output response. That is the raw model.

You'll see it in OpenAI’s quickstart, it is simply a handful of lines where you make a client and send it one string, then print what comes back..

from openai import OpenAI

client = OpenAI()

response = client.responses.create(
    model="gpt-5.6",
    input="Write a one-sentence bedtime story about a unicorn.",
)

print(response.output_text)

Everything else you picture when you think of an LLM is built on top of that one call. The looping conversation, the chat that remembers your name when you tell it, the agent that uses tools, none of that is the raw LLM model. It's still a bunch of code in the background calling the model over and over and stitching the separate answers into what feels like a single system that remembers you and acts on its own.

This is not the theory of how the weights work. Attention and transformers, the architecture that makes next-token prediction possible, is covered better than I could in the papers and videos linked at the bottom. This module is the layer above that, the plumbing that turns one stateless call into something that can hold a conversation and take actions.

A handful of mechanics carry most of it, and each is simpler and stranger than it looks:

• Tokens. The model works in tokens, chunks of text often smaller than a word. It never sees the clean row of letters you typed, which the “how many r’s in strawberry” test made famous.
• The loop. One call answers once. A chatbot is that same call, run again by your code for each new message.
• Context. The model has no memory. What looks like memory is a list of messages your code re-sends on every call.
• Tools. The model never runs a tool. It writes a request, in text, and a harness you built runs it and feeds the result back.

There is a famous test for language models, where you ask how many r’s are in “strawberry”, and for a long time, the good models got it wrong and answered two. This became the go-to example of an LLM being surprisingly dumb.

The model has seen “strawberry” in millions of sentences and probably knows everything about it. But before any of the actual text ever reaches the model, a tokenizer breaks the word into tokens, which are chunks of text. Each one of those tokens is mapped to an ID number. By the time it arrives, “strawberry” has become a small handful of tokens. The model works from those chunks and no longer has the ten letters s-t-r-a-w-b-e-r-r-y laid out in a row to step through. At least not a non-reasoning model.

The easy conclusion is that the chunks are the problem. The model works in pieces, the thinking goes, so it never sees the r’s clearly enough to count them. But that's not what goes wrong. The letters are recoverable from the tokens which you can see by probing a model and confirming that the spelling is correct (Fu et al., 2024). A non-reasoning model (the kind that just answers you right off the bat), can spell “strawberry” perfectly well. What it can’t do is reliably count the letters, and that stays true for whichever chunk a letter falls in.

Why a non-reasoning model can’t, and how a reasoning model can, comes down to something separate from the chunks, namely how the model produces an answer, one piece at a time. The fix even has a name and a price, reasoning tokens. Both are easier to follow once you know what a token actually is, so that is where we go next.

A model reads text in chunks, most of them shorter than a word. A token is one of those chunks, and every token comes from a fixed vocabulary, a numbered list of all the chunks a tokenizer knows, each with its own ID. Splitting text into tokens is really just looking each piece up in that list. What goes in the list is the design choice. You could fill it with whole words, or with single characters, and both extremes cause real problems, which is why most models land on something in between.

Whole words look like the clean choice, but they fail on two counts. A word-level vocabulary needs an entry for every word the model might ever meet, in every language it might see. That list runs into the millions, and the model carries a slot for each one, so most of them stay rare and poorly learned. Anything off the list is worse off still: a new product name or a word in a language you did not plan for has no token at all, and its meaning is simply gone.

Single characters fix the coverage problem, since any text is just letters and symbols you already have. What they cost is length. A word that was one unit becomes five or ten, so every input gets several times longer in tokens. A transformer’s work grows faster than the length of what it reads, because its attention step compares every token with every other one. So longer inputs take more work to process and fill the context window faster. Characters are exact and expensive.

This comes down to one thing, the size of the vocabulary. Characters give you a tiny one, so everything is spelled out in many tokens; whole words would need an enormous one you cannot build. Between those extremes, the exact size still matters. The bigger the vocabulary, the more common sequences each get their own token, so the same text comes out as fewer tokens, which is cheaper and faster to run. But you cannot raise it without limit. The model carries a learned entry for every token in its vocabulary, so more entries means more parameters and memory, and the rarer a token, the less training it gets to learn a good one. Efficiency pulls the size up and model cost pulls it back, so providers settle on a size in between.

Subword tokenization is the middle ground most models use. The tokenizer scans a large amount of text and learns the pieces that show up most often. It starts from single characters and keeps merging the most frequent neighboring pair into one new token, again and again, until it has a vocabulary of common chunks. Common words end up as a single token. Rare or unseen strings fall back to a few smaller pieces, down to single bytes if they have to, so nothing is ever truly unknown. The vocabulary stays bounded, commonly around a hundred thousand tokens where a word list would need many millions, and sequences stay short. Common words and common word-pieces get their own entry; anything rarer is assembled from the smaller pieces the list does have.

You can see all of this on one sentence. Here is “A capybara ate my strawberry” run through OpenAI’s tokenizer:

Two things stand out. Common words come through as single tokens, while a rarer word like “capybara” is built from smaller pieces the vocabulary already has. And the split shifts with the surrounding text. “Strawberry” is a single token here, but lift it out on its own, with no space in front of it, and it drops to three, something like st, raw, berry. Either way the model works from these pieces, the same thing “strawberry” showed at the top, now across a whole sentence.

A short-looking prompt can still be a lot of tokens. Code and non-English text often break into far more tokens than their length suggests, so when the count matters, measure the tokens directly.

Splitting your text into tokens is only half the story. The model also outputs tokens, but it makes them one at a time. It reads everything so far (the whole input + every output token at t=n) and predicts the single next token, then adds that token on and runs again over the longer text. A hundred tokens of output is a hundred of those passes, one after another, each a full run through the model.

Input and output run through the hardware differently. The input prompt is read in parallel. These models run on GPUs, built to do enormous amounts of parallel arithmetic at once, so the model takes your whole input in a single pass and works every token of it at the same time. Output cannot work that way. As I mentinoed above, each token is generated from everything written so far, so the model cannot start one until the previous is done (that is what autoregressive means), and the tokens come out in sequence, one pass each, with much of the GPU’s parallel power going unused between them. That difference is what the price reflects. Input is cheap per token, and output costs several times more. For example, Claude Sonnet 5 is $2 per million input tokens and $10 per million output.

Back at the strawberry test, we blamed the miscount on something other than the chunking and never said what. It comes down to the one-at-a-time (autoregressive) way the model produces an answer. A non-reasoning model answers in a single pass, and a single pass does a fixed amount of work. Counting letters, especially one with repeated letters, is more step-by-step work than reliably fits in a non-reasoning model, so the model answers from a rough sense of the word and miscounts. “Strawberry” has two r’s back to back, exactly the case it gets wrong, so it lands at two.

A reasoning model handles it by working the problem out on the page first. It spends tokens to spell the word into its letters and count them one at a time before it commits to an answer. Writing the steps out moves the work out of that single shot and into the tokens themselves, where each step is something the next can build on. Imagine the model now being able to reason and think of each letter seperately BEFORE it gives you a final answer. You do not need a special model for this. Tell a plain model to spell the word out and count, and it will usually get there too; what a reasoning model adds is doing that on its own, by default.

You can watch it on a word the model has never met. Ask a reasoning model to count the r’s in something like “ungarbarrylirryburirira.” It cannot recall an answer, so it lays the string out, u n g a r b a r r y l i r r y b u r i r i r a, and works through it letter by letter. A strong one gets it right, even though it is a hard case, so it may land on a wrong tally first and fix it before giving you a final output.

That reasoning mechanism is not free. All of is considered to be output tokens, generated before the answer, but still billed at the same output rate whether or not the interface shows them to you (it still costs as much as any other output token even if it is not shown in the final output). So a reasoning model buys a better answer with more tokens and more latency. Reach for it when a problem needs working through, and leave it off when it does not.

BTW, It should go without saying, that when the task is purely mechanical, like counting the number of Rs in a word, don't go for LLMs. A model reasoning through characters is slow and can still land on the wrong number. string.count("r") is instant and exact. Save the model for the parts of the job that need actually need a model. The strawberry example was simply an exmaple to show the inner workings of tokens.

Every section so far has been about one call to the model, a single API request. You send a block of text, and it sends one back. That request is over. A chatbot clearly does more than that. You type, it replies, you type again, it replies again, for as long as you keep going. The model doesn’t change between those replies, and it isn’t running on its own in between. Each new reply is your a piece of code (usually a while loop) calling the model again.

That is the loop. Your code sends the user’s message and shows the reply that comes back, then waits for the next message and sends again. Each turn is a full call from scratch, one block of text in and one out, exactly like the very first one. The model never runs this loop; it answers once and stops, every time, and your code is what decides to call it again. (This is a different loop from the one inside a single call, where the model writes its answer one token at a time. That inner loop is the model’s. This outer one is yours.)

Written out, the loop is short:

from openai import OpenAI

client = OpenAI()

while True:
    user_message = input("you: ")  # wait for the next message

    # a full call from scratch, sending only this turn's message
    response = client.responses.create(
        model="gpt-5.6",
        input=user_message,
    )

    print("model:", response.output_text)

Run it in its simplest form, sending only the newest message each turn, and you get a chatbot that answers everything and remembers nothing. Tell it “my name is Ada” and it replies. Ask it “what’s my name?” on the next turn and it has no idea, because that second call never saw the first. Every turn, the model wakes up with no past.

So the loop gets you a chatbot that keeps answering. It still has no idea who it is talking to. Making each call carry what came before is the next piece.

The model does not remember anything you told it earlier. It keeps nothing between calls. Each one stands alone, and the server holds no thread of the conversation after it answers. The only thing the next call sees is what your code puts into it.

So how does a chatbot remember your name? It's because your code keeps the conversation as a list of messages, one entry per turn, and sends the whole list on every call. When you type your next message, the app appends it to the list and hands the model everything from the start again. Every call, the model rereads the whole history from scratch, because you sent it. The rereading is all the memory there is. Some providers store that list for you and call it a thread or a session, but underneath it is the same list, re-sent on every call.

from openai import OpenAI

client = OpenAI()
history = []

while True:
    history.append({"role": "user", "content": input("you: ")})

    response = client.responses.create(
        model="gpt-5.6",
        input=history,            # the whole conversation, resent every turn
    )

    print("model:", response.output_text)
    history += response.output    # add the reply so the next call can see it

That is the manual version, and it is what happens in the background. Keep the list, resend it. Newer OpenAI SDKs add a shortcut. You send only the new message and the id of the last response, and OpenAI keeps the earlier turns on its own servers.

from openai import OpenAI

client = OpenAI()
last_id = None

while True:
    user_message = input("you: ")

    response = client.responses.create(
        model="gpt-5.6",
        input=user_message,              # only the new message
        previous_response_id=last_id,    # OpenAI already holds the rest
    )

    print("model:", response.output_text)
    last_id = response.id

Nothing underneath changed. OpenAI is storing the same list and feeding it to the model for you. You still pay for every turn in it, and it can still outgrow the window. Not every provider offers this, so the manual version is the one that always works.

That list is the context. For any single call, everything the model can use is in it: the system instructions at the top, and every message in the conversation so far, including this one. Nothing else is in scope. If a fact is not in the list, the model does not have it, no matter how many times you told it in some earlier chat.

The context window is the size limit on that list, measured in tokens. A conversation long enough to pass the window has to lose something, so your code drops or summarizes the oldest turns. That is why a long chat starts to forget how it began. The beginning fell out of the window, or the app trimmed it to fit.

For you, that has direct consequences:

• Memory is your job. The model will not hold state for you. If you want it to remember something, it has to be in the list you send.
• You pay for the whole list every turn. Each call re-sends the full history, so a long conversation costs more per message as it grows.
• “It forgot” is usually a context problem. The fact was trimmed to fit the window, or it was never put in the list to begin with.

By now the chatbot holds a conversation and remembers what you told it. It still can’t *do* anything. Ask it for today’s weather or a row from your database, and it can only produce text, with no way to go and look. Everything a model emits is text. It never calls an API or runs code.

Tools are how it acts anyway, and the model still does not act. When you give a model tools, one of its possible replies becomes a request to use one, naming the tool and the arguments to call it with. Your code reads that request and runs the real function, then sends the result back on the next call. The model reads the result and keeps going. It decides which tool and when. Your code is what runs it.

Getting a model to do this is older than any real support for it. The early trick was to spell the format out in the prompt: to search, write a line like Action: search("…") and stop. The model wrote that line as ordinary text, and your code pulled the tool name and arguments out of it with regular expressions and ran the tool. It worked, and it was fragile. The model would drift off the format — a stray word, an argument that would not parse — and the parser broke. Nothing enforced the shape. You were trusting the model to format text exactly right every time (ReAct, Yao et al., 2022).

Models are trained for it now. You send a list of tools, each with a name and a description of what it does, plus a schema its arguments must fit. When the model wants one, it returns the call as structured data, the tool name and the arguments already in the shape the schema asked for, and flags the reply as a tool call. Your code reads the fields straight off and runs the function, then returns the result. No scanning, no guessing at the format. Because the model was trained on this, it holds the shape far more reliably than one coaxed into a text convention. OpenAI and Anthropic both expose it the same way, as function calling and tool use. Training a model to choose the tool and its arguments goes back to work like Toolformer.

Under both, old and new, the model only ever emits text. A tool call is text too, structured text the model learned to produce, filling the same reply it would otherwise answer in. It runs nothing. Your harness runs the tool and owns whatever it touches. And it all rides the same loop from before: the model asks for a tool, your code runs it and adds the result to the list, the model reads the result and either answers or asks again. An agent is that loop, left to run until the model stops asking.

Step back, and under every chatbot and every agent sits the same stateless call from the start, one block of text in and one back, with nothing kept afterward. Everything else was your code wrapping that call: a loop that carries the conversation forward by resending it each turn, and tools that turn the model’s requests into actions your code runs. The model reads and writes all of it in tokens, which is why you measure and pay for everything by the token.

That is what makes these systems debuggable. When a chatbot forgets what you told it a minute ago, the fault is the list you sent, and the weights are fine. When it miscounts the letters in a word, the model is reading tokens, and no prompt fully fixes it. Once you know which layer a behavior comes from, you know where to look.