Observability & Traceability

By the end of the last module I could tell you exactly when my scroll down button detector was wrong. The dataset gave me a recall number and the exact screenshots it missed. What it couldn't give me was the reason. A test case shows that the model answered “no button” when a button was there. It can’t show what the model was thinking on the way to that answer, and for a component that reads an image and returns yes or no, the reason lives inside that reasoning. The output alone throws it away.

So I recorded it. I traced the detector’s runs through Langfuse and read back what the model actually reasoned on each screenshot it missed. The failures stopped being a list of wrong answers and became a list of wrong reasons.

That is what told me the fix. On the misses, the model’s own reasoning showed it never considered that the scroll down button could blend into the background. When a user asked for markdown output, the button sat on almost the same shade as the text behind it, and the model, reasoning as though every button stands out, never looked for a faint one. I could not have guessed that from a yes/no answer. I could read it straight off the trace.

This looks small because the example is one screenshot in and one answer out. Fixing it for every screenshot is not small, and without a record of the reasoning it is much harder, because the judgment happens inside a vision model. There is no classifier to open up, no threshold to print. The reasoning is the only thing you can inspect, and if you don’t record it, a wrong answer is all you keep. That is what this module is about: making an AI system’s work visible enough that when it fails, you can see which step failed and why, instead of guessing from the output.

A trace is the record of one run, broken into the steps the system took. Each step is one operation, a single model or tool call, and the trace keeps its input and output plus the exact call it made in between, each stamped with when it happened. Line the steps up in order and you can see the exact path the run took.

Take the scroll down button detector from the last module, the component that reads one screenshot and answers whether a scroll down button is on the page. Its whole run is a single model call, so its trace is just that call: the screenshot it saw, and the reasoning and answer it gave back. Nothing else happens inside it, so that is the entire trace.

The detector was one small component inside a much larger system, though. A real run passes through many components like it, and each one makes its own calls that can go wrong in their own way. They all need the same recording, so that when the system fails you can open any component and see what it did, instead of only the final answer it produced.

In the agent, one step’s record is concrete: the screenshot and prompt that went in, what the model decided, whatever tool or action it fired, and the result that came back. Read the steps top to bottom and you have every input and output, and every call the system made, in the order they ran.

The tool I used for observability and tracing was Langfuse. It records every run as a tree of steps I can open one at a time, so a task that failed deep in a long run is a few clicks from the exact step that broke, and the reasoning inside it. Without that record, a failed run is one wrong result at the end, with no way back to the step that caused it.

Everything so far describes a shape that every tracing tool shares: a run, recorded as a tree of steps, each step a model or tool call with its inputs and outputs. What changes from one tool to the next is only the vocabulary on the boxes.

The tool I used, Langfuse, calls the whole run a trace and each step inside it an observation, and it groups related runs into a session, so one user conversation is a single session made of many traces. OpenTelemetry, the vendor-neutral standard most tools build on, calls each step a span and the tree of them a trace. The labels differ. The nesting is the same, so once you can read one trace viewer, you can read any of them.

That is why it is worth trying more than one of them. You are learning the structure every one is built on, so pick whichever you like and go deeper.

A trace only helps if the right things are inside it, and the common mistake is logging too little: the final answer and maybe an error, with nothing kept about the middle. Go back to the system diagram. Every arrow between those boxes is data moving from one component to the next, and every box is a decision. If you keep only the endpoints, a failure in the middle is invisible again. Here is what to capture at each step, and the rule that decides it.

Log this at every step

• Every input and output, as the real thing. The actual screenshot a component saw, and the text or document the next one produced from it. Log the image itself, not a note that says “an image was passed.” When a component misbehaves, you need to see exactly what went in and what came out.
• The full prompt, exactly as it reached the model. Every variable already filled in, the way the model actually received it. The bug often lives in what got interpolated into the prompt, and only the filled-in version shows it to you.
• The model’s raw reasoning and output. The chain of thought and the unparsed response, before your code trims it down to a yes or no. That reasoning is the part that told me the detector assumed every button stands out.
• Which model, and its settings. The model name and version, and the settings you set, like temperature. So you can catch a step that quietly fell back to a smaller model, or a setting that made it wander.
• Every tool call, with its arguments and its raw result. Which tool the agent called, what it passed in, and what came back before any cleanup. Agents fail by calling the wrong tool or handing it the wrong argument, and the final answer never shows you that.
• Cost and latency, per step. The tokens and dollars a step spent, and how long it took. This is how you find the one slow or expensive step later, instead of guessing at the whole run.
• Failures and retries. When a step failed and tried again, and how many times. A step that only worked on its third try is a step that mostly does not work, and a trace that hides the retries hides that.
• Enough labels to slice by later. What kind of input it was, and a version tag for your prompt or system. So you can pull every failure on markdown inputs, or everything since your last change, instead of scrolling through runs by hand.
• Keep secrets out. API keys and personal user data don’t belong in a trace, since a trace gets stored and shared. Redact them where the data enters, before any of it is written down.

A trace records how long each step took and how many tokens it used, so it answers two questions the final output never does: where is this run slow, and where is it expensive. A single QA task runs many model calls, so its latency and its cost are both the sum across every step. The trace breaks that sum apart and shows which step took most of it.

Latency is what the user feels. An agent that takes 5 minutes to answer feels broken, even when the answer is right. Open the trace and the slow step is the longest bar, usually one model call doing the heavy work, sometimes a tool call waiting on a slow API. You optimize that one step, because the trace told you where the time actually went.

Cost is real money. Every model call spends tokens, and tokens have a price, so a run that quietly makes ten calls costs about ten times what a single call would. The trace shows the tokens each step used, so you can find the one prompt that is far too long, or the step that reaches for an expensive model when a cheap one would do.

And watch these across many runs. One run can be fast by luck. Track the P95, the slow-tail latency that only some of your users hit, because that is the number that decides whether the system feels reliable.

In a run with many steps, each step hands its output to the next. The screen reader’s text goes to the planner, the planner’s decision goes to the action, and on down the line. When something goes wrong, the step that visibly fails is usually not the step that caused it. A wrong answer early gets passed forward, and every step after it does something reasonable with bad input, until the mistake finally shows up at the end.

Follow the scroll down button detector through a real run. Remember what it does: it looks at a screenshot and answers whether there is more conversation below. Suppose it says “no button” when a button is actually there. That wrong answer does not stay put. The planner reads “no button” as “this is the whole conversation,” so it stops scrolling and treats a half-loaded page as the full chat. The verifier checks the screen after the action, sees nothing wrong, and the run keeps going, so the agent produces a QA result for a conversation it never finished reading.

What you see is a wrong result at the very end, and the last step looks fine: given what it was handed, it did exactly the right thing. So did the step before it. If you only look at the step where the failure showed up, each step looks correct on its own, and you get nowhere.

The trace lets you walk backwards instead. You open the failing run and read each step’s input and reasoning in order, from the end toward the start, asking one thing at each hop: was this step right, given what it actually received? The verifier was right about a half-loaded page. The planner was right to stop, given a “no button.” The detector was the first step that was wrong about its own input, and its reasoning shows why. That is the hop that caused everything after it.

The per-step logging from the last section is what makes this walk-back possible. A trace that keeps only the final output can tell you the run failed. One that keeps every hop lets you find the one decision that failed, several steps before the symptom showed up.

Walking the trace back gets you to the broken step. On the scroll agent that was the scroll down button detector, and the trace showed why: its reasoning assumed a scroll down button always stands out, so on a markdown background it never looked for a faint one. You do two things with that.

First, you fix the step. Because the trace told you the reason, you change the one thing that caused it. I rewrote the detector’s prompt to tell it that a scroll down button can sit on almost the same shade as the text, and where to look for it. Without the reasoning in front of me, I would have been editing the prompt and hoping.

Second, you take the exact input that failed and add it as a test case to the ground-truth dataset from the last module. That screenshot, labeled with the right answer, now lives in the eval. The next time you run it, the regression check tells you whether the fix worked, and every run after that tells you if the same failure ever creeps back.

Those two habits are one loop. Observability tells you why a single run failed and which step to fix. Evaluation tells you how often it fails across the whole dataset, and it catches the failure if it returns. The trace finds the bug once. The dataset makes sure it stays gone. So you build a raw system with tracing on from the start, watch real runs to see what it gets wrong, fix the step the trace points to, and add that test case to the eval. Then you do it again.

You have read how a trace works. Now put one into a system and use it to catch a bug. The point is to feel the difference between a wrong answer at the end and a trace that shows you which step caused it.

The assignment

1. 01Take a small multi-step system, three or four steps at most. It has to pass one step’s output to the next, so a mistake early can travel forward. A tiny agent works, or the scroll down button detector wired to a step that acts on its answer. Keep it small enough that a single run’s trace fits on your screen.
2. 02Add tracing to every step. Describe each step to an AI coding tool and have it record, for that step, the input it received, the model’s reasoning and raw output, which model and tool it called, and how long it took. Any tool works, from Langfuse to a plain JSON log you print yourself.
3. 03Run it on 15 to 20 inputs, including ones you expect to fail. Open the traces and read a few end to end, so the shape stops being abstract.
4. 04Find a failing run and walk it back. Read each step from the end toward the start, asking one thing at each hop: was this step right, given what it actually received? Stop at the first step that was wrong about its own input. That first wrong step is the cause you are after, wherever in the run the failure happened to surface.
5. 05Close the loop. Fix that step, then take the input that failed and add it as a test case to a small ground-truth dataset. Run it again and confirm two things in the trace: the step now reasons correctly, and the run ends right.

You’re done when:

• Your trace records what each step received and the reasoning it produced, at every step of the run.
• You can point to the exact step that first went wrong, using its reasoning in the trace.
• You fixed that step, and the input that failed is now a test case in your eval.
• The trace shows cost and latency per step, and you can name the slowest one.