Datasets & Evaluation

I was building an AI native Computer-Use Agent, which in this case was used to QA chatbots directly from the UI. It basically reads the screen and decides what to click or type. One small part of it had to answer a yes/no question: is there more conversation below what I can see, or is this the whole thing? Most chat UIs scroll on their own. Some don’t. They leave a small scroll button at the bottom of the thread, and if the agent misses it, it reads a half-loaded page and acts on incomplete information. So I built a detector for that button.

In early testing it looked done. I gave it screenshots, and it found the scroll down button when one was there and said no when it wasn’t. On the ones I initially tried, it worked fine.

Then it started getting things wrong. For example, when a user asked the LLM to output something in markdown format (such as a system prompt), the background that the text was sitting on changed to the same background as the scroll down button, so my system missed it. It incorrectly classified an image, that had a scroll down button, as not having one (which is what we call a false negative). I only caught this one by watching the agent work through real chats and noticing the call it got wrong. And that was a real problem: seeing one failure told me the detector could be wrong, but not how often, or for which pattern of cases.

That is what this module is about. You have an AI component that works on the inputs you checked by hand and fails on the ones you didn’t. You have no measurement of the gap between them. When you change it to fix one test case, you might break another, and looking at a few screenshots won’t tell you which.

What got me out of it is the whole subject of this lesson, and it was boring. I collected the screenshots the detector got wrong, labeled what the correct answer actually was, and turned them into a fixed dataset I could run the detector against whenever I wanted. Now a change produced a number. I could see whether it helped overall, whether it broke any test case that used to pass, and whether it fixed the exact ones that had been failing. The rest of this lesson builds that: a ground-truth dataset that covers the test cases that matter, including the ones your system gets wrong, and an evaluation you run every time you change something.

Start with one test case. For the scroll down button detector, a test case is a single screenshot plus the answer that should come back: is the scroll down button present, or not. The screenshot is the input the model actually sees. The label is the correct answer, decided by a person, not by the model. A test case is that pair, an input and its verified answer, and the dataset is a pile of them.

The dataset holds both kinds of test case. The hard ones it got wrong, and the ordinary chats with an obvious scroll down button or an obvious end of conversation, each labeled with the answer you expect. You want the easy ones in there for a specific reason: when you later change the model to catch the test cases where the scroll down button sits on top of text, you need to know whether you broke one of the plain ones it was already getting right. If those test cases aren’t in the dataset, you can’t see that happen.

What makes the dataset worth having is that it also holds the test cases that break the model. A dataset built only from clean, easy test cases always passes, whatever you change, so its score can’t tell you whether a change helped or hurt. Mine started being useful the moment I added the test cases where the scroll down button overlapped the text, and the ones where it sat on an unusual background, each with its correct label, because the score could finally move.

I didn’t invent these test cases. I pulled them from real runs of the agent against actual chatbots, and kept the ones it got wrong. Then I labeled each one by hand, which for this task means looking at the screenshot and deciding whether a scroll down button is really there. That labeled pile is the ground truth: a fixed record of the right answer for each input, independent of what the model currently says about it.

With the dataset in place, the detector stops being something I check by eye and becomes something I run against a fixed list of known answers. Every test case has a right answer, so every run produces a count: how many it got right, and exactly which ones it missed. That count is what the next part of this lesson is built on.

The dataset gives you a count on every run: how many the detector got right. Turning that count into a score sounds simple, but the obvious way to do it hides the failure you care about. Three numbers are worth keeping straight.

Accuracy: how often it’s right overall

The share of all test cases where the model’s answer matched the label. In real chats, most of the time there’s no scroll button, so suppose nine out of ten test cases are labeled “no button.” A detector that answers “no button” every time, without even looking, is right nine times out of ten. Ninety percent accuracy, and it has never once done its job, because the test cases with a real button are the ones it misses.

Recall: of the real buttons, how many it caught

Of all the test cases that truly have a button, the fraction the detector found. The failure I opened with was a recall miss: a real button was there and the detector said no. For this system, recall is the number I watched, because a miss is the expensive error. When the detector misses a real button, the agent thinks it has seen the whole conversation, so it reads a half-loaded page and acts on incomplete information.

Precision: of the ones it flagged, how many were real

Of all the test cases the detector called “button present,” the fraction that really had one. A precision failure is a false positive: it claims a button that isn’t there, and the agent wastes a scroll. On this task that’s cheap. On another, it might be the error that hurts most.

A regression is a test case that used to pass and now fails. You'll hear people say “regression dataset” as if it's a second dataset you build. The difference between a regression dataset and your ground-truth dataset is only this: how you use the one dataset you already have.

The same ground-truth dataset does two jobs:

• Score where you stand now. Run it once against the current system, and read the metrics from the last section (recall and precision), plus the exact test cases it's getting wrong.
• Catch what a change broke. Run it again after a change, then compare the new run to the previous one, case by case. The test cases that flipped from pass to fail are the regressions.

The case-by-case comparison is the part that matters. A single total score can sit still while a change quietly trades one test case for another: it fixes two failures and breaks two that used to pass, and the number doesn't budge. Only comparing case by case shows you the swap.

And the dataset grows. Every test case you fix stays in it, so the same failure can't come back later without the comparison catching it. Over time it holds every problem you've already solved, and every run re-checks the new version against all of them at once.

Now the pieces connect. You have a dataset, a number that matters (recall), and a way to compare versions (regression). Here is the loop I actually ran to fix the low-contrast misses.

Run it

A run is simple to describe, and that's the whole point: if you can describe it precisely, you can hand it to an AI and read what comes back. A run does four things:

1. 01For each test case in the dataset, show the screenshot to the detector and take its yes/no.
2. 02Compare that answer to the label.
3. 03Count how many real buttons it caught versus missed. That ratio is recall.
4. 04Keep every missed test case aside, because that list is what you read next.

You don't have to write this by hand. The skill worth building is describing it exactly, then letting an AI produce the script. A prompt like this is enough:

Load scroll_button.jsonl. Each row has an image and a label, "button" or
"no button". Run my detector on each image and compare its answer to the
label. Report recall (of the rows labeled "button", the fraction the
detector also called "button") and list every row where the label was
"button" but the detector answered "no button".

What matters here is what you asked for: the right metric, and the failures kept separate. Ask for plain accuracy instead, and the AI writes you a clean, confident script that hides the exact problem you're chasing. My first run came back with a recall I wasn't happy with, and a list of the misses. That list is what you read next.

Read the failures

I opened the misses and looked at what the model answered and why. They clustered. Almost all of them were the low-contrast test cases, where the button sat on the same shade as the chat behind it. The pattern told me the fix: the model needed to be told those faint buttons exist and where they tend to show up.

Fix and run again

I rewrote the prompt to give it that context, then ran the same dataset again. For the change to count, two things had to be true:

• Recall up: it now caught the buttons it used to miss.
• Regression clean: the easy test cases it already handled still passed.

Both held, so the change shipped. The fixed test cases stayed in the dataset, so that failure can't come back without the next run catching it.

A dataset can pass every run and still be useless. A run can only check the test cases you put in it, so if they're weak, a high score means nothing. Four things make a dataset good.

Coverage: does it hold the ways your system fails?

Coverage is whether the dataset contains the test cases that break the system. My first version was all clean chats with obvious buttons, so it kept passing while the agent kept misreading real ones. It only became useful once every failure I'd seen (the low-contrast button, the button over text) had test cases of its own. A gap in coverage is a failure you'll ship without knowing it's there.

Balance: enough of the test cases that are rare but matter

In real chats, most of the time there's no button, so if you just sample live traffic you get a dataset that's about ninety percent “no button.” Recall is measured on the few real buttons, and with only a handful of them, one wrong answer moves the number a lot. Balance means deliberately adding enough of the rare test cases that matter (the real buttons, and the hard versions of them) so the number is steady enough to trust.

Honesty: test cases you're not sure you pass

The temptation is to fill the dataset with test cases you know the system handles, because a high number feels like progress. A dataset like that can't teach you anything, because it will always pass. A good dataset is full of test cases you're not sure about, including ones you expect to fail. Before a test case goes in, ask one question: could this fail, and would I want to know? If the answer is no, leave it out.

Size: small enough to actually read

Start smaller than feels right. Thirty to eighty test cases you chose and labeled by hand, and can re-read in an afternoon, beat five thousand scraped ones you've never opened. The danger of a big dataset is that you stop checking it: the labels drift, a few go wrong, and you end up trusting a number built on test cases you never verified. Grow it once the labels are solid and you need more weight on a specific slice.

Coverage only helps if you can actually find the test cases that break your system. They come from three places, and I trust them in roughly this order.

Real runs, especially the failures

The best test cases are the ones your system already got wrong on real input. That's where mine came from: I watched the agent run against actual chatbots, kept the screenshots it misjudged, and labeled the right answer by hand. These are worth the most because they're real. They carry the messy, unusual input you'd never think to invent, which is exactly the input that breaks things. Every failure you hit in production or testing should end up here as a test case, so it can't come back unnoticed.

Test cases you write by hand

When you know a failure mode but haven't seen it yet, write the test case yourself. I knew a scroll button on a busy background would be hard before it ever failed, so I could have built test cases for it up front instead of waiting. This is where a domain expert helps: someone who knows the system can write the ten inputs they know are hard. It's slower than collecting real ones, but it lets you cover a failure before it costs you.

Synthetic data, when real data is hard to get

Sometimes you can't collect enough real examples. The data is scarce or private, so you can't just go collect more. When that happens, you generate your own, which is what synthetic data means: examples you build to stand in for the real ones you don't have.

I ran into this on an internship. I was fine-tuning a YOLO model for one specific business use case, and the images came from a hospital, so real data was slow and hard to get. We had a handful of real examples and needed far more to train on, so I built a pipeline that generated synthetic images imitating the real ones, in many variations, and trained on those. I made the data myself because I couldn't get enough of the real thing.

The same technique applies far beyond images. Any time real data is scarce, you can generate examples to fill the gap. For an eval, that means asking an AI to produce test cases of a kind you don't have enough of. But two rules always hold. Keep most of your data real and representative, because synthetic data only helps if it matches reality, and matching reality is hard to get right. And label generated test cases yourself: if the same model writes the input and decides the answer, you've tested nothing.

Every section so far leaned on one thing: the detector gave an answer a person could write down ahead of time and check against. Button or no button, compared to the label, right or wrong. When you have that, matching against the label is the whole job, and most of this lesson assumes you do.

But some systems don’t hand you a clean answer. When the output is open-ended text, or there’s no fixed answer to compare against, the label check has nothing to match, and you need another way to score. Two tools cover most of it.

An LLM as the judge

The first is to have a strong model do the grading. You give it the output your system produced and the answer you were hoping for, and it returns a verdict with a reason. I used this on a project that mapped a user’s input to a required output. I had a ground-truth dataset of what each output should look like, but a language model is non-deterministic: run the same input twice and the wording comes back different, even when the meaning is right. Exact match was useless, because a correct answer almost never matched my reference word for word. The judge reads for meaning instead, so it could mark each output good or bad the way a person would, without me doing it by hand.

The catch is that you’re now using one language model to grade another, and the judge is as unreliable as the thing it’s grading. Its verdict can drift, or come back confidently wrong. So it helps to also score the output a way that has no model in it.

A deterministic cross-check

That’s where BLEU and ROUGE come in. Both compare the output to a reference by counting the runs of consecutive words the two share. ROUGE leans toward recall, BLEU toward precision, the same two ideas from the metrics section. The number is deterministic, so it can’t wander the way a judge can. Its blind spot is that it only sees word overlap: a correct answer worded differently scores low, and a fluent but wrong answer that reuses the reference’s words scores high. So I never trusted it alone. I ran it next to the judge, and when the two disagreed, that output was worth opening by hand.

When the judge is all you have

Sometimes even a reference is out of reach. The scroll agent’s verification step was like that: after it clicks to scroll, did the page actually move? The way to tell is the screenshot right after the click, so I had a vision model look at it and decide. There’s no overlap metric for “did this screenshot change the way it should have,” so the judge is the only option, and a tricky one. To make the call it has to detect the scroll down button itself, the exact problem the rest of this lesson is about. It can miss for the same reasons the detector did, and report a scroll that never happened.

None of this gets you out of the work the lesson started with. A judge is a scoring function, and it can be wrong, so you check it the same way you check anything else: run it against a ground-truth dataset a person already labeled, and see how often it agrees. It drops into the same run, read, fix, run again loop the detector used, in place of the label check. You still need the dataset.

You’ve read how an eval works. Now build one. Take a component you have and turn it into something you can measure, the same way I turned the scroll detector into a dataset I could run. Use the scroll detector as a ready-made target, or swap in any AI component of your own that returns a checkable answer.

The assignment

1. 01Pick a component with a checkable answer. A yes/no or small-label output works best: the scroll detector, or any classifier of your own. You need to be able to look at the output and say whether it’s right.
2. 02Collect 20 to 40 test cases. Each is an input plus a label a person verified. Pull them from real runs where you can, and write the rest by hand. Include the ones your system gets wrong, plus some easy ones it should pass.
3. 03Run the dataset. Describe the run to an AI coding tool, the way the loop section did: for each test case, take the model’s answer, compare it to the label, report recall and precision, and list every miss.
4. 04Read the misses and fix one thing. Look at what failed and find the pattern. Change the one thing the pattern points to, usually the prompt.
5. 05Run the same dataset again. Confirm two numbers moved the right way: recall went up, and nothing that used to pass now fails.

You’re done when:

• Your dataset includes test cases the system gets wrong, plus some easy ones it should pass.
• You can state recall and precision as numbers, before and after.
• Your fix raised recall without breaking a test case that used to pass.
• The test cases you fixed stayed in the dataset.