MLGuerrillaStart with M1 →

The AI-engineering job market, measured

Learn to build the production AI systems employers actually ask for.

There are 22 modules live right now, free to read, with no account, and more to come. Each one takes a real piece of an AI system, like the ground-truth dataset, the evals, the tracing, the harness or the guardrails, and walks you through building it the way it gets built at work, including the failure cases and the trade-offs.

Start with M1 · Datasets & Evaluation →See all 22 modules

Free · no account · over 12 hours of reading · save any section as you go

Inside the modules

The DatasetM1 · Datasets & EvaluationFour inputs feed one component: a support ticket, a photo, an audio waveform and a strip of video frames. The component holds a fixed list of five labels. Out of it comes a score for every label, reading 0.91 for refund, 0.06 for damaged, 0.02 for delivery change, 0.01 for where is my order and 0.00 for other, with refund marked as the winner. A note underneath says the labels are fixed before the call and the model cannot return anything outside the list.M13 · ClassificationThe sentence “A capybara ate my strawberry” split into seven tokens, each shown with its numeric ID. The common words A, ate, my, and strawberry are single tokens; the rarer word capybara fans out into three pieces: cap, y, bara.M3 · How LLMs Actually Run ThingsA figure titled "It describes the scene correctly and misreads the small signs", from Qwen2.5-VL-3B on the street photo the vision articles use, asked four questions at temperature 0. On the left is the photo, a crosswalk in Amsterdam with people walking across it, shops behind them, 1920 by 1445 pixels. On the right are four question and answer cards. Asked what is happening in this photo, it answered "People are walking across a crosswalk in a city", marked correct. Asked exactly how many people are in the photo, it answered 7, marked correct, with a note that YOLO found 7 too. Asked whether there is any text visible and what it says, it answered with a list, RAVANAA BHAVAN, FOU FOW RAMER, Pasta e Pizza, SARAYANAA BHAVAN and FOU FOW RAMER again, and this card is outlined in terracotta and marked "two signs misread". Asked whether the bicycle is left or right of the stroller, it answered "Right", with a note that the boxes overlap so neither is true. A panel below, titled "what the signs actually say", shows a crop of the shopfronts and lists the truth. The small hanging sign SARAVANAA BHAVAN was read as SARAYANAA BHAVAN. The round sign FOU FOW RAMEN was read as FOU FOW RAMER, twice. RAVANAA BHAVAN matches the window sign, where a pole hides the SA, and Pasta e Pizza was read exactly. The caption reads: the wrong sign readings come in the same tone as the right ones.Models · Vision-language models (VLMs)A line chart for one task that succeeds 75 percent of the time, tried k times from 1 to 8 with each try independent. pass@k, the chance at least one try succeeds, rises from 75 percent to 98 percent at three tries and close to 100 percent after that. pass^k, the chance every try succeeds, drawn in terracotta, falls from 75 percent to about 42 percent at three tries, 0.75 cubed, and to about 10 percent at eight. A side panel shows a real benchmark, tau-bench with GPT-4o on retail tasks in 2024, solving about 61 percent on one try and under 25 percent on all eight tries, averaged over many tasks.M9 · Harness & ReliabilityThe support agent's state architecture. A customer exchanges messages with the support agent app, which has two tools, look_up_order and issue_refund. The app reads and writes four stores it owns, each labeled with a general group and the agent's version. Conversation history is the conversation, read by session id. User-specific memory is the remembered facts, read by customer id. Reference knowledge is the knowledge base copy, searched by meaning. Work in progress is the refund run progress, read by run id or status. Separately, operational records, the customer and order records, are owned by the order system and reached through the agent's tools.M6 · Data & StateA scatter chart of six configurations, with cost per resolved ticket across the bottom and pass rate on the 200-question dataset up the side. Sonnet 5 at low effort sits at $0.894 and 96%, and Sol at medium effort at $1.009 and 97%; both are terracotta and joined by a dashed frontier line. Sol at low effort is $0.969 at 96%, Terra at low effort $0.937 at 94%, Haiku 4.5 $0.930 at 91%, and Luna at low effort $1.003 at 86%, all in black below the line. Three cards underneath say that Sonnet 5 low is on the frontier at $0.894 a ticket and Sol medium buys one more point for 12 cents, that Sol low, Terra low and Haiku 4.5 are beaten on both axes, and that Luna costs 18 times less per token than Sol and the most per ticket, because 14% of its conversations reach a person.M12 · LLMOps & CostA figure titled "It transcribes speech well, and it writes words over silence too", from openai/whisper-small at 242M parameters on a Mac CPU, using public-domain LibriSpeech audio. The upper card, three clips of real speech, notes that the reference transcript has no punctuation or casing and that Whisper adds both. The first clip is 5.9 seconds of audio transcribed in 2.6 seconds, giving "Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel." The second is 4.8 seconds transcribed in 1.4 seconds, giving "Nor is Mr. Quilter's manner less interesting than his matter." The third is 12.5 seconds transcribed in 1.7 seconds, giving "He tells us that at this festive season of the year, with Christmas and roast beef looming before us, symbolies drawn from eating and its results occur most readily to the mind." The lower card, outlined in terracotta and titled ten seconds with no speech in them, notes that the same model was given audio containing nothing to transcribe. Given silence it wrote the word "you". Given quiet noise it wrote a long run of repeated Georgian letters. The caption reads: nothing in the output marks the last two as different from the first three.Models · WhisperTwo bar charts of the same next-token probabilities, sharp at a low Temperature and flat at a high TemperatureM4 · The Model as a ComponentA confusion matrix for a check, two by two, with the check's verdict down the side and the truth across the top. Top left, the check passed a good answer, 81.7% of traffic, marked costs nothing. Top right, the check passed a bad answer, 2.1%, marked in terracotta as a bad answer reaches the customer and costs a handoff at 2.50 dollars. Bottom left, the check flagged a good answer, 4.3%, marked as a needless escalation costing 0.1443 dollars. Bottom right, the check flagged a bad answer, 11.9%, marked as working correctly. Underneath, two arrows show that the two error cells cost 17 times different amounts, which is why the threshold does not sit in the middle.M15 · VerificationA nested diagram labelled in one tool’s terms: a SESSION box contains three TRACE boxes, and each TRACE box holds several OBSERVATION rows.M2 · Observability & TraceabilityA figure titled "An LLM's output is a probability for every token in its vocabulary". The top panel shows the prompt "The customer was charged twice for the same order, so this support ticket should go to the" going into Qwen2.5-3B, and a horizontal bar chart of the ten most likely next tokens with their probabilities: " billing" 30.2% in terracotta, then " ____" 7.8%, " __" 4.0%, " ___" 3.7%, " complaint" 3.6%, " Billing" 3.6%, " [" 2.6%, " issue" 2.5%, " fraud" 2.5% and " customer" 2.3%. A note says these ten add up to 62.9%, and the other 37.1% is spread across the remaining 151,926 tokens. The bottom panel shows eight generation steps at temperature 0, each picking the top token and adding it to the input: " billing" 30.2%, " department" 65.4%, "." 47.5%, " A" 24.7%, "." 94.2%, " Correct" 65.5%, " B" 99.8% and "." 100%. A bracket under the first three steps says the model completes the sentence, and a bracket under the last five says it is now writing a quiz. The caption says that with no chat template around the input, nothing marks where an answer should end, so it keeps going.Models · Large language models (LLMs)A three-circle diagram of the Agents Rule of Two applied to v2. Circle A, reads untrusted text, holds gift-card messages, delivery instructions, help-center passages and the customer's messages. Circle B, reaches private data, holds any order record by id with its contact details, and remembered facts. Circle C, changes something or sends data out, holds issue_refund up to $50 with no person, remember_fact and hand_off, and rendered images and links. Where A and C overlap without B, a label marks the gift-card refund, which used no private data. The center, where all three overlap, is filled terracotta and labeled v2, and a legend says it stands for one where are my flowers conversation in v2 as proposed. Two cards below the circles summarize the rules. The lethal trifecta, from Simon Willison in June 2025, is all three with C meaning a way to send data out, so an attacker can get the agent to read private data and send it to them. The Agents Rule of Two, from Meta AI in October 2025, allows at most two per session, and with all three a person or another reliable check has to supervise the agent.M10 · AI Security & Guardrails
Two bar charts side by side. The left chart, titled knowledge-update accuracy on the LongMemEval knowledge-update subset, oracle split, n equals 78, shows three pairs of bars. For gpt-4.1-mini, every session in the context reaches 82 percent in grey and a 300-character memory the agent maintains itself reaches 63 percent in terracotta. For gpt-4.1 the pair is 91 and 64 percent. For gpt-5.4 it is 92 and 77 percent. A dashed line across the chart is labelled as where full context saturates, 92 percent. The right chart, titled history against budget for gpt-4.1-mini at n equals 25, shows three bars. Two sessions at a 300-character memory reaches 68 percent in grey. Forty-eight sessions at 300 characters falls to 28 percent in terracotta. Forty-eight sessions at 7,150 characters stays at 28 percent in terracotta. A line under it reads 24 times the history at the same budget, then 24 times the budget at that history. The meta line says all numbers come from Patel 2026, Supersede, arXiv 2606.27472, a single-author preprint, with the left chart from its Table 1 and the right chart from its Table 2.M16 · MemoryA 128,000-token window drawn as a fixed-width horizontal bar, carved left to right into reserved output (4,000), the fixed system prompt and tools (about 6,000), the reference policy (2,000), and the remaining space for conversation history (about 115,000), with a dashed working-cap line marking 20,000 of history well short of the ceilingM5 · Prompt & Context EngineeringA figure titled "Images and captions share a space, and they sit in different parts of it", built from the same five images and five captions embedded with CLIP ViT-B/32 and placed on the one axis that separates them. On the left, ten labeled rows each carry a dot at its position along the first principal component, which accounts for 47.7% of the variance. The five images, the street photo, the stroller crop, the bicycle crop, the car crop and the hat crop, all sit as terracotta dots on the left side. The five captions, a busy city crosswalk, a baby stroller, a bicycle, a small black car and a striped knitted hat, all sit as dark dots on the right side. A shaded band between the two groups is labeled "the gap", and a note says only the dot's position is data and the rows are spread out so the labels fit. On the right, four measured averages over the same ten vectors are shown as bars: between two images 0.657, between two captions 0.655, image to its own caption 0.295 highlighted in terracotta, and any image to any caption 0.229. A note says two images look far more alike to CLIP than an image and its own caption do. The caption reads: this is why a raw image-caption score never reaches 1, and why 0.3 is a good match rather than a weak one.Models · CLIPThree results on the same scale of extra policy questions resolved within 24 hours per 100, agent-first group minus team-only group. The first, plus 6 with a range from plus 1 to plus 11, is real with its size still loosely pinned, so ship it and keep a holdout. The second, plus 6 with a range from minus 1 to plus 13, is drawn in terracotta because the range includes zero, which makes it inconclusive. The third, plus 0.4 with a range from plus 0.2 to plus 0.6, is real and below the 5 points agreed in advance as worth acting on.M8 · ExperimentationAccuracy, Precision, Recall confusion matrixM1 · Datasets & EvaluationThree rows comparing the same arrival rate at different call latencies, each against a pool of 32 slots drawn as small squares. Row one, 4 seconds per call in an ordinary hour at the peak: 1.54 turns per second times 5.8 seconds per turn is 9 in flight, and 9 of the 32 slots are filled. Row two, 12 seconds per call on the evening of the 12th: the same 1.54 turns per second times 17.5 seconds per turn is 27 in flight, and 27 slots are filled. Row three, outlined in terracotta, 12 seconds per call with up to 9 tries per failing call: the same arrival rate and turn time give more than 32 in flight, every slot is filled in terracotta, and the note reads pool full, new turns queue.M11-1 · Production, Deployment & Scale, Part 1A 3×3 grid of reasoning steps, t=0 to t=8. Each step-card shows everything the model has written so far plus one new token, spelling “strawberry” one letter at a time; the running r-count ticks 0, 1, 2, 3 as the r’s appear. A final banner reads: “strawberry” has 3 r’s. These are output tokens the model writes for itself, having been handed none of them as input.M3 · How LLMs Actually Run ThingsA figure titled "How attention updates one token". The sentence "The customer was charged twice" is split into five token cards. "charged" is highlighted in terracotta as the token being updated, and "twice" is greyed out and marked as masked, because a decoder can't look at later tokens. Under each of the first four tokens is its match score, the query of "charged" dotted with that token's key and divided by the square root of the key length: The 0.0, customer 2.0, was 0.3 and charged 1.2. Softmax turns these into weights shown as bars: The 8%, customer 57%, was 10% and charged 25%, adding up to 100%. A final row shows the new vector for "charged" as 0.08 times the value of The, plus 0.57 times the value of customer, plus 0.10 times the value of was, plus 0.25 times the value of charged. The formula softmax(QKᵀ / √d_k) V sits in the top corner, and the caption says most of the new vector comes from "customer", so it carries who was charged into the next layer.Models · Transformer and attentionSeven rungs compared on added latency and what each one needs. Rules add under one millisecond and need nothing. Embeddings plus kNN add about ten milliseconds and need a few hundred labelled requests. A trained router adds five to fifty milliseconds and needs thousands of graded pairs. A typed-decision model adds a claimed seventy to five hundred milliseconds and needs nothing. An LLM router adds five hundred to four thousand milliseconds and needs nothing. A provider-native router is included in the call and gives up control. A cascade adds no routing latency at all, and pays for a second call on the share that escalates. A band underneath gives the routing cost per sixty thousand conversations: zero for rules and embeddings, fifty cents for the typed-decision model and $3.12 for the LLM router.M14 · RoutingFour stacked rows and a side panel. Outcome is what the support team wants, policy questions answered without a person and first-reply time, measured on live traffic. Quality is whether the answers are right, matching the help-center article on a labeled set, measured on the eval set. Guardrails are what can't get worse, wrong policy claims and refund requests never handed off, measured on both. The ship bar, outlined in terracotta, lists at least 95 percent of 200 eval answers matching the article, 100 percent of refund mentions handed to a person, five cents or less per conversation, and a first reply faster than last February's thirty hours. The side panel shows tickets deflected struck out, replaced by got what they asked for and didn't come back a day later.M7 · AI Product FramingA trace-viewer timeline. A 4.23-second run named rag-chat-pipeline, with nested rows: a retrieve-documents span holding embed-query, vector-db-search (with a cache-miss event), and rerank-results, then a generate-answer span holding a generate-answer model call and a stream-complete event. Each row has a duration bar on a 0-to-4-second axis.M2 · Observability & TraceabilityYour app calls one function, complete(), which is the only place that talks to a provider; behind it OpenAI, Anthropic, or a self-hosted open-weights model can be swapped by configM4 · The Model as a Component

25 of the 162 figures drawn for the lessons and the model articles. Each one opens the page it belongs to.

From M1 · Datasets & Evaluation

I was building an AI native Computer-Use Agent, which in this case was used to QA chatbots directly from the UI. It reads the screen and decides what to click or type. One small part of it had to answer a yes/no question. Is there more conversation below what I can see, or is this the whole thing?

In early testing it looked done. Then it started getting things wrong. When a user asked the LLM to output something in markdown, the background the text sat on turned the same color as the scroll button, and the detector missed it. Seeing one failure told me the detector could be wrong, and it told me nothing about how often, or for which pattern of cases.

Example of a hard test case
This is an example where the bg of the scroll down button has the same color as the text background

Read the rest of M1 →

What the sweep found

The areas AI-engineering postings ask for.

  • Datasets & Evals
  • Observability & Traceability
  • LLM Application Engineering
  • RAG & Retrieval Engineering
  • Agentic Systems & Tool Use
  • Prompt & Context Engineering
  • Production AI Systems
  • AI Security, Safety & Guardrails
  • AI Infrastructure & Inference
  • LLMOps / AI Dev Lifecycle
  • Working as a Production AI Engineer

The whole curriculum

Every module the course covers, with the written ones filled in. The foundation comes first, the rails run through everything under them, and each capability repeats the same loop on its own subject.

Foundation

Rails

Capabilities

Compose

  • M25 AI System Design & Trade-offs, being written
  • M26 AI Infrastructure & Inference, being written
  • M31 Orchestration & Multi-Agent Systems, being written
  • M27 Capstone · Build One End-to-End, being written

Enterprise

  • M28 Enterprise, Compliance & Governance, being written

Professional

  • M29 Working as a Production AI Engineer, being written

start here22 written11 still being written

What you walk away with

You walk away with production-ready AI knowledge.

Production-ready goes beyond what an AI hobbyist builds in their projects. It means you learn what makes good AI systems employers hire for. Evals, observability, security, harnesses and much much more.

Each module ends with something you can put in front of a hiring manager. Three of them, from modules that are live now:

  • M1 · Datasets & Evaluation

    A labeled test set of 20 to 40 cases, with your recall and precision before and after a change, and the list of misses that shows the pattern you found.

  • M2 · Observability & Traceability

    The trace of one failing run with every step's input and reasoning, the step that went wrong, the fix, and that failing input added to your eval set.

  • M13 · Classification

    A classifier written down as decisions: the label set, what counts as one case, the threshold that auto-acts, the band a person reviews, and the cost per decision.

Who this is for

  • Software engineers who want to move into AI engineering.
  • CS students and new grads aiming at AI roles out of school.
  • Anyone who has shipped a demo and been asked how they know it works.

This covers building systems on top of models. Training foundation models from scratch is a different field and I don't cover it.

Who is writing this

I started taking AI seriously in 2022, did an R&D internship at an AI lab as the first undergrad they had taken, and became an Applied AI Engineer straight out of undergrad. Today I work in big tech on AI systems, including computer-use agents and AI tools for developer productivity, and I'm doing a master's in computer science and data science alongside it. The modules are the material I wanted when I was trying to get in.

The longer version →

Questions

Is this another bootcamp?

No cohort, no schedule, no 12-week promise. Modules you enter by need.

What does it cost?

Nothing right now. Every written module is readable in full, with no account and no email required. If that changes I'll say so on this page before it does.

I already work as a software engineer.

Start at M1 for the eval discipline, then jump to whatever you're building. The foundation modules assume you can already write and ship code, so they spend their time on the parts that are new: evals, tracing, harnesses, guardrails.

Will this get me a job?

It gets you the artifacts and the vocabulary. The market runs on two clocks. Postings ask for 2–8 years of software engineering but only 1–3 years of LLM-specific work, most commonly 2, because the field is barely two years old (five role families, n=493, measured 27 July 2026). What this makes you is the candidate with a working eval harness when everyone else brought a demo.

Do I need a degree?

31.2% of postings mention one. 68.8% don't.

Every posting counted

One dot = one posting. Hover for the receipt.

Start with the module everything else builds on.

M1 is 8 minutes. It ends with a labeled test set and a number for how often your system is wrong.

Start with M1 · Datasets & Evaluation →See all 22 modules

Follow along

I post the work as it ships: new modules, what the data says, and what I got wrong on the way. The Discord is open if you want to follow along or ask something.

Join the Discord →Watch on TikTok →

Coming soon

New modules go live as I write them. Get each one in your inbox the day it ships. No spam, just the next lesson.