Software Engineering with Large Language Models: Course 1
I made this music video with Claude Code over two days. In Module 1 I show you each step of the build and what the language model did at that step. Module 1 is free to read, to do and to watch, and the text of every other module is free to read.
Start Module 1See every module
A course in three parts: the vocabulary, the history that produced large language models, and a practicum in building, breaking and defending real software with them.
- Edition
- Autumn 2026
- Reading window
- Sep 18 – Nov 17, 2026
- Format
- Lecture notes + labs
- Length
- 34,991 words
Contents
- Module 1 · Making a music video with an agent
- Module 2 · The agent harness: tools and skills under oversight
- Module 3 · Semantic labeling and the data behind models
- Module 4 · From symbols to statistics (1943–2012)
- Module 5 · The Transformer and the scaling era (2013–2026)
- Module 6 · Harness engineering
- Module 7 · From brief to release
- Module 8 · Verification through testing and evaluation
- Module 9 · Laboratory: making games
- Module 10 · Laboratory: interfaces for people and for agents
- Module 11 · Laboratory: reverse engineering
- Module 12 · Network protection
- Module 13 · Cyber security for builders and their agents
- Module 14 · Shipping and operating software
- Module 15 · Local models and fine-tuning on your own hardware
- Module 16 · Capstone and assessment
- How the practical work runs
- Assignment 1 · Karel the agent
- Assignment 2 · Warm-ups for the label drill and the pyramid
- Assignment 3 · Bricks
- Assignment 4 · Sand and fire with byte decoding
- Assignment 5 · The infinite story
- Assignment 6 · A search engine served securely
- Section problems
- Starter code and tools
- Unit overview
- Build 1 · By hand without AI
- Build 2 · With an agent
- Hand code and AI code compared
- Timing worksheet
Course overview
Course description
I write most of my software now by directing language models. A model reads the files, runs the commands and checks its own work. My part is to state the goal, supply the context, limit what the model may do and verify the result. Typing code is now a small share of the job. In this course I teach that work as an engineering subject. Part I defines the vocabulary, Part II covers the history of the technology and how it works, and Parts III to V are practice. I compiled the material from sixty days of working practice in the field, from 18 September to 17 November 2026: engineering write-ups, tool documentation, published agent skills, incident reports and recorded talks. Where a figure comes from a vendor or from a single practitioner and has not been reproduced, I say so.
I restate the methods from those sources in my own words. Part II explains the technology in enough depth that you can derive the design choices of the later modules and do not have to memorise them. Part III is a practicum. It begins with the agent harness, the workflow from specification to release, and verification. It continues with applied laboratories in games, front-end craft, reverse engineering, network protection, security, operations and local models.
Part IV adds practical work: six assignments and a set of weekly section problems. You do this work with agents and language models, and the assignments and section problems are placed beside the modules that they exercise.
Prerequisites
You should be able to program in at least one language and to use a shell and Git. The course assumes no background in machine learning, and Module 5 states the mathematics that it needs. If you have not used a shell and Git, first take an introductory tools course that covers the command line, the development environment, debugging and profiling, version control, and packaging and shipping code, in that order.
Learning goals
By the end of the course you should be able to:
- Define and correctly use the working vocabulary of the field, including harness, context window, tool call, skill, subagent, semantic labeling, calibration and evaluation.
- Explain how a Transformer language model produces text, and derive from that mechanism why context is a budget, why models state falsehoods fluently, and why instructions are advisory.
- Configure an agent harness for a real repository: instruction files, skills, tools, hooks, permissions and a sandbox.
- Take a feature from a one-page brief to a verified release using agents, with evidence at each gate.
- Design tests and evaluations that measure behaviour, and detect tests that only restate the implementation.
- Build a small game, reverse engineer an undocumented device or protocol, segment and filter a network, and audit an application for security defects, in each case using an agent and checking its claims.
- Estimate the cost, latency and memory of a design that uses hosted or local models before you build it.
Course structure
| Part | Modules | Subject |
|---|---|---|
| I | 1–3 | Definitions and core vocabulary: the model, the harness and semantic labeling |
| II | 4–5 | History: from symbolic AI to statistical learning, then from the Transformer to agents |
| III | 6–16 | Practicum: harness, workflow, verification, games, front end, reverse engineering, network protection, security, operations, local models and the capstone |
| IV | with 1–14 | Practical work: six assignments and weekly section problems |
| V | with 7–9 | The remake unit: you build one game by hand and again with an agent, then time and compare the two builds |
Coursework and grading
Each module has objectives, lecture notes, and either a laboratory with a deliverable or a problem set. We grade a laboratory on evidence: the artifact, the record of how you verified it, and a written list of what you did not check.
Course materials
Each of the sixteen modules has a slide deck for a 60-minute class. A deck has 31 to 33 slides, or 23 for the capstone, and each slide has the teacher's notes under it. The deck sits in a frame under the module's heading, and the link above the frame opens the deck as a standalone page. A deck shows one slide at a time. The arrow keys move between slides, and N shows the notes.
Each assignment in Part IV has a starter kit to download, and so do the section problems and Part V. Starter code and tools lists the kits, the commands that run them, and the editors and online tools that we checked.
In Part V you build one game twice, by hand and with an agent, and time both builds. The unit ends with the two files side by side and a worksheet of measured times.
Course rules
Three rules apply in every module.
Rule 1. The checker is never the finder. Whatever produced a claim, a second and independent process must try to disprove the claim before anyone acts on it.
Rule 2. Behaviour that must always hold belongs in the harness. A sentence in a prompt is advice that the model usually follows. A hook, a permission or a validator is a guarantee.
Rule 3. Measure before you cut, and state what you did not measure. This rule applies equally to tests, cost, performance and security findings.
Part I · Definitions and core vocabulary
Module 1 · Making a music video with an agent
We are recording the Module 1 episode. Until it is up, you can watch C0der Bøi, the music video whose making this module explains step by step.
Module 1 is free to read on this page, to do as the project at the end of the module and to watch as an episode. The text, the project and the episode cover the same material.
Objectives
By the end of this module you should be able to make a twenty-second music video with an agent and name the tool, the cost and the check for each step. You should also be able to define the terms that describe a language model as a software component and state the engineering constraint that each term imposes.
The music video
I made C0der Bøi, a pop-punk music video of 2 minutes 50 seconds, with Claude Code on 14 and 15 October 2026. Claudita, a cartoon personification of Claude, sings the song. The song is a parody in the style of a 2002 pop-punk single, with an original melody and original lyrics. Claude Code wrote and ran every script in the pipeline, and I gave notes on each cut.
| Measure | C0der Bøi, version 2 |
|---|---|
| Length | 2 minutes 50 seconds, 1080p at 24 frames per second |
| Song takes | 55 numbered takes from one prompt; take 55 is the one in the video |
| Shots | 143, with an average length of about 1.2 seconds |
| Lip-synced shots | 15 |
| Video generation | about $56 of credit on fal, across versions 1 and 2 |
| Other generation | small charges on Google (music, stills and the listening judge) and ElevenLabs (sound effects), not itemised |
| Rendering | on a Mac, at no charge |
Claude Opus 5.5, running in Claude Code, wrote the concept, the lyrics, the edit and every script. Google Lyria 3.5 made the music and the vocals. Gemini 3 Pro Image (Nano Banana Pro) made the character sheets and the keyframe stills. Kling v3 turbo on fal turned stills into video clips, Kling Lip Sync matched Claudita's mouth to the vocals, and ByteDance Seedance 2.0 made the performance shots. Gemini 2.5 Pro listened to the song takes and scored them. Whisper measured the lyric timings and librosa the beat grid, and ffmpeg and a canvas page in headless Chrome rendered the film.
In sections 1.1 to 1.4 we explain each property of a language model with a moment from this build. In section 1.5 we set out the pipeline step by step, and in section 1.6 we give the cost and the rules on likeness and copyright.
1.1 The model as a component
Definition (large language model, LLM). A large language model is a neural network, almost always a Transformer, that is trained to assign probabilities to the next token of a sequence. Its weights (also called parameters) are the numbers that training fixes. Running the model is called inference.
A model keeps no state between calls. When it answers, it can draw on two sources: its weights, which training fixed at the training cutoff, and the text sent with the request.
| Term | Definition | Constraint it imposes | In C0der Bøi |
|---|---|---|---|
| Token | The unit that a model reads and writes: a word piece from a fixed vocabulary of roughly 50,000 to 250,000 entries. English prose runs near 0.75 words per token. | Providers count cost, speed and limits in tokens. | Each time Claude Code reread the 143-row edit list, it sent every row to the model again as tokens. |
| Context window | The maximum number of tokens in one request, input and output together. | The window is a fixed budget that the instructions, the files, the tool results and the answer share. | The session ran long enough that Claude Code compacted it and replaced the early conversation with a summary. |
| Prompt | Everything sent to the model for one inference. | The prompt is the only channel for task-specific information. | The 742-word music prompt held the genre, the arrangement and the full lyrics, and Lyria had no other source for them. |
| System prompt | The part of the prompt that the application sets before the user speaks. | The system prompt sets the role and the rules. It is text, and nothing enforces it. | Claude Code sends its own instructions and the project's CLAUDE.md file with every request. Nothing checks that the agent follows them. |
| Sampling | The choice of the next token from the predicted distribution. Temperature scales how random the choice is. | The same prompt can produce different outputs. Run anything important more than once. | One song prompt produced 55 numbered takes, which differed in melody, genre and structure. |
| Training cutoff | The date after which the model saw no data. | You must supply anything newer through retrieval or documentation. | An agent's knowledge of model names and prices stops at its training cutoff. Give it the current documentation and price list of each video model. |
| Hallucination | Fluent output that is false or unsupported. | Training optimises the model to continue text plausibly. Plausibility and truth are different properties. | When asked for an older Coder Boy, the video's main character, the image model changed his face and his ethnicity and added a beard. |
| Calibration | The agreement between stated confidence and observed accuracy. | You cannot use an uncalibrated confidence figure as a threshold. | The first scoring script rated lyric coverage and hook clarity, and it could not hear that the takes had drifted toward country. The agent added a Gemini listening judge, and I checked the judge's ratings against my own listening. |
1.2 The context budget
A chat application or an agent resends the whole conversation with every request. The agent appends each tool result and runs the model again.
A long session gets slower and more expensive with every step, because the input grows. Quality falls before the session reaches the limit, because irrelevant text in the window competes with the relevant text for the model's attention. At the limit you must choose what to drop. Consider a local model served with a 4,096-token window that the instructions, the input and the answer share. That model can summarise, classify and rewrite a page of text and extract information from it. It cannot hold a source file and reason about it. When you design a system that uses a model, choose a task that fits the window before you decide anything else.
The agent kept the working state of C0der Bøi in files instead of in the conversation. The edit list was in scripts/edl.py, the word timings in src/lyrics.json and the beat grid in src/timing.json, and the agent read each file when a step needed it. When the session was compacted, those files still held every detail that the summary dropped.
1.3 Three kinds of model
Practitioners now distinguish three classes of model by what each class emits and how it was tuned.
- Chat models are pretrained and then tuned on human preference to follow instructions and produce agreeable prose. Chat models are general and easy to use, but their confidence is poorly calibrated.
- Reasoning models receive further training with rewards that a program can check automatically, for example whether a test passes or a proof verifies. A reasoning model spends a variable budget of hidden deliberation before it answers. Reasoning models suit multi-step work, and they are the slowest and most expensive class per call.
- Decision models emit no prose. Given a state and a question with a fixed set of answers, a decision model returns a probability for each answer in a single forward pass. The typical question types are a yes/no proposition, a choice among described options and a score on an ordered rubric. Because the model generates nothing token by token, a decision takes from milliseconds to a few hundred milliseconds and costs a small fraction of a chat call. Published training recipes add a calibration term (a Brier loss) to the usual cross-entropy, so that you can use the probability as a threshold: act above the threshold and escalate to review below it.
Design rule. Use ordinary code for control flow, a decision model for classification, filtering and routing, and a reasoning model only where the task needs reasoning. Place a deterministic guard, written in code, after any model whose output triggers an action.
Treat vendor figures for decision models, such as speed-ups of two orders of magnitude or "zero hallucination", as marketing until you reproduce them on your own labelled set. A decision model cannot invent text, but it can still choose the wrong label.
In the C0der Bøi pipeline, Claude Opus 5.5 ran as the agent, and it is a reasoning model. The pipeline also used four generative media models, which fall outside the three classes: Lyria 3.5, Gemini 3 Pro Image, Kling and Seedance. Each generative model samples an output from a prompt, so it needs the same repeated runs and checks as a chat model. The Gemini listening judge was a chat model that answered one fixed question about each take. That is the job of a decision model, so the judge's ratings needed the check that a decision model needs: a comparison of its ratings with a human listener's.
1.4 Adapting a model
| Term | Meaning |
|---|---|
| In-context learning | Performing a new task from the instructions and examples in the prompt, with no change to the weights. |
| Retrieval-augmented generation (RAG) | Fetching relevant documents at request time and placing them in the context. |
| Embedding | A vector that represents a text, such that the vectors of texts with similar meanings lie close together. Semantic search is built on embeddings. |
| Fine-tuning | Continuing the training of a model on your own data. Continued pretraining uses raw text, supervised fine-tuning uses example conversations, and preference optimisation (DPO) uses pairs of better and worse answers. |
| LoRA | Low-rank adaptation, which trains a small set of added weights and leaves the base model frozen. QLoRA applies the same method to a quantised base model. |
| Quantisation | Storing weights at lower precision (commonly 4 or 8 bits) to reduce memory use at a small cost in quality. |
| Distillation | Training a small model to imitate a large model. |
Try the prompt first, then retrieval, then fine-tuning. Each step costs more than the step before it and is harder to undo. Private retrieval has one property that the other two methods lack. A general model contains only what was published, so knowledge that never reached the public record can enter the system only through the context that you supply.
The C0der Bøi build fine-tuned no model, and every adaptation was in the context. The music prompt named the genre and listed what to leave out ("no twang, no slide, no acoustic guitar, nothing country"). For each new still of Coder Boy, the request passed his original character sheet as the only reference image, and the prompt began with "EXACTLY the same face/ethnicity".
1.5 The pipeline
Each step below writes a file that the next step reads. The script names are the ones in the C0der Bøi project.
- Brief and concept. I described the song, the story and the look in a few messages. The agent wrote the concept and the shot ideas into the project's documents.
- Lyrics. The agent drafted the lyrics, and two subagents reviewed them, one for humour and one for craft and for originality against the parodied song. The parody needed a repeating hook, which became the rhyme of "coder boi" with "over, boi".
- Song.
generate_music.pysent the music prompt and the lyrics to Lyria 3.5 and saved each take.score_takes.pymeasured the vocal onset, the hook clarity, the lyric coverage and the length of the bridge.judge_avril.pyasked Gemini 2.5 Pro to listen to three sections of each take against a reference excerpt. Take 55 was the take that the judge rated best and that I picked. - Timing.
analyze_audio.pyused librosa for the beat grid and the energy of each section, and Whisper for word timestamps.build_lyrics.pyaligned the written lyrics to the transcribed words, filled the gaps and wrote the result tosrc/lyrics.json. - Characters and keyframes.
gen_image.pymade a character sheet for each person, then a keyframe still for each shot, always with the character sheet as the reference image. Before any video was generated, the agent showed me the stills side by side. - Clips.
fal_video.pysent each keyframe and a motion prompt to Kling v3 turbo on fal and downloaded a clip of four or five seconds. Seedance rejected most photoreal inputs as likenesses of real people, so the live-action shots used Kling. - Lip-sync.
lipsync_kling.pymade a performance clip of Claudita singing and then passed it, with the matching slice of the song, to Kling Lip Sync.sync_check.pycross-correlated the clip's audio with the song slice to find the offset.mouth_sync.pycompared the mouth opening in each frame with the loudness of the isolated vocal, andjudge_sync.pyasked Gemini to watch the clip with sound. - Edit.
edl.pyholds the edit decision list: one row for each shot, with its start time, its clip and its tags. The start times snap to the beat grid. - Compositor and render.
src/film.jsdraws each frame on an HTML canvas, with the camera moves, the colour grade, the lyric typography and the on-screen jokes.render.mjsopens that page in headless Chrome through Playwright, steps it frame by frame and pipes the frames to ffmpeg.mix_audio.pymixes the sound effects from ElevenLabs under the song. - Notes and recut. I watched each cut and gave notes in plain words. A note about the cut changed the edit list or the compositor, and re-rendering cost nothing. A note about a face, a performance or the song required new generations, which cost money.
1.6 Cost and rights
| Item | Cost | Source of the figure |
|---|---|---|
| Kling v3 turbo image-to-video | $0.112 per second of 720p video, about $0.45 for a four-second clip | the fal price at the time of the build |
| Video generation for the whole video | about $56 | the fal balance before version 1 and after version 2 |
| Music, stills and the listening judge | small, not itemised | the Google bill |
| Sound effects and voices | small, not itemised | the ElevenLabs bill |
| Rendering | $0 | it ran on a Mac |
Generation is the only step that costs money. A note that changes only the edit costs nothing to apply, and a note that needs new clips costs money, so settle the song and the stills before you generate clips.
We take four rules on rights from this build and from FRONT1ER, an earlier video made with the same pipeline.
- Write an original melody and original lyrics. The music prompt asked for an original melody and said "No artist impersonation". A parody may imitate the style of an era, but it must not copy the recording, the melody or the words.
- Do not depict a real person. Seedance refuses inputs that look like real people, even fictional characters drawn in a photoreal style. Do not try to get around that check.
- Design objects that are clearly your own. During FRONT1ER, fal's copyright check rejected a generated robot that looked like a Gundam mech, after fal had already billed the generation time.
- Read each provider's terms before you sell or monetise a generated song or video. The terms differ between providers and change over time.
Problem set
- Use a tokenizer library to count the tokens in three of your own source files. State how many files of that size fit in a 200,000-token window alongside a 5,000-token instruction file and an 8,000-token reserve for the answer.
- A classifier reports a confidence of 0.9 on 1,000 items and is right on 780 of them. Is the classifier over-confident or under-confident, and what threshold policy would you set?
- Choose a task. Write the same request three ways: for in-context learning, for retrieval and as a specification for a fine-tuning dataset. Say which version you would ship and why.
- Generate five takes of one short song from the same prompt. Before you listen, write three checks that a script can run on each take and one check that only a listener can make. Rank the five takes with the script and by ear, and report where the two rankings agree.
- C0der Bøi runs at about 150 beats per minute, and its 143 shots last about 1.2 seconds each on average. How many beats long is the average shot? If every cut must fall on a beat, what is the shortest shot you can have, and how many shots of that length fit in 20 seconds?
Project · A twenty-second music video
Make a music video between 19 and 21 seconds long with an agent. You choose the song, the story and the look. We mark the project on the eight checks in plan.py, on your credits and on your notes.
The starter code is in module-1-music-video.zip. In the kit, plan.py runs the eight checks and beats.py prints a beat grid. render.py uses ffmpeg to cut the shots on the beats and add the lyrics, burned into the picture when your ffmpeg supports it and as a caption track otherwise. The kit also holds a sample plan that renders with no files at all, a credits template and prompt templates taken from C0der Bøi.
You can take the free path or the generated path. On the free path you use a song that you made or have the right to use and your own photos or drawings as the stills, and the project costs nothing. On the generated path you use paid models for the song, the stills and the clips. In October 2026 five four-second clips from Kling v3 turbo cost about $2.25.
- Write the brief: the length, the song, the story, the look and the budget. Give it to your agent with the starter kit.
- Find the tempo and the first beat of the song, and print the grid with
beats.py. - Have the agent fill in
plan.jsonwith the shots and the lyric lines placed on beats, and runplan.pyafter each change. - Look at every still before the agent generates any clip from it.
- Render with
render.py, watch the result and give notes. Keep each note, word for word, innotes.md.
Hand in out/video.mp4, plan.json, credits.md with the tool and the cost of each part, and notes.md with each note and what it changed.
Practical work
Further practical work for this module is in Part IV: Assignment 2 · Warm-ups for the label drill and the pyramid and the Section problems.
Module 2 · The agent harness: tools and skills under oversight
We make this module's lesson video when 25 people sign up for it.
If you don't want to wait, you can buy the video now. The price is what it costs us to make it, plus 20%. We start on it the day you buy it, we send it to you the day it is done, and then we post it on this page for everyone.
What the $16.95 pays for
| Narration, 32,865 characters | $9.86 |
| 6 generated clips of 5 seconds | $3.36 |
| 6 keyframe stills for those clips | $0.90 |
| Rendering on our own Macs | $0.00 |
| Cost to make | $14.12 |
| Plus 20% | $2.83 |
| Price | $16.95 |
|---|
The agent runs on a flat subscription and rendering runs on our own Macs, so the price includes no charge for the agent or for rendering.
Objectives
By the end of this module you should be able to define the parts of the system that surround the model and place each rule or capability in the part that can enforce it.
2.1 Harness and agent
Definition (harness). A harness is the program that wraps a model. It assembles the prompt, sends it to the model, interprets the output, executes any tool that the model requested, appends the result and repeats. The harness also owns permissions, sandboxing, memory and the user interface.
Definition (agent). An agent is a model that runs in a harness loop with tools and pursues a goal over many steps. In each pass of the loop the agent reads the context, decides, acts through a tool and observes the result. The loop repeats until the agent finishes or is stopped.
The core loop fits in about two hundred lines of code. Harnesses differ in everything else: which tools they provide, how they manage context over hours of work, and what they ask of the human and when. Two harnesses that run the same model produce different work. When a better model runs in the same harness, earlier scaffolding often becomes unnecessary. A harness that is too strict for the model it wraps wastes the model. A harness that is too loose lets the model do damage that nobody reviews.
2.2 The parts
| Term | Definition |
|---|---|
| Tool | A function that the model may ask the harness to run, described by a name, a description and an input schema. The model sees only that description when it decides whether to call the tool. |
| Tool call | Model output that the harness interprets as a request to run a tool. The model itself never executes anything. |
| Model Context Protocol (MCP) | An open protocol through which a server offers tools and data to any compatible harness, either locally over standard input/output or remotely over HTTP. |
| Instruction file | A file that the harness loads in full at the start of every session (AGENTS.md, CLAUDE.md). It holds the recurring rules of a project, such as how to run the checks and what never to do. |
| Skill | A folder that contains a SKILL.md file and optional scripts, references and examples. Only the skill's name and one-line description are always in context. The body loads when the task matches the description. |
| Subagent | A second agent that the first agent starts with a fresh context of its own. The subagent returns only its conclusion. |
| Hook | Code that the harness runs at a fixed point, for example before every tool call. A hook can allow, modify or block the action. |
| Sandbox | An environment enforced by the operating system that limits what the code run by the agent can read, write and reach on the network. |
| Compaction | Replacing the older part of a conversation with a summary to free context. |
| Memory | Any store that the harness reads back into context in a later session. |
| Context engineering | Deciding what goes into the window, in what form, and when. |
| Progressive disclosure | Keeping a small index always loaded and fetching detail on demand. |
2.3 Choice of mechanism
An instruction file, a skill and a hook differ in cost and in strength. Choosing the wrong mechanism is the commonest configuration error.
An instruction file costs tokens on every request and is advisory. It suits a short list of standing facts. A skill costs almost nothing until the agent uses it (published packs report always-loaded descriptions of 175 to 205 tokens), and it suits a procedure that is needed occasionally. A hook costs no context and is deterministic.
In one reported case, a team wanted a cheap model to summarise routine file reads, so that the files were not loaded into the expensive model's context. The team wrote this requirement as a rule in the instruction file. The agent read the rule and opened the files anyway. The team then implemented the rule as a hook that intercepts the read call, and the rule held. The case illustrates Rule 2 of this course: if a behaviour must always hold, do not rely on a prompt to enforce it.
2.4 Oversight vocabulary
- Human in the loop. A person approves each consequential action before it happens.
- Human on the loop. The agent acts, and a person monitors it and can intervene.
- Autonomous or unattended. No person watches during the run. A person reviews the work afterwards.
As an agent becomes more autonomous, the person controls less of what it does but remains just as accountable for the result. A sound policy states which mode applies to each class of action. It places the irreversible classes (spending money, publishing, deleting, sending on someone's behalf) under human in the loop.
Definition (vibe coding). Vibe coding is building software by describing what you want and accepting the model's changes without reading the code. The term was coined in early 2025 for throwaway projects. Its defining feature is that nobody reads the diff. Vibe coding is not a synonym for AI-assisted engineering, in which the engineer reads, tests and reviews the changes.
2.5 The standard components of an agent product
Products built on agents converge on the same parts, in the way that web applications converged on a database, a users table and sign-in pages. The parts are the loop, connectors and tools, context management, memory, a sandbox, web search, computer use (operating a screen), background or scheduled execution, and notifications. In a design review, confirm that each part exists or is deliberately absent, and then ask what the product does beyond that baseline.
Problem set
- Take five rules from an instruction file that you use. Classify each rule as an instruction, a skill or a hook, and move any rule that is in the wrong place.
- Write a tool description for a function that issues refunds. Then write three requests on which the model should call the tool and three on which it must not.
- Consider an agent that has access to your email. List every class of action that the agent can take, and assign an oversight mode to each class.
Practical work
The practical work for this module is in Part IV: Assignment 1 · Karel the agent.
Module 3 · Semantic labeling and the data behind models
We make this module's lesson video when 25 people sign up for it.
If you don't want to wait, you can buy the video now. The price is what it costs us to make it, plus 20%. We start on it the day you buy it, we send it to you the day it is done, and then we post it on this page for everyone.
What the $18.28 pays for
| Narration, 36,556 characters | $10.97 |
| 6 generated clips of 5 seconds | $3.36 |
| 6 keyframe stills for those clips | $0.90 |
| Rendering on our own Macs | $0.00 |
| Cost to make | $15.23 |
| Plus 20% | $3.05 |
| Price | $18.28 |
|---|
The agent runs on a flat subscription and rendering runs on our own Macs, so the price includes no charge for the agent or for rendering.
Objectives
By the end of this module you should be able to define semantic labeling, design a label schema, build a labeling pipeline and measure its quality.
3.1 Definition
Definition (semantic labeling). Semantic labeling is the task of assigning to each item of data (a sentence, an image region, a video clip, a transaction, a span of code) a label that states what the item means in terms of a schema chosen in advance. Annotation is the general activity. Semantic segmentation is the special case in which every pixel is labelled. Named-entity labeling and intent labeling are the cases for text.
Supervised training consumes labels. Evaluation is impossible without labels, because a model's accuracy is defined against labelled examples. You can filter a labelled corpus before you search it, which improves retrieval. Operations depend on labels too: routing a support ticket, categorising a transaction and flagging a message that needs a reply are all labeling tasks.
3.2 Schema design
The schema limits the quality of the labels more than the labeler does.
- Close the label set. A fixed list of mutually exclusive categories, such as a tax code list or a ticket queue list, reduces the task to a choice. Open-ended labels drift.
- Define each label by a test, and give a positive and a negative example. If two trained people disagree about an item, the definition is incomplete.
- Provide an explicit escape. Make "Other" or "out of scope" a label in the schema. Published benchmarks show small classifiers that score well on in-scope intents and collapse on out-of-scope detection. That collapse is the failure that matters in production.
- Separate what is observed from what is inferred. "A person is holding an object" is an observation. "Armed" is the result of a rule applied to two observations (a weapon box that overlaps a person box). Keep derived states such as "armed" in code.
- Record provenance with every label. Record who or what produced the label, under which schema version and at what confidence.
3.3 Pipelines
Detection then classification. For images and video, a small specialised detector that runs locally finds and crops each object. A general classifier then labels each crop with a probability for each class. One published inspection demo reports about 270 ms and 280 input tokens per crop, at a cost of roughly three cents per thousand images. Budget the latency, the tokens and the cost per item before you choose the classifier.
Detection, tracking and counting. To count moving objects, the pipeline must keep the identity of each object across frames. A detector finds the objects, a tracker assigns each object a persistent ID, and the count increases by one when a track crosses a line. Without tracking, the pipeline counts the same object in every frame.
Model labeling with a human audit. A cheap multimodal classifier with a fixed schema can label in bulk. One practitioner reports tagging every goal of a hockey season (who is on screen, the shot type, the celebration) for under two dollars. That report includes no accuracy check. Before you trust a bulk run, label a random sample by hand and measure the agreement with the model's labels.
Human review of novel items. A scheduled pipeline that sorts bank transactions into tax categories stores every decision, asks the person only about merchants that it has not seen, and exports a deterministic report. The closed label set keeps the model's task simple, and the stored decisions make the pipeline consistent.
3.4 Measuring label quality
- Inter-annotator agreement. Have two labelers label the same items. Report Cohen's kappa, which corrects raw agreement for chance. When kappa is below about 0.6, revise the schema before you pay for more labeling.
- Gold set. A gold set is a few hundred items that have been labelled carefully and adjudicated. Score every labeler, human or model, against the gold set, and never use it for training.
- Calibration check. Bin the predictions by stated confidence, and compare each bin's stated confidence with its observed accuracy.
- Confusion matrix. A confusion matrix shows which pairs of labels the labeler confuses. Rewrite the definitions of those labels.
3.5 The supply chain for labels
The training of frontier models depends on human expertise that model developers buy: specialist annotators, graded reasoning traces, and constructed environments in which a program can score a model's actions automatically. Public job postings and procurement notices for annotators and data indicate which capabilities those developers are building next. Every accepted correction in your own product (an edited transcript, an overridden category, a fixed draft) is a labelled example that you already own. A system that stores these corrections can be adapted to its users later.
Laboratory 1
Choose a corpus of at least 500 items that you care about, such as support messages, commit messages or photographs. Write a closed schema of five to ten labels, with a definition and examples for each label. Label 100 items yourself, and have a second person label the same 100 items. Compute kappa, revise the schema, and repeat until kappa exceeds 0.7. Then label the remaining items with a model and score the model against your 100 labelled items. Report the accuracy, the confusion matrix, the cost per item and the cases that you could not decide.
Practical work
The practical work for this module is in Part IV: Assignment 2 · Warm-ups for the label drill and the pyramid and Assignment 4 · Sand and fire with byte decoding.
Part II · How we got to language models
Module 4 · From symbols to statistics (1943–2012)
We make this module's lesson video when 25 people sign up for it.
If you don't want to wait, you can buy the video now. The price is what it costs us to make it, plus 20%. We start on it the day you buy it, we send it to you the day it is done, and then we post it on this page for everyone.
What the $17.98 pays for
| Narration, 35,710 characters | $10.72 |
| 6 generated clips of 5 seconds | $3.36 |
| 6 keyframe stills for those clips | $0.90 |
| Rendering on our own Macs | $0.00 |
| Cost to make | $14.98 |
| Plus 20% | $3.00 |
| Price | $17.98 |
|---|
The agent runs on a flat subscription and rendering runs on our own Macs, so the price includes no charge for the agent or for rendering.
Objectives
By the end of this module you should be able to place any current technique on a seventy-year timeline, explain why each earlier approach stalled, and state the three ingredients whose arrival in the same decade made deep learning work.
4.1 Two founding ideas
The field began with the idea that thinking is the manipulation of symbols by rules. With his 1936 machine, Turing showed that a small set of mechanical operations can compute anything that is computable. In his 1950 paper he turned the question "can machines think?" into a behavioural test conducted in text. The 1956 Dartmouth workshop gave the research programme its name. The Logic Theorist, the General Problem Solver and later LISP-based systems all pursued intelligence as a search through a space of symbolic expressions.
The second idea has competed with the symbolic idea ever since. It holds that thinking is what networks of simple units do when experience adjusts the strengths of their connections. McCulloch and Pitts (1943) modelled the neuron as a threshold logic gate. Rosenblatt's perceptron (1958) added a learning rule, which nudges each weight in proportion to the error on each example. For a linearly separable problem the rule provably converges.
Shannon supplied a third thread. In 1948 he defined the information content of a message in terms of the probability of its symbols. In 1951 he measured the entropy of printed English by asking people to guess the next letter of a text. A language model in the modern sense is a machine that plays Shannon's guessing game. It is a function that assigns a probability to the next symbol, given the symbols so far.
Definition (language model). A language model is a probability distribution over sequences of tokens, usually factorised left to right: P(x₁…xₙ) = ∏ P(xₜ | x₁…xₜ₋₁). Everything that a large language model does at inference is a repeated evaluation of the right-hand term.
4.2 Limits of the first two waves
In Perceptrons (1969), Minsky and Papert proved that a single-layer perceptron cannot represent a function that is not linearly separable. The standard example is XOR. A multi-layer network can represent such a function, but at the time nobody had a practical way to train the hidden layers. Funding moved to symbolic systems.
Symbolic AI then produced expert systems, which were hand-written rule bases such as MYCIN for diagnosing infections and XCON for configuring VAX orders. Expert systems worked inside narrow domains and failed outside them. The builders had to extract knowledge from experts one rule at a time, a limit known as the knowledge-acquisition bottleneck. The rules interacted in ways that their authors could not predict. When an input fell outside the rules, nothing in the system degraded gracefully. The second "AI winter" followed in the late 1980s.
A system whose competence is bounded by what its builders can write down by hand stops improving when the builders run out of time. A system whose competence is bounded by data and compute keeps improving as the data and the compute grow. Sutton later named this pattern the bitter lesson: across seventy years of results, general methods that scale with computation have overtaken methods that encode human insight about the domain.
4.3 Backpropagation and the long wait
Rumelhart, Hinton and Williams (1986) popularised backpropagation. The method computes the error at the output and then applies the chain rule backwards through the layers to obtain the gradient of the error with respect to every weight. In principle, backpropagation solved the problem of training the hidden layers. LeCun's convolutional networks (1989–1998) read handwritten digits on bank cheques. Recurrent networks handled sequences by feeding their own hidden state back in. Hochreiter and Schmidhuber's Long Short-Term Memory (1997) added gated cells that let gradients survive across long spans.
Through the 1990s and 2000s these networks were mostly outperformed by support vector machines, boosted trees and graphical models. Those methods had better theory and lower compute demands. Neural networks needed three ingredients that did not yet exist together.
| Ingredient | What was missing | What supplied it |
|---|---|---|
| Data | Labelled sets at the scale of millions | The web and ImageNet (2009), a set of 14 million labelled images |
| Compute | Cheap dense linear algebra | Graphics processors programmed for general use (CUDA, 2007) |
| Technique | Stable training of deep stacks | ReLU activations, better initialisation and dropout, and later batch normalisation, layer normalisation and residual connections |
4.4 Statistical language processing before neural networks
Language technology turned to statistics earlier than vision did. From the late 1980s IBM's speech and translation groups treated speech and translation as problems of noisy-channel decoding, in which the system chooses the output sentence that maximises P(output) × P(input | output). The first factor, P(output), is a language model. In those systems the language model was an n-gram model, which estimates the probability of a word from counts of the previous two to four words. Smoothing methods (Katz back-off, Kneser–Ney) handle the sequences that never appeared in training.
N-gram models have two limits. They cannot use context beyond a few words. They also treat every word as an unrelated symbol, so evidence about "cat" teaches an n-gram model nothing about "kitten". Bengio and colleagues (2003) addressed the second limit with a neural probabilistic language model, which maps each word to a learned vector and predicts the next word from the vectors of the previous few words. Words that are used in similar contexts end up with similar vectors, so the model generalises across those words. Every embedding in use today descends from this idea of replacing discrete symbols with learned continuous vectors.
4.5 The three ingredients in 2012
In 2012 AlexNet (Krizhevsky, Sutskever and Hinton), a convolutional network trained on two consumer graphics cards, cut the ImageNet top-5 error from about 26% to about 15%. Nothing in AlexNet was conceptually new. The result was the first large public demonstration that depth, data and GPU compute together beat a decade of hand-engineered features. Within three years deep networks held the best results in speech recognition, vision and translation.
Discussion questions
- Expert systems failed because of the knowledge-acquisition bottleneck. Identify the equivalent bottleneck in a present-day agent that depends on hand-written tool descriptions and instruction files. Is the agent's bottleneck the same failure or a different one?
- Shannon used human guessers to estimate the entropy of English at roughly one bit per letter. In the same terms, what does a model's per-token loss measure? Why is a lower loss evidence of something that looks like understanding?
- Restate the bitter lesson as a falsifiable prediction about a tool you use now.
Module 5 · The Transformer and the scaling era (2013–2026)
We make this module's lesson video when 25 people sign up for it.
If you don't want to wait, you can buy the video now. The price is what it costs us to make it, plus 20%. We start on it the day you buy it, we send it to you the day it is done, and then we post it on this page for everyone.
What the $18.94 pays for
| Narration, 38,369 characters | $11.52 |
| 6 generated clips of 5 seconds | $3.36 |
| 6 keyframe stills for those clips | $0.90 |
| Rendering on our own Macs | $0.00 |
| Cost to make | $15.78 |
| Plus 20% | $3.16 |
| Price | $18.94 |
|---|
The agent runs on a flat subscription and rendering runs on our own Macs, so the price includes no charge for the agent or for rendering.
Objectives
By the end of this module you should be able to trace the line from word vectors to tool-using agents, name the paper that introduced each step, and explain what changed mechanically at each step.
5.1 Word vectors and sequence-to-sequence
Mikolov et al. (2013) trained word embeddings, known as word2vec, on billions of words with a shallow network. They showed that directions in the vector space carry meaning: the offset from "man" to "woman" approximates the offset from "king" to "queen". GloVe (2014) produced similar vectors from co-occurrence statistics.
Sequence-to-sequence learning (Sutskever, Vinyals and Le, 2014; Cho et al., 2014) made neural translation practical. An encoder LSTM reads the source sentence into a single fixed vector. A decoder LSTM generates the translation from that vector one word at a time. The fixed vector is a bottleneck, and translation quality fell as sentences grew longer.
5.2 Attention
Bahdanau, Cho and Bengio (2014) removed the bottleneck. At each step their decoder computes a weighted average over all the encoder states instead of reading one summary vector. The decoder learns to assign the weights according to relevance. The model learns the alignment between source words and target words without being told which words align. Attention began as an add-on to recurrent networks.
5.3 The Transformer
In "Attention Is All You Need", Vaswani et al. (2017) removed recurrence altogether. In a Transformer every position attends directly to every other position in the same layer, which is called self-attention. You should be able to reproduce the following mechanics on a whiteboard.
- The model maps each token to a vector and adds positional information to that vector, because attention by itself is blind to order.
- Each layer projects every token vector into a query, a key and a value. The attention weight from position i to position j is the softmax of the dot product of i's query and j's key, scaled by the square root of the dimension. The output at i is the weighted sum of the values.
- Several attention heads run in parallel, so that the layer can track different relations (syntax, coreference, position) at once.
- A position-wise feed-forward network follows each attention block. Residual connections and layer normalisation wrap the attention block and the feed-forward network.
- In a decoder, a causal mask prevents a position from attending to later positions, which makes left-to-right generation consistent with training.
Two consequences of this design explain the next eight years. First, training parallelises across all positions of a sequence, so a Transformer uses GPUs far more efficiently than a recurrent network does. Second, the path between any two tokens has length one, so long-range dependencies are easy to learn. The cost is that attention is quadratic in sequence length, which is why context length is a budget (Module 1).
5.4 Pretraining one model for many tasks
Before 2018 each task had its own model, which was trained on its own labelled set. Two papers changed the unit of work from "train a model" to "adapt a pretrained model".
- GPT (Radford et al., 2018) is a Transformer decoder that was pretrained to predict the next token on unlabelled books and then fine-tuned on each task.
- BERT (Devlin et al., 2018) is a Transformer encoder that was pretrained to fill in masked tokens using context from both sides and then fine-tuned. BERT set the best result on eleven language-understanding benchmarks at once.
Both pretraining objectives are self-supervised: the labels are the text itself, so the supply of training data is the written record. Subword tokenisation with byte-pair encoding (Sennrich et al., 2016) fixed the vocabulary at tens of thousands of pieces that can spell any string, which removed the out-of-vocabulary problem.
5.5 Scale and the prompt interface
GPT-2 (2019, 1.5 billion parameters) wrote coherent paragraphs. GPT-3 (2020, 175 billion parameters) showed that a large enough model performs a new task from a description and a few examples placed in its input, with no weight update. This behaviour is called in-context learning, and it moved the interface from fine-tuning to the prompt. Kaplan et al. (2020) measured loss and found that it falls as a smooth power law in parameters, data and compute. This result let research laboratories plan their budgets around capability. Hoffmann et al. (2022) corrected the recipe by showing that earlier large models were under-trained for their size and that parameters and training tokens should grow together.
5.6 Alignment to instructions
A raw pretrained model continues the text it is given and does not follow requests. Three techniques produced assistant behaviour.
- Instruction tuning. The developers fine-tune the model on pairs of an instruction and the desired response.
- Reinforcement learning from human feedback (Christiano et al., 2017; Ouyang et al., 2022). The developers collect human preferences between outputs, train a reward model on those preferences, and then optimise the language model against the reward model.
- Written principles. The developers train the model against a document of stated values and behaviours (Bai et al., 2022), which makes the training target open to inspection.
ChatGPT (November 2022) combined a capable model, tuning of this kind and a chat interface. It reached a hundred million users in about two months.
5.7 From text predictor to agent
Each step from chat to the systems that you use in this course's practicum was small, and each step is still visible in current tools.
| Year | Step | What changed mechanically |
|---|---|---|
| 2020 | Retrieval-augmented generation (Lewis et al.) | The system fetches documents at query time and places them in the context instead of relying on what the weights memorised. |
| 2022 | Chain-of-thought prompting (Wei et al.) | A prompt that asks for intermediate reasoning raises accuracy on multi-step problems. |
| 2022 | ReAct (Yao et al.) | The model interleaves reasoning text with actions and the observed results of those actions in one loop. |
| 2023 | Function calling in commercial APIs | The model emits a structured request, and the surrounding program runs the request and returns the result. |
| 2023–24 | Long context and multimodal input | The context holds hundreds of thousands of tokens, and the model accepts images, audio and documents as input. |
| 2024 | Model Context Protocol and computer use | The protocol is a standard wire format for tools and data sources. With computer use, a model operates a screen. |
| 2024–25 | Reasoning models | The models are trained to spend a variable budget of hidden deliberation before they answer. |
| 2025–26 | Coding agents and harnesses | A model runs in a loop with a shell, a file system and version control, and works for hours on one task. |
Through all of these steps the model stayed a next-token predictor. The change was in what the surrounding program allows the model's tokens to do. When the surrounding program interprets a token sequence as "run this command", the predictor becomes an actor. The program that does the interpreting is the harness, and it becomes half of the system (Part III).
5.8 Parallel lines of work
- Open weights. The developers of Llama, Mistral, Qwen, DeepSeek and other models publish the weights, and the models run on local hardware. With quantisation (storing weights in 4–8 bits instead of 16) and efficient runtimes, a capable model runs on a laptop (Module 15).
- Diffusion models (Ho et al., 2020) took over image, audio and video generation. They are trained with a different objective, which is to learn to reverse a gradual noising process.
- Interpretability. Researchers in interpretability work on reading the internal computations of trained networks. Their work ranges from attention-head circuits to dictionary learning over activations, and it is the field's attempt to replace behavioural testing with inspection.
5.9 Three regularities for planning
Three regularities in this history are stable enough to plan on. First, a capability that needs careful prompting in one model generation becomes default behaviour in the next generation, so scaffolding that you build to compensate for a weakness has a short life. Second, the cost of a fixed level of capability falls by a large factor each year, so a design that is too expensive now is worth prototyping. Third, evaluation lags capability. By the time a benchmark is widely reported it is close to saturated, and private, task-specific evaluation is the only kind of evaluation that stays informative (Module 8).
Problem set
- Implement scaled dot-product attention for one head in fewer than twenty lines of NumPy. Verify that, with a causal mask, the output at position t does not change when you alter the tokens after t.
- Train a character-level bigram model and a one-layer Transformer on the same small corpus. Report the held-out loss of each model and explain the gap in terms of context length.
- Choose one row of the table in 5.7 and find the earliest tool you use that depends on the step in that row. Describe what would break in the tool if that step were removed.
- A colleague argues that a model "only predicts the next token" and therefore cannot plan. Write a one-page response that takes the mechanism seriously and still explains multi-step behaviour.
Practical work
The practical work for this module is in Part IV: Assignment 2 · Warm-ups for the label drill and the pyramid.
Part III · The coding practicum
Module 6 · Harness engineering
We make this module's lesson video when 25 people sign up for it.
If you don't want to wait, you can buy the video now. The price is what it costs us to make it, plus 20%. We start on it the day you buy it, we send it to you the day it is done, and then we post it on this page for everyone.
What the $18.83 pays for
| Narration, 38,072 characters | $11.43 |
| 6 generated clips of 5 seconds | $3.36 |
| 6 keyframe stills for those clips | $0.90 |
| Rendering on our own Macs | $0.00 |
| Cost to make | $15.69 |
| Plus 20% | $3.14 |
| Price | $18.83 |
|---|
The agent runs on a flat subscription and rendering runs on our own Macs, so the price includes no charge for the agent or for rendering.
Objectives
By the end of this module you should be able to configure an agent harness for a real repository, write instruction files, skills and tool descriptions that a model can act on, and place every hard rule where it is enforced.
6.1 A working mental model
Treat the model as the processor, the harness as the operating system, and skills and plugins as the applications. The processor is replaced every few months, so anything you write against one model's quirks has a short life, while a well-kept operating layer carries over. Teams that ship products on agents report that they treat the model as replaceable, route different jobs to different models, and keep their prompts portable in preference to fine-tuning. To test a design, ask whether it gets better or worse when a stronger model is released next month.
A harness specialised for one industry has four parts beyond the model: connectors to that industry's data, evaluations written by people who know the work, rules for what the agent may and may not touch, and a clean screen on which a person reviews the result. Practitioners who build such harnesses estimate that the model is about a fifth of the system.
6.2 Managing context over a long session
Use six operations to manage context, in this order:
- Clear. Start a new session for an unrelated task. Old context that you carry forward costs tokens and accuracy.
- Rewind. When the agent has gone wrong, roll the session back to the last good point. A corrective message leaves the mistake in the window as an example.
- Delegate. Send searches, web research and wide file reading to a subagent, which returns only the conclusion.
- Disclose progressively. Keep an index in context and fetch detail on demand. One company reports 800 internal tool servers and more than 5,000 tools behind a gateway. No model can hold that many definitions, so the agent first finds the right server, then lists the server's tools, then loads one schema, and then calls the tool. A command-line tool with 3,000 generated operations solves the same problem with a search command, which takes a need stated in plain language and returns the matching commands.
- Compact. Summarise older turns. Compaction is lossy, so write decisions and open questions to a file first.
- Externalise. Write durable results to files, an issue tracker or a ledger. Do not leave them only in the transcript.
Documentation written for models, such as llms.txt or one Markdown page per component, is cheaper for a model to read than HTML. It is also the remedy when a dependency is newer than the model's training cutoff.
6.3 Writing instruction files
The model reads the instruction file on every request, so each line must be worth the tokens it costs. A good rule has three parts: a trigger, a required behaviour and a prohibited behaviour. The following three rules have proved their worth in public repositories.
- Scope. Before adding complexity to handle a speculative race condition or an unlikely security edge case, state the concrete failure mode, its likelihood and the complexity cost, and get agreement. Do not expand scope silently. Under this rule the model must present its defensive over-engineering as a proposal, which a person can decline.
- Tests. Do not write change-detector tests (Module 8 defines them).
- Age. Review any instruction older than four months. Instructions go stale as the models and the code change, and a stale rule is worse than no rule.
One way to produce a first draft is to have the model read the engineering writing and the open-source code that you most respect, and distil the instruction file from those exemplars. Whatever the source, keep the file short and move procedures into skills.
6.4 Anatomy of a skill
A skill is a folder of plain text. It does not run by itself. The agent reads it when the task matches the skill's description. The published skills that work well share the following structure.
| Part | Purpose |
|---|---|
| Name and description | The only text of the skill that is always in context, which lists the phrases that should trigger the skill and the requests that should not |
| One standard | A single sentence that defines a good result |
| Named failure modes | A short vocabulary for what goes wrong, against which the agent can check its work |
| Steps | Most of the steps come before the agent produces any output. |
| References | Topic files that the agent loads only when it needs them |
| Gold examples | Complete worked results to copy from, and bad/good pairs used only for diagnosis |
| Script | A build, measurement or verification program that the skill runs |
| Evidence and handoff format | A statement of what the agent must report, including what it did not check |
Several design practices are worth copying. Give each skill one narrow job, and add a small router skill that recommends which skill to use. Keep audit and review skills separate from build skills. Write a "learn" skill that, at the end of a session, turns the user's corrections into written rules with reasons. Where a skill needs memory across sessions, give it an explicit state-export command.
Develop a skill the way it will be judged. Give the same short prompt to two fresh agents, one with the skill and one without, and compare the results side by side. When something is wrong, write down what is wrong, change the skill, and run the whole comparison again. Do not fix the output by hand, because a hand fix leaves the skill unchanged.
A skill is code that you run with your own permissions. Know which channel each skill came from and whether that channel still publishes updates. When a marketplace stops updating third-party packs, the installed copies are stranded on old versions. Package names are squatted. Read a skill before you install it.
6.5 Tools and interfaces designed for a model
The model decides whether to call a tool from the tool's metadata alone. That metadata is therefore a selection surface, and it deserves the care that you give a public API.
- Give each tool an action-oriented name, a description that says when to use it, accurate annotations and a real input schema.
- Include negative cases in the acceptance tests: requests on which the tool must not fire.
- Enforce authorisation on the server for every request. The model is not a security principal, so never trust it to apply access rules.
- Validate the schemas with an inspector before you test in a host application.
Agents now use command-line tools as much as people do. One infrastructure provider measured agents at about a quarter of its CLI usage in March 2026 and at nearly half by autumn. The agents used almost twice as many distinct commands a day as humans. The provider redesigned its CLI for agents according to a short set of rules:
- Generate the CLI from the same API schema that drives the documentation, so that coverage is complete and the verbs are consistent.
- Make JSON the default output and condense it for agents. A table costs tokens, and the agent must parse it.
- Ship command search inside the tool.
- Prefer typed configuration (a TypeScript config with a schema) to untyped files, so that a language server checks the agent's edits.
- When behaviour must change, introduce a new tool. Models have absorbed years of documentation for the old tool, and a change to a familiar tool works against those learned priors.
6.6 Routing work to the cheapest adequate model
In a coding session a large share of the tokens goes to reading whole files, writing boilerplate and updating configuration. Delegation reduces that share. On the read path, a cheap model reads the file and returns a summary, so the main context receives the gist. On the write path, a cheap model generates the boilerplate and writes it straight to disk, so the main model never emits those output tokens. The claim that this pattern saves ninety per cent is second-hand and unverified. The trade-off is not in doubt: the cheap model missed a thread-safety defect that the strong model found quickly. Route work by task type. Delegate reading and boilerplate, and keep concurrency reasoning, debugging, review and design on the strong model. Provide a way to bypass delegation for files that the main model must read in full.
Instrument every run so that you can attribute the token spend to a task and an outcome. For a small team a flat subscription is hard to beat, so measure any orchestration that you build to save money against the subscription.
6.7 Permissions and sandboxes for attached machines
A persistent machine attached to an agent (installed tools, logged-in sessions, a real browser) makes the agent far more capable, because you authenticate once and most command-line tools expect credentials in standard places. The machine also extends the blast radius to everything it can reach, and it exposes the agent to instructions hidden in any page the agent reads. People complain that approval prompts stall long runs. The complaints are fair as a usability report, and they list exactly the controls that a careful design keeps: prompts for credentials, limits on self-modification, and allowlists.
The engineering answer is to isolate the agent by environment. Run a computer-use agent inside a guest virtual machine, so that the agent can see only the guest's files and sessions. Run a no-confirmation mode only inside a container or a VM. For anything that executes untrusted code, the sandbox must disable outside networking, pass a sanitised and allowlisted environment, enforce resource limits and restrict writes to assigned scratch paths. File-edit permission rules do not constrain shell commands, so a deny rule on a directory protects the directory only when the sandbox enforces the same rule.
Laboratory 2
Prepare a repository that you own for agent work. Submit four items: an instruction file of fewer than 60 lines; one skill with a description, a standard, named failure modes, a script and a handoff format; one hook that blocks a class of action; and a permissions file. Then run the two-agent comparison from 6.4 on three tasks. Report where the skill helped, where it made no difference, and what you pruned as a result.
Practical work
The practical work for this module is in Part IV: Assignment 1 · Karel the agent and the Section problems.
Module 7 · From brief to release
We make this module's lesson video when 25 people sign up for it.
If you don't want to wait, you can buy the video now. The price is what it costs us to make it, plus 20%. We start on it the day you buy it, we send it to you the day it is done, and then we post it on this page for everyone.
What the $18.45 pays for
| Narration, 37,029 characters | $11.11 |
| 6 generated clips of 5 seconds | $3.36 |
| 6 keyframe stills for those clips | $0.90 |
| Rendering on our own Macs | $0.00 |
| Cost to make | $15.37 |
| Plus 20% | $3.08 |
| Price | $18.45 |
|---|
The agent runs on a flat subscription and rendering runs on our own Macs, so the price includes no charge for the agent or for rendering.
Objectives
By the end of this module you should be able to take a feature from idea to verified release with agents, using written artifacts and approval gates, and identify the constraint that limits your team once building is fast.
7.1 The brief
When building is cheap, the fastest way to test an idea is often to build it. You then plan in build-and-review rounds, typically about three, and the planning document shrinks to one page:
- A goal that is deliberately more ambitious than your first instinct.
- A few concrete use cases.
- The place of the feature in the existing code, and the design considerations.
- The observable evidence by which you will judge that the feature works.
- Explicit latitude for the agent to exceed the brief.
Run the build unattended, review the result the next morning, discard freely, and send the work back to the agent. Expect a mixed result, in which some parts are good, some are poor, and some show that the idea was wrong. The method leaves out three things: review cost, integration, and the maintenance of a larger scope than you planned. Plan for those three separately.
At the scale of a single task, give every item in the task tracker four fields: Why (purpose), Done (the finished state), Access (tools and accounts needed) and Who (agent or person). A morning job reads the day's tasks and completes the tasks assigned to the agent. For each task assigned to a person, the job writes a brief that does as much of the work as possible in advance. The four fields cost about thirty seconds of writing per task.
7.2 Gated phases
For work with consequences, separate thinking from typing and require human approval after each thinking phase.
| Phase | Artifact | Gate |
|---|---|---|
| Orient | An index of the project, so that agents look things up instead of rereading it | None |
| Specify | Written requirements, gathered one question at a time | Human approval |
| Architect | Data flow, failure handling, retries and idempotency | Human approval |
| Plan | Small test-first tasks with exact file paths | Human approval |
| Build | A failing test, then the code, then a refactoring | Automated checks |
| Review | One pass that checks the change against the specification and one pass that checks that the code is sound | Serious findings block the merge. |
| Ship | A deployment with rollback and monitoring, which runs first in staged or paper mode | Human approval |
In a legacy-modernisation toolkit published this season, each command writes a named artifact (an intent file, a preflight report, a topology map, business rules as Given/When/Then cards with file and line citations, a phased brief). The toolkit builds nothing before the brief is approved, and it reserves six gates for a person and never ticks them automatically. It links the legacy source read-only, puts new code in a separate directory, and lists any instructions that it finds inside the analysed code without following them.
7.3 Parallel agents and isolation
Run several agents on the same task to compare the outcomes, or on different tasks that do not overlap. Give each agent its own Git worktree so that the agents' changes cannot collide. A person who supervises a fleet of agents needs four things from the tooling: the state of each agent at a glance (working, finished or waiting for input), a notification when an agent needs attention, worktree isolation, and scoped control, so that an agent can act only inside its own workspace. Agent tooling leaves residue behind: stale sessions, abandoned worktrees, background processes and old skills. Schedule a cleanup that reports first and deletes only after review.
7.4 Audit prompts that find real defects
Four audit prompts recur because each prompt targets a structural source of bugs.
- State machines. Find every state machine in the application, whether explicit (enumerations, status columns) or implicit (combinations of flags and nullable timestamps). Document the states, the invariants and a transition table in one file. Identify the invalid states that the code can actually reach, and give a concrete path as evidence for each, including paths through retries, double submissions and replayed events. Rank the invalid states by impact, fix each with a regression test, and keep the document current. Because the prompt requires a reachable path, speculative findings stay out of the report.
- Compensating code. Find the workarounds, retries, defensive checks and adapter layers that exist only to make up for a flaw elsewhere, then propose redesigns under which that code would never have been written.
- Whole-project audit on each new model. Read the entire project, find what is broken, inconsistent, inefficient or unused, fix what you are sure of, and list the rest for review. Because the prompt splits the findings by confidence, the person keeps the judgment calls. You can also use the same prompt to compare models on your own code.
- Reference audit. Read a named best-practice reference, then report, in ranked order, the most valuable improvements available in this codebase now.
7.5 Code review and ownership
Practitioners currently dispute whether the pull request is obsolete when agents write the code. One side holds that it is obsolete, for three reasons: a human reviewer rarely catches what several models catch when they check the change repeatedly, agents can run the full suite before they push, and continuous integration originally meant integrating to the trunk continuously. The other side holds that unreviewed changes accumulate into systems that nobody understands.
Both sides accept two principles, and this course adopts them: code is judged on its merits regardless of how it was produced, and a named person is accountable for every change. Route human attention by risk. Run automated review and compute a risk score on every change, and require human review where the score or the blast radius is high. Instead of reading diffs, a person can supervise by structure. One approach is to draw the system as a navigable diagram and to colour each component by two metrics: a change-risk score, which combines cyclomatic complexity with test coverage, and a mutation score. The diagram also marks the dependencies that point from an inner layer to an outer layer. The person spots the unhealthy structure and directs the agent to it.
7.6 Constraints upstream and downstream of the build
When a team has mastered fast building, the bottleneck moves to both sides of the build step. Upstream, the constraints are product capacity, a shortage of good ideas and rigid process. Downstream, the constraints are review capacity, verification and delivery of the result to customers. A downstream obstacle propagates upstream, because people who cannot ship their ideas stop having ideas. Remedies reported to work include engineers who own more of the requirements, automated review with risk routing, preview environments in which tool-equipped agents verify the result, and a leader who is willing to reorganise around the new constraint.
Structured multi-stage pipelines, called software factories, raise the floor for a large organisation but do not necessarily raise the ceiling for its best engineers. People who have built several factories give the following advice. Fix the fundamentals first: repository readiness, fast checks and previews. Begin with a narrow factory for bugs, security or interface polish, because a narrow factory is easier to evaluate. Measure every component. Do not rebuild a general coding agent. Manufacturing solved related coordination problems decades ago. The concepts to borrow from manufacturing are a limit on work in progress, stopping the line on a defect, and standardised work.
An internal tool that is built quickly imposes a cost on whoever must check it. When users do not trust a dashboard that duplicates the system of record, they verify it against the system of record by hand, and the work is done twice. Capture the domain constraints before you build, treat a demonstration as a demonstration, and give every tool an owner.
Laboratory 3
Take one feature through 7.1 and 7.2. Submit the one-page brief, the three approved artifacts, the diff and a review record. Then run prompts 1 and 2 from 7.4 on the same repository. Submit the state-machine document, which must contain at least one reachable invalid state and the fix for it.
Practical work
The practical work for this module is in Part IV: Assignment 3 · Bricks.
Module 8 · Verification through testing and evaluation
We make this module's lesson video when 25 people sign up for it.
If you don't want to wait, you can buy the video now. The price is what it costs us to make it, plus 20%. We start on it the day you buy it, we send it to you the day it is done, and then we post it on this page for everyone.
What the $18.87 pays for
| Narration, 38,187 characters | $11.46 |
| 6 generated clips of 5 seconds | $3.36 |
| 6 keyframe stills for those clips | $0.90 |
| Rendering on our own Macs | $0.00 |
| Cost to make | $15.72 |
| Plus 20% | $3.15 |
| Price | $18.87 |
|---|
The agent runs on a flat subscription and rendering runs on our own Macs, so the price includes no charge for the agent or for rendering.
Objectives
By the end of this module you should be able to decide whether a test or an evaluation measures behaviour, build graders that are themselves checked, and read reported results critically.
8.1 Verification as the scarce skill
Generation is cheap, and judgment is the bottleneck. Verifying code can be harder than writing it, and the people most in demand on agent teams are the people who can write evaluations. Every claim in this course that something works depends on a method of verifying it.
8.2 Criteria for keeping a test
A model writes a test for every small change. If nobody prunes the suite, it fills with tests that restate the implementation, always pass, and break on every refactor. Projects report deleting tens or hundreds of thousands of lines of such tests with little change in coverage.
Definition (change-detector test). A change-detector test is a test that mirrors the implementation, usually by mocking collaborators and asserting that specific calls were made. It fails when the implementation changes although the behaviour has not changed, and it passes when a collaborator's real behaviour breaks. The term dates from 2015.
Before you add a test, put it through a gate of four questions. If you cannot answer one of the questions, do not add the test yet.
- What observable behaviour, invariant or independent contract does the test protect?
- What credible regression would make the test fail?
- Why does the existing coverage not already catch that failure? Each contract has one primary test, placed at the strongest boundary.
- Does the test need a seam in production code (an export, a flag or an injection hook) that no production caller needs? If it does, move the test to the real boundary.
Show that a regression test fails on the code before the fix, for the intended reason, and passes on the code after the fix. A regression test that has never failed proves only that the mock works.
Reject the following patterns on sight: probes with no assertions; self-comparisons; copied fixtures and export lists; exact greps of source text; tests of private predicates that a boundary test already covers; expected values generated by the code under test; mocks that implement the behaviour being asserted; tests whose only effect is to keep a test-only export alive; and negative controls that pass for an unrelated reason.
Keep two kinds of test even when they look like tests of the implementation: tests that independently enforce a public API, protocol, configuration, migration, storage, security or release contract, and call-order tests where the order of calls is observable behaviour. Do not delete a test because it is slow. When you remove a test, record the evidence: the test's name, what the test can detect, the stronger proof that remains, and the validation command that you ran afterwards.
Whether to rely on end-to-end tests alone is contested. A narrower rule can be defended: prefer tests at the boundary that a user or a caller sees, and have each end-to-end test finish by producing a repeatable artifact that can be inspected.
8.3 Tests that use a model
Agent-driven end-to-end frameworks mix four kinds of step in one test: deterministic navigation, a goal stated in plain language that the agent works out how to achieve, a model-judged assertion, and an exact locator with a deterministic assertion. The principle is to use the model where a script would be brittle and to use code where exactness matters. The frameworks control cost with action caching: they record the concrete actions that the agent took and replay them without a model call while the actions still match the application. In automated runs the cache is read-only. Tests refer to credentials by name, and the framework fills them in from the environment, so the credentials never appear in test code or in a transcript.
A scheduled visual review agent is a cheap addition. The agent visits the application, reports at most five findings, and attaches a screenshot to each finding. The cap forces the agent to prioritise, and the screenshot is the evidence. To review motion, film it, because a single screenshot cannot show a one-frame jump or text that overflows during a transition.
8.4 Calibrating a grader
An evaluation is a program and has bugs. A web-platform guidance project published the cleanest calibration loop of the season:
- Experts write the guidance and a demonstration.
- The project builds tasks on real applications. Each task has a golden solution and an automated grader.
- The grader must pass every golden solution and fail the unmodified application every time. If it does not, the grader is wrong.
- The prompts imitate real requests and never name the technique, so the test measures whether the agent finds the technique.
- Each run compares an agent with the guidance against the same agent without the guidance.
Opportunity is one hundred per cent minus the unguided pass rate. Uplift is the guided pass rate minus the unguided pass rate. Where a model already passes without the guidance, the maintainers delete the guidance to save tokens. The maintainers report an uplift of 27 to 36 percentage points across three agents on about 210 tasks. These figures are the maintainers' own.
The legacy-modernisation toolkit applies the same idea to equivalence proofs. The toolkit re-derives its checks on each run: a clean rebuild, in which zero executed tests counts as a failure; a run of the old and new systems on identical inputs, with the output compared byte for byte and every masked field listed; at least ten unseen inputs that cover boundaries and malformed records; a deliberate one-line break that must turn the tests red, and a one-byte change that must trip the comparator; and a rule that every critical requirement is named by a test that actually ran. The verdicts are proven, partly proven and not proven. When the old system cannot be executed, partly proven is the highest verdict available.
8.5 Reading results
- Distrust a single metric. A public library of fifty trading strategies includes one strategy with a hundred per cent win rate, an infinite profit factor and a risk-adjusted return near zero. Always report several measures and the worst case.
- Validate out of sample. In walk-forward testing you train on one window and test on the next window, with a gap between the windows so that information cannot leak. The library labels even its walk-forward results as diagnostics and does not present them as independent evidence.
- Check the claim against its source. The same library was promoted as a set of leaked proprietary strategies. It is an openly licensed teaching set, and most of its strategies perform poorly.
- Fix the protocol. Good model cards state the sampling settings, give the average of three runs with the ranges, say that the network was disabled for code execution, and list what was not measured. They also warn that small differences prove nothing.
- Repeat. One good output does not establish reliability. A security audit harness reports that a single run found roughly half of what repeated runs found in total.
- Weigh who is reporting. Treat a vendor benchmark that was run on the vendor's own tasks as an advertisement until it is reproduced.
8.6 Checking a claim before acting on it
Analysis of Competing Hypotheses is a method developed for intelligence analysts in the 1970s, and it transfers directly to checking a claim. List at least three explanations, one of which is that the source is wrong. For each piece of evidence, record which explanations it supports and which explanations it contradicts. Evidence consistent with every explanation has no diagnostic value. Prefer the explanation with the least evidence against it, name one check that you can perform yourself, and decide whether to act, wait or verify. A model that judges its own claim has the same blind spots that it had when it made the claim, so the external check is the step that adds information.
8.7 Measuring prose
Model-written prose has recognisable faults, and you can count them. A model-written sentence opens with contrast, negation or hedging instead of the main clause. It relies on abstract nouns and metaphorical verbs, so the reader cannot tell who does what. Short parallel sentences and nominalisation make the prose sound weightier than its content. One rule repairs these faults: name the subject, the object and the action. The effective review question is "what does this sentence mean?", followed by "then write that". Apply the same discipline to the reports that an agent writes for you, because those reports are the main channel through which you learn what the agent did.
Problem set
- Apply the four-question gate to twenty tests from a project that you know. Report how many tests pass the gate. Delete three tests and record the evidence that removal requires.
- Build a grader for one task and run the calibration in 8.4. Report the grader's pass rate on the golden solutions and on the unmodified base.
- Take a published benchmark claim from the last month. Identify the protocol, who ran the benchmark, what was not measured, and one explanation of the result other than the explanation offered with the claim.
Practical work
The practical work for this module is in Part IV: Assignment 3 · Bricks, Assignment 4 · Sand and fire with byte decoding and the Section problems.
Module 9 · Laboratory: making games
We make this module's lesson video when 25 people sign up for it.
If you don't want to wait, you can buy the video now. The price is what it costs us to make it, plus 20%. We start on it the day you buy it, we send it to you the day it is done, and then we post it on this page for everyone.
What the $17.34 pays for
| Narration, 33,942 characters | $10.19 |
| 6 generated clips of 5 seconds | $3.36 |
| 6 keyframe stills for those clips | $0.90 |
| Rendering on our own Macs | $0.00 |
| Cost to make | $14.45 |
| Plus 20% | $2.89 |
| Price | $17.34 |
|---|
The agent runs on a flat subscription and rendering runs on our own Macs, so the price includes no charge for the agent or for rendering.
Objectives
By the end of this module you should be able to build a small complete game with an agent. In building the game you will practise explicit state machines, deterministic simulation, generated content and the verification of interactive software.
9.1 Games as a laboratory
A game is the most compact program that exercises everything in this course. It has a real-time loop, state that must never become invalid, content pipelines, input handling, performance limits measured in milliseconds, and a result that a person can judge in seconds. A defect in a game is visible immediately, so a game is a good place to learn how far you can trust an agent's unverified claim of "done".
9.2 The architecture to ask for
State the architecture in the brief. An agent left to choose the architecture will produce a single file that mixes everything.
- Fixed-timestep simulation. Advance the game state in constant steps (for example 60 per second). Render as often as the display allows, and interpolate between states. A simulation that advances by the measured frame time behaves differently on every machine and cannot be replayed.
- Determinism. Draw all randomness from one seeded generator. The simulation reads its inputs and never reads the clock.
- Explicit state machines. Give each screen (title, playing, paused, game over) and each entity (idle, moving, hit, dead) an enumerated state and a transition table. Game bugs arise in implicit states that are built from combinations of flags. The state-machine audit of Module 7 applies directly.
- Separation of simulation and presentation. The simulation is a pure function from state and input to the next state. The rendering code and the sound code read the state and never write it.
- Data-driven content. Keep levels, enemies and tuning values in data files that the agent can edit without touching code.
With these five points in place, verification is cheap. Record the seed and the input stream of a session, and you can replay the session exactly. A replay that ends in a different state after a code change is a regression test for which you wrote no assertion by hand.
9.3 A model inside the loop
A language model is too slow for a frame budget of sixteen milliseconds. A small decision model is fast enough. One open implementation of an encoder-based classifier answers a short question in 7 to 14 milliseconds on a laptop. In a demonstration, the classifier plays Snake by asking three questions per move. Over 2,400 moves it reaches about 75 moves a second and does not die. A visible safety layer, written in code, can override any unsafe move. The same pattern applies outside games: put a fast classifier in the hot path and a deterministic guard after it, and keep a reasoning model for planning between rounds.
9.4 Content from data
Agents are good at assembling game content from open data. One project generated a coast-to-coast driving route of 5,285 kilometres for a clicker game. The route came from open map geometry sampled every ten metres, which is about half a million points. A second project built a teaching site for board assembly, with thirteen real electronics boards, part-by-part assembly, and deliberate failure modes such as a reversed capacitor. The two projects used the same method: find an open dataset, have the agent write a repeatable build that converts the dataset, and commit the build with its output so that you can regenerate the content.
Packaged skills now produce interactive illustrations as single self-contained HTML files. In the workflow of such a skill, the agent models the object from all sides, builds the internal view from the way the real machine works, and checks its own output from several angles and at phone width before it finishes. Budget for the cost. The author quotes fifteen to thirty minutes and 200,000 to 400,000 tokens per figure.
9.5 Performance measured against the hardware
Benchmark against what the hardware can do, and never against a slower competitor. If you are measuring startup in seconds, you have chosen the wrong reference point. Arrange the program so that locally cached media is on screen within a few display refreshes of launch. At 60 Hz each refresh takes about 17 milliseconds. Arrange the server so that one query returns everything the first screen needs, and send that query in parallel with graphics initialisation. The target is a game that responds before the player's hand can reach the next control.
Outside games, a speech-recognition decoder spent most of its time on per-item bookkeeping in Python. Each step was rewritten as a fixed handful of tensor operations, shared values were computed once per utterance, and the search was stopped exactly when no candidate could still win. These changes cut the time to decode a twelve-second clip from 8.4 seconds to 2.1 and peak memory from 37 GB to 2.6 GB, with output identical to the reference. Accept an optimisation only when its output is identical to the reference.
9.6 Software the player can change
Some personal applications now include a text box wired to an agent. The agent implements whatever the user types, immediately and without review. One user of such an application submitted about 150 changes in a few days. In a single-player game or a personal tool, the text box lets the user mod the software by conversation. The proponents of this design state its costs themselves: nobody knows how maintainable such software is over time, the design does not obviously extend to software shared by people who want different things, and an input field that rewrites the program is remote code execution by design. Such an application needs a sandbox, version control behind every change, and rollback.
Laboratory 4
Build a complete small game (an arcade, puzzle or text game) that runs in a browser.
- Specify the five architectural points of 9.2 in the brief.
- Submit the state-machine document that the audit prompt produces, with its transition table.
- Submit a recorded replay (the seed and the inputs) and show that the replay still reproduces the session after a refactor.
- Have a second agent play-test the game with worst-case inputs: held keys, rapid pause and resume, a window resize, a hidden tab and a 320-pixel-wide screen.
- Report the frame time at the 50th and 99th percentiles and the time from load to the first interactive frame.
- List what you did not test.
Do not reproduce the name, characters, art or levels of an existing commercial game. You may build in an existing genre, but the design must be your own.
Practical work
The practical work for this module is in Part IV: Assignment 3 · Bricks and Assignment 5 · The infinite story.
Module 10 · Laboratory: interfaces for people and for agents
We make this module's lesson video when 25 people sign up for it.
If you don't want to wait, you can buy the video now. The price is what it costs us to make it, plus 20%. We start on it the day you buy it, we send it to you the day it is done, and then we post it on this page for everyone.
What the $17.34 pays for
| Narration, 33,942 characters | $10.19 |
| 6 generated clips of 5 seconds | $3.36 |
| 6 keyframe stills for those clips | $0.90 |
| Rendering on our own Macs | $0.00 |
| Cost to make | $14.45 |
| Plus 20% | $2.89 |
| Price | $17.34 |
|---|
The agent runs on a flat subscription and rendering runs on our own Macs, so the price includes no charge for the agent or for rendering.
Objectives
By the end of this module you should be able to produce interface code that uses the current web platform, looks designed, and can be read by software agents as well as by people.
10.1 Dated front ends in agent output
Older code dominates the training data, so a model defaults to the patterns that were common five years ago. It writes scripts where the platform now has a native feature, uses pixel values and physical directions, and animates everything. Even when a model knows a modern API, it has seen few dense examples of that API. The remedy is to add context: give the agent a reference that it retrieves before it writes. Two public packs take this approach. One pack is a set of 47 CSS practices that installs as a skill. The other pack gives platform-wide guidance, with a tiny always-loaded description and local search over some 200 guides. Every entry in both packs has the same shape: when to use the practice, the code, why the code works, the rules and failure modes, and browser support with version numbers. That shape is a good template for any technical reference meant for a model.
10.2 Foundations that prevent most layout defects
*, *::before, *::after { box-sizing: border-box; min-width: 0; }
:root {
color-scheme: light dark;
scrollbar-gutter: stable;
text-wrap: pretty;
overflow-wrap: break-word;
-webkit-text-size-adjust: 100%;
}
body { margin: 0; min-height: 100svh; }
img, svg, video { display: block; max-width: 100%; height: auto; }
input, select, textarea { font-size: max(16px, 1rem); }
min-width: 0 lets flex and grid children shrink below the width of their content. The default value, auto, is the commonest cause of horizontal overflow. 100svh keeps content from being hidden behind a phone's toolbars. A stable scrollbar gutter stops the page from shifting sideways. An input with a font size of 16 pixels or more stops iOS from zooming on focus.
The following rules are of the same kind.
- Logical properties. Write
padding-inlineandmargin-block-end, so that right-to-left layouts mirror without extra code. - Colour in OKLCH. Define a base colour, and derive hover states and tints from it with
color-mix(in oklch, …). Lightness in OKLCH is perceptual, so scales and contrast are predictable. Write greys with huenone, because a hue of zero is red and pulls every mix towards red. - One token per colour. Define one token per colour for both themes with
light-dark(a, b). Switch themes by changingcolor-scheme, and never by redefining tokens. - Fluid sizes. Set fluid sizes with
clamp(min, rem + vw, max). Always include aremterm in the preferred value. For type, keep the maximum within 2.5 times the minimum so that text can still double in size under zoom. - Intrinsic grids. A grid defined with
repeat(auto-fit, minmax(min(100%, 16rem), 1fr))needs no media query. - Container queries. A container query sizes a component by the slot it sits in instead of by the viewport.
overflow: clip. Useoverflow: clipin place ofhidden, becausehiddencreates a scroll container and breaks sticky positioning.- Spacing. Set spacing on the parent element. Use
gap, or for uncontrolled prose use.prose > * + * { margin-block-start: 1em }. Do not add spacer elements.
10.3 Interaction and motion
- Never remove the focus outline. Use
:focus-visiblewith a visible ring. - Apply hover styles only inside
@media (hover: hover) and (pointer: fine), and give touch feedback with:active. - Extend small targets to a 44-pixel hit area with a pseudo-element.
- Write the state change first, and then add transitions only inside
@media (prefers-reduced-motion: no-preference). - Use an ease-out curve for elements that enter and leave. Never use
ease-infor interface motion, and never usetransition: all. Use these durations: 100–160 ms for a press, 125–200 ms for small popovers, 150–250 ms for menus, and 200–500 ms for dialogs and drawers. - Decide first whether an animation is warranted at all. An animation gives feedback and helps the user understand a state change. Do not animate keyboard-triggered actions or repeated tooltips.
- Prefer native behaviour to script:
<dialog>,popover,<details>, scroll snapping for carousels,:has()for parent state and:user-invalidfor form feedback.
10.4 Giving an agent taste
Generic output is a problem of context, and four remedies are in use.
- Reference components as source. A component registry delivers code that the agent copies into the repository and adapts to your tokens, so you own the code. Several registries also expose a remote tool server and a Markdown page for each component.
- Reference screens. Have the model compare its design against real examples retrieved from a library of shipped application screens, critique itself against the examples, and then delete the examples.
- Design skills. A design suite provides one skill for each narrow job: to decide on an animation, to review existing motion, to make a page feel native on a phone, to choose a component library from a trusted list, and to stress the interface with worst-case data.
- Named failure modes. One skill for marketing interfaces names five failure modes: too much, wireframe, boxed, flat and bland. The skill sets a word budget of three to seven real strings per screen, requires design at the displayed size, and does not hand over the work until a script has filmed the running result at two widths and a gate has passed.
10.5 Pages that agents can read
A growing share of the readers of a page are programs. To make a page readable by programs, follow the accessibility practices and add a few more.
- Fix the accessibility tree: name every icon button, label every form field, describe every image, and never put essential information only in an image.
- Put the facts (price, specification, proof) in the HTML that the server delivers, because many crawlers do not run scripts.
- Use semantic elements (
nav,header,button) for the functional parts of the page. - State in plain sentences what the thing is, who it is for and what it does, and put a date on facts that change.
- Publish an
llms.txtindex and Markdown versions of the documentation. robots.txtand the sitemap are public files. Listing a path as disallowed advertises the path and does not protect it. Use authentication ornoindexfor pages that must not be found.- Add
og:videoalongsideog:imagewhere a link preview should play.
A page can also offer its functions to an in-browser agent as tools, declared through form attributes or script. Treat those tools like any public API.
Laboratory 5
Take an existing page of your own. (a) Run a reference audit of the page against a modern CSS reference and apply the five most valuable changes. (b) Make the page pass at widths of 320, 768 and 1440 pixels, with the keyboard only, and with reduced motion. (c) Load the page with scripts disabled and through a text-only fetch, and record which facts are missing. (d) Submit before-and-after screenshots at each width and a list of what you did not check.
Practical work
The practical work for this module is in Part IV: Assignment 6 · A search engine served securely.
Module 11 · Laboratory: reverse engineering
We make this module's lesson video when 25 people sign up for it.
If you don't want to wait, you can buy the video now. The price is what it costs us to make it, plus 20%. We start on it the day you buy it, we send it to you the day it is done, and then we post it on this page for everyone.
What the $18.60 pays for
| Narration, 37,436 characters | $11.24 |
| 6 generated clips of 5 seconds | $3.36 |
| 6 keyframe stills for those clips | $0.90 |
| Rendering on our own Macs | $0.00 |
| Cost to make | $15.50 |
| Plus 20% | $3.10 |
| Price | $18.60 |
|---|
The agent runs on a flat subscription and rendering runs on our own Macs, so the price includes no charge for the agent or for rendering.
Objectives
By the end of this module you should be able to recover the behaviour of software or a device from the artifact alone, state each finding with its evidence and its limits, and prove a reconstruction against the original.
11.1 Scope and law
Reverse engineering is the method behind interoperability work, security research, preservation and repair. It is also how software is cloned. Work only on what you own or are authorised to test, and respect licence terms and local law. Agents can now analyse and rebuild commercial software cheaply, which is one reason that value is moving from code to data and distribution.
11.2 The method
- State the question. A question such as "How does feature X compute its result" can be answered. A request such as "Understand this program" cannot.
- Inventory without executing. List the file type, sections, strings, imports, embedded resources and signatures.
- Static analysis. Disassemble and decompile the target. Follow cross-references from a known string or API call to the code of interest.
- Dynamic analysis. Run the target under observation in an isolated environment. Observe its system calls, the files it touches and its network traffic, and use a debugger at the function that you found statically.
- Hypothesise and test. Write down what you believe the code does and an experiment that would show that you are wrong.
- Reconstruct. Reimplement the behaviour.
- Prove. Compare the reconstruction with the original on many inputs.
Record every finding with its evidence and its limitation: what you observed, with what tool, and what you could not determine.
11.3 The toolchain
Ghidra is the open-source framework that the NSA's research directorate maintains. It provides disassembly, decompilation, graphing and scripting for many processors and formats. You can use it interactively or headless and extend it in Java and Python. Ghidra's own documentation warns that some versions carry known vulnerabilities. A tool that parses hostile input is part of your attack surface, so analyse untrusted samples in a virtual machine.
Agents now drive engines such as Ghidra through a tool server. One open package gives a coding agent a single interface to many target types and returns each result with its evidence.
| Target | What is recovered | Engine |
|---|---|---|
| Native binaries | Pseudocode, assembly, strings, symbols, cross-references | Ghidra, Hopper or IDA |
| JavaScript and Electron apps | Modules, imports, source maps, routes, inter-process messages | none needed |
| Android packages | Manifest, classes, decompiled methods | JADX, Apktool |
| .NET assemblies | Metadata, intermediate language | static only |
| WebAssembly | Sections, imports and exports, text form | WABT |
| Firmware images | Regions and extracted file systems | Binwalk, Unblob |
| Network captures | Requests, responses, payloads | HAR files, mitmproxy |
| Running processes | Output, exit status, file system effects | a pseudo-terminal |
Static inspection never runs the target. Runtime capture runs the target with your permissions, which is why you use a sandbox.
11.4 The standard of proof
In one published case study, a sound-panning calculation was recovered from an old game and reimplemented. The reimplementation passed 3,205 test cases taken from the original x86 code, and it then compiled to the same 63 bytes as the original function. In a second case study, a calculation from a 16-bit DOS game was reconstructed in C++ and compared with the output of the historical compiler. Two tests distinguish a correct reconstruction from one that is only plausible: differential testing against the original, and byte-identical output where a compiler is available.
11.5 Hardware and radio
Much of the apparent magic in devices comes from one simple signal. In one demonstration, animated shapes jump between two small screens as the two boards approach each other. The demonstration uses no proximity sensor. Each board is a microcontroller with a display. The two boards exchange short packets over a connectionless radio protocol, and each board reads the received signal strength of the packets from the other board. The software maps rising signal strength to animation progress and triggers the jump at a threshold. Signal strength is noisy and depends on orientation and obstacles, so it gives relative closeness but not distance. For a device of this kind, list every sensor and radio it has, and then find the cheapest signal that explains the behaviour.
The same approach works for consumer appliances. To recover the control protocol of an air conditioner or of a thirty-year-old vehicle bus, capture the traffic while you press each control, and then write a thin client. An agent shortens the client-writing step to minutes. The capture and the hypothesis remain your work.
11.6 Black-box audits of what devices send
A peer-reviewed audit of smart televisions shows how to study a device that you cannot open. Put the device behind a network that you control, capture its traffic, and vary one condition at a time: logged in or logged out, opted in to tracking or opted out, different input sources, and different countries. The audit found that content recognition continued while the set was used as a plain display for another device over HDMI. It also found that the privacy opt-out did stop the traffic and that behaviour differed by country. The public reporting on the audit adds capture intervals of 10 to 500 milliseconds, depending on the maker. Claims from individual rooted devices, such as the claim that a set borrows the microphone of a connected speaker, are unverified. Treat those claims as hypotheses and test them with the same method.
11.7 Replacing a closed toolchain
One open project builds, signs, installs and ships iPhone applications from Linux, with no Mac. The project shows how to take a vendor toolchain apart, and its lessons apply generally.
- The irreducible step. One step may have no substitute. The platform SDK exists only inside the vendor's distribution, and you must obtain it once.
- Version matching. Treat version matching as a hard constraint. The SDK must come from the release whose compiler matches the installed toolchain.
- Environment failures. Most failures come from the environment. A system compiler that is found ahead of the toolchain's own compiler produces baffling errors. Swapping the toolchain changes absolute paths, and a registered SDK still points to the old paths.
- Offline validation. Validate the build offline before you upload it. The release script checks the bundle layout, property keys, architecture, icons, entitlements and every sealed hash, and it stops on any failure.
- Reimplementation and comparison. Reimplement a vendor tool and compare its output with the original's. The project judges its stand-ins for the vendor's asset and interface compilers by whether they produce byte-identical output on real applications.
- The findings log. Keep a findings log. In this project a numbered file records what broke and how it was fixed, with dates and the exact versions on which each claim was verified. The file also says which flows have not been re-run on the newest versions.
Laboratory 6
Choose one target: (a) an application you are licensed to use that is built on a web or Electron stack, (b) a device you own, or (c) a binary from a public set of reverse-engineering exercises. Answer one stated question about the behaviour of the target. Submit the question, the findings log with the evidence and limits for each entry, a reconstruction, and a differential test of the reconstruction against the original on at least 100 inputs. State what the agent did and what you verified yourself.
Practical work
The practical work for this module is in Part IV: Assignment 4 · Sand and fire with byte decoding.
Module 12 · Network protection
We make this module's lesson video when 25 people sign up for it.
If you don't want to wait, you can buy the video now. The price is what it costs us to make it, plus 20%. We start on it the day you buy it, we send it to you the day it is done, and then we post it on this page for everyone.
What the $18.57 pays for
| Narration, 37,359 characters | $11.21 |
| 6 generated clips of 5 seconds | $3.36 |
| 6 keyframe stills for those clips | $0.90 |
| Rendering on our own Macs | $0.00 |
| Cost to make | $15.47 |
| Plus 20% | $3.10 |
| Price | $18.57 |
|---|
The agent runs on a flat subscription and rendering runs on our own Macs, so the price includes no charge for the agent or for rendering.
Objectives
By the end of this module you should be able to design a network, at home or for a small service, in which a compromised device or account can reach as little as possible, and to verify the design by observation.
12.1 Threat model
Consumer devices are computers with sensors. They are rarely patched, and they report to their makers. Anything that listens on a public address is scanned within minutes. Some search engines index exposed cameras, storage boxes and printers, many of which have default passwords. A thief who steals or phishes a credential can do whatever that identity is allowed to do. To protect a network you therefore limit what each device and account can reach: by network segment, by name resolution, by exposed ports and by identity.
Makers have an economic reason to collect what their devices report. Television prices fell about 97 per cent between 2000 and 2022, and one maker reported a quarterly loss of $6.7 million on devices against a profit of $115.8 million from advertising and data. The device is where the maker collects the data. According to one unverified report, a set without its own microphone will look for a microphone on a connected peripheral. A device's sensors include the sensors of everything attached to it.
12.2 Segmentation
Put televisions, cameras, speakers and appliances on a separate network (a router's guest or IoT network, or a VLAN) that cannot initiate connections to laptops, phones and storage. The purpose is containment: a compromised device sees only the other devices on its own network. Verify the separation by scanning the trusted network from a device on the untrusted network. Until you have run that scan, you do not know that the rule works.
12.3 Filtering at name resolution
Most device telemetry begins with a DNS lookup. A filtering resolver that serves the whole network therefore blocks the telemetry of devices on which you can install nothing.
- Choose a hosted filtering resolver or a self-hosted one.
- Enable a maintained blocklist and the vendor-specific tracking lists for the makers of the devices you own.
- Configure the router to use the resolver over an encrypted transport (DNS over HTTPS or DNS over TLS) where it is supported.
- Block outbound port 53 to every destination except the router, because some devices hard-code their own resolver to bypass filtering.
- Read the query log. The log shows how often each device calls home. When an update or an app breaks, the log shows the blocked name that you need to allow.
Turn off the collection on the device itself as well. Every brand of television gives the setting a different name (viewing information services, live plus, viewing data, use info from TV inputs). The published audit confirms that opting out stops the traffic. Also opt out of interest-based advertising, reset the advertising identifier, disable voice recognition and remove unused applications.
12.4 Traffic monitoring
On a workstation, a per-application outbound monitor shows which programs connect out, how often they connect and how much traffic they send. Unexpected connections are a reason to uninstall the program. On a server, log every inbound connection attempt and alert on any attempt from outside the expected set. You can audit any device by placing it behind a network you control and capturing its traffic, as in Module 11.
12.5 A server with no open ports
For a server that you manage yourself, the following pattern exposes the smallest surface:
- Administrative access goes only over a private mesh network, and the SSH daemon listens only on the mesh address.
- Public traffic arrives through an outbound tunnel from the server to a reverse proxy at a content network, so the server accepts no inbound connections.
- The firewall closes every inbound port on the public address.
Nothing then answers on the public address, so the server is no longer exposed to scanning or brute force. You must still patch the server, secure the application and keep backups that you have restored at least once.
12.6 Segmentation by identity
Network location is a weak basis for trust. Authenticate every request, and then authorise each identity only for the specific pages, functions and endpoints that it needs. An intruder who holds a valid account can then reach only what that one account is authorised for. When abuse is detected, suspend that account automatically. The damage is then limited to one identity, and the intruder is outside again.
An organisation usually exposes a function to another organisation by allowlisting IP addresses. An IP allowlist is brittle, because it breaks when people work remotely and when users share an egress address. The durable alternative is identity-based access, which uses short-lived credentials bound to a device or a key.
12.7 Residual channels
Two isolated processes may still have a path between them after you remove their shared files and network links. With a timing side channel, one process signals another by varying how long a shared resource, such as a cache, takes to respond. One second-hand report this season describes agents that exchanged information through such a channel. The report is unverified, but the mechanism is well established. When you claim that the agents in a multi-agent system are isolated, state which channels you considered.
Laboratory 7
Work on a network you administer. (a) Inventory every device, with its maker and its purpose. (b) Move the untrusted devices to a separate segment, and prove from inside that segment that the trusted segment is unreachable. (c) Deploy a filtering resolver and block outbound port 53. After 48 hours, report the ten most frequently blocked names and the device that asked for each. (d) Scan your public address from outside and account for every open port. Submit your results from before and after the changes.
Practical work
The practical work for this module is in Part IV: Assignment 6 · A search engine served securely.
Module 13 · Cyber security for builders and their agents
We make this module's lesson video when 25 people sign up for it.
If you don't want to wait, you can buy the video now. The price is what it costs us to make it, plus 20%. We start on it the day you buy it, we send it to you the day it is done, and then we post it on this page for everyone.
What the $18.76 pays for
| Narration, 37,888 characters | $11.37 |
| 6 generated clips of 5 seconds | $3.36 |
| 6 keyframe stills for those clips | $0.90 |
| Rendering on our own Macs | $0.00 |
| Cost to make | $15.63 |
| Plus 20% | $3.13 |
| Price | $18.76 |
|---|
The agent runs on a flat subscription and rendering runs on our own Macs, so the price includes no charge for the agent or for rendering.
Objectives
By the end of this module you should be able to audit an application and an agent deployment for security defects with a method that separates finding from confirming, and to harden the accounts, secrets and dependencies that the work rests on.
13.1 The baseline
Twenty controls cover the defects that quickly built applications most often ship with.
| Area | Control |
|---|---|
| Secrets | Keep keys in the environment or in a secret store, never in source. Rotate any key that was ever committed, because deleting the key from the current tree leaves it in the history. |
| Access | Validate all input on the server. Check authorisation on every route. Use row-level security in the database. |
| Authentication | Hash passwords with a slow algorithm. Rate-limit sign-in and sign-up. Set cookies to be secure and HTTP-only. Log failed sign-ins. |
| Web | Use HTTPS everywhere. Restrict cross-origin requests to your own origins. Use parameterised queries. Escape output. Add CSRF protection. Restrict uploads by type and size. |
| Operations | Show no stack traces in production. Keep dependencies updated. Back up the database daily. Set a spending cap on every metered API key. |
Check the daily backups and the spending caps yourself. An agent's statement that the backups run and that a cap is set is a claim until you have checked it.
13.2 Client-side evidence of the server interface
In October 2026 a national research institute published an advisory that named a pattern behind a run of data leaks. The attacker obtains a published mobile application, analyses it to learn which servers it talks to and how the interface works, and then attacks the server with requests built for that interface. The application itself need not contain a flaw. An operator may think of the backend as private, but the client that ships to every user describes the backend in full, and agents have made the analysis cheap.
The attacks target these defect classes, in order of frequency: broken object-level authorisation (a request for another user's record succeeds because the server checks only the identifier), server-side request forgery, remote code execution, SQL injection and misconfiguration. The recommended response is to inventory every public interface, review the access logs for bulk retrieval, test authorisation first, shut down the interfaces that are not needed, and reissue credentials wherever exposure is possible.
13.3 A method for auditing with agents
A large infrastructure company has published an open audit skill, which is the single-repository form of its internal vulnerability-discovery system. The skill is the best published structure for agent-run security review. It has six phases:
- Reconnaissance. Agents map the architecture, the trust boundaries and the input surfaces. They write an architecture document and a coverage ledger that lists the units to examine.
- Coverage-led hunting. Each isolated hunter agent is assigned units from the ledger. Separate critic agents look for gaps in the coverage.
- Validation. Every candidate finding goes to a fresh agent, whose job is to disprove the finding.
- Structured output. The findings are written to a file, and the file is checked against a schema.
- Independent verification. New agents verify the final claims against the source.
- Reporting. The reports are derived from the verified records and the ledger.
The agent that checks a finding is never the agent that found it. Deterministic validators, which are small scripts with their own tests, run after every change to the ledger and the findings. Each finding receives one of three verdicts: confirmed (the finding has a complete trace through the source and a bounded observed result), needs validation (the record names the exact unresolved fact and assigns no severity) or rejected (the finding was disproved and is kept on record). A finding receives a severity only when it has an impact. When the first layer of defence holds, a missing second layer is recorded as a hardening note. Runs are additive: each run uses the earlier ledger to target the gaps. The authors report that a single run found about half of what repeated runs found. Target code is executed only inside an operating-system sandbox with no outside network. Without a sandbox, the lead stays at needs validation.
The skill's attack-class references cover memory safety; prompt injection and tool misuse; request framing, caching and authentication protocols; client-side injection and messaging trust; dependencies, build systems, signing and updates; cloud identity, infrastructure code and serverless; resource exhaustion and operator spend; tenancy, export, backup, deletion and restore; and local inter-process communication.
13.4 Securing the agent itself
An agent that has system access gives an attacker a new entry point.
Prompt injection. Any text that the agent reads (a web page, a file, a comment in the code under analysis, a support message) may contain instructions, and the model has no reliable way to tell those instructions from yours. The defences are structural. Treat analysed content as data, and list any instructions found in it. Require approval for irreversible actions. Restrict network egress. Separate the agent that reads untrusted content from the agent that holds credentials. Offensive agents are exposed to the same weakness. One untested idea is a decoy file in a server root. The file tells an intruding agent that its test is complete and that it should patch its entry point and disconnect.
Credentials. Keep secrets out of any text that a model can read. Reference each secret by name and inject it at run time. Enter keys in your own terminal, never in a chat. Audit skill folders, tool-server configuration and instruction files for embedded keys. Some people argue that, as models improve, keeping credentials outside the agent's machine will cost more in capability than it buys in safety. Keep the conservative default until that argument has been shown to hold against injection. One reported refusal is not a demonstration.
Stored authorisations. A hosted service that holds users' OAuth grants to other systems is a central store of long-lived tokens. An attacker who compromises the service gains every connected account. Prefer short-lived tokens and proof-of-possession schemes. In a proof-of-possession scheme, a hardware-protected key that cannot be copied must sign each request.
The runtime. Patch the agent's own command-line tool and its language runtime as you patch any other software.
Scope. An agent that was told it could use any key in a keychain produced a 36-minute film and a bill above $200. Blanket access is an anti-pattern even when the result is good. Set a budget, a length limit and an explicit list of the credentials that the agent may use.
Oversight of collection. Before you grant an agent read access to mail and accounts for an audit, weigh that access as a risk of its own.
13.5 Cost as an attack
Every model call costs money, so abuse of an AI product is a direct financial attack on its operator. The eight measures below come from one company's experience. No single measure is sufficient.
- Make free trials cost something: require a card and pre-authorise it.
- Verify that requests carry valid credentials and that they come from the genuine client application.
- Allowlist the models that you offer and deny the rest by default, because abusers go for the newest and most expensive models.
- Enforce hard limits per account and per organisation, and allow no overage until payment has cleared.
- Bill sooner. Billing in arrears and grace periods let abusers spend on cards that never clear.
- Use the payment provider's fraud rules.
- Watch cost daily and be ready to suspend accounts or pause sign-ups.
- Restrict trials to verified business domains and cap the number of trials per domain.
You can inflict the same cost on yourself. On one serverless platform a scheduled alarm re-armed itself in a loop and produced trillions of storage operations and a bill near $10,800. The bill was paid in full after an automated support agent answered the appeal in circles. Give every usage-billed primitive that can schedule itself a termination condition and a loop guard, and set up budget alerts and hard limits before you deploy.
13.6 Supply chain
Every dependency is the unpaid work of some maintainer, and it is part of the attack surface of whoever uses it. An analysis of 23 foundational projects found that in 11 of them one or two people did the regular work. One volunteer made 218 of the 251 changes in a year to the time-zone database that runs on some four billion phones. The maintainer of a compression library was socially engineered over years into accepting a contributor, and the contributor inserted a backdoor. An engineer who noticed unusual processor use caught the backdoor in March 2024. A widely deployed cryptographic library was running on about $2,000 a year in donations in 2014, when a flaw in it exposed roughly 17 per cent of secure servers.
Keep the public surface and the dependency list of anything you publish small, because each dependency becomes a dependency of every adopter. Prefer a little duplicated code to a trivial package. Pin versions, and verify the checksums of the models and binaries that you download. Include maintainer health in your dependency inventories. Treat "paste this link and let the agent install it" as both a supply-chain risk and an injection risk.
13.7 Open-source intelligence for self-audit
Run reconnaissance against yourself and your organisation before someone else does.
- Usernames. Search tools check a name across hundreds or thousands of sites and follow each newly found identifier recursively. If you reuse one username, the tools link every account that has it.
- Domains. Certificate-transparency logs list every certificate ever issued, and therefore every subdomain. Collection tools separate passive sources, which never contact the target, from active sources.
- Devices. Search engines for connected devices index what your addresses expose.
- Email addresses. An outsider can guess a company's address format and verify the guess. A breach-notification service acts as an oracle that reveals whether an address exists.
- Web artefacts. Client-side bundles, advertising transparency registers, domain registration and payment identifiers each attribute a site to its operator. In one reported investigation these sources were chained in twenty minutes to identify the operator of a cloned product and to file abuse reports with each intermediary.
- Media. A recording that contains a passing aircraft can be located by matching the Doppler curve of the aircraft's engine tone against public flight-tracking data.
We treat the ethics of these tools as part of the method. Attribution is frequently wrong, publishing an individual's identity does harm that cannot be undone, and the same tools enable stalking. Use the tools on yourself, and on anyone else only with written authorisation.
13.8 Personal security as engineering hygiene
Whoever controls an engineer's accounts controls everything that the engineer can deploy.
- Use a password manager with a unique generated password for every service. The password manager then becomes the asset to protect.
- Use hardware security keys as the second factor on email, source control and banking, and store two or three keys apart. Codes sent to a phone number fall to SIM swapping. Add a SIM PIN regardless.
- Give random strings as the answers to security questions.
- Turn on full-disk encryption and automatic updates on every machine.
- Never follow a link in an email to a service. Navigate to the service yourself. Disable automatic image loading, because tracking pixels report when you open a message.
- Use a separate virtual card for each merchant, each card with its own limit.
- Use end-to-end encrypted messaging with disappearing messages.
- Keep personal accounts off employer-managed machines.
- Know which tier of each AI tool you use and what its data terms are. Contracts now forbid placing confidential material into consumer tools that may train on it.
- Take a periodic inventory of every service you have signed up for, every connected application and old key, every connector granted to an AI tool and every vulnerable dependency. Write the inventory to a sheet with the risk, the urgency and the action for each item.
Laboratory 8
Audit an application you built. (a) Check the twenty controls of 13.1 and record the evidence for each control. (b) Analyse your own client as an outsider would, and list every endpoint that the client reveals. Test object-level authorisation on each endpoint with a second account. (c) Run a phased agent audit with a separate finder and verifier. Submit the findings under the three verdicts, including the rejected findings. (d) Run two of the self-audits in 13.7. (e) List what you did not examine.
Practical work
The practical work for this module is in Part IV: Assignment 5 · The infinite story and Assignment 6 · A search engine served securely.
Module 14 · Shipping and operating software
We make this module's lesson video when 25 people sign up for it.
If you don't want to wait, you can buy the video now. The price is what it costs us to make it, plus 20%. We start on it the day you buy it, we send it to you the day it is done, and then we post it on this page for everyone.
What the $19.34 pays for
| Narration, 39,487 characters | $11.85 |
| 6 generated clips of 5 seconds | $3.36 |
| 6 keyframe stills for those clips | $0.90 |
| Rendering on our own Macs | $0.00 |
| Cost to make | $16.11 |
| Plus 20% | $3.23 |
| Price | $19.34 |
|---|
The agent runs on a flat subscription and rendering runs on our own Macs, so the price includes no charge for the agent or for rendering.
Objectives
By the end of this module you should be able to choose an architecture and a hosting model from actual constraints, design interfaces that last, ship with evidence, and explain what separates a demonstration from a product.
14.1 Architecture and load
When you split an application into many services, function calls inside one process become calls across a network, and network calls fail in ways that local calls do not. In the last decade, teams adopted hundreds of services for theoretical scale while serving a few hundred users, and delivered software more slowly as a result. Start with one deployable unit. Divide the unit when a measured constraint requires it: a team boundary, a scaling limit or an isolation requirement.
We also study two smaller patterns from the season's practice:
- An event table and a schedule. A complete event-driven pipeline has three tables (entities, people and dated events about an entity), a scheduled job that fills the events table, and an agent that starts when an event arrives. In a public-data variant the core step is a spatial join of events (storms, permits, sales) to properties. The database does the de-duplication and scoring, and a model is used only at the end to classify and draft.
- State machines as owned objects. An implicit state machine spread across database rows can be made into an addressable unit that owns its state and processes one message at a time. The claimed benefits are observability, isolated testing and pushed updates. This advice comes from a vendor of such a platform. Find the state machines first (Module 7) and decide afterwards.
14.2 Interfaces that last
The following rules come from an engineer who built operating-system interfaces used by thousands of developers:
- Design the call site first. Write the code that an adopter will write, then implement the interface behind it. Accept an ugly declaration if it makes usage clean.
- Keep types opaque across layers. Expose the capability the next layer needs and nothing more, so that the implementation can change without edits in every caller.
- Do not use stringly-typed parameters. Use enumerations and typed values, which the compiler and the editor can check.
- Treat errors as part of the interface. A user must be able to diagnose an error and recover from it.
- Make expensive operations explicit methods and document them as expensive. A property that silently performs I/O is a trap.
- Use progressive disclosure. An adopter should understand the simplest use in seconds and find more power as the need arises.
- Keep the public surface small. A public symbol can never be removed. Add capability by reusing existing types in new ways.
- Minimise dependencies. Each dependency becomes a dependency of every adopter.
For service clients, adopt two patterns. First, expose paginated results as an asynchronous sequence that the caller iterates, and hide the page fetching. Second, retry with exponential backoff inside the client library. Never put a server-side SDK or its service-account keys into a client application.
14.3 Hosting and cost
Commodity infrastructure is cheap to run yourself. You need a virtual server at a few dollars a month, a database on that server, a content network in front and object storage for bulk files. In one reported case, consolidating five subscriptions saved about $155 a month. Storing 100 GB of video in object storage with free egress costs a little over a dollar a month, so keep bulk media out of the application server. In exchange you take on the operations work: patching, monitoring, backups and restores.
Three terms are becoming standard for buyers with stricter needs. Under bring your own cloud, the vendor's software runs in the customer's account. Under bring your own key, the customer controls the encryption keys in a single-tenant environment. Under bring your own inference, the customer supplies the model endpoint.
Unit economics. A personal assistant agent is reported to cost $3,000 to $7,000 per user per year to run, depending on scale and model choice. This figure is second-hand. Revenue per user must cover the inference cost of an always-on agent, and this requirement favours either narrow high-value uses or much cheaper models. Model the cost per session before you build.
Machines for agents. Persistent virtual machines sold for agents offer per-second billing, disk snapshots and forking from a snapshot. Compare a full machine for each agent with managed browser sessions by session-minutes. A stopped machine can still be billed for storage.
14.4 Build or buy
One current claim is that internal builds now beat purchased software. Agents have changed some terms of the comparison and left other terms unchanged.
| Criterion | What changed | What did not |
|---|---|---|
| Build cost | A long agent run can produce a focused version. | Someone must still specify the software and verify it. |
| Scale | You serve one organisation. A vendor serves its whole customer base. | |
| Fixing defects | You can reproduce a defect and fix it at once. A defect in a vendor's product waits for the vendor. | You are now on call. |
| Reliability | You can measure your own uptime. A vendor reports its own figures. | |
| Switching | Moving thousands of people off an existing tool is painful, even when the replacement is perfect. | |
| Ownership | You own the security, compliance and maintenance of code that nobody wrote by hand. |
If professional software can be built for the price of the tokens, its market value falls toward that price. The value then moves to distribution, data, switching cost and operation.
14.5 From demonstration to product
You can build a prototype of a customer-support inbox in an afternoon. To own one you must handle every item on the following list, which makes a good requirements catalogue for any networked application:
- Email: deliverability, bounces, sender authentication records and reputation drift; threading that breaks when a subject changes; nested forwards; separating a new reply from quoted text; auto-responder loops.
- Collaboration: two people drafting the same reply; internal notes that must stay internal; drafts synchronised across devices; a "saved" indicator that was wrong.
- Time: snooze across time zones and daylight-saving changes, including a local time that does not exist.
- Identity: one customer who appears as several; several people who share an address; merge and un-merge.
- Messaging and webhooks: duplicated, reordered and missing deliveries; retrying a failed send without sending twice; rate limits discovered at the first spike.
- Real time: reconnect storms after a restart; half-open connections; in-order delivery; read state across tabs.
- The AI layer: scanned PDFs; stale facts; conflicting sources and which source wins; a full re-index when the embedding model changes; injection in scraped pages; an agent that promises a feature that does not exist; a retry that refunds twice; a hand-off to a person after which the agent actually stops; re-evaluation after every model update.
- Operations: malware uploads; deletion requests that reach logs and backups; backups and tested restores as separate features; background jobs that stop silently; full disks; monitoring, and monitoring of the monitoring.
A vendor wrote the list to argue for buying, but every item on it is a real requirement.
Idempotency. When the outcome of an operation is unknown, as with a publish request that timed out, check the outcome before you retry, or the operation happens twice. Read-back verification. After you write through an interface, read the result and compare it with what you intended, because forms show fields that the system did not save. Separate evidence for separate states. Paid, label created, accepted by the carrier, delivered and paid out are five facts, and each fact has its own proof.
14.6 Release engineering
- Validate offline before you upload. A release script checks everything that can be checked locally and stops at the first failure.
- Pin and verify. Install dependencies from a lockfile. Pin every downloaded model and binary to a commit and verify it by checksum. Make the build run without the network after setup.
- Sign and notarise desktop software. An unsigned application forces its users past an operating-system warning.
- Plan the monitoring before the deploy. Write down which signals will show that the change is healthy, then watch those signals during the rollout.
- Keep a record of what was verified on which versions.
Continuous integration economics. On hosted runners, billed time and elapsed time are different numbers, and most of the waste is the difference between the two. A five-way matrix that finishes in two minutes is billed for ten minutes or more. A five-second job is billed for a full minute. A merge reruns everything that the change already passed. The procedure below applies to any pipeline:
- Measure the total billed minutes by workflow and by job, with the median and the 90th-percentile elapsed time.
- Learn what each check protects before you propose any change. The longest dependency chain is the critical path, and the pipeline gets faster only when you shorten that chain.
- Look for the known kinds of waste: suites that run twice on the same commit; path filters that match nearly every change; many tiny jobs, each of which pays for setup and has its billed time rounded up; missing dependency caches; a rerun of everything on every push, with no cancellation; jobs with no timeout; and expensive operating-system runners used where they are not needed.
- Estimate each saving from the measured data before you change anything, and state the risk.
- Make the changes in small steps, and add a comment that explains each change so that nobody undoes it.
Caching and removing duplicate work improve both speed and cost. Splitting the work into parallel shards makes the pipeline faster and costs more. Say whether you are optimising for speed or for cost.
14.7 Deciding what to build
An internal platform team has no market signal, so the team has work only when an engineer identifies it. Eleven signals come from four sources:
- Incident reviews, the cost bill and the team's own toil.
- Conversations with users, uses of the platform that it was not designed for, and joint prototypes.
- Stated objectives, things that management repeats, and teams that never migrated.
- The wider industry, which supplies two signals: writing a description of the current system to expose expired decisions, and importing lessons from fields that are ahead.
The strongest signal is a use that the platform was not designed for, because such a use is a prototype that is already running. Do not assemble the backlog from the loudest signals.
For a product, you must learn as fast as you build. Recruit a small group of people who already have the problem. Ask about their last real attempt, and never about their opinion of a feature. Keep an evidence ladder in view: agreeing to talk shows only a willingness to talk, completing one real task shows that the product can help once, and choosing the product again when the need recurs is the first evidence of value. For an agent, the sign of value is that another real assignment finishes correctly under the intended supervision.
One project stores national law as Git history, with one commit for each law that takes effect. The history yields the text in force on any date and the law behind every line. The commits were not made in chronological order, so each commit carries an in-force date and the tools order by that field. Valid time and recorded time are different, and a data model must say which one it means.
Laboratory 9
Ship one small service to the public internet. Submit the following: a one-page architecture note that justifies the choice of one deployable unit or of more than one; the interface, designed call site first; the server, with no inbound ports open (Module 12); a spending cap and a billing alert, with a screenshot of each; a release script that validates before it deploys; a restore from backup that you performed and timed; and a list of the items in 14.5 that apply to your service, with each item marked handled, deferred or not applicable.
Practical work
The practical work for this module is in Part IV: Assignment 6 · A search engine served securely and the Section problems.
Module 15 · Local models and fine-tuning on your own hardware
We make this module's lesson video when 25 people sign up for it.
If you don't want to wait, you can buy the video now. The price is what it costs us to make it, plus 20%. We start on it the day you buy it, we send it to you the day it is done, and then we post it on this page for everyone.
What the $19.58 pays for
| Narration, 40,153 characters | $12.05 |
| 6 generated clips of 5 seconds | $3.36 |
| 6 keyframe stills for those clips | $0.90 |
| Rendering on our own Macs | $0.00 |
| Cost to make | $16.31 |
| Plus 20% | $3.27 |
| Price | $19.58 |
|---|
The agent runs on a flat subscription and rendering runs on our own Macs, so the price includes no charge for the agent or for rendering.
Objectives
By the end of this module you should be able to decide when a model should run on hardware that you control, size the model to that hardware, adapt the model to a task and measure the result.
15.1 Reasons to run a model locally
Confidentiality. Some data must not leave the premises, by contract or by choice. Privacy then follows from where the computation runs and does not depend on a policy. Cost. Once the hardware exists, you pay no charge per token. Latency and availability. A local model needs no network round trip and works offline. Control. A local model is fixed and does not change under you. Against these advantages, weigh the gap in capability between a local model and the best hosted models, and the cost of owning and operating the hardware.
15.2 The memory budget
Rule. Memory required = weights + key-value cache for the context + runtime overhead.
Weights stored at 4 bits per parameter take about half a gigabyte per billion parameters, so a 27-billion-parameter model needs roughly 13.5 GB before it holds any context. Context length is a memory cost as well as a property of the model, because the cache grows with every token that it holds. Use this arithmetic to test claims. One configuration advertised this season put a 27-billion-parameter model at "4-bit quality" in an 8.65 GB file, with a 256,000-token context, on a 16 GB card. The file size implies about 2.5 bits per weight, and a cache for that context needs memory that the claim does not account for. For any local setup, ask how much model fits, how much context can be kept and how much computation the runtime can avoid.
The published cards for one large open-weight model show the trade-off. A 2.9-bit build is about 90 GB smaller than a 4-bit build and holds nearly three times the cached context, at about half the throughput under heavy concurrency. All of the builds score within a point of the original model on three standard test sets.
Speculative decoding is the main way to avoid computation. A cheap drafter proposes several next tokens, and the main model verifies them in one batched pass. Accepted tokens cost no separate forward pass, and the output is unchanged when verification is exact. The gain depends on how often the draft is accepted.
15.3 Sizing a model to the machine
A well-built local application checks the machine's memory at launch and downloads the model that fits. One offline reference assistant publishes its tiers with their scores on its own 22-question test, which was graded blind against official guidance:
| Machine memory | Model class | Score | Download |
|---|---|---|---|
| 128 GB and above | Largest open model | 21/22 | 80 GB |
| 32–96 GB | 27B class | 21/22 | 18 GB |
| 16–31 GB | 8B class | 14/22 | 6 GB |
The test is small and self-run. It is still the right kind of test, because it uses a handful of questions from the actual task and the grader does not know which model answered. The same application reports about 100 watts while answering and 5 to 40 seconds per question, which gives 100 to 300 answers per battery charge.
Small models make more errors, so the application relies on citation for reliability: it lists the articles used for every answer and opens the source at the passage. Corpus licences travel with the data and must be shown.
15.4 Small specialised models
The most useful local models are often not chat models.
- Decision classifiers of a few hundred million parameters answer in milliseconds and use less than a gigabyte of memory (Module 1).
- Forecasting models of 5 to 209 million parameters take a history of values and return a probabilistic forecast of future values, with quantiles, in 10 to 140 milliseconds on a processor.
- Speech. A complete voice agent now runs on a phone. It combines on-device recognition, a small on-device language model, a 32-million-parameter encoder that decides which character the user addressed, and a 31-million-parameter speech synthesiser quantised to 8 bits on the processor. The agent generates each line while it speaks the previous line, which hides latency. Native speech-to-speech models are trained on audio alone and can listen and speak at once. They keep tone, emphasis and timing that the transcribe-then-synthesise pipeline discards. The claims made for these models this season are not yet backed by published weights or benchmarks.
- Vision. Open recorders run object detection on ordinary camera feeds entirely on local hardware.
- Models built into the operating system. One desktop system is reported to ship a terminal command for its on-device model, with piping, schema-constrained JSON output and a local server that speaks the common chat interface. The model's 4,096-token window restricts it to summarising, extracting, classifying and rewriting.
A non-text model can hallucinate too. The makers of a lip-reading dictation tool found that its recogniser confidently produced fluent sentences when shown a still mouth. The fix was to measure mouth motion and never send still stretches to the model. Gate a model's input on evidence that there is something to recognise.
15.5 Fine-tuning
A small open model that is fine-tuned on good proprietary labels can beat a frontier generalist on one narrow task, and it is cheaper and faster. One exchange reports this result for fraud review with a 9-billion-parameter model, but the exchange published no figures. The result is plausible where labelled data is plentiful and the task is fixed.
A guided fine-tuning agent published this season sets out the phases:
- Goal. Question the goal until it is concrete.
- Compute. Detect the local hardware or prepare a rented machine.
- Base model. Choose a base model that fits the goal, the hardware and the licence.
- Data. Find, import, write or generate the data. Preview exactly what the model will learn. Clean out empty, duplicate and broken records.
- Recipe. Propose hyperparameters only within the published guidelines for that model, and estimate memory, time and cost first.
- Training. Run the training detached, with a live loss curve.
- Taste test. Run the same prompts against the tuned model and the base model, side by side.
- Export. Export an adapter or merged weights, with a model card and licence files.
Training objectives can be chained. Supervised fine-tuning alone suits an instruction-following base model. Continued pretraining followed by supervised tuning teaches a domain and then a dialogue form. Supervised tuning followed by preference optimisation sharpens style. To make the preference pairs, sample two answers and have a person, or a judged rubric with human spot checks, pick the better answer.
Copy the fine-tuning agent's guardrails into any agent that spends money. Every action that costs money, installs software, sends data out or publishes requires a confirmation that the model cannot bypass. A real GPU operation runs first, to catch a broken driver. Secrets never pass through the conversation. The agent's documentation gives worked examples. A 2-billion-parameter model was tuned to a writing voice from 826 examples in about two epochs. An image adapter was trained from 160 captioned photographs, and the checkpoint at step 500 was chosen over the final checkpoint by inspection. Roughly an hour on a rented 24 GB card cost about forty cents.
15.6 Interpretability
Interpretability is the study of what the individual parts of a network compute. A polysemantic neuron appears to do several unrelated jobs. Superposition is the hypothesis that a network stores more features than it has neurons by overlapping the features. A sparse autoencoder is a tool for separating such features. A long informal essay published this season argues the opposite of the prevailing view. The essay claims that, with better tools, most neurons in modern networks are legible one at a time, that earlier illegibility came from architecture and tooling, and that a neuron's negative side, meaning what suppresses the neuron, often carries the distinction that matters. The essay's evidence includes rising fractions of interpretable neurons across network generations, and ablations in which removing a handful of suppressing neurons changed an answer more than removing twenty obvious neurons.
The author calls the argument a working hypothesis, the percentages depend on thresholds, and the key results await a paper. What you can use now is the method for studying any unit. Look at the unit's strong and weak activations, at what suppresses the unit and at its connections. Turn the unit up and see what the model writes. Test the explanation on held-out cases. Remove the unit before you trust a label.
In a related experiment design, a model is trained only on text from before a cutoff date and is then tested on whether it can derive what was discovered afterwards. This design separates recall from derivation, which ordinary benchmarks cannot do.
Laboratory 10
Choose a narrow task with at least 500 labelled examples (Laboratory 1 will serve). (a) Compute the memory budget for two candidate local models on your machine and check each budget against observed use. (b) Write a 20-question test that is graded blind. (c) Score a hosted model, a local model, and the local model after fine-tuning or with retrieval. (d) Report accuracy, latency at the 50th and 95th percentiles, cost per thousand items, and energy if you can measure it. (e) State which model you would deploy.
Module 16 · Capstone and assessment
We make this module's lesson video when 25 people sign up for it.
If you don't want to wait, you can buy the video now. The price is what it costs us to make it, plus 20%. We start on it the day you buy it, we send it to you the day it is done, and then we post it on this page for everyone.
What the $15.87 pays for
| Narration, 29,837 characters | $8.96 |
| 6 generated clips of 5 seconds | $3.36 |
| 6 keyframe stills for those clips | $0.90 |
| Rendering on our own Macs | $0.00 |
| Cost to make | $13.22 |
| Plus 20% | $2.65 |
| Price | $15.87 |
|---|
The agent runs on a flat subscription and rendering runs on our own Macs, so the price includes no charge for the agent or for rendering.
The project
Build, verify, secure and ship one piece of software that a real person uses, and work with agents throughout. The software must include a component from each of at least three of the laboratory modules (9 to 15).
Deliverables
- The brief. Submit the one-page brief and the approved specification, architecture and plan.
- The harness configuration. Submit the instruction file, skills, hooks and permissions, with a note on what each enforces.
- The software. Submit the deployed software with its release script.
- The verification record. Include tests that pass the four-question gate, a calibrated grader or replay test, and the result of an independent audit in which finder and verifier were separate.
- The security record. Include the baseline controls with evidence, the endpoint inventory and the spending cap.
- The cost model. State the tokens and money spent building the software, and the cost to run it per user per month.
- The findings log. Number and date each entry, and give the versions on which each claim was verified.
- A statement of what was not checked.
Grading
| Weight | Criterion |
|---|---|
| 25% | The software works for its user, as shown by a second real use after the first. |
| 25% | Verification: the evidence is independent and repeatable, and you state its limits honestly. |
| 20% | Security and operations: the project applies least privilege, exposes no surface without a reason and has a tested restore. |
| 15% | Harness: each rule is placed where it is enforced, and the context is used economically. |
| 15% | Writing: the brief, the log and the report name the subject, the object and the action. |
If a project claims more than its evidence supports, we grade the project on the evidence.
Part IV · Practical work
How the practical work runs
The practical work in this part consists of six assignments, set about ten days apart, and a problem section every week. All of it is written for work with agents and language models. The modules, laboratories and problem sets of Parts I to III stand as they are. This part adds a second, more gradual track of hands-on work that runs beside them.
The first assignment starts with a robot that knows four commands. Each later assignment adds one idea and reuses all the earlier ideas.
| Weeks | Assignment | Idea added | Modules |
|---|---|---|---|
| 1–2 | 1 · Karel the agent | Decomposition under a restricted harness | 2, 6 |
| 2–3 | 2 · The label drill and the pyramid | Hand traces, named constants and input checks | 1, 3, 5 |
| 4–5 | 3 · Bricks | A build in stages that are each playable | 7, 8, 9 |
| 6–7 | 4 · Sand and fire | Tests written first, thresholds and decoding | 3, 8, 11 |
| 7–8 | 5 · The infinite story | A model inside the program | 9, 13 |
| 9–10 | 6 · Search engine | An index, a server and a ranking you can defend | 10, 12, 13, 14 |
Four habits apply to every assignment.
- Warm up first. Each assignment opens with a few problems that take minutes. Do them before the large problem. You do some of them by hand, with no model and no interpreter.
- Build in stages. A large problem is a list of numbered steps. Each step ends in something that runs, and you check it before you begin the next step.
- Many worlds. A solution that passes on one input has not yet been shown to work. Every problem names several starting states, including the smallest state and the awkward ones, and the same prompts must pass on all of them.
- Name the numbers. Define every limit, count, threshold and size as a named constant at the top of the file. Do not write a bare number inside a prompt or a loop.
Each assignment ends with an optional extension of your own design. An extension earns credit only if the required parts already pass. For every problem, submit the prompts, the session log, the commands that verified the result and a list of what you did not check.
Assignment 1 · Karel the agent
This assignment accompanies Modules 2 and 6.
Karel is a robot on a grid. It can move, turn left, put down a marker and pick one up. A beginner programs Karel with those commands, simple tests, loops and named helper functions, and with nothing else. The language has no stored values and no shortcut out of a loop. A beginner can therefore solve a hard problem only by splitting it into small named pieces. In this assignment you put an agent under the same kind of restriction.
The restricted harness. For every problem, use one instruction file of at most 30 lines, plain prompts that you type yourself, and the built-in tools for reading files, editing files and running one test command. Do not use skills, hooks, subagents, connected tool servers, web access or background tasks. A world is a branch of a small repository of your own that has a known starting state. Make a fresh branch for each run, so that you can discard a failed run whole.
Decomposition. Before you prompt, write each solution as a list of named subtasks, in the way that a Karel program is a list of small helper functions. Give each subtask its own prompt. We grade the decomposition: a single prompt that asks for the whole problem earns no credit, even when the result is correct, because no step of it can be checked.
- Fetch and return. One configuration value is stored in one file. Have the agent copy the value into a named section of the README. The run must end with a clean working tree and exactly one commit. Use at least two subtasks. First the agent finds the value and reports its file and line. Then the agent makes the change. Show with
git show --statthat the commit touches only the two intended files. This problem has a fixed world in which nothing varies. The agent must still do the trivial job completely, including the return to a clean state. - Repair without duplication. Every module of one kind in the world should have a test file of a known shape, and some of the test files are missing. The agent adds a test where one is missing and leaves every existing test untouched. Run the same prompts on at least three worlds: the default world, a world with a single module, and a world in which no test is missing. In the world with no missing test, the correct result is an empty diff. Verify each result with commands. Reading the diff does not count as verification.
- Cover the grid. The agent writes a one-paragraph summary of every source file in a repository that is several times larger than its context window. The agent must read each file exactly once, and file content must never fill more than half the window (1.2). Get one directory right before you attempt the whole repository. The edge worlds are a repository of one small file, a repository with a single file larger than half the window, and a repository whose file count does not divide evenly into your chunks. Submit a coverage table (file, chunk, read count) that a script generates from the session log.
- Find the midpoint. A test passes on an old commit and fails on the newest commit. The agent finds the commit that broke the test by testing the midpoint of the remaining range at each step. Cleaning up counts as much as finding the commit. Compare the repository before and after the run. The only difference we allow is a note that names the commit. A run fails if it leaves an open bisect session, a leftover branch or a scratch file. Solve the problem on histories of 2, 9 and 200 commits.
Extension. Design a world of your own that defeats your prompts from problem 2 or problem 3. Then fix the prompts without making them specific to that world.
Starter code. assignment-1-karel.zip holds a text-mode grid robot with a by-hand warm-up and two scripts. make_worlds.py builds a fresh practice repository for each of the four problems and their edge worlds, and verify_world.py checks problems 1, 2 and 4 with commands. To practise Karel itself first, open the online Karel IDE, which has English menus and takes programs in Python. The kit's browser/ folder holds the warm-up and its four worlds as files that the IDE loads. The other editors are listed under Starter code and tools.
Assignment 2 · Warm-ups for the label drill and the pyramid
This assignment accompanies Modules 1, 3 and 5.
You begin this assignment with small warm-ups that each exercise one idea: tracing by hand, counting down, repeating a random draw, refusing an impossible input and calling a library function. You then write two larger programs: a drill that repeats until the person who takes it gets three items right in a row, and a pyramid that you build row by row from shared constants.
Warm-ups. Write your hand-worked answers in by-hand.txt before you run anything.
- Trace by hand. A session starts with a 4,000-token instruction file and a 150,000-token budget. Each turn adds a 600-token request, 9,000 tokens of tool output and a 1,200-token reply. Write the running total turn by turn, and name the turn on which the session exceeds the budget. Then state which of the three per-turn costs you would cut first and how you would cut it (6.2).
- Countdown. Write a loop that asks a model to summarise the same passage in 10 sentences, then in 9 sentences, and so on down to 1. Record how many sentences the model returns each time. An instruction is advice (Rule 2). Count how often the model follows this instruction.
- Ten runs. Set
NUM_RUNS = 10and define the temperature as a constant. Send one open-ended prompt ten times at temperature 1 and ten times at temperature 0. Report the number of distinct answers in each set. Write one sentence on what the two counts mean for a test that calls a model once. - Refuse the impossible. Write
calibration_gap(confidences, outcomes), which returns the stated confidence minus the observed accuracy for each bin. The function rejects a confidence below 0 or above 1 with a clear message. The function must not clamp such a value and must not compute with it. - By hand and then by library. On paper, compute the cosine similarity of (3, 4) and (4, 3), and the cosine similarity of (1, 0) and (−1, 0). Then compute the cosine similarity in code for two pairs of sentence embeddings: one pair you consider close and one pair you consider far apart.
The label drill. The label drill is the gate to Laboratory 1. Take the schema from Laboratory 1, and use the first 40 items that you labelled there as an answer key. Write a program that shows a person a random item from the key, takes the person's label and says whether the label matched the key. On a miss, the program shows the expected label and its definition. The program stops when the person reaches CORRECT_IN_A_ROW = 3 or after MAX_ITEMS = 30, and reports which condition stopped it. A miss resets the streak to zero. Only people who pass the drill label the 100 shared items in Laboratory 1. Report how many items each annotator needed and which labels caused the resets. Rewrite the definitions of those labels.
The pyramid. The problem set of Module 5 asks for a bigram model, one attention head and a one-layer Transformer. Build the three models as a pyramid. Start with the bottom row, and build each row on the row below it. Define the corpus path, the context length and the train and held-out split once, as constants that all rows share. Leave the held-out set untouched until a row reports its loss. Row 1 counts character pairs. Row 2 is the attention head with its causal-mask check, and Row 3 is the Transformer. Then change one constant, the context length, rebuild all three rows and report what moved.
Extension. Add a fourth row to the pyramid. Predict the held-out loss of the fourth row before you train it, and report how far the prediction was from the measured loss.
Starter code. assignment-2-warm-ups.zip holds by-hand.txt, a file for each warm-up, the drill's constants and report with a sample answer key, and the pyramid's shared constants with row 1. The model.py in the kit is a stand-in that runs with no account. Replace the stand-in before you produce the numbers you hand in. The editors are listed under Starter code and tools.
Assignment 3 · Bricks
This assignment accompanies Modules 7, 8 and 9.
You open this assignment with one small function. Before any code exists, you write the expected behaviour of the function as examples, including what the function returns when nothing matches. You then build a brick-and-paddle game in a fixed order: the scene that does not move, one moving object, the player's input, collisions and, last, the rules for lives and winning. After every step you can play the game in a small way. This order is the gated build of 7.2, reduced to the scale of one afternoon.
Warm-up. Write the examples for first_failing(results) before any code exists. Given a list of test results, the function returns the name of the first failing test, or None when every test passes. Include the empty list and a list in which every test passes. Give an agent only the examples, have the agent write the function, and run the examples. Then change one example so that it is wrong, and confirm that the run fails (8.2).
The staged build. Build the game of Laboratory 4 in five stages. If you have already finished Laboratory 4, you may build any small real-time program instead. Each stage consists of one brief, one agent session, one commit and one recorded check.
- The static scene. Lay out everything that does not move from constants: the counts, sizes, gaps and offsets. To check this stage, change two constants and confirm that the layout is still centred and still fits.
- One moving object. Add one object that has a velocity and reflects off the edges. To check this stage, run the game twice with a fixed seed and confirm that the object follows the same path both times.
- Input. Add one object that follows the player and cannot leave the play area. To check this stage, hold the input at each extreme.
- Collisions. Handle the collisions of the moving object with everything else. To check this stage, run a recorded replay in which the object arrives at a corner.
- Rules. Set
LIVES = 3, and add a win condition and a loss condition. To check this stage, run one replay for each ending.
Do not start a stage until the previous stage's check passes, and do not let a later session rewrite an earlier stage without running that stage's check again. At the end, run the five checks again on the final commit and submit them.
Extension. Add one feature of your own, with a stage and a check of its own. Report whether any of the five earlier checks broke when you added the feature.
Starter code. assignment-3-bricks.zip holds the warm-up, the Bricks engine, an empty game.py and check.py, which checks each of the five stages through the game's two functions. In Part V you build the same game twice and time each build. The editors are listed under Starter code and tools.
Assignment 4 · Sand and fire with byte decoding
This assignment accompanies Modules 3, 8 and 11.
In this assignment you write the expected results before you write the code. The sand simulation is a grid of cells. A grain falls straight down when the cell below is free, and diagonally when it is blocked. For every rule you first write out the expected results, including the moves that go out of bounds and the moves that a corner blocks. In the fire exercise each item has one score, and you set one cut-off. An item that scores above the cut-off is flagged for a person, and every other item passes unreviewed. In two short problems you decode a compact log and pair up two lists.
Warm-ups.
- Zip. Write
pair(predictions, gold), which returns a new list that holds one[prediction, expected]pair for each position. Write the examples first. The examples cover three items, one item, no items at all, and two lists of different lengths. The function reports lists of different lengths as an error. Add one example that shows that the call leaves both arguments unchanged. - Decode. An agent's log arrives in a compact form, in which a repeat count follows each event letter. Write the decoder. Then give the decoder input that breaks the format: a count with no letter, a letter with no count, and a count of two digits. Decide what the decoder should do with each input, and write your decisions down before you run it.
Sand. In this problem you write the tests first. Choose a small grid that has rules: a falling-sand toy, a cellular automaton or the board of the game from Assignment 3. Before any implementation exists, write the expected result of each rule yourself as executable examples. Write the examples in three groups: the ordinary move, every out-of-bounds move and every blocked move. Give an agent the examples and the rule descriptions and nothing else. The agent must not edit the examples. Enforce this restriction with a permission or a hook (Rule 2), because a sentence in the prompt is only advice. When the agent reports success, apply the test of 8.2: break one rule in the implementation by hand and confirm that an example fails. Report the number of examples you wrote, the number that the first implementation failed, and any rule that your examples did not pin down.
Fire. A threshold on a score is a classifier. Take 200 items that each have a numeric score, such as the confidence of the model's labels from Laboratory 1 or a spam score on your own mail. Define THRESHOLD as a constant. Your program highlights the items above THRESHOLD for a person and passes the items below THRESHOLD without review. Label 50 items by hand. Then report, at three values of the threshold, how many items the program highlights, how many of the highlighted items deserved a highlight, and how many items that deserved a highlight the program missed. Choose a value and defend it in terms of what a miss costs and what a false alarm costs (3.4). A filter of this kind usually ships with one fixed number in it. In this exercise you see that a person chose that number.
Bytes. In this problem you decode an unknown format, which prepares you for Laboratory 6. Take a file format or a message that you did not design and for which you have no documentation: a save file, a configuration export or a packet from a device that you own. Use the method of 11.2. State one hypothesis about the encoding, write a decoder for that hypothesis, and test the decoder with a round trip: decode the bytes, re-encode the result and compare the bytes. Record each hypothesis that failed and the evidence that ruled it out.
Extension. Add one new kind of cell to the grid. Write the rules of the new cell as examples before you implement them.
Starter code. assignment-4-sand-fire-bytes.zip holds the two warm-ups, a grid class with an examples file for you to fill and a rules file for the agent, a generator for 200 scored items, and a 49-byte sample file with a round-trip harness. The editors are listed under Starter code and tools.
Assignment 5 · The infinite story
This assignment accompanies Modules 9 and 13.
In this assignment you build a text adventure that has no fixed end. The story is a data file that holds a plot summary and a set of scenes. Each scene has text and a list of choices, and each choice points to another scene by the key of that scene. In the warm-up you write a program that walks the file and lists the keys that choices point to but that do not exist. You then fill each missing scene at the moment the player reaches it, by asking a language model for a new scene in the same structure. The assignment closes with written questions on what the model's choices reveal and where the model should not be trusted. This is the one assignment in which a model runs inside the program, so the program checks every reply before the player sees it.
- Warm-up on dead ends. Write the story file for a short adventure of your own, with at least eight scenes. Write the walker that lists every choice that points to a missing scene. Add two more checks to the walker: one for scenes that no choice reaches and one for loops that have no exit.
- Generate on demand. When the player reaches a missing key, ask a model for the scene. Include in the request the plot, the key, one example scene and a statement of the required structure (9.3, 9.4).
- Reply validation. Validate every reply before the game uses it. A valid reply parses, has the text and has a list of choices. Every choice has text and a key, and the number of choices is within a limit that you define as a constant. When validation fails, retry up to
MAX_RETRIEStimes, and then show the player a fallback scene that you have written. The game must not crash and must not show the player a raw reply. Save each accepted scene, so that a second visit shows the same scene. - Player input. Check the player's entry against the choices on screen, and ask again until the entry is one of those choices. Never place the player's text inside a prompt.
- The credential. Read the credential for the model from the environment or a secret store, keep it out of every file under version control, and give it a spending cap (13.1, 13.5). Search the repository history for the credential and show the result.
- Replay. Record a session as the list of choices that the player made and the scenes that the model generated. Show that a replay of the record reproduces the session without a call to the model.
Written questions. Answer each question in a paragraph, and base each answer on your own runs.
- Generate forty scenes and tally every character name that the model chose, with the most frequent name first. What do the names at the top of the tally have in common, and what would a reader from another part of the world notice is missing? What did you change, if anything, and where did you make the change: in the prompt, in the data or in the code?
- Which decisions in your game did you leave to the model, and which decisions did you keep? State the rule that you used to decide.
- Suppose that you move the same loop of request, validation and retry into a program whose output decides something about a person, such as which support ticket is answered first or whether a refund is approved. Which of your checks from steps 3 to 6 would still be enough, and which failures would no check of that kind catch?
Extension. Generate an illustration for each new scene. Apply the same validation and the same cap, and show a placeholder when generation fails.
Starter code. assignment-5-infinite-story.zip holds a three-scene story file, the walker, the game loop with its constants, and a stand-in model that returns a malformed reply about once in three calls. The kit contains no credential, and its .gitignore excludes key files. The editors are listed under Starter code and tools.
Assignment 6 · A search engine served securely
This assignment accompanies Modules 10, 12, 13 and 14.
In the last assignment you build a search engine from nothing. A folder holds a few hundred text files. Your program reads every file and builds an index that maps each word to the files that contain it. The index ignores case and the punctuation around a word. The program answers a query of several words with the files that contain every word of the query. You then put the engine behind a small web server. In the written questions you consider what happens to a search engine when someone can pay to change the order of its results.
Warm-up. Write in_both(first, second), which returns the items that appear in both lists, each item once, in the order of the first list. Write the examples first. One example should hold duplicates, because a person who types a query word twice still expects each result once. To answer a multi-word query, the engine applies in_both to the index.
- Index your own corpus. Use a folder of a few hundred text files that you have the right to use, such as your notes, the documentation of your project or your sent mail. Build the word index on three files first, and print the whole index so that you can check it by eye. Then build the index on the whole folder. Decide what counts as a title for your files, and state your decision.
- Query. Return the files that contain every word of the query. Then build a second engine on embeddings of the same files. Write 20 queries with the answers that you expect, and score the two engines on them. Report the queries on which exact words win and the queries on which meaning wins.
- Serve. Put the engine behind a small web server that has one page and one search endpoint. Bind the server to the local machine only. If the server must be reached from outside, reach it through a tunnel or a private network, and open no inbound port (12.5). Limit the query length and the request rate. Treat a query as data, and never place a query inside a shell command, a file path or a prompt.
- Readable by agents. The results page must work with scripts disabled and through a text-only fetch (10.5). Give each result a stable address.
- Audit. Run the endpoint inventory and the finder-and-verifier audit of 13.3 against what you built, which is a server that reads files by name and takes text from strangers.
Written questions.
- Add a way to boost one file to the top of any result. Describe how a user could tell that the file was boosted, and state what you would have to show on the page for the ranking to remain honest.
- A ranking serves goals that pull against each other: the result that best matches the words, the result from the most reliable file, an order that the user can have explained, and a first page that does not say the same thing ten times. Put the four goals in order for your engine, and state what the engine's users lose because of the goal you put last.
Extension. Add ranking. Order the results by a measure that you choose, state the measure on the page, and show one query for which the measure gives a worse order than the unranked list.
Starter code. assignment-6-search.zip holds the warm-up, a sixteen-file corpus to index first, search.py, and serve.py, which binds to 127.0.0.1 and caps the query at 200 characters. The editors are listed under Starter code and tools.
Section problems
The section problems cover strings, files and nested dictionaries, and classes. A section is a set of short problems that you work in a small group in under an hour, and you write the expected results of each problem before the code.
Strings (with Modules 1 and 8).
- Keep only. Write
keep_only(text, allowed), which returns the characters oftextthat are inallowed, in their original order. The comparison ignores case. Usekeep_onlyto strip a model's reply down to the digits of an answer, and list three replies for which the stripped result is wrong. - Count the differences. Run one prompt twice at temperature 0, and line the two outputs up character by character. Write the function that reports how many characters do not match the character at the same position in the other output, and report the count. Two runs rarely return outputs of the same length, so decide first what the function does with the extra characters.
- Number the lines. Write a function that takes a list of lines and returns a list of
[index, line]pairs. Explain why a tool that shows a file to a model usually numbers the lines, and what goes wrong in an edit request when the tool does not number them (6.5). - Doubling. Write a function that repeats each character of a string, so that every character appears twice. Count the tokens of ten strings before and after the doubling. The character count doubles exactly, and the token count does not. Explain the difference with the material of 1.1.
Files and nested dictionaries (with Modules 6 and 14).
- Usage ledger. A log has one line for each model call. Each line has three fields separated by tabs: the task, the model and the tokens used. Build a dictionary keyed by model, in which each value is a dictionary from task to total tokens. Write a helper that adds one call to the ledger and creates the inner dictionary when the model is new. Write the examples for the helper first. The examples cover an empty ledger, a second task for a model that is already in the ledger, and a repeat of a model and task that are already in the ledger. The repeat adds to the total and creates nothing. Print the three most expensive pairs of model and task. Then move one task to a cheaper model and state the saving (6.6, 14.3).
- Fastest one-off. A latency log records the milliseconds that each call took, with one call on each line, and many values repeat. Find the lowest latency that no other call shares. Skip zero and negative readings, because they are faulty. Before you write the code, state the result for an empty log and the result for a log in which every value repeats.
Classes (with Modules 6 and 8).
- A budget object. Write a class
ContextBudgetthat is created with a window size. The class has methods that add a named item of a given token count, report what remains and report the largest item. Write the test for the class in a separate file that imports the class and uses only its methods. Then have an agent implement the class again from the test file alone, and compare the two implementations. The two implementations disagree wherever the test file did not say what the class should do.
Starter code. section-problems.zip holds a file for each group of problems with one example for each function, a 120-line usage log, an 80-line latency log, and ContextBudget with its test file. The editors are listed under Starter code and tools.
Starter code and tools
Every assignment in this part has a starter kit, and so do the section problems and the remake unit of Part V. A starter kit is a zip of Python files written for this course. Each kit runs as downloaded. An unfinished file prints TODO: and the name of the function that you must write, or it prints the count of examples that pass, which starts at zero.
Downloads
| Kit | Goes with | Contents |
|---|---|---|
| assignment-1-karel.zip | Assignment 1 | A text-mode grid robot with four worlds and a by-hand warm-up; the same warm-up and worlds as files for the online Karel IDE; a script that builds the practice Git repositories for problems 1 to 4; a verifier for problems 1, 2 and 4 |
| assignment-2-warm-ups.zip | Assignment 2 | by-hand.txt, the four warm-up programs, the label drill with a sample answer key, the pyramid's shared constants, a 14,500-character corpus and row 1 |
| assignment-3-bricks.zip | Assignment 3 | The first_failing warm-up; the Bricks engine, an empty game.py, and a check for each of the five stages |
| assignment-4-sand-fire-bytes.zip | Assignment 4 | pair and decode; a grid class with an examples file and an empty rules file; a generator for 200 scored items; a 49-byte file in an undocumented format with a round-trip harness |
| assignment-5-infinite-story.zip | Assignment 5 | A three-scene story file, the dead-end walker, the game loop, and a stand-in model that returns malformed replies on purpose |
| assignment-6-search.zip | Assignment 6 | in_both, a sixteen-file corpus, the indexer and query program, and a server bound to 127.0.0.1 |
| section-problems.zip | Section problems | The four string functions, a 120-line usage log, an 80-line latency log, and ContextBudget with its test file |
| remake-bricks.zip | Part V | The Bricks engine and checks, a hand folder and an ai folder, the brief, a stopwatch, a code comparison script and the worksheet |
Running a kit
The kits need Python 3.9 or later and no other package. Assignment 1 also needs Git. Unzip the kit, open a terminal in its folder, and run the first command in its README.md.
unzip assignment-3-bricks.zip
cd assignment-3-bricks
python3 check.py
On Windows the command is python or py. The two Bricks kits open a window: play.py uses tkinter, which comes with Python from python.org and with Thonny. Every check, the stopwatch and the comparison script run in a terminal and open no window.
Each kit that calls a model includes a stand-in model in model.py that needs no account and no network, so you can write and test the loops first. The numbers that you hand in must come from a real model, and your program must read the model's credential from the environment (13.1).
Editors and online tools
We opened each address below in Chrome on 17 November 2026, and each one loaded a working page. No tool in the table needs an account.
| Tool | Runs | Description and use |
|---|---|---|
| Reeborg's World | In the browser | An online Karel IDE with English menus that takes programs in Python. It has a world, a code editor and a run button. The Assignment 1 kit's browser/ folder holds the warm-up and its four worlds as files that the IDE loads. Use it to practise decomposition by hand before Assignment 1. |
| Online Python | In the browser | An editor with a Run button and a terminal pane. Use it for the single-file warm-ups and the section problems. |
| Python Tutor | In the browser | A tool that steps through a short program one line at a time and draws its variables. Use it to check a hand trace after you have written the trace. |
| Pyodide console | In the browser | A Python prompt. Use it to try one function. |
| Thonny | Installed | A beginner's editor that includes its own Python and tkinter. It is the shortest route to the Bricks window. |
| PyCharm | Installed | A full Python IDE with a debugger and a test runner. |
| Visual Studio Code | Installed | A general editor with a Python extension. |
| Python | Installed | The interpreter itself, with tkinter. |
The browser tools hold one program at a time and have no Git. They suit the warm-ups, strings.py and any single function. Run the multi-file kits, the agent work and everything that needs a repository on your own machine.
For the by-hand build of Part V, turn off AI completion in the editor that you use. Thonny ships without AI completion.
Part V · The remake unit
Unit overview
This unit accompanies Modules 7, 8 and 9 and follows Assignment 3. It takes two sessions, in weeks 5 and 6.
In this unit you build Bricks twice. Bricks is the brick-and-paddle game from Assignment 3, and both builds must satisfy the same contract. You write Build 1 by hand, without a language model. An agent writes Build 2 from a one-page brief. You time both builds with the same stopwatch, run the same five stage checks on both, and measure both files with one script. At the end of the unit you have two versions of game.py, a timing log, a completed worksheet and a written comparison of the two versions.
The starter kit for this unit is remake-bricks.zip. The editors are listed under Starter code and tools.
| Fixed for both builds | Different between the two builds |
|---|---|
engine.py, which provides the window, the 60-steps-a-second clock and the replay runner | The author of game.py |
The contract, which defines the two functions new_game and step and a state with seven keys | The sources that the author may consult |
The five stages and check.py, which checks each stage through new_game and step | The work that is on the clock |
stopwatch.py and compare.py |
new_game(seed, config) returns the starting state, which holds the wall, the paddle, the ball resting on the paddle, three lives and the status "ready". step(state, target_x, launch) advances the game by one sixtieth of a second. game.py contains only the rules of the game. It opens no window, reads no clock and prints nothing (9.2). As a result, check.py can run game.py without a display, and a recorded session replays exactly.
Do Build 1 before Build 2. If you have already seen the agent's collision code, you can reproduce that code from memory, and your Build 1 time then measures recall.
Measures
| Measure | How the measure is taken |
|---|---|
| Time to first working version | Working minutes from start to the first run of check.py that prints 5 of 5 stages pass |
| Total time | Working minutes from start to stop. Pauses do not count. |
| Lines of code | compare.py counts every line that is not blank, a comment or part of a docstring. |
| Defects found | The number of bug entries in the log. Log one entry for each defect, whether you find it through a check, a replay, playing or reading. |
| Time spent fixing | Working minutes between each bug entry and its fixed entry |
| Stage checks passing | The number of stages that check.py passes, from 0 to 5 |
Every figure comes either from timing-log.csv, to which stopwatch.py only appends, or from a script that you run on the two files. We do not accept a figure typed from memory (Rule 3).
Build 1 · By hand without AI
Write Build 1 without a language model of any kind. Do not use a chat assistant, an agent or AI completion in the editor. You may use the Python documentation, a search engine and your own notes. Work individually.
python3 stopwatch.py start hand
python3 check.py hand # after each stage
python3 stopwatch.py first hand # when check.py first prints 5 of 5
python3 stopwatch.py bug hand "the ball sticks inside the paddle"
python3 stopwatch.py fixed hand
python3 stopwatch.py pause hand # a break; resume with: resume hand
python3 stopwatch.py stop hand
- Read the contract in the kit's
README.mdand the emptyhand/game.py. Reading is on the clock. - Write stage 1, the static scene, and run
python3 check.py hand. The check stops at the first stage that fails and prints the reason. - Write stages 2 to 5 in order: one moving object, input, collisions and rules. Run the check after each stage. Do not start a stage while an earlier stage fails (7.2).
- Open the window with
python3 play.py handand play one game until you win or lose. Log anything that is wrong as abug, with a description that a stranger could act on. - Record one session with
python3 play.py hand --record run.json. In Build 2 you will replay this recording on the agent's file. - Stop the clock when all five stages pass and you would hand in the file as it stands.
The stage 4 check places the ball inside the paddle, which happens in play when the paddle moves into the ball. The check fails a build in which the ball reverses on every step and never leaves the paddle.
Build 2 · With an agent
Use any coding agent, and run it under the harness of Module 6. The agent must not see hand/game.py.
python3 stopwatch.py start ai
python3 check.py ai
python3 stopwatch.py first ai
python3 stopwatch.py bug ai "catches every error in step and carries on"
python3 stopwatch.py fixed ai
python3 stopwatch.py stop ai
- Start the clock, then edit
ai/BRIEF.md. The kit's brief states the goal, the files to read, the constraints and the definition of done (7.1). Writing the brief is on the clock. - Give the agent the brief and the folder. The agent writes
ai/game.pyand runscheck.py aiitself. Keep the session log. - Log
firstwhencheck.pyfirst prints5 of 5. - Read every line of
ai/game.pybefore you accept the file. Your name goes on the file, and a named person is accountable for every change (7.5). Reading is on the clock. - Run
python3 compare.pyandpython3 play.py ai --replay run.json. The replay gives Build 2 the seed and the inputs that you recorded from Build 1. If the final states differ, the two builds disagree about a rule. - Work through the review table in
WORKSHEET.md. Log each finding as abugentry. Then fix the finding, or write down why it stays. - Stop the clock when you would put your name to the file.
Treat the agent's own report as a claim. The checks that the agent ran are the checks that it was given. Rule 1 applies here: you, the replay and compare.py are the second process.
Hand code and AI code compared
The two files in this section pass the same five stage checks. The left column shows the course's reference build, which was written stage by stage in the plain style that we require of the by-hand build. The right column shows the output of one agent session on 17 November 2026. The agent was given the kit's ai/BRIEF.md and the kit folder. The agent's file appears as the agent wrote it, and the page build reads the excerpts from the two files. In your own comparison, use your two files in place of the two columns.
Measurements of the two files
The following table is the output of python3 compare.py on the two files.
| Measure | Reference build | Agent build |
|---|---|---|
| Lines in the file | 170 | 208 |
| Lines of code | 126 | 133 |
| Comment lines | 8 | 13 |
| Docstring lines | 6 | 38 |
| Words of explanation per 10 lines of code | 7.1 | 4.1 |
| Functions | 12 | 13 |
| Functions with a docstring | 0 | 12 |
| Longest function (lines) | 21 | 23 |
| Deepest nesting | 2 | 3 |
| Functions defined but never called here | 0 | 0 |
| try / except blocks | 0 | 0 |
| ... that catch everything | 0 | 0 |
| isinstance / hasattr / .get( checks | 0 | 0 |
| Type hints on functions | 0 | 0 |
| Bare numbers inside functions | 1 | 0 |
| Repeated lines | 2 | 3 |
The two files hold nearly the same amount of code: 126 lines against 133. The files differ in length by 38 lines, and 32 of the 38 are docstrings. The one bare number in the reference build is the index in brick[3]. Use the counts to decide where to read. When you have read a counted line, decide whether that line is a defect.
Divergence on the same seed and inputs
We recorded five sessions on the reference build, with seeds 1 to 5 and a scripted paddle that follows the ball loosely, and replayed each session on the agent build. All five replays ended in a state that differed from the final state of the recording. The reference build and the agent build each pass stage 2, which requires that one seed gives one path. Each build is therefore deterministic, and the differences come from the rules of the two builds.
The serve. With seeds 1, 2 and 3, one build serves the ball to the left and the other build serves it to the right.
Reference build
def serve(state):
config = state["config"]
rng = state["rng"]
sideways = rng.uniform(config["speed_x_min"], config["speed_x_max"])
if rng.random() < 0.5:
sideways = -sideways
state["ball"]["vx"] = sideways
state["ball"]["vy"] = -config["speed_y"]
state["status"] = "playing"
Agent build
def _launch_ball(state):
"""Send the resting ball upward, left or right as the seeded rng decides."""
config, rng = state["config"], state["rng"]
sideways = rng.uniform(config["speed_x_min"], config["speed_x_max"])
state["ball"]["vx"] = sideways * rng.choice((-1, 1))
state["ball"]["vy"] = -config["speed_y"]
state["status"] = "playing"
Each build first draws the speed with rng.uniform. The reference build then picks the direction with rng.random() < 0.5, and the agent build picks the direction with rng.choice((-1, 1)). The contract says only that the direction is "chosen by rng". A replay recorded on one build cannot be used on the other build until the contract names the call.
The walls. With seed 5 the two builds agree for 84 steps. On step 85, at the right wall, they diverge by 0.08 of a pixel.
Reference build
def bounce_off_walls(state):
config = state["config"]
ball = state["ball"]
r = config["ball_r"]
# Put the ball back inside and point it away from the wall. Setting the
# sign (not flipping it) means a ball that is still outside next step
# cannot be turned round a second time.
if ball["x"] - r < 0:
ball["x"] = r
ball["vx"] = abs(ball["vx"])
if ball["x"] + r > config["width"]:
ball["x"] = config["width"] - r
ball["vx"] = -abs(ball["vx"])
if ball["y"] - r < 0:
ball["y"] = r
ball["vy"] = abs(ball["vy"])
Agent build
def _bounce_off_walls(ball, config):
"""Reflect the ball off the two side walls and the top."""
radius = config["ball_r"]
far_right = config["width"] - radius
if ball["x"] < radius:
ball["x"] = radius + (radius - ball["x"])
ball["vx"] = abs(ball["vx"])
elif ball["x"] > far_right:
ball["x"] = far_right - (ball["x"] - far_right)
ball["vx"] = -abs(ball["vx"])
if ball["y"] < radius:
ball["y"] = radius + (radius - ball["y"])
ball["vy"] = abs(ball["vy"])
The reference build puts the ball back against the wall. The agent build reflects the ball by the distance that it overshot the wall, which is the more exact rule. Stage 2 accepts both rules, because the wall check in stage 2 requires only that the ball stays inside the play area.
The bricks. With seed 4 the two builds agree for 79 steps. On step 80 each build removes the first brick. The reference build then sends the ball back down. The agent build reads the contact as a hit on the side of the brick, reverses the sideways direction of the ball and lets the ball carry on upward. The ball then removes a second brick on step 81 and a third brick on step 84.
Reference build
def hit_brick(state):
ball = state["ball"]
r = state["config"]["ball_r"]
for brick in state["bricks"]:
if overlaps(ball, r, brick[0], brick[1], brick[2], brick[3]):
state["bricks"].remove(brick)
ball["vy"] = -ball["vy"]
return # one brick per step
Agent build
def _bounce_off_bricks(state):
"""Remove at most one brick the ball overlaps and turn the ball away from it."""
ball, bricks = state["ball"], state["bricks"]
hit = _find_hit_brick(ball, bricks, state["config"]["ball_r"])
if hit is None:
return
dx, dy = _offset_from_rect(ball, bricks.pop(hit))
if abs(dx) > abs(dy):
ball["vx"] = math.copysign(ball["vx"], dx) # hit a side
elif dy != 0:
ball["vy"] = math.copysign(ball["vy"], dy) # hit the top or bottom
else:
ball["vy"] = -ball["vy"] # centre already inside
The contract says that the ball "bounces off" the bricks and that a hit brick is removed, "at most one per step". The agent build obeys both sentences of the contract and adds a rule of its own, which depends on the face of the brick that the ball hit. No stage check covers a side hit. The agent stated this gap in its report, in the part that lists what the checks do not cover.
What to look for in the agent's file
Read the agent's file against the list in the table and write down the line number of each finding. The third column records what this one run showed. A different agent, brief or day gives different findings, so fill in the third column from your own agent's file.
| Look for | Why it matters | In this run |
|---|---|---|
| A rule that the brief did not ask for | The rule has no check, so nothing shows that the rule works. | The agent added a rule for side hits on bricks and a rule that removes the most deeply overlapped brick first. |
| A rule that the brief asked for and the file lacks | The checks cover examples of the contract, and the sentences of the contract say more than the examples. | We found none when we read the file against the contract. |
| An error that the code catches and ignores | The game carries on in a state that nobody designed. | None. The file has 0 try / except blocks. |
| A check for something that cannot happen | The check hides which inputs the author thought were possible. | None. The file has 0 isinstance, hasattr or .get( calls. |
| A function that nobody calls | The function was written for a version of the file that no longer exists. | None |
| A comment or docstring that repeats the code | The reader reads the same thing twice and learns no reason for the code. | The agent build has docstrings on 12 of 13 functions. The reference build has no docstrings on its functions and has 8 comment lines, each of which gives a reason. |
| Text copied from the starter that is no longer true | The stale text shows that the author did not reread the file as a whole. | Line 2 still reads "This is the file you write." |
| A bare number that belongs in the config | The next person who changes the layout will miss the number. | None. The count is 0. |
| A claim in the report that you cannot confirm | The report is the agent's own account of its work. | The agent reported that it found and fixed one defect. The log shows the bug entry and the fixed entry 11 seconds apart. |
Read the reference build against the same list. While probing, the agent found that paddle_x was an integer when the paddle sat against a wall and a float everywhere else, and the agent fixed the inconsistency. The reference build still has the inconsistency: its replay prints 0 where the replay of the agent build prints 0.0. The stage checks pass both builds.
Improving the agent's file
- The contract. Close the contract before you change either build. Each disagreement above arises where the contract left a point open. Write the missing sentences: which call draws the direction, what a wall bounce does with the overshoot, and whether a brick has sides. Then change both builds to match the completed contract.
- Checks. Give every rule that you keep a check. If side hits stay, write the example first: a ball that arrives at the left face of a brick, and the velocity that you expect after the hit. Show that the example fails when the rule is broken (8.2).
- Deletions. Delete whatever the brief did not ask for and the check does not need. This includes the docstrings that restate a function's name and the stale header.
- Re-runs. After each change, run
python3 check.py aiand then the replay. If a fix changes the replay, the fix has changed the game. - The brief. Put each finding in the brief. You keep the brief after the session ends. The next run then starts from a sentence such as "the direction is drawn with one call to
rng.random()".
Timing worksheet
Fill in this worksheet from the output of python3 stopwatch.py report. The report command reads timing-log.csv, counts the lines in each game.py and runs the stage checks. The kit's WORKSHEET.md holds the same table, with the review table and the defect table below it.
| Measure | By hand | With AI | Hand ÷ AI |
|---|---|---|---|
| Time to first working version (min) | |||
| Total time (min) | |||
| Lines of code in game.py | |||
| Defects found | |||
| Time spent fixing (min) | |||
| Stage checks passing (of 5) |
Then split the total time of the AI build into the following parts, using the log and the session.
| Writing the brief | Waiting for the agent | Reading its code | Fixing | Total |
|---|---|---|---|---|
One measured run
The agent build shown above was timed with the kit's stopwatch. The first table gives the log of that run.
| Event | Clock time | Working minutes from start |
|---|---|---|
start | 06:37:15 | 0.0 |
first: check.py ai prints 5 of 5 | 06:39:46 | 2.5 |
bug: paddle_x an integer at the walls | 06:40:25 | 3.2 |
fixed | 06:40:36 | 3.4 |
stop | 06:40:43 | 3.5 |
| Measure | With AI |
|---|---|
| Time to first working version | 2.5 min |
| Total time | 3.5 min |
| Lines of code in game.py | 133 |
Runs of check.py that failed before the first 5 of 5, according to the agent's report | 0 |
| Defects found | 1 |
| Time spent fixing | 0.2 min |
| Stage checks passing | 5 of 5 |
The 3.5 minutes cover the agent's work alone. The brief was already written when the clock started, and nobody read the code while the clock ran. The review in the section above, which found the three disagreements and the stale header, came afterwards and is not in the 3.5 minutes. Your Build 2 time includes writing the brief and reviewing the code, and the worksheet splits the Build 2 total for that reason.
After this run we added one check to the kit's stage 4: a ball that overlaps two bricks may remove only one of the two bricks in a step. The reference build and the agent build pass the new check.
We print no by-hand time here. The reference build was not written against the stopwatch, so a time for the reference build would be invented. Fill in the by-hand column from your own measurements.
Reading the worksheet
- Divide the by-hand time by the AI time, once for the first working version and once for the total. The two ratios are the measured speed-up of the AI route on this task.
- Find the largest cell in the split of the AI build. If the largest cell is reading or fixing, verification took more of the AI build's time than any other part (8.1).
- Compare the number of defects you found with the number of defects that the checks found. A defect that you found only by reading or by replay is a defect that the checks passed.
- For the class as a whole, collect the two ratios from every student and report the median and the range. The ratios of one student come from one run.
References
Reading list
Foundations and history
- Turing, A. M. (1950). Computing machinery and intelligence. Mind, 59(236).
- Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27.
- Shannon, C. E. (1951). Prediction and entropy of printed English. Bell System Technical Journal, 30.
- McCulloch, W. S., and Pitts, W. (1943). A logical calculus of the ideas immanent in nervous activity. Bulletin of Mathematical Biophysics, 5.
- Rosenblatt, F. (1958). The perceptron: a probabilistic model for information storage and organization in the brain. Psychological Review, 65(6).
- Minsky, M., and Papert, S. (1969). Perceptrons. MIT Press.
- Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). Learning representations by back-propagating errors. Nature, 323.
- Hochreiter, S., and Schmidhuber, J. (1997). Long short-term memory. Neural Computation, 9(8).
- Bengio, Y., Ducharme, R., Vincent, P., and Jauvin, C. (2003). A neural probabilistic language model. Journal of Machine Learning Research, 3.
- Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). ImageNet classification with deep convolutional neural networks. NeurIPS.
- Sutton, R. (2019). The bitter lesson. Essay.
The Transformer and the scaling era
- Mikolov, T., et al. (2013). Efficient estimation of word representations in vector space. arXiv:1301.3781.
- Sutskever, I., Vinyals, O., and Le, Q. V. (2014). Sequence to sequence learning with neural networks. NeurIPS.
- Bahdanau, D., Cho, K., and Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv:1409.0473.
- Sennrich, R., Haddow, B., and Birch, A. (2016). Neural machine translation of rare words with subword units. ACL.
- Vaswani, A., et al. (2017). Attention is all you need. NeurIPS.
- Radford, A., et al. (2018). Improving language understanding by generative pre-training. OpenAI.
- Devlin, J., et al. (2018). BERT: pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805.
- Brown, T., et al. (2020). Language models are few-shot learners. NeurIPS.
- Kaplan, J., et al. (2020). Scaling laws for neural language models. arXiv:2001.08361.
- Hoffmann, J., et al. (2022). Training compute-optimal large language models. arXiv:2203.15556.
- Christiano, P., et al. (2017). Deep reinforcement learning from human preferences. NeurIPS.
- Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS.
- Bai, Y., et al. (2022). Constitutional AI: harmlessness from AI feedback. arXiv:2212.08073.
- Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. NeurIPS.
- Wei, J., et al. (2022). Chain-of-thought prompting elicits reasoning in large language models. NeurIPS.
- Yao, S., et al. (2022). ReAct: synergizing reasoning and acting in language models. arXiv:2210.03629.
- Ho, J., Jain, A., and Abbeel, P. (2020). Denoising diffusion probabilistic models. NeurIPS.
- Hu, E., et al. (2021). LoRA: low-rank adaptation of large language models. arXiv:2106.09685.
- Rafailov, R., et al. (2023). Direct preference optimization. NeurIPS.
- Leviathan, Y., Kalman, M., and Matias, Y. (2023). Fast inference from transformers via speculative decoding. ICML.
Method and measurement
- Heuer, R. J. (1999). Psychology of Intelligence Analysis. Center for the Study of Intelligence. (Analysis of Competing Hypotheses.)
- Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1).
- Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1).
- Ohno, T. (1988). Toyota Production System: Beyond Large-Scale Production. Productivity Press.
- Goldratt, E. M. (1984). The Goal. North River Press. (Theory of constraints.)
- Martin, R. C. (2017). Clean Architecture. Prentice Hall. (The dependency rule.)
- Nystrom, R. (2014). Game Programming Patterns. Genever Benning. (Game loop, state, update method.)
- Spolsky, J. (2006). The development abstraction layer. Essay.
- Graham, P. (2008). You weren't meant to have a boss. Essay.
- Google Testing Blog (2015). Change-detector tests considered harmful.
- Anselmi, G., et al. (2024). Watching TV with the second-party: a first look at automatic content recognition tracking in smart TV ecosystems. arXiv:2409.06203.
- MIT CSAIL (2026). The Missing Semester of Your CS Education. Course notes.
Sources
In addition to the works above, the practicum draws on engineering write-ups, tool and skill documentation, incident accounts and recorded talks from the sixty days ending 17 November 2026. The text names a tool or a standard where you need the name to find it. A figure that the text attributes to a vendor, a maintainer or a single practitioner is that source's own figure, and we have not reproduced it independently for this course.