Software Engineering with Language Models
Module 2 · The agent harness: tools and skills under oversight
- Class time
- 60 minutes
- Slides
- 31, each with teacher’s notes
- Goes with
- Problem set 2; Assignment 1 (Part IV)
Where this module sits
- Module 1 defined the model, which keeps no state between calls and has the prompt as its only channel.
- This class defines the harness, which is the program around the model.
- You locate each rule or capability in the part that can enforce it.
- In section 2.3 we apply Rule 2 of the course to a reported case.
- Module 3 follows with labels, and Module 6 returns to the harness in Part III.
Objectives
- Define harness and agent, and trace one pass of the agent loop.
- Define the twelve parts of a harness and say what each controls.
- Place a rule in an instruction file, a skill or a hook according to cost and strength.
- Assign an oversight mode to each class of action an agent can take.
- Distinguish vibe coding from AI-assisted engineering.
- Review an agent product against the nine standard components.
Agenda
- 4 min: where we are and the objectives.
- 12 min: 2.1 harness, agent and the loop.
- 16 min: 2.2 the twelve parts.
- 9 min: 2.3 instruction file, skill or hook.
- 12 min: 2.4 oversight and 2.5 standard components.
- 7 min: common mistakes, work due and recap.
2.1 The harness
- "Harness": the program that wraps a model.
- It assembles the prompt.
- It sends the prompt to the model.
- It interprets the output.
- It executes any tool the model requested, appends the result, and repeats.
- Permissions, sandboxing, memory and the user interface also belong to it.
The agent
- "Agent": a model running in a harness loop with tools, pursuing a goal over many steps.
- Step 1: read context.
- Step 2: decide.
- Step 3: act through a tool.
- Step 4: observe the result.
- Repeat until done or stopped.
Diagram of the agent loop
+--------------------+
+--> | read context |
| | decide (one call) |
| +--------------------+
| | tool call? no --> done
| v yes
| +--------------------+
| | harness runs tool |
+--- | result appended |
+--------------------+
Worked example: one task through the loop
goal: fix the failing date test
1 model -> call run_tests()
2 harness -> runs it, appends output
3 model -> call read_file(dates.py)
4 harness -> appends 80 lines
5 model -> call edit_file(dates.py)
6 harness -> applies edit, appends ok
7 model -> call run_tests()
8 harness -> appends "all passed"
9 model -> "Fixed", no tool: done
Differences between harnesses
- About two hundred lines of code are enough for the core loop.
- Harnesses differ in which tools exist.
- They differ in how context is managed over hours.
- They differ in what is asked of the human and when.
- Two harnesses running the same model produce different work.
- A harness too strict for its model wastes the model, and a harness too loose produces unreviewed damage.
2.2 Tool and tool call
- Tool: a function the model may ask the harness to run.
- A tool is described by a name, a description and an input schema.
- The model sees only that description when deciding whether to call it.
- Tool call: model output that the harness interprets as a request to run a tool.
- The model never executes anything itself.
The model's view of a tool
name: close_ticket
description: |
Close one support ticket that
the user names by its id.
Not for closing many tickets.
Not for reading a ticket.
input:
ticket_id: string, required
reason: string, required
In-class exercise: a tool description
- The made-up function is
send_invoice(customer_id, amount). - In pairs, take four minutes to write its name, description and input schema.
- Write two requests on which it should fire.
- Write two requests on which it must not fire.
- Swap with another pair and try to make their description misfire.
MCP and the instruction file
- Model Context Protocol (MCP): an open protocol for offering tools and data to a harness.
- A server offers tools and data to any compatible harness.
- The connection is local over standard input/output or remote over HTTP.
- Instruction file: a file loaded in full at the start of every session.
- Two examples of instruction files are
AGENTS.mdandCLAUDE.md. - An instruction file holds recurring project rules: how to run checks and what never to do.
A skill on disk and in context
release-notes/
SKILL.md body: loads on match
scripts/ optional
references/ optional
examples/ optional
Always in context:
name + one-line description
Loaded when the task matches:
the body of SKILL.md
Three terms for control
| Term | Definition | Effect |
|---|---|---|
| Subagent | A second agent started by the first, with its own fresh context | Returns only its conclusion |
| Hook | Code the harness runs at a fixed point, for example before every tool call | Can allow, modify or block the action |
| Sandbox | An environment enforced by the operating system | Limits what agent-run code can read, write and reach on the network |
Four terms for managing context
| Term | Definition | Follows from |
|---|---|---|
| Compaction | Replacing older conversation with a summary | The window is a hard budget |
| Memory | Any store the harness reads back into context in a later session | The model keeps no state between calls |
| Context engineering | Deciding what goes into the window, in what form, and when | Irrelevant text lowers quality before the limit |
| Progressive disclosure | A small index always loaded; detail fetched on demand | Always-loaded text is paid for on every request |
2.3 Three mechanisms for rules
| Mechanism | Context cost | Strength | Suits |
|---|---|---|---|
| Instruction file | Tokens on every request | Advisory | A short list of standing facts |
| Skill | Almost nothing until used | Advisory | A procedure needed occasionally |
| Hook | None | Deterministic | A behaviour that must always hold |
Worked example: the file-read rule
- The goal is for a cheap model to summarise routine file reads.
- The purpose is to keep file contents out of the expensive model's context.
- The first attempt was a rule written in the instruction file.
- The agent read the rule and then opened the files anyway.
- The second attempt was a hook that intercepts the read call.
- With the hook, the rule held.
Sketch of the hook
def before_tool_call(call):
if call.name == "read_file":
text = read(call.path)
note = cheap_summary(text)
return replace(call, note)
return allow(call)
Worked example: placing five rules
| Rule (made-up) | Place | Reason |
|---|---|---|
Tests run with ./check.sh | Instruction file | Standing fact, needed every session |
| How to cut a release, twelve steps | Skill | Procedure, needed occasionally |
Never edit files under migrations/ | Hook | Must always hold |
| Use British spelling in documentation | Instruction file | Standing preference; a lapse is cheap |
| Never push to the main branch | Hook or permission | Must always hold |
2.4 Oversight vocabulary
- Human in the loop: each consequential action waits for a person's approval.
- Human on the loop: the agent acts while a person monitors and can step in.
- Autonomous or unattended: nobody watches the run, and review comes afterwards.
- As autonomy rises, control thins.
- Accountability does not move.
A policy per action class
- A sound policy states which mode applies to each action class.
- Irreversible classes go in the first mode, which is human in the loop.
- Spending money is irreversible.
- Publishing is irreversible.
- Deleting is irreversible.
- Sending on someone's behalf is irreversible.
Ask the room: oversight for a documentation agent
- In this made-up case, an agent maintains a team's documentation site.
- Its action classes are reading pages, drafting an edit, publishing a page, deleting a page and emailing subscribers.
- Assign each class one mode.
- Give the reason for each in one sentence.
- Name who answers for an unattended action that goes wrong.
Vibe coding
- "Vibe coding": describing what you want and accepting the model's changes without reading the code.
- The term was coined in early 2025.
- It was coined for throwaway projects.
- Its defining feature is that the diff is not read.
- AI-assisted engineering is a different practice, in which people read, test and review the changes.
2.5 The nine standard components
1 the loop
2 connectors and tools
3 context management
4 memory
5 a sandbox
6 web search
7 computer use (operating a screen)
8 background or scheduled execution
9 notifications
In-class exercise: a product review
- Pick an agent product you use.
- Mark each of the nine components as present, absent or unknown.
- For each absent component, say whether the absence looks deliberate.
- Name one thing the product does above the baseline.
- Work alone for three minutes, then compare with a neighbour.
Common mistakes
- Do not put a must-always rule in the instruction file. Use a hook or a permission.
- Do not put a long occasional procedure in the instruction file. Write it as a skill.
- Do not say the model runs tools. The model requests a tool and the harness executes it.
- Do not write a tool description that restates the name. Say when to call the tool and when not to.
- Do not call all AI-assisted work vibe coding. The test is whether the diff is read.
- Do not treat autonomy as a transfer of accountability. Accountability does not move.
Problem set 2: the three questions
- Q1: take five rules from an instruction file you use and classify each as instruction, skill or hook.
- Q1: move every misplaced rule to the right mechanism.
- Q2: write a tool description for a function that issues refunds.
- Q2: write three requests on which the tool should fire and three on which it must not.
- Q3: list every action class of an agent with access to your email and assign each an oversight mode.
Assignment 1 (Part IV): Karel the agent
- The restricted harness allows one instruction file of at most 30 lines and plain typed prompts.
- The tools are the built-in ones for reading files, editing files and running one test command.
- A world is a branch with a known starting state. Make a fresh branch for each run.
- Each solution is a list of named subtasks, with one prompt per subtask.
- The four problems are Fetch and return; Repair, never duplicate; Cover the grid; and Find the midpoint.
- For each problem, hand in the prompts, the session log, the verifying commands and what was not checked.
Recap
- The harness assembles the prompt, runs the loop and executes every tool.
- An agent is a model in that loop with tools and a goal over many steps.
- The model sees a tool as a name, a description and an input schema.
- An instruction file and a skill are advisory, and a hook is deterministic.
- A person approves every irreversible action before it happens.
- Accountability stays with people at every level of autonomy.
Before next class
- Read Module 3, "Semantic labeling and the data behind models".
- Finish Problem set 2 with the instruction file you brought.
- Set up the restricted harness and a small repository for Assignment 1.
- Write the subtask list for problem 1 of Assignment 1 before typing any prompt.
- Choose a corpus of at least 500 items you care about for Laboratory 1.