Syllabus Module 2 · The agent harness: tools and skills under oversight
NSW 3011 / 31
Software Engineering with Language Models

Module 2 · The agent harness: tools and skills under oversight

Class time
60 minutes
Slides
31, each with teacher’s notes
Goes with
Problem set 2; Assignment 1 (Part IV)
Module 22 / 31

Where this module sits

  • Module 1 defined the model, which keeps no state between calls and has the prompt as its only channel.
  • This class defines the harness, which is the program around the model.
  • You locate each rule or capability in the part that can enforce it.
  • In section 2.3 we apply Rule 2 of the course to a reported case.
  • Module 3 follows with labels, and Module 6 returns to the harness in Part III.
Module 23 / 31

Objectives

  • Define harness and agent, and trace one pass of the agent loop.
  • Define the twelve parts of a harness and say what each controls.
  • Place a rule in an instruction file, a skill or a hook according to cost and strength.
  • Assign an oversight mode to each class of action an agent can take.
  • Distinguish vibe coding from AI-assisted engineering.
  • Review an agent product against the nine standard components.
Module 24 / 31

Agenda

  • 4 min: where we are and the objectives.
  • 12 min: 2.1 harness, agent and the loop.
  • 16 min: 2.2 the twelve parts.
  • 9 min: 2.3 instruction file, skill or hook.
  • 12 min: 2.4 oversight and 2.5 standard components.
  • 7 min: common mistakes, work due and recap.
Module 25 / 31

2.1 The harness

  • "Harness": the program that wraps a model.
  • It assembles the prompt.
  • It sends the prompt to the model.
  • It interprets the output.
  • It executes any tool the model requested, appends the result, and repeats.
  • Permissions, sandboxing, memory and the user interface also belong to it.
Module 26 / 31

The agent

  • "Agent": a model running in a harness loop with tools, pursuing a goal over many steps.
  • Step 1: read context.
  • Step 2: decide.
  • Step 3: act through a tool.
  • Step 4: observe the result.
  • Repeat until done or stopped.
Module 27 / 31

Diagram of the agent loop

      +--------------------+
 +--> | read context       |
 |    | decide (one call)  |
 |    +--------------------+
 |        | tool call?  no --> done
 |        v yes
 |    +--------------------+
 |    | harness runs tool  |
 +--- | result appended    |
      +--------------------+
Module 28 / 31

Worked example: one task through the loop

goal: fix the failing date test
1 model   -> call run_tests()
2 harness -> runs it, appends output
3 model   -> call read_file(dates.py)
4 harness -> appends 80 lines
5 model   -> call edit_file(dates.py)
6 harness -> applies edit, appends ok
7 model   -> call run_tests()
8 harness -> appends "all passed"
9 model   -> "Fixed", no tool: done
Module 29 / 31

Differences between harnesses

  • About two hundred lines of code are enough for the core loop.
  • Harnesses differ in which tools exist.
  • They differ in how context is managed over hours.
  • They differ in what is asked of the human and when.
  • Two harnesses running the same model produce different work.
  • A harness too strict for its model wastes the model, and a harness too loose produces unreviewed damage.
Module 210 / 31

2.2 Tool and tool call

  • Tool: a function the model may ask the harness to run.
  • A tool is described by a name, a description and an input schema.
  • The model sees only that description when deciding whether to call it.
  • Tool call: model output that the harness interprets as a request to run a tool.
  • The model never executes anything itself.
Module 211 / 31

The model's view of a tool

name: close_ticket
description: |
  Close one support ticket that
  the user names by its id.
  Not for closing many tickets.
  Not for reading a ticket.
input:
  ticket_id: string, required
  reason: string, required
Module 212 / 31

In-class exercise: a tool description

  • The made-up function is send_invoice(customer_id, amount).
  • In pairs, take four minutes to write its name, description and input schema.
  • Write two requests on which it should fire.
  • Write two requests on which it must not fire.
  • Swap with another pair and try to make their description misfire.
Module 213 / 31

MCP and the instruction file

  • Model Context Protocol (MCP): an open protocol for offering tools and data to a harness.
  • A server offers tools and data to any compatible harness.
  • The connection is local over standard input/output or remote over HTTP.
  • Instruction file: a file loaded in full at the start of every session.
  • Two examples of instruction files are AGENTS.md and CLAUDE.md.
  • An instruction file holds recurring project rules: how to run checks and what never to do.
Module 214 / 31

A skill on disk and in context

release-notes/
  SKILL.md      body: loads on match
  scripts/      optional
  references/   optional
  examples/     optional

Always in context:
  name + one-line description
Loaded when the task matches:
  the body of SKILL.md
Module 215 / 31

Three terms for control

TermDefinitionEffect
SubagentA second agent started by the first, with its own fresh contextReturns only its conclusion
HookCode the harness runs at a fixed point, for example before every tool callCan allow, modify or block the action
SandboxAn environment enforced by the operating systemLimits what agent-run code can read, write and reach on the network
Module 216 / 31

Four terms for managing context

TermDefinitionFollows from
CompactionReplacing older conversation with a summaryThe window is a hard budget
MemoryAny store the harness reads back into context in a later sessionThe model keeps no state between calls
Context engineeringDeciding what goes into the window, in what form, and whenIrrelevant text lowers quality before the limit
Progressive disclosureA small index always loaded; detail fetched on demandAlways-loaded text is paid for on every request
Module 217 / 31

2.3 Three mechanisms for rules

MechanismContext costStrengthSuits
Instruction fileTokens on every requestAdvisoryA short list of standing facts
SkillAlmost nothing until usedAdvisoryA procedure needed occasionally
HookNoneDeterministicA behaviour that must always hold
Module 218 / 31

Worked example: the file-read rule

  1. The goal is for a cheap model to summarise routine file reads.
  2. The purpose is to keep file contents out of the expensive model's context.
  3. The first attempt was a rule written in the instruction file.
  4. The agent read the rule and then opened the files anyway.
  5. The second attempt was a hook that intercepts the read call.
  6. With the hook, the rule held.
Module 219 / 31

Sketch of the hook

def before_tool_call(call):
    if call.name == "read_file":
        text = read(call.path)
        note = cheap_summary(text)
        return replace(call, note)
    return allow(call)
Module 220 / 31

Worked example: placing five rules

Rule (made-up)PlaceReason
Tests run with ./check.shInstruction fileStanding fact, needed every session
How to cut a release, twelve stepsSkillProcedure, needed occasionally
Never edit files under migrations/HookMust always hold
Use British spelling in documentationInstruction fileStanding preference; a lapse is cheap
Never push to the main branchHook or permissionMust always hold
Module 221 / 31

2.4 Oversight vocabulary

  • Human in the loop: each consequential action waits for a person's approval.
  • Human on the loop: the agent acts while a person monitors and can step in.
  • Autonomous or unattended: nobody watches the run, and review comes afterwards.
  • As autonomy rises, control thins.
  • Accountability does not move.
Module 222 / 31

A policy per action class

  • A sound policy states which mode applies to each action class.
  • Irreversible classes go in the first mode, which is human in the loop.
  • Spending money is irreversible.
  • Publishing is irreversible.
  • Deleting is irreversible.
  • Sending on someone's behalf is irreversible.
Module 223 / 31

Ask the room: oversight for a documentation agent

  • In this made-up case, an agent maintains a team's documentation site.
  • Its action classes are reading pages, drafting an edit, publishing a page, deleting a page and emailing subscribers.
  • Assign each class one mode.
  • Give the reason for each in one sentence.
  • Name who answers for an unattended action that goes wrong.
Module 224 / 31

Vibe coding

  • "Vibe coding": describing what you want and accepting the model's changes without reading the code.
  • The term was coined in early 2025.
  • It was coined for throwaway projects.
  • Its defining feature is that the diff is not read.
  • AI-assisted engineering is a different practice, in which people read, test and review the changes.
Module 225 / 31

2.5 The nine standard components

1 the loop
2 connectors and tools
3 context management
4 memory
5 a sandbox
6 web search
7 computer use (operating a screen)
8 background or scheduled execution
9 notifications
Module 226 / 31

In-class exercise: a product review

  • Pick an agent product you use.
  • Mark each of the nine components as present, absent or unknown.
  • For each absent component, say whether the absence looks deliberate.
  • Name one thing the product does above the baseline.
  • Work alone for three minutes, then compare with a neighbour.
Module 227 / 31

Common mistakes

  • Do not put a must-always rule in the instruction file. Use a hook or a permission.
  • Do not put a long occasional procedure in the instruction file. Write it as a skill.
  • Do not say the model runs tools. The model requests a tool and the harness executes it.
  • Do not write a tool description that restates the name. Say when to call the tool and when not to.
  • Do not call all AI-assisted work vibe coding. The test is whether the diff is read.
  • Do not treat autonomy as a transfer of accountability. Accountability does not move.
Module 228 / 31

Problem set 2: the three questions

  • Q1: take five rules from an instruction file you use and classify each as instruction, skill or hook.
  • Q1: move every misplaced rule to the right mechanism.
  • Q2: write a tool description for a function that issues refunds.
  • Q2: write three requests on which the tool should fire and three on which it must not.
  • Q3: list every action class of an agent with access to your email and assign each an oversight mode.
Module 229 / 31

Assignment 1 (Part IV): Karel the agent

  • The restricted harness allows one instruction file of at most 30 lines and plain typed prompts.
  • The tools are the built-in ones for reading files, editing files and running one test command.
  • A world is a branch with a known starting state. Make a fresh branch for each run.
  • Each solution is a list of named subtasks, with one prompt per subtask.
  • The four problems are Fetch and return; Repair, never duplicate; Cover the grid; and Find the midpoint.
  • For each problem, hand in the prompts, the session log, the verifying commands and what was not checked.
Module 230 / 31

Recap

  • The harness assembles the prompt, runs the loop and executes every tool.
  • An agent is a model in that loop with tools and a goal over many steps.
  • The model sees a tool as a name, a description and an input schema.
  • An instruction file and a skill are advisory, and a hook is deterministic.
  • A person approves every irreversible action before it happens.
  • Accountability stays with people at every level of autonomy.
Module 231 / 31

Before next class

  • Read Module 3, "Semantic labeling and the data behind models".
  • Finish Problem set 2 with the instruction file you brought.
  • Set up the restricted harness and a small repository for Assignment 1.
  • Write the subtask list for problem 1 of Assignment 1 before typing any prompt.
  • Choose a corpus of at least 500 items you care about for Laboratory 1.