Software Engineering with Language Models
Module 6 · Harness engineering
- Class time
- 60 minutes
- Slides
- 33, each with teacher’s notes
- Goes with
- Laboratory 2; Assignment 1 (Part IV); Section problems (Part IV)
Where this module sits
- Module 2 defined the harness, its parts and the choice between an instruction, a skill and a hook.
- Module 5 called the harness half of the system.
- In this class you configure a harness for a real repository.
- Rule 2 applies throughout the hour: hard rules go where they are enforced.
- In Laboratory 2 you apply the material to a repository you own.
- Module 7 then follows one feature from a brief to a verified release.
Objectives
- Set up an agent harness on a real repository.
- Apply the six context operations in the order 6.2 gives.
- Write instruction rules with a trigger, a required behaviour and a prohibited behaviour.
- Build a skill from its eight parts and test it by comparison.
- Write tool metadata that a model can select from, and test it with negative cases.
- Place every hard rule in a hook, a permission or a sandbox.
Agenda
- 7 min: the mental model and the stronger-model test.
- 10 min: six context operations and progressive disclosure.
- 6 min: instruction files.
- 13 min: skills, with the two-agent comparison.
- 9 min: tools and command lines for agents.
- 15 min: routing, sandboxes, sorting rules, work due, recap.
6.1 A working mental model in three layers
+------------------------------+
| skills, plugins: applications|
+------------------------------+
| harness: operating system |
+------------------------------+
| model: processor |
+------------------------------+
swapped every few months: model
carries over: harness and above
Ask the room: the stronger-model test
- The test asks whether a design improves or worsens when a stronger model ships next month.
- Case A: a prompt that tells the model to count list items twice.
- Case B: a hook that blocks any write outside the project directory.
- Case C: a skill whose script runs the build and reports failures.
- Classify each case as better, worse or unaffected.
6.1 A harness for one industry
- A specialised harness has four parts beyond the model.
- Connectors reach that field's data.
- People who know the work write the evaluations.
- Access rules state what the agent may touch and what it may not.
- A person reviews the result on a clean screen.
- Practitioners estimate the model at about a fifth of the system.
6.2 Context operations 1 to 3
- There are six operations, and you reach for them in a fixed order. These are the first three.
- Clear: start a new session for an unrelated task.
- Carrying old context forward costs tokens and accuracy.
- Rewind: when the agent has gone wrong, undo to the last good point.
- A correction typed as a message keeps the mistake in the window as an example.
- Delegate: send searches, web research and wide file reading to a subagent, which returns only the conclusion.
6.2 Context operations 4 to 6
- Disclose progressively: hold an index in context and load detail when needed.
- Compact: summarise older turns.
- Compaction loses information, so first write decisions and open questions to a file.
- Externalise: put durable results in files, an issue tracker or a ledger.
- Durable results are never kept only in the transcript.
- Documentation written for models is an llms.txt file and a Markdown page per component.
Worked example: one tool among 5,000
reported: 800 tool servers,
over 5,000 tools,
a gateway in front
1. find the right server
2. list that server's tools
3. load one tool's schema
4. call the tool
in context: two short lists and
one schema
6.3 Writing instruction files
- The instruction file is read on every request, so each line has to justify its tokens.
- A good rule has a trigger, a required behaviour and a prohibited behaviour.
- Scope: before adding speculative complexity, state the failure mode, its likelihood and its cost, and get agreement.
- Tests: write no change-detector tests, a term Module 8 defines.
- Age: review any instruction older than four months.
- Keep the file short and move procedures into skills.
Worked example: a vague rule rewritten
before: Be careful with complexity.
after:
trigger: about to add code for a
speculative race or an
unlikely security case
required: state failure mode,
likelihood, cost; ask
prohibited: growing scope silently
6.4 The skill folder
release-notes/
SKILL.md name, description,
standard, steps
references/ loaded when needed
examples/ gold results
scripts/
check.sh verifies the result
6.4 Anatomy of a skill (parts 1 to 4)
| Part | Purpose |
|---|---|
| Name and description | Always in context, and nothing else is; states which phrases trigger the skill and which requests must not |
| One standard | A single sentence that defines a good result |
| Named failure modes | A few named ways the work goes wrong, for the agent to check against |
| Steps | Most of them come before any output is produced |
6.4 Anatomy of a skill (parts 5 to 8)
| Part | Purpose |
|---|---|
| References | Topic files, loaded only when needed |
| Gold examples | Complete worked results to copy from; bad and good pairs used only for diagnosis |
| Script | A build, measurement or verification program that the skill runs |
| Evidence and handoff format | The items a report must contain, among them what went unchecked |
6.4 Design practices
- Give each skill one narrow job.
- A small router skill recommends which skill to use.
- Keep audit and review skills separate from build skills.
- A "learn" skill turns end-of-session corrections into written rules with reasons.
- Give a skill that needs memory across sessions an explicit state-export command.
In-class exercise: the two-agent comparison
- Give one short prompt to two fresh agents.
- One agent has the skill; the other does not.
- Compare the two results side by side.
- When something is wrong, write down what is wrong.
- Change the skill.
- Run the whole comparison again.
6.4 Skills as code you run
- A skill is code, and it runs with your permissions.
- Know which channel each skill came from.
- Know whether that channel still updates.
- A marketplace that stops updating third-party packs leaves installs stranded on old versions.
- Package names are squatted.
- Read a skill before installing it.
6.5 Tool metadata as a selection surface
- A model decides whether to call a tool from the tool's metadata alone.
- Give each tool an action-oriented name and a description that says when to use it.
- Make the annotations accurate and the input schema real.
- Acceptance tests cover negative cases too: requests that must not trigger the tool.
- The server enforces authorisation on each request.
- Check schemas with an inspector first, then test inside a host application.
Worked example: a test of one tool description
weak:
name: orders
desc: Handles order data.
better:
name: search_orders
desc: Use when the user asks
about an existing order
by number, email or date.
Not for creating orders.
6.5 Agents as command-line users
- Agents now use command-line tools as much as people do.
- At one infrastructure provider, agents were about a quarter of CLI usage in March 2026.
- By autumn the share was nearly half.
- Agents ran almost twice as many distinct commands per day as human users.
- The provider redesigned its CLI for that audience.
6.5 Five rules for an agent-facing CLI
- Generate the CLI from the API schema that drives the documentation.
- Output JSON by default, in a condensed form for agents.
- Ship command search inside the tool.
- Prefer typed configuration to untyped files.
- For a change in behaviour, ship a new tool.
6.6 Routing work to the cheapest adequate model
- A large share of session tokens goes to whole-file reads, boilerplate and configuration updates.
- On the read path a cheap model reads the file and returns a summary.
- On the write path a cheap model produces the boilerplate and saves it directly to disk.
- A claimed ninety per cent saving is second-hand and unverified.
- The cheap model missed a thread-safety defect that the strong model found quickly.
- Instrument each run so that token spend can be traced to a task and an outcome.
Worked example: routing five tasks
| Task | Route | Reason |
|---|---|---|
| Find where a setting is parsed in a long file | Cheap, read path | Reading; a summary answers it |
| Generate ten handler stubs of one shape | Cheap, write path | Boilerplate, written to disk |
| Decide whether a cache is safe under two threads | Strong | Concurrency reasoning |
| Debug a failing test | Strong, full read | Debugging; the escape applies |
| Review the final diff | Strong | Review |
6.7 Attached machines and blast radius
- A persistent machine has installed tools, logged-in sessions and a real browser.
- It makes an agent far more capable, because authentication is done once.
- Most command-line tools look for credentials in standard places.
- The blast radius becomes everything that machine can reach.
- The agent is exposed to instructions hidden in any page it reads.
- Keep three controls: prompts for credentials, limits on self-modification and allowlists.
6.7 Isolation by environment
host machine (your files, sessions)
+----------------------------------+
| guest VM or container |
| agent runs here |
| no outside network |
| allowlisted environment |
| resource limits |
| writes: scratch paths only |
+----------------------------------+
Ask the room: prompt or harness
- "Use British spelling in documentation."
- "Never write to the release branch."
- "Prefer small functions."
- "Never read files under secrets/."
- "Run the test command before reporting that work is done."
- Place each rule in an instruction file, a skill, a hook, a permission or a sandbox.
Common mistakes
- Correcting a wrong turn by message. Fix: rewind to the last good point.
- Letting the instruction file grow. Fix: prune it and move procedures into skills.
- Writing a hard rule as a sentence. Fix: put it in a hook, a permission or a sandbox.
- Repairing a skill's output by hand. Fix: change the skill and rerun the comparison.
- Trusting a file deny rule to stop the shell. Fix: put the sandbox behind it.
- Building around one model's quirks. Fix: apply the stronger-model test.
Laboratory 2
- Get a repository you own ready for agents to work in.
- Deliver an instruction file under 60 lines.
- Deliver one skill with a description, a standard, named failure modes, a script and a handoff format.
- Deliver one hook that blocks a class of action, and a permissions file.
- Run the 6.4 two-agent comparison on three tasks.
- Report where the skill helped, where it changed nothing, and what was pruned.
Assignment 1 (Part IV): Karel the agent
- The restricted harness is one instruction file of at most 30 lines and plain prompts.
- The tools read files, edit files and run one test command.
- Use no skills, hooks, subagents, tool servers, web access or background tasks.
- Each solution is a list of named subtasks with one prompt per subtask.
- There are four problems, and each runs on fresh branches called worlds.
- Hand in the prompts, the session log, the verifying commands and the list of what was not checked.
Section problems (Part IV): problems 3 and 5 to 7
- A small group works short problems in under an hour and writes the expected results before the code.
- Problem 3 (number the lines) asks why file tools number lines for a model (6.5).
- Problem 5 (usage ledger) totals tokens by model and task from a tab-separated log (6.6).
- Problem 6 (fastest one-off) finds the lowest latency that no other call shares.
- Problem 7 (a budget object) is a ContextBudget class tested only through its methods.
- The starter kit has a usage log of 120 lines, a latency log of 80 lines, and ContextBudget with its test file.
Recap
- The model is the replaceable processor, and the harness is the layer that carries over.
- The context operations in order are clear, rewind, delegate, disclose, compact and externalise.
- An instruction rule has a trigger, a required behaviour and a prohibited behaviour.
- A skill has eight parts and is developed by two-agent comparison.
- Tool metadata is a selection surface, and authorisation is enforced on the server.
- Hard rules go in hooks, permissions and sandboxes (Rule 2).
Before next class
- Start Laboratory 2 with the instruction file, then the hook.
- Begin Assignment 1 with problem 1, the fixed world.
- Read Module 7, sections 7.1 to 7.6.
- Choose one feature to take through Laboratory 3.
- Write its goal in one sentence, and make the goal more ambitious than your first instinct.