Syllabus Module 6 · Harness engineering
NSW 3011 / 33
Software Engineering with Language Models

Module 6 · Harness engineering

Class time
60 minutes
Slides
33, each with teacher’s notes
Goes with
Laboratory 2; Assignment 1 (Part IV); Section problems (Part IV)
Module 62 / 33

Where this module sits

  • Module 2 defined the harness, its parts and the choice between an instruction, a skill and a hook.
  • Module 5 called the harness half of the system.
  • In this class you configure a harness for a real repository.
  • Rule 2 applies throughout the hour: hard rules go where they are enforced.
  • In Laboratory 2 you apply the material to a repository you own.
  • Module 7 then follows one feature from a brief to a verified release.
Module 63 / 33

Objectives

  • Set up an agent harness on a real repository.
  • Apply the six context operations in the order 6.2 gives.
  • Write instruction rules with a trigger, a required behaviour and a prohibited behaviour.
  • Build a skill from its eight parts and test it by comparison.
  • Write tool metadata that a model can select from, and test it with negative cases.
  • Place every hard rule in a hook, a permission or a sandbox.
Module 64 / 33

Agenda

  • 7 min: the mental model and the stronger-model test.
  • 10 min: six context operations and progressive disclosure.
  • 6 min: instruction files.
  • 13 min: skills, with the two-agent comparison.
  • 9 min: tools and command lines for agents.
  • 15 min: routing, sandboxes, sorting rules, work due, recap.
Module 65 / 33

6.1 A working mental model in three layers

+------------------------------+
| skills, plugins: applications|
+------------------------------+
| harness: operating system    |
+------------------------------+
| model: processor             |
+------------------------------+
swapped every few months: model
carries over: harness and above
Module 66 / 33

Ask the room: the stronger-model test

  • The test asks whether a design improves or worsens when a stronger model ships next month.
  • Case A: a prompt that tells the model to count list items twice.
  • Case B: a hook that blocks any write outside the project directory.
  • Case C: a skill whose script runs the build and reports failures.
  • Classify each case as better, worse or unaffected.
Module 67 / 33

6.1 A harness for one industry

  • A specialised harness has four parts beyond the model.
  • Connectors reach that field's data.
  • People who know the work write the evaluations.
  • Access rules state what the agent may touch and what it may not.
  • A person reviews the result on a clean screen.
  • Practitioners estimate the model at about a fifth of the system.
Module 68 / 33

6.2 Context operations 1 to 3

  • There are six operations, and you reach for them in a fixed order. These are the first three.
  • Clear: start a new session for an unrelated task.
  • Carrying old context forward costs tokens and accuracy.
  • Rewind: when the agent has gone wrong, undo to the last good point.
  • A correction typed as a message keeps the mistake in the window as an example.
  • Delegate: send searches, web research and wide file reading to a subagent, which returns only the conclusion.
Module 69 / 33

6.2 Context operations 4 to 6

  • Disclose progressively: hold an index in context and load detail when needed.
  • Compact: summarise older turns.
  • Compaction loses information, so first write decisions and open questions to a file.
  • Externalise: put durable results in files, an issue tracker or a ledger.
  • Durable results are never kept only in the transcript.
  • Documentation written for models is an llms.txt file and a Markdown page per component.
Module 610 / 33

Worked example: one tool among 5,000

reported: 800 tool servers,
          over 5,000 tools,
          a gateway in front
1. find the right server
2. list that server's tools
3. load one tool's schema
4. call the tool
in context: two short lists and
            one schema
Module 611 / 33

6.3 Writing instruction files

  • The instruction file is read on every request, so each line has to justify its tokens.
  • A good rule has a trigger, a required behaviour and a prohibited behaviour.
  • Scope: before adding speculative complexity, state the failure mode, its likelihood and its cost, and get agreement.
  • Tests: write no change-detector tests, a term Module 8 defines.
  • Age: review any instruction older than four months.
  • Keep the file short and move procedures into skills.
Module 612 / 33

Worked example: a vague rule rewritten

before: Be careful with complexity.
after:
  trigger:  about to add code for a
            speculative race or an
            unlikely security case
  required: state failure mode,
            likelihood, cost; ask
  prohibited: growing scope silently
Module 613 / 33

6.4 The skill folder

release-notes/
  SKILL.md      name, description,
                standard, steps
  references/   loaded when needed
  examples/     gold results
  scripts/
    check.sh    verifies the result
Module 614 / 33

6.4 Anatomy of a skill (parts 1 to 4)

PartPurpose
Name and descriptionAlways in context, and nothing else is; states which phrases trigger the skill and which requests must not
One standardA single sentence that defines a good result
Named failure modesA few named ways the work goes wrong, for the agent to check against
StepsMost of them come before any output is produced
Module 615 / 33

6.4 Anatomy of a skill (parts 5 to 8)

PartPurpose
ReferencesTopic files, loaded only when needed
Gold examplesComplete worked results to copy from; bad and good pairs used only for diagnosis
ScriptA build, measurement or verification program that the skill runs
Evidence and handoff formatThe items a report must contain, among them what went unchecked
Module 616 / 33

6.4 Design practices

  • Give each skill one narrow job.
  • A small router skill recommends which skill to use.
  • Keep audit and review skills separate from build skills.
  • A "learn" skill turns end-of-session corrections into written rules with reasons.
  • Give a skill that needs memory across sessions an explicit state-export command.
Module 617 / 33

In-class exercise: the two-agent comparison

  1. Give one short prompt to two fresh agents.
  2. One agent has the skill; the other does not.
  3. Compare the two results side by side.
  4. When something is wrong, write down what is wrong.
  5. Change the skill.
  6. Run the whole comparison again.
Module 618 / 33

6.4 Skills as code you run

  • A skill is code, and it runs with your permissions.
  • Know which channel each skill came from.
  • Know whether that channel still updates.
  • A marketplace that stops updating third-party packs leaves installs stranded on old versions.
  • Package names are squatted.
  • Read a skill before installing it.
Module 619 / 33

6.5 Tool metadata as a selection surface

  • A model decides whether to call a tool from the tool's metadata alone.
  • Give each tool an action-oriented name and a description that says when to use it.
  • Make the annotations accurate and the input schema real.
  • Acceptance tests cover negative cases too: requests that must not trigger the tool.
  • The server enforces authorisation on each request.
  • Check schemas with an inspector first, then test inside a host application.
Module 620 / 33

Worked example: a test of one tool description

weak:
  name: orders
  desc: Handles order data.
better:
  name: search_orders
  desc: Use when the user asks
        about an existing order
        by number, email or date.
        Not for creating orders.
Module 621 / 33

6.5 Agents as command-line users

  • Agents now use command-line tools as much as people do.
  • At one infrastructure provider, agents were about a quarter of CLI usage in March 2026.
  • By autumn the share was nearly half.
  • Agents ran almost twice as many distinct commands per day as human users.
  • The provider redesigned its CLI for that audience.
Module 622 / 33

6.5 Five rules for an agent-facing CLI

  1. Generate the CLI from the API schema that drives the documentation.
  2. Output JSON by default, in a condensed form for agents.
  3. Ship command search inside the tool.
  4. Prefer typed configuration to untyped files.
  5. For a change in behaviour, ship a new tool.
Module 623 / 33

6.6 Routing work to the cheapest adequate model

  • A large share of session tokens goes to whole-file reads, boilerplate and configuration updates.
  • On the read path a cheap model reads the file and returns a summary.
  • On the write path a cheap model produces the boilerplate and saves it directly to disk.
  • A claimed ninety per cent saving is second-hand and unverified.
  • The cheap model missed a thread-safety defect that the strong model found quickly.
  • Instrument each run so that token spend can be traced to a task and an outcome.
Module 624 / 33

Worked example: routing five tasks

TaskRouteReason
Find where a setting is parsed in a long fileCheap, read pathReading; a summary answers it
Generate ten handler stubs of one shapeCheap, write pathBoilerplate, written to disk
Decide whether a cache is safe under two threadsStrongConcurrency reasoning
Debug a failing testStrong, full readDebugging; the escape applies
Review the final diffStrongReview
Module 625 / 33

6.7 Attached machines and blast radius

  • A persistent machine has installed tools, logged-in sessions and a real browser.
  • It makes an agent far more capable, because authentication is done once.
  • Most command-line tools look for credentials in standard places.
  • The blast radius becomes everything that machine can reach.
  • The agent is exposed to instructions hidden in any page it reads.
  • Keep three controls: prompts for credentials, limits on self-modification and allowlists.
Module 626 / 33

6.7 Isolation by environment

host machine (your files, sessions)
+----------------------------------+
| guest VM or container            |
|   agent runs here                |
|   no outside network             |
|   allowlisted environment        |
|   resource limits                |
|   writes: scratch paths only     |
+----------------------------------+
Module 627 / 33

Ask the room: prompt or harness

  • "Use British spelling in documentation."
  • "Never write to the release branch."
  • "Prefer small functions."
  • "Never read files under secrets/."
  • "Run the test command before reporting that work is done."
  • Place each rule in an instruction file, a skill, a hook, a permission or a sandbox.
Module 628 / 33

Common mistakes

  • Correcting a wrong turn by message. Fix: rewind to the last good point.
  • Letting the instruction file grow. Fix: prune it and move procedures into skills.
  • Writing a hard rule as a sentence. Fix: put it in a hook, a permission or a sandbox.
  • Repairing a skill's output by hand. Fix: change the skill and rerun the comparison.
  • Trusting a file deny rule to stop the shell. Fix: put the sandbox behind it.
  • Building around one model's quirks. Fix: apply the stronger-model test.
Module 629 / 33

Laboratory 2

  • Get a repository you own ready for agents to work in.
  • Deliver an instruction file under 60 lines.
  • Deliver one skill with a description, a standard, named failure modes, a script and a handoff format.
  • Deliver one hook that blocks a class of action, and a permissions file.
  • Run the 6.4 two-agent comparison on three tasks.
  • Report where the skill helped, where it changed nothing, and what was pruned.
Module 630 / 33

Assignment 1 (Part IV): Karel the agent

  • The restricted harness is one instruction file of at most 30 lines and plain prompts.
  • The tools read files, edit files and run one test command.
  • Use no skills, hooks, subagents, tool servers, web access or background tasks.
  • Each solution is a list of named subtasks with one prompt per subtask.
  • There are four problems, and each runs on fresh branches called worlds.
  • Hand in the prompts, the session log, the verifying commands and the list of what was not checked.
Module 631 / 33

Section problems (Part IV): problems 3 and 5 to 7

  • A small group works short problems in under an hour and writes the expected results before the code.
  • Problem 3 (number the lines) asks why file tools number lines for a model (6.5).
  • Problem 5 (usage ledger) totals tokens by model and task from a tab-separated log (6.6).
  • Problem 6 (fastest one-off) finds the lowest latency that no other call shares.
  • Problem 7 (a budget object) is a ContextBudget class tested only through its methods.
  • The starter kit has a usage log of 120 lines, a latency log of 80 lines, and ContextBudget with its test file.
Module 632 / 33

Recap

  • The model is the replaceable processor, and the harness is the layer that carries over.
  • The context operations in order are clear, rewind, delegate, disclose, compact and externalise.
  • An instruction rule has a trigger, a required behaviour and a prohibited behaviour.
  • A skill has eight parts and is developed by two-agent comparison.
  • Tool metadata is a selection surface, and authorisation is enforced on the server.
  • Hard rules go in hooks, permissions and sandboxes (Rule 2).
Module 633 / 33

Before next class

  • Start Laboratory 2 with the instruction file, then the hook.
  • Begin Assignment 1 with problem 1, the fixed world.
  • Read Module 7, sections 7.1 to 7.6.
  • Choose one feature to take through Laboratory 3.
  • Write its goal in one sentence, and make the goal more ambitious than your first instinct.