Software Engineering with Language Models
Module 16 · Capstone and assessment
- Class time
- 60 minutes
- Slides
- 23, each with teacher’s notes
- Goes with
- The capstone project; laboratory modules 9 to 15
Where this module sits
- Modules 1 to 8 supplied vocabulary, mechanism, harness, workflow and verification.
- Modules 9 to 15 are the laboratory modules.
- Laboratories are graded on evidence.
- The capstone is one piece of software that a real person uses.
- This class covers the project, the submission and the grading.
Objectives
- State the project requirement, including the three-module rule.
- List the eight submission items and say what each contains.
- Predict a grade from evidence, using the five weighted criteria.
- Choose a project with a real user and a recurring need.
- Leave with an outline of the brief, a module list and an ordered plan.
Agenda
- 3 min: where this module sits and the objectives.
- 5 min: the project and the three-module rule.
- 13 min: the eight submission items.
- 7 min: grading.
- 12 min: three worked examples on choosing, rejecting and marking.
- 20 min: planning exercises, the cost model, common mistakes and the recap.
The project
- Build, verify, secure and ship one piece of software.
- A real person uses it.
- You work with agents throughout.
- It includes a component from at least three of the laboratory modules 9 to 15.
The seven laboratory modules
9 making games
10 interfaces for people and agents
11 reverse engineering
12 network protection
13 cyber security
14 shipping and operating software
15 local models, fine-tuning, hardware
at least three must contribute
The eight submission items
- The brief, with the approved specification, architecture and plan.
- The harness configuration.
- The software, deployed, with its release script.
- The verification record.
- The security record.
- The cost model.
- The findings log.
- What was not checked.
Items 1 and 2 on plans and harness
- The brief is one page.
- You submit the approved versions of the specification, architecture and plan.
- The harness configuration has an instruction file, skills, hooks and permissions.
- A note states what each of the four parts enforces.
Items 3 and 4 on software and verification
- The software is deployed.
- Its release script is submitted with it.
- The tests pass the four-question gate.
- A calibrated grader or a replay test is included.
- An independent audit result is included, with finder and verifier separate.
Items 5 and 6 on the security record and the cost model
- Item 5 gives the baseline controls, each with evidence.
- Item 5 gives the endpoint inventory.
- Item 5 gives the spending cap.
- Item 6 gives the tokens and money spent building the project.
- Item 6 gives the cost to run the project per user per month.
Items 7 and 8 on the log and the limits
- The findings log is numbered.
- Each entry is dated.
- Each claim names the versions it was verified on.
- Item 8 lists what was not checked.
Grading
| Weight | Criterion |
|---|---|
| 25% | It works for its user, shown by a second real use after the first |
| 25% | Verification: evidence is independent, repeatable and honest about limits |
| 20% | Security and operations: least privilege, no exposed surface without reason, tested restore |
| 15% | Harness: rules placed where they are enforced; context used economically |
| 15% | Writing: the brief, the log and the report name the subject, the object and the action |
Reading the grading table
- Use and verification together carry 50 per cent.
- Security and operations carry 20 per cent.
- Harness and writing carry 15 per cent each.
- A project that claims more than its evidence supports is graded on the evidence.
Worked example: choosing a project
- Begin with a real person who has a recurring need.
- In this made-up case a club secretary searches past meeting notes each month.
- The need recurs monthly, which gives the occasion for a second real use.
- The components are a readable results page (10), no open ports (12), an audit (13) and a release script (14).
- Check each of the eight items against the project before committing.
Worked example: a proposal to turn down
- The made-up proposal is a story game with model-written scenes, "for anyone".
- It has no particular user, so no second real use can be shown.
- It draws on one laboratory module, 9, where three are required.
- It makes a model call per scene and has no cost model or spending cap.
- To repair it, find a player, serve the game (12, 14), audit it (13) and cap the spend.
Worked example: marking a submission
criterion evidence mark
user 25% one use, no 2nd partial
verify 25% finder = checker low
secure 20% cap, no restore partial
harness 15% hook + note full
writing 15% clear log full
graded on the evidence
Ask the room: evidence of a second use
- "She said she would definitely use it again."
- "He opened it twice during the demonstration."
- "She returned the following week with a new task and completed it."
- "The agent finished another real assignment under the intended supervision."
- Place each statement on the evidence ladder of 14.7.
In-class exercise: an outline of the brief
- The goal, more ambitious than the first instinct.
- A few concrete use cases.
- Where it fits, and the design considerations.
- How success will be judged: the observable evidence.
- The latitude the agent has to exceed the brief.
In-class exercise: the order of the work
phase artifact gate
specify requirements you approve
architect flows, failures you approve
plan small tasks you approve
build test, then code checks
review two passes findings
ship deploy, monitor you approve
The cost model in outline
build
tokens used
money spent
run, per user per month
sessions per month
x cost per session
+ share of hosting
= cost per user per month
Common mistakes
- A project with no real user. Instead, choose the person first.
- One use reported as proof. Instead, record the second real use, with its date.
- An agent that audits its own work. Instead, separate the finder and the verifier.
- Rules written only in the instruction file. Instead, use hooks and permissions, with a note.
- An empty "not checked" list. Instead, write what was not measured.
- Reports full of abstract nouns. Instead, name the subject, the object and the action.
Recap
- The project is one piece of software that a real person uses, built with agents throughout and drawing on three laboratory modules.
- The eight items are plans, harness, software, verification, security, cost, log and limits.
- A second real use and independent, repeatable verification carry 50 per cent.
- Security and operations carry 20 per cent, the harness 15 and writing 15.
- A project that claims more than its evidence supports is graded on the evidence.
- "What was not checked" is a submitted item.
After the course
- Keep the findings log going, with dates and versions.
- Re-run the evaluation after every model update.
- Repeat the audit, because a single run found about half.
- Restore from backup again, and time it.
- Watch cost daily while the service is live.
- Renew the inventory of services, keys and connectors.