Syllabus Module 16 · Capstone and assessment
NSW 3011 / 23
Software Engineering with Language Models

Module 16 · Capstone and assessment

Class time
60 minutes
Slides
23, each with teacher’s notes
Goes with
The capstone project; laboratory modules 9 to 15
Module 162 / 23

Where this module sits

  • Modules 1 to 8 supplied vocabulary, mechanism, harness, workflow and verification.
  • Modules 9 to 15 are the laboratory modules.
  • Laboratories are graded on evidence.
  • The capstone is one piece of software that a real person uses.
  • This class covers the project, the submission and the grading.
Module 163 / 23

Objectives

  • State the project requirement, including the three-module rule.
  • List the eight submission items and say what each contains.
  • Predict a grade from evidence, using the five weighted criteria.
  • Choose a project with a real user and a recurring need.
  • Leave with an outline of the brief, a module list and an ordered plan.
Module 164 / 23

Agenda

  • 3 min: where this module sits and the objectives.
  • 5 min: the project and the three-module rule.
  • 13 min: the eight submission items.
  • 7 min: grading.
  • 12 min: three worked examples on choosing, rejecting and marking.
  • 20 min: planning exercises, the cost model, common mistakes and the recap.
Module 165 / 23

The project

  • Build, verify, secure and ship one piece of software.
  • A real person uses it.
  • You work with agents throughout.
  • It includes a component from at least three of the laboratory modules 9 to 15.
Module 166 / 23

The seven laboratory modules

 9  making games
10  interfaces for people and agents
11  reverse engineering
12  network protection
13  cyber security
14  shipping and operating software
15  local models, fine-tuning, hardware
at least three must contribute
Module 167 / 23

The eight submission items

  1. The brief, with the approved specification, architecture and plan.
  2. The harness configuration.
  3. The software, deployed, with its release script.
  4. The verification record.
  5. The security record.
  6. The cost model.
  7. The findings log.
  8. What was not checked.
Module 168 / 23

Items 1 and 2 on plans and harness

  • The brief is one page.
  • You submit the approved versions of the specification, architecture and plan.
  • The harness configuration has an instruction file, skills, hooks and permissions.
  • A note states what each of the four parts enforces.
Module 169 / 23

Items 3 and 4 on software and verification

  • The software is deployed.
  • Its release script is submitted with it.
  • The tests pass the four-question gate.
  • A calibrated grader or a replay test is included.
  • An independent audit result is included, with finder and verifier separate.
Module 1610 / 23

Items 5 and 6 on the security record and the cost model

  • Item 5 gives the baseline controls, each with evidence.
  • Item 5 gives the endpoint inventory.
  • Item 5 gives the spending cap.
  • Item 6 gives the tokens and money spent building the project.
  • Item 6 gives the cost to run the project per user per month.
Module 1611 / 23

Items 7 and 8 on the log and the limits

  • The findings log is numbered.
  • Each entry is dated.
  • Each claim names the versions it was verified on.
  • Item 8 lists what was not checked.
Module 1612 / 23

Grading

WeightCriterion
25%It works for its user, shown by a second real use after the first
25%Verification: evidence is independent, repeatable and honest about limits
20%Security and operations: least privilege, no exposed surface without reason, tested restore
15%Harness: rules placed where they are enforced; context used economically
15%Writing: the brief, the log and the report name the subject, the object and the action
Module 1613 / 23

Reading the grading table

  • Use and verification together carry 50 per cent.
  • Security and operations carry 20 per cent.
  • Harness and writing carry 15 per cent each.
  • A project that claims more than its evidence supports is graded on the evidence.
Module 1614 / 23

Worked example: choosing a project

  • Begin with a real person who has a recurring need.
  • In this made-up case a club secretary searches past meeting notes each month.
  • The need recurs monthly, which gives the occasion for a second real use.
  • The components are a readable results page (10), no open ports (12), an audit (13) and a release script (14).
  • Check each of the eight items against the project before committing.
Module 1615 / 23

Worked example: a proposal to turn down

  • The made-up proposal is a story game with model-written scenes, "for anyone".
  • It has no particular user, so no second real use can be shown.
  • It draws on one laboratory module, 9, where three are required.
  • It makes a model call per scene and has no cost model or spending cap.
  • To repair it, find a player, serve the game (12, 14), audit it (13) and cap the spend.
Module 1616 / 23

Worked example: marking a submission

criterion     evidence          mark
user 25%      one use, no 2nd   partial
verify 25%    finder = checker  low
secure 20%    cap, no restore   partial
harness 15%   hook + note       full
writing 15%   clear log         full
graded on the evidence
Module 1617 / 23

Ask the room: evidence of a second use

  • "She said she would definitely use it again."
  • "He opened it twice during the demonstration."
  • "She returned the following week with a new task and completed it."
  • "The agent finished another real assignment under the intended supervision."
  • Place each statement on the evidence ladder of 14.7.
Module 1618 / 23

In-class exercise: an outline of the brief

  1. The goal, more ambitious than the first instinct.
  2. A few concrete use cases.
  3. Where it fits, and the design considerations.
  4. How success will be judged: the observable evidence.
  5. The latitude the agent has to exceed the brief.
Module 1619 / 23

In-class exercise: the order of the work

phase      artifact         gate
specify    requirements     you approve
architect  flows, failures  you approve
plan       small tasks      you approve
build      test, then code  checks
review     two passes       findings
ship       deploy, monitor  you approve
Module 1620 / 23

The cost model in outline

build
  tokens used
  money spent
run, per user per month
  sessions per month
  x cost per session
  + share of hosting
  = cost per user per month
Module 1621 / 23

Common mistakes

  • A project with no real user. Instead, choose the person first.
  • One use reported as proof. Instead, record the second real use, with its date.
  • An agent that audits its own work. Instead, separate the finder and the verifier.
  • Rules written only in the instruction file. Instead, use hooks and permissions, with a note.
  • An empty "not checked" list. Instead, write what was not measured.
  • Reports full of abstract nouns. Instead, name the subject, the object and the action.
Module 1622 / 23

Recap

  • The project is one piece of software that a real person uses, built with agents throughout and drawing on three laboratory modules.
  • The eight items are plans, harness, software, verification, security, cost, log and limits.
  • A second real use and independent, repeatable verification carry 50 per cent.
  • Security and operations carry 20 per cent, the harness 15 and writing 15.
  • A project that claims more than its evidence supports is graded on the evidence.
  • "What was not checked" is a submitted item.
Module 1623 / 23

After the course

  • Keep the findings log going, with dates and versions.
  • Re-run the evaluation after every model update.
  • Repeat the audit, because a single run found about half.
  • Restore from backup again, and time it.
  • Watch cost daily while the service is live.
  • Renew the inventory of services, keys and connectors.