Syllabus Module 13 · Cyber security for builders and their agents
NSW 3011 / 33
Software Engineering with Language Models

Module 13 · Cyber security for builders and their agents

Class time
60 minutes
Slides
33, each with teacher’s notes
Goes with
Laboratory 8; Assignments 5 and 6 (Part IV)
Module 132 / 33

Where this module sits

  • Module 12 limited what a compromised device or account can reach.
  • After section 12.5 you still have three duties: patching, a secure application and restored backups.
  • In this class you audit the application and the agent deployment around it.
  • You also harden the accounts, secrets and dependencies the work rests on.
  • In Module 14 you ship a service with a spending cap and a timed restore.
Module 133 / 33

Objectives

  • Check an application against the twenty controls of 13.1 and record evidence for each.
  • List the endpoints your own client reveals and test authorisation on each.
  • Run a phased agent audit in which the finder and the verifier are separate.
  • File every finding under one of three verdicts, including the rejected findings.
  • Place structural defences around an agent that reads untrusted text.
  • Harden the accounts, secrets and dependencies the work rests on.
Module 134 / 33

Agenda

  • 4 min: where this module sits and the objectives.
  • 10 min: the baseline of twenty controls.
  • 8 min: the client as documentation of the server.
  • 14 min: the six-phase audit, the verdicts and a worked finding.
  • 10 min: securing the agent and cost as an attack.
  • 14 min: supply chain, self-reconnaissance, hygiene, the laboratory and the recap.
Module 135 / 33

The baseline of twenty controls

AreaControls
SecretsKeys in environment or a secret store, never in source. Rotate any key that was ever committed
AccessServer-side validation of all input. Authorisation check on every route. Row-level security in the database
AuthenticationPasswords hashed with a slow algorithm. Rate limits on sign-in and sign-up. Secure, HTTP-only cookies. Failed sign-ins logged
WebHTTPS everywhere. Cross-origin requests restricted to your own origins. Parameterised queries. Output escaping. CSRF protection. Uploads restricted by type and size
OperationsNo stack traces in production. Dependencies updated. Daily database backups. A spending cap on every metered API key
Module 136 / 33

Reading the baseline

  • A deleted key remains in history, so you rotate any key that was committed.
  • Validation runs on the server and covers all input.
  • The authorisation check applies to every route.
  • The database enforces row-level security.
  • The spending cap applies to every metered API key.
  • You check the backups and the cap yourself, because an agent's report is a claim.
Module 137 / 33

Ask the room: claim or evidence

  • An agent reports: "Backups run daily and the spending cap is set."
  • State what you must see before you record either control as met.
  • Name the one action that shows a backup is usable.
  • Name the place where you read a spending cap, given that the repository does not hold it.
Module 138 / 33

The client as documentation of the server

  • An October 2026 advisory from a national research institute named an attack pattern.
  • Step 1: obtain a published mobile application.
  • Step 2: analyse it to learn its servers and how the interface works.
  • Step 3: attack the server with requests built for that interface.
  • The application itself need not contain a flaw.
  • Agents have made the analysis cheap.
Module 139 / 33

What a shipped client reveals

client app  ------>  server
every user           operator thinks
holds a copy         it is private
    |
    v
anyone can read the copy and learn
  - which servers it talks to
  - how the interface works
    v
requests built for that interface
Module 1310 / 33

Five defect classes in order of frequency

  1. Broken object-level authorisation, where the server checks only the identifier.
  2. Server-side request forgery.
  3. Remote code execution.
  4. SQL injection.
  5. Misconfiguration.
Module 1311 / 33

Worked example: a response to the advisory

  1. Inventory every public interface.
  2. Review access logs for bulk retrieval.
  3. Test authorisation first.
  4. Shut down interfaces that are not needed.
  5. Reissue credentials where exposure is possible.
Module 1312 / 33

An audit method in six phases

  1. Reconnaissance: map architecture, trust boundaries and input surfaces, and write a coverage ledger.
  2. Coverage-led hunting: isolated hunters take ledger units, and critics look for gaps.
  3. Validation: a fresh agent tries to disprove each candidate.
  4. Structured output: the findings go in a file checked against a schema.
  5. Independent verification: new agents check the final claims against the source.
  6. Reporting: the reports are derived from the verified records and the ledger.
Module 1313 / 33

The audit as a pipeline

1 recon: architecture doc + ledger
        |
2 hunters take ledger units
  critics look for gaps
        |
3 fresh agent tries to disprove
4 findings file <- schema check
5 new agents verify against source
        |
6 report from records + ledger
Module 1314 / 33

Three verdicts

VerdictRequiresSeverity
ConfirmedA complete trace through the source plus a bounded observed resultRequires impact
Needs validationThe exact unresolved fact, namedNone assigned
RejectedA disproofNone; kept on record
Module 1315 / 33

Rules of the audit method

  • The checker is never the finder.
  • Validator scripts with their own tests run after every change to the ledger or the findings.
  • Severity requires impact.
  • A missing second layer is a hardening note when the first layer holds.
  • Runs are additive: each run uses the earlier ledger to target gaps.
  • Target code runs only in an operating-system sandbox with no outside network.
Module 1316 / 33

Worked example: three outcomes for one candidate

  • A hunter reports a candidate: "the export route returns any user's file".
  • A fresh agent traces the route, handler and query in the source.
  • A complete trace plus a bounded result in the sandbox means confirmed.
  • A trace that stops at unread middleware means needs validation, with the open fact named.
  • If the middleware checks ownership on every path, the candidate is rejected and recorded.
Module 1317 / 33

The attack-class references

  1. Memory safety.
  2. Prompt injection and tool misuse.
  3. Request framing, caching and authentication protocols.
  4. Client-side injection and messaging trust.
  5. Dependencies, build systems, signing and updates.
  6. Cloud identity, infrastructure code and serverless.
  7. Resource exhaustion and operator spend.
  8. Tenancy, export, backup, deletion and restore.
  9. Local inter-process communication.
Module 1318 / 33

Securing the agent against prompt injection

  • Instructions can arrive in any text the agent reads.
  • The model has no reliable way to tell those instructions from the operator's.
  • Treat analysed content as data and list any instructions found in it.
  • Require approval for irreversible actions.
  • Restrict network egress.
  • Separate the agent that reads untrusted content from the agent that holds credentials.
Module 1319 / 33

Separation of the reader from the credential holder

untrusted text      your instructions
(page, file, message)      |
       |                   v
       v            agent B: acts
agent A: reads      holds credentials
no credentials      never reads the
limited egress      untrusted text
       |                   ^
       +---- data only ----+
  summary + instructions found
Module 1320 / 33

Credentials and the agent's runtime

  • No secret appears in text a model can read.
  • You reference secrets by name, and they are injected at run time.
  • You enter keys in your own terminal, never in a chat.
  • Audit skill folders, tool-server configuration and instruction files for embedded keys.
  • Prefer short-lived tokens and proof-of-possession to stored long-lived grants.
  • Patch the agent's command-line tool and language runtime.
Module 1321 / 33

In-class exercise: the scope of an agent's access

  • In the case, an agent was told it may use any key in a keychain.
  • The agent produced a 36-minute film.
  • The bill was above $200.
  • Write the three limits that should have been set before the run.
  • Then weigh read access to mail and accounts as a risk of its own.
Module 1322 / 33

Eight measures against cost attacks

  1. Make free trials cost something: require a card and pre-authorise it.
  2. Verify that requests carry valid credentials and that they come from the genuine client application.
  3. Allowlist the models offered and deny the rest by default.
  4. Enforce hard limits per account and per organisation, with no overage before payment clears.
  5. Bill sooner.
  6. Use the payment provider's fraud rules.
  7. Watch cost daily and be ready to suspend accounts or pause sign-ups.
  8. Restrict trials to verified business domains and cap trials per domain.
Module 1323 / 33

Worked example: the alarm that re-armed itself

  • A scheduled alarm on a serverless platform re-armed itself in a loop.
  • The loop produced trillions of storage operations.
  • The bill was near $10,800.
  • The appeal went to an automated support agent, which answered in circles.
  • The bill was paid in full.
  • Before deployment the job needs a termination condition, a loop guard, budget alerts and hard limits.
Module 1324 / 33

The supply chain and its maintainers

  • Of 23 foundational projects analysed, 11 had one or two regular maintainers.
  • A time-zone database sits on some four billion phones.
  • One volunteer made 218 of its 251 changes in a year.
  • A compression library's maintainer was socially engineered over years.
  • The resulting backdoor was caught in March 2024 because of unusual processor use.
  • A cryptographic library ran on about $2,000 a year before its 2014 flaw.
Module 1325 / 33

Supply chain practices

  • For anything you publish, keep the public surface and the dependency list small.
  • Prefer a little duplicated code to a trivial package.
  • Pin versions.
  • Verify checksums of downloaded models and binaries.
  • Include maintainer health in dependency inventories.
  • "Paste this link and let the agent install it" is both a supply-chain risk and an injection risk.
Module 1326 / 33

Open-source intelligence on yourself

  • Usernames: search tools check one name across hundreds or thousands of sites.
  • Domains: certificate-transparency logs list every certificate, and so every subdomain.
  • Devices: search engines index what your addresses expose.
  • Email addresses: company formats are guessable and verifiable.
  • Web artefacts attribute a site to its operator, and a recording can be located from its contents.
  • Use these tools on yourself, or with written authorisation.
Module 1327 / 33

In-class exercise: your own hygiene score

  1. A password manager with a unique generated password per service.
  2. Hardware security keys on email, source control and banking, and a SIM PIN.
  3. Random strings as answers to security questions.
  4. Full-disk encryption and automatic updates on every machine.
  5. Navigate to services yourself, and turn automatic image loading off.
  6. A separate virtual card per merchant, with its own limit.
  7. End-to-end encrypted messaging with disappearing messages.
  8. Personal accounts kept off employer-managed machines.
  9. Know each AI tool's tier and data terms.
  10. A periodic inventory.
Module 1328 / 33

Common mistakes

  • Deleting a committed key. Instead, rotate it.
  • Accepting an agent's word on backups and caps. Instead, check both yourself.
  • Letting the finder confirm its own finding. Instead, have a fresh agent try to disprove it.
  • Giving an unproven lead a severity. Instead, file it as needs validation with the open fact named.
  • Discarding rejected findings. Instead, keep them on record.
  • Granting an agent every credential. Instead, set a budget, a length limit and a named list.
Module 1329 / 33

Laboratory 8: what is due

  • (a) The twenty controls of 13.1, with recorded evidence for each.
  • (b) Every endpoint your client reveals, with authorisation tested on each with a second account.
  • (c) A phased agent audit with separate finder and verifier.
  • (c) Findings under the three verdicts, including the rejected findings.
  • (d) Two of the self-audits in 13.7.
  • (e) A list of everything left unexamined.
Module 1330 / 33

Assignment 5 · The infinite story: what is due

  • A text adventure whose missing scenes a model writes on demand.
  • Every reply validated before use, with bounded retries and a written fallback scene.
  • Player input checked against the choices and never placed inside a prompt.
  • The model credential outside version control, with a spending cap.
  • A replay of a recorded session that makes no model call.
  • Hand in the prompts, the session log, the verifying commands and the list of what was not checked.
Module 1331 / 33

Assignment 6 · A search engine served securely: what is due

  • A word index over a few hundred of your own text files.
  • A second engine on embeddings, with both engines scored on 20 queries.
  • A server bound to the local machine, with no inbound port opened.
  • Limits on query length and request rate, and a query always treated as data.
  • The endpoint inventory and the finder-and-verifier audit of 13.3, run against your server.
  • Hand in the prompts, the session log, the verifying commands and the list of what was not checked.
Module 1332 / 33

Recap

  • Record evidence for the twenty controls of 13.1 and check the last two yourself.
  • The shipped client describes the server, so test authorisation with a second account.
  • The six audit phases keep finder and checker apart, and each finding has one of three verdicts.
  • Defences against prompt injection are structural.
  • Set hard limits and budget alerts before deployment.
  • Pin versions, keep the dependency list small, and run reconnaissance on yourself.
Module 1333 / 33

Before next class

  • Begin Laboratory 8 with part (a), the evidence for the twenty controls of 13.1.
  • Check your backup by restoring it, and your spending cap by viewing it.
  • Start the inventory sheet from 13.8, with columns for risk, urgency and action.
  • Read Module 14, sections 14.1 to 14.3.
  • Bring one small service you could ship for Laboratory 9.