Software Engineering with Language Models
Module 13 · Cyber security for builders and their agents
- Class time
- 60 minutes
- Slides
- 33, each with teacher’s notes
- Goes with
- Laboratory 8; Assignments 5 and 6 (Part IV)
Where this module sits
- Module 12 limited what a compromised device or account can reach.
- After section 12.5 you still have three duties: patching, a secure application and restored backups.
- In this class you audit the application and the agent deployment around it.
- You also harden the accounts, secrets and dependencies the work rests on.
- In Module 14 you ship a service with a spending cap and a timed restore.
Objectives
- Check an application against the twenty controls of 13.1 and record evidence for each.
- List the endpoints your own client reveals and test authorisation on each.
- Run a phased agent audit in which the finder and the verifier are separate.
- File every finding under one of three verdicts, including the rejected findings.
- Place structural defences around an agent that reads untrusted text.
- Harden the accounts, secrets and dependencies the work rests on.
Agenda
- 4 min: where this module sits and the objectives.
- 10 min: the baseline of twenty controls.
- 8 min: the client as documentation of the server.
- 14 min: the six-phase audit, the verdicts and a worked finding.
- 10 min: securing the agent and cost as an attack.
- 14 min: supply chain, self-reconnaissance, hygiene, the laboratory and the recap.
The baseline of twenty controls
| Area | Controls |
|---|---|
| Secrets | Keys in environment or a secret store, never in source. Rotate any key that was ever committed |
| Access | Server-side validation of all input. Authorisation check on every route. Row-level security in the database |
| Authentication | Passwords hashed with a slow algorithm. Rate limits on sign-in and sign-up. Secure, HTTP-only cookies. Failed sign-ins logged |
| Web | HTTPS everywhere. Cross-origin requests restricted to your own origins. Parameterised queries. Output escaping. CSRF protection. Uploads restricted by type and size |
| Operations | No stack traces in production. Dependencies updated. Daily database backups. A spending cap on every metered API key |
Reading the baseline
- A deleted key remains in history, so you rotate any key that was committed.
- Validation runs on the server and covers all input.
- The authorisation check applies to every route.
- The database enforces row-level security.
- The spending cap applies to every metered API key.
- You check the backups and the cap yourself, because an agent's report is a claim.
Ask the room: claim or evidence
- An agent reports: "Backups run daily and the spending cap is set."
- State what you must see before you record either control as met.
- Name the one action that shows a backup is usable.
- Name the place where you read a spending cap, given that the repository does not hold it.
The client as documentation of the server
- An October 2026 advisory from a national research institute named an attack pattern.
- Step 1: obtain a published mobile application.
- Step 2: analyse it to learn its servers and how the interface works.
- Step 3: attack the server with requests built for that interface.
- The application itself need not contain a flaw.
- Agents have made the analysis cheap.
What a shipped client reveals
client app ------> server
every user operator thinks
holds a copy it is private
|
v
anyone can read the copy and learn
- which servers it talks to
- how the interface works
v
requests built for that interface
Five defect classes in order of frequency
- Broken object-level authorisation, where the server checks only the identifier.
- Server-side request forgery.
- Remote code execution.
- SQL injection.
- Misconfiguration.
Worked example: a response to the advisory
- Inventory every public interface.
- Review access logs for bulk retrieval.
- Test authorisation first.
- Shut down interfaces that are not needed.
- Reissue credentials where exposure is possible.
An audit method in six phases
- Reconnaissance: map architecture, trust boundaries and input surfaces, and write a coverage ledger.
- Coverage-led hunting: isolated hunters take ledger units, and critics look for gaps.
- Validation: a fresh agent tries to disprove each candidate.
- Structured output: the findings go in a file checked against a schema.
- Independent verification: new agents check the final claims against the source.
- Reporting: the reports are derived from the verified records and the ledger.
The audit as a pipeline
1 recon: architecture doc + ledger
|
2 hunters take ledger units
critics look for gaps
|
3 fresh agent tries to disprove
4 findings file <- schema check
5 new agents verify against source
|
6 report from records + ledger
Three verdicts
| Verdict | Requires | Severity |
|---|---|---|
| Confirmed | A complete trace through the source plus a bounded observed result | Requires impact |
| Needs validation | The exact unresolved fact, named | None assigned |
| Rejected | A disproof | None; kept on record |
Rules of the audit method
- The checker is never the finder.
- Validator scripts with their own tests run after every change to the ledger or the findings.
- Severity requires impact.
- A missing second layer is a hardening note when the first layer holds.
- Runs are additive: each run uses the earlier ledger to target gaps.
- Target code runs only in an operating-system sandbox with no outside network.
Worked example: three outcomes for one candidate
- A hunter reports a candidate: "the export route returns any user's file".
- A fresh agent traces the route, handler and query in the source.
- A complete trace plus a bounded result in the sandbox means confirmed.
- A trace that stops at unread middleware means needs validation, with the open fact named.
- If the middleware checks ownership on every path, the candidate is rejected and recorded.
The attack-class references
- Memory safety.
- Prompt injection and tool misuse.
- Request framing, caching and authentication protocols.
- Client-side injection and messaging trust.
- Dependencies, build systems, signing and updates.
- Cloud identity, infrastructure code and serverless.
- Resource exhaustion and operator spend.
- Tenancy, export, backup, deletion and restore.
- Local inter-process communication.
Securing the agent against prompt injection
- Instructions can arrive in any text the agent reads.
- The model has no reliable way to tell those instructions from the operator's.
- Treat analysed content as data and list any instructions found in it.
- Require approval for irreversible actions.
- Restrict network egress.
- Separate the agent that reads untrusted content from the agent that holds credentials.
Separation of the reader from the credential holder
untrusted text your instructions
(page, file, message) |
| v
v agent B: acts
agent A: reads holds credentials
no credentials never reads the
limited egress untrusted text
| ^
+---- data only ----+
summary + instructions found
Credentials and the agent's runtime
- No secret appears in text a model can read.
- You reference secrets by name, and they are injected at run time.
- You enter keys in your own terminal, never in a chat.
- Audit skill folders, tool-server configuration and instruction files for embedded keys.
- Prefer short-lived tokens and proof-of-possession to stored long-lived grants.
- Patch the agent's command-line tool and language runtime.
In-class exercise: the scope of an agent's access
- In the case, an agent was told it may use any key in a keychain.
- The agent produced a 36-minute film.
- The bill was above $200.
- Write the three limits that should have been set before the run.
- Then weigh read access to mail and accounts as a risk of its own.
Eight measures against cost attacks
- Make free trials cost something: require a card and pre-authorise it.
- Verify that requests carry valid credentials and that they come from the genuine client application.
- Allowlist the models offered and deny the rest by default.
- Enforce hard limits per account and per organisation, with no overage before payment clears.
- Bill sooner.
- Use the payment provider's fraud rules.
- Watch cost daily and be ready to suspend accounts or pause sign-ups.
- Restrict trials to verified business domains and cap trials per domain.
Worked example: the alarm that re-armed itself
- A scheduled alarm on a serverless platform re-armed itself in a loop.
- The loop produced trillions of storage operations.
- The bill was near $10,800.
- The appeal went to an automated support agent, which answered in circles.
- The bill was paid in full.
- Before deployment the job needs a termination condition, a loop guard, budget alerts and hard limits.
The supply chain and its maintainers
- Of 23 foundational projects analysed, 11 had one or two regular maintainers.
- A time-zone database sits on some four billion phones.
- One volunteer made 218 of its 251 changes in a year.
- A compression library's maintainer was socially engineered over years.
- The resulting backdoor was caught in March 2024 because of unusual processor use.
- A cryptographic library ran on about $2,000 a year before its 2014 flaw.
Supply chain practices
- For anything you publish, keep the public surface and the dependency list small.
- Prefer a little duplicated code to a trivial package.
- Pin versions.
- Verify checksums of downloaded models and binaries.
- Include maintainer health in dependency inventories.
- "Paste this link and let the agent install it" is both a supply-chain risk and an injection risk.
Open-source intelligence on yourself
- Usernames: search tools check one name across hundreds or thousands of sites.
- Domains: certificate-transparency logs list every certificate, and so every subdomain.
- Devices: search engines index what your addresses expose.
- Email addresses: company formats are guessable and verifiable.
- Web artefacts attribute a site to its operator, and a recording can be located from its contents.
- Use these tools on yourself, or with written authorisation.
In-class exercise: your own hygiene score
- A password manager with a unique generated password per service.
- Hardware security keys on email, source control and banking, and a SIM PIN.
- Random strings as answers to security questions.
- Full-disk encryption and automatic updates on every machine.
- Navigate to services yourself, and turn automatic image loading off.
- A separate virtual card per merchant, with its own limit.
- End-to-end encrypted messaging with disappearing messages.
- Personal accounts kept off employer-managed machines.
- Know each AI tool's tier and data terms.
- A periodic inventory.
Common mistakes
- Deleting a committed key. Instead, rotate it.
- Accepting an agent's word on backups and caps. Instead, check both yourself.
- Letting the finder confirm its own finding. Instead, have a fresh agent try to disprove it.
- Giving an unproven lead a severity. Instead, file it as needs validation with the open fact named.
- Discarding rejected findings. Instead, keep them on record.
- Granting an agent every credential. Instead, set a budget, a length limit and a named list.
Laboratory 8: what is due
- (a) The twenty controls of 13.1, with recorded evidence for each.
- (b) Every endpoint your client reveals, with authorisation tested on each with a second account.
- (c) A phased agent audit with separate finder and verifier.
- (c) Findings under the three verdicts, including the rejected findings.
- (d) Two of the self-audits in 13.7.
- (e) A list of everything left unexamined.
Assignment 5 · The infinite story: what is due
- A text adventure whose missing scenes a model writes on demand.
- Every reply validated before use, with bounded retries and a written fallback scene.
- Player input checked against the choices and never placed inside a prompt.
- The model credential outside version control, with a spending cap.
- A replay of a recorded session that makes no model call.
- Hand in the prompts, the session log, the verifying commands and the list of what was not checked.
Assignment 6 · A search engine served securely: what is due
- A word index over a few hundred of your own text files.
- A second engine on embeddings, with both engines scored on 20 queries.
- A server bound to the local machine, with no inbound port opened.
- Limits on query length and request rate, and a query always treated as data.
- The endpoint inventory and the finder-and-verifier audit of 13.3, run against your server.
- Hand in the prompts, the session log, the verifying commands and the list of what was not checked.
Recap
- Record evidence for the twenty controls of 13.1 and check the last two yourself.
- The shipped client describes the server, so test authorisation with a second account.
- The six audit phases keep finder and checker apart, and each finding has one of three verdicts.
- Defences against prompt injection are structural.
- Set hard limits and budget alerts before deployment.
- Pin versions, keep the dependency list small, and run reconnaissance on yourself.
Before next class
- Begin Laboratory 8 with part (a), the evidence for the twenty controls of 13.1.
- Check your backup by restoring it, and your spending cap by viewing it.
- Start the inventory sheet from 13.8, with columns for risk, urgency and action.
- Read Module 14, sections 14.1 to 14.3.
- Bring one small service you could ship for Laboratory 9.