Software Engineering with Language Models
Module 14 · Shipping and operating software
- Class time
- 60 minutes
- Slides
- 33, each with teacher’s notes
- Goes with
- Laboratory 9; Assignment 6 and Section problems (Part IV)
Where this module sits
- From Module 12 you have the server with no open ports.
- From Module 13 you have the baseline controls and the spending cap.
- This class covers architecture, interfaces, hosting cost and release engineering.
- Section 14.5 lists what the owner of a product owns that a demonstration lacks.
- In Module 15 you apply cost reasoning to models on your own hardware.
Objectives
- Choose one deployable unit or several from a measured constraint.
- Design an interface call site first, with a small public surface.
- Estimate hosting cost and cost per session before building.
- Answer a build-or-buy question with what agents changed and what they did not change.
- List what the owner of a product owns that a demonstration lacks.
- Ship with a script that validates first, and measure a pipeline before trimming it.
Agenda
- 4 min: where this module sits and the objectives.
- 7 min: the cost of a service boundary and two patterns.
- 10 min: interfaces that last and a worked call site.
- 12 min: hosting, cost per session and build or buy.
- 11 min: demonstrations and products, and three named properties.
- 16 min: release engineering, CI economics, what to build, the laboratory and the recap.
The cost of a service boundary
- A split into services turns in-process function calls into network calls.
- Network calls fail in ways local calls do not.
- Teams adopted hundreds of services while serving a few hundred users.
- Those teams paid in delivery speed.
- Start with one deployable unit.
- Divide it on a measured constraint: a team boundary, a scaling limit or an isolation requirement.
The event table and schedule pattern
entities people
|
v
events (dated, about an entity)
^
| rows added by
scheduled job
new event row --> agent triggered
State machines as owned objects
- Implicit state machines are spread across database rows.
- Each state machine can be made into an addressable unit.
- Each unit owns its state.
- Each unit processes one message at a time.
- The claimed benefits are observability, isolated testing and pushed updates.
- The advice comes from a vendor of such a platform.
Rules 1 to 4 for interfaces that last
- Design the call site first.
- Keep types opaque across layers.
- No stringly-typed parameters.
- Errors are part of the interface.
Rules 5 to 8 for interfaces that last
- Expensive operations are explicit methods, documented as expensive.
- Progressive disclosure.
- Keep the public surface small.
- Minimise dependencies.
Worked example: a call site written first
want = Status.OPEN
async for inv in client.invoices(want):
show(inv.number, inv.total)
try:
await client.export_all()
except QuotaError as err:
wait(err.retry_after)
Two patterns and one prohibition for service clients
- Paginated results are an asynchronous sequence the caller iterates.
- Page fetching is hidden from the caller.
- Retry with exponential backoff goes inside the client library.
- A server-side SDK never goes into a client application.
- The SDK's service-account keys never go into a client application either.
Hosting and cost
- A virtual server costs a few dollars a month, and a database can run on it.
- A content network sits in front, and object storage holds bulk files.
- One reported consolidation of five subscriptions saved about $155 a month.
- 100 GB of video in object storage with free egress costs just over a dollar a month.
- Running it yourself means owning patching, monitoring, backups and restores.
- Machines sold for agents bill per second, and stopped machines can still bill for storage.
Three "bring your own" terms
| Term | Meaning |
|---|---|
| Bring your own cloud | The vendor's software runs in the customer's account |
| Bring your own key | The customer controls the encryption keys, in a single-tenant environment |
| Bring your own inference | The customer supplies the model endpoint |
In-class exercise: cost per session
- A personal assistant agent is reported to cost $3,000 to $7,000 per user per year.
- The figure is second-hand.
- The figure depends on scale and model choice.
- Revenue per user has to cover the inference cost of an always-on agent.
- Convert the range to a monthly figure, then to a cost per session.
What agents changed in the build-or-buy decision
- Build cost: one long agent run can produce a focused version.
- Scale: you serve one organisation, and a vendor serves its whole base.
- Fixing defects: you can reproduce and fix a defect at once.
- A vendor's defect waits on the vendor.
- Reliability: you can measure your own uptime, and a vendor's figure is self-reported.
What agents did not change in the build-or-buy decision
- Someone must still specify the build and verify it.
- Fixing your own defects puts you on call.
- Switching: moving thousands of people off a tool is painful.
- Ownership: you still carry security, compliance and maintenance for code nobody wrote by hand.
- Software that can be built for the price of tokens falls in market value toward that price.
- Value moves to distribution, data, switching cost and operation.
The difference between a demonstration and a product
- A prototype of a customer-support inbox takes an afternoon.
- Email: deliverability, bounces, sender authentication records, reputation drift.
- Email: broken threading, nested forwards, quoted text, auto-responder loops.
- Collaboration: two people drafting one reply; internal notes; synchronised drafts.
- Time: snooze across time zones and daylight-saving changes.
- Identity: one customer appearing as several; shared addresses; merge and un-merge.
The other four headings of the list
- Messaging and webhooks: duplicated, reordered and missing deliveries; rate limits.
- Real time: reconnect storms, half-open connections, ordering, read state across tabs.
- AI layer: scanned PDFs, stale facts, conflicting sources, re-indexing, injected pages.
- AI layer: false promises, double refunds, hand-off to a person, re-evaluation.
- Operations: malware uploads, deletion requests, silent job failures, full disks.
- Operations: backups and tested restores; monitoring of the monitoring.
Three named properties
- Idempotency: an operation whose outcome is unknown is checked before it is retried.
- Read-back verification: after a write, read the result and compare it with what you intended.
- Separate states need separate evidence.
- Paid, label created, accepted by the carrier, delivered and paid out are five facts.
Worked example: the publish that timed out
- The publish request times out. The outcome is unknown.
- Check first: ask the system whether the item now exists.
- If it exists, read it back and compare with what was intended.
- If it does not exist, send once more, then read back.
- Record which state is proven, and by what evidence.
Release engineering
- Validate offline, then upload, and stop on the first failure.
- Dependencies come from a lockfile.
- Downloaded models and binaries are pinned to a commit and verified by checksum.
- Builds need no network after setup.
- Sign and notarise desktop software.
- Plan the monitoring before the deploy, and record what was verified on which versions.
Worked example: the bill for a test matrix
job elapsed billed
5-way matrix 2 min 10+ min
tiny job 5 s 1 min
merge rerun repeats what passed
billed and elapsed time differ
most waste lies between them
Five steps for continuous integration
- Measure total billed minutes by workflow and job, with median and 90th-percentile elapsed time.
- Learn what each check protects and find the critical path.
- Look for known waste.
- Estimate each saving from the measured data, and state the risk.
- Make changes in small steps and comment each one.
In-class exercise: waste in a pipeline
runs on: every push, every path
cancel older runs: no
jobs:
unit tests
unit tests again, same commit
12 lint jobs, about 5 s each
runner: costly OS image, all jobs
dependency cache: none
timeout: none
Eleven signals from four sources for platform work
- Incident reviews, the cost bill and the team's own toil.
- Conversations with users, uses the platform was not designed for and joint prototypes.
- Stated objectives, things management repeats and teams that never migrated.
- From the wider industry: a written description of the current system, and lessons from fields that are ahead.
- The strongest signal is a use the platform was not designed for.
- The failure to avoid is a backlog assembled from the loudest signals.
The evidence ladder for product learning
- Recruit a small group who already have the problem.
- Ask about their last real attempt, never their opinion of a feature.
- Rung 1 is agreeing to talk, which shows willingness to talk.
- Rung 2 is one real task completed, which shows the product can help once.
- Rung 3 is being chosen again when the need recurs, which is the first evidence of value.
- For an agent, the sign of value is another real assignment that finishes correctly under the intended supervision.
Ask the room: the date that orders a history
- A project keeps national law as Git history.
- Each law taking effect is one commit.
- The history yields the text in force on any date.
- Commits were made out of chronological order.
- Name the date the tools must sort by, and where it is stored.
Common mistakes
- Splitting into services for theoretical scale. Instead, keep one unit until a measured constraint requires a split.
- Implementing first. Instead, write the adopter's call site, then build behind it.
- Retrying an operation whose outcome is unknown. Instead, check, then retry.
- Believing a "saved" indicator. Instead, read back and compare.
- Trimming a pipeline by intuition. Instead, measure billed minutes and learn what each check protects.
- Counting a backup as a restore. Instead, perform a restore and time it.
Laboratory 9: what is due
- A one-page architecture note justifying one deployable unit or more.
- The interface, designed call-site first.
- The server with no inbound ports open.
- A spending cap and a billing alert, each shown in a screenshot.
- A release script that validates before deploying, and a restore performed and timed.
- The 14.5 items that apply, each marked handled, deferred or not applicable.
Assignment 6 · A search engine served securely: what is due
- An index and query engine over your own files, scored on 20 queries.
- One page and one search endpoint, bound to the local machine.
- Outside access by tunnel or private network, with no inbound port opened.
- Limits on query length and request rate.
- A results page that works with scripts disabled, and a stable address per result.
- Hand in the prompts, the session log, the verifying commands and the list of what was not checked.
Section problems: what is due
- Problem 5, the usage ledger: tokens per task, grouped by model.
- Write the examples for its helper before the helper.
- Print the three most expensive pairs and move one task to a cheaper model.
- Problem 6, the fastest one-off: the lowest latency no other call shares.
- State the result for an empty log before writing code.
- Hand in the prompts, the session log, the verifying commands and the list of what was not checked.
Recap
- Keep one deployable unit until a measured constraint requires a split.
- Design the call site first and keep the public surface small.
- Model the cost per session before building.
- Owning a product means owning the eight headings of 14.5.
- Check before retrying, read back after writing, prove each state separately.
- Validate offline before upload, and measure billed minutes before cutting a check.
Before next class
- Begin Laboratory 9 with the one-page architecture note.
- Perform one restore from backup and write down how long it took.
- Start section problem 5, the usage ledger.
- Read Module 15, sections 15.1 to 15.3.
- Find your machine's memory size and bring the number.