Syllabus Module 14 · Shipping and operating software
NSW 3011 / 33
Software Engineering with Language Models

Module 14 · Shipping and operating software

Class time
60 minutes
Slides
33, each with teacher’s notes
Goes with
Laboratory 9; Assignment 6 and Section problems (Part IV)
Module 142 / 33

Where this module sits

  • From Module 12 you have the server with no open ports.
  • From Module 13 you have the baseline controls and the spending cap.
  • This class covers architecture, interfaces, hosting cost and release engineering.
  • Section 14.5 lists what the owner of a product owns that a demonstration lacks.
  • In Module 15 you apply cost reasoning to models on your own hardware.
Module 143 / 33

Objectives

  • Choose one deployable unit or several from a measured constraint.
  • Design an interface call site first, with a small public surface.
  • Estimate hosting cost and cost per session before building.
  • Answer a build-or-buy question with what agents changed and what they did not change.
  • List what the owner of a product owns that a demonstration lacks.
  • Ship with a script that validates first, and measure a pipeline before trimming it.
Module 144 / 33

Agenda

  • 4 min: where this module sits and the objectives.
  • 7 min: the cost of a service boundary and two patterns.
  • 10 min: interfaces that last and a worked call site.
  • 12 min: hosting, cost per session and build or buy.
  • 11 min: demonstrations and products, and three named properties.
  • 16 min: release engineering, CI economics, what to build, the laboratory and the recap.
Module 145 / 33

The cost of a service boundary

  • A split into services turns in-process function calls into network calls.
  • Network calls fail in ways local calls do not.
  • Teams adopted hundreds of services while serving a few hundred users.
  • Those teams paid in delivery speed.
  • Start with one deployable unit.
  • Divide it on a measured constraint: a team boundary, a scaling limit or an isolation requirement.
Module 146 / 33

The event table and schedule pattern

entities       people
    |
    v
events (dated, about an entity)
    ^
    | rows added by
scheduled job

new event row --> agent triggered
Module 147 / 33

State machines as owned objects

  • Implicit state machines are spread across database rows.
  • Each state machine can be made into an addressable unit.
  • Each unit owns its state.
  • Each unit processes one message at a time.
  • The claimed benefits are observability, isolated testing and pushed updates.
  • The advice comes from a vendor of such a platform.
Module 148 / 33

Rules 1 to 4 for interfaces that last

  1. Design the call site first.
  2. Keep types opaque across layers.
  3. No stringly-typed parameters.
  4. Errors are part of the interface.
Module 149 / 33

Rules 5 to 8 for interfaces that last

  1. Expensive operations are explicit methods, documented as expensive.
  2. Progressive disclosure.
  3. Keep the public surface small.
  4. Minimise dependencies.
Module 1410 / 33

Worked example: a call site written first

want = Status.OPEN
async for inv in client.invoices(want):
    show(inv.number, inv.total)

try:
    await client.export_all()
except QuotaError as err:
    wait(err.retry_after)
Module 1411 / 33

Two patterns and one prohibition for service clients

  • Paginated results are an asynchronous sequence the caller iterates.
  • Page fetching is hidden from the caller.
  • Retry with exponential backoff goes inside the client library.
  • A server-side SDK never goes into a client application.
  • The SDK's service-account keys never go into a client application either.
Module 1412 / 33

Hosting and cost

  • A virtual server costs a few dollars a month, and a database can run on it.
  • A content network sits in front, and object storage holds bulk files.
  • One reported consolidation of five subscriptions saved about $155 a month.
  • 100 GB of video in object storage with free egress costs just over a dollar a month.
  • Running it yourself means owning patching, monitoring, backups and restores.
  • Machines sold for agents bill per second, and stopped machines can still bill for storage.
Module 1413 / 33

Three "bring your own" terms

TermMeaning
Bring your own cloudThe vendor's software runs in the customer's account
Bring your own keyThe customer controls the encryption keys, in a single-tenant environment
Bring your own inferenceThe customer supplies the model endpoint
Module 1414 / 33

In-class exercise: cost per session

  • A personal assistant agent is reported to cost $3,000 to $7,000 per user per year.
  • The figure is second-hand.
  • The figure depends on scale and model choice.
  • Revenue per user has to cover the inference cost of an always-on agent.
  • Convert the range to a monthly figure, then to a cost per session.
Module 1415 / 33

What agents changed in the build-or-buy decision

  • Build cost: one long agent run can produce a focused version.
  • Scale: you serve one organisation, and a vendor serves its whole base.
  • Fixing defects: you can reproduce and fix a defect at once.
  • A vendor's defect waits on the vendor.
  • Reliability: you can measure your own uptime, and a vendor's figure is self-reported.
Module 1416 / 33

What agents did not change in the build-or-buy decision

  • Someone must still specify the build and verify it.
  • Fixing your own defects puts you on call.
  • Switching: moving thousands of people off a tool is painful.
  • Ownership: you still carry security, compliance and maintenance for code nobody wrote by hand.
  • Software that can be built for the price of tokens falls in market value toward that price.
  • Value moves to distribution, data, switching cost and operation.
Module 1417 / 33

The difference between a demonstration and a product

  • A prototype of a customer-support inbox takes an afternoon.
  • Email: deliverability, bounces, sender authentication records, reputation drift.
  • Email: broken threading, nested forwards, quoted text, auto-responder loops.
  • Collaboration: two people drafting one reply; internal notes; synchronised drafts.
  • Time: snooze across time zones and daylight-saving changes.
  • Identity: one customer appearing as several; shared addresses; merge and un-merge.
Module 1418 / 33

The other four headings of the list

  • Messaging and webhooks: duplicated, reordered and missing deliveries; rate limits.
  • Real time: reconnect storms, half-open connections, ordering, read state across tabs.
  • AI layer: scanned PDFs, stale facts, conflicting sources, re-indexing, injected pages.
  • AI layer: false promises, double refunds, hand-off to a person, re-evaluation.
  • Operations: malware uploads, deletion requests, silent job failures, full disks.
  • Operations: backups and tested restores; monitoring of the monitoring.
Module 1419 / 33

Three named properties

  • Idempotency: an operation whose outcome is unknown is checked before it is retried.
  • Read-back verification: after a write, read the result and compare it with what you intended.
  • Separate states need separate evidence.
  • Paid, label created, accepted by the carrier, delivered and paid out are five facts.
Module 1420 / 33

Worked example: the publish that timed out

  1. The publish request times out. The outcome is unknown.
  2. Check first: ask the system whether the item now exists.
  3. If it exists, read it back and compare with what was intended.
  4. If it does not exist, send once more, then read back.
  5. Record which state is proven, and by what evidence.
Module 1421 / 33

Release engineering

  • Validate offline, then upload, and stop on the first failure.
  • Dependencies come from a lockfile.
  • Downloaded models and binaries are pinned to a commit and verified by checksum.
  • Builds need no network after setup.
  • Sign and notarise desktop software.
  • Plan the monitoring before the deploy, and record what was verified on which versions.
Module 1422 / 33

Worked example: the bill for a test matrix

job           elapsed   billed
5-way matrix  2 min     10+ min
tiny job      5 s       1 min
merge rerun   repeats what passed

billed and elapsed time differ
most waste lies between them
Module 1423 / 33

Five steps for continuous integration

  1. Measure total billed minutes by workflow and job, with median and 90th-percentile elapsed time.
  2. Learn what each check protects and find the critical path.
  3. Look for known waste.
  4. Estimate each saving from the measured data, and state the risk.
  5. Make changes in small steps and comment each one.
Module 1424 / 33

In-class exercise: waste in a pipeline

runs on: every push, every path
cancel older runs: no
jobs:
  unit tests
  unit tests again, same commit
  12 lint jobs, about 5 s each
runner: costly OS image, all jobs
dependency cache: none
timeout: none
Module 1425 / 33

Eleven signals from four sources for platform work

  • Incident reviews, the cost bill and the team's own toil.
  • Conversations with users, uses the platform was not designed for and joint prototypes.
  • Stated objectives, things management repeats and teams that never migrated.
  • From the wider industry: a written description of the current system, and lessons from fields that are ahead.
  • The strongest signal is a use the platform was not designed for.
  • The failure to avoid is a backlog assembled from the loudest signals.
Module 1426 / 33

The evidence ladder for product learning

  • Recruit a small group who already have the problem.
  • Ask about their last real attempt, never their opinion of a feature.
  • Rung 1 is agreeing to talk, which shows willingness to talk.
  • Rung 2 is one real task completed, which shows the product can help once.
  • Rung 3 is being chosen again when the need recurs, which is the first evidence of value.
  • For an agent, the sign of value is another real assignment that finishes correctly under the intended supervision.
Module 1427 / 33

Ask the room: the date that orders a history

  • A project keeps national law as Git history.
  • Each law taking effect is one commit.
  • The history yields the text in force on any date.
  • Commits were made out of chronological order.
  • Name the date the tools must sort by, and where it is stored.
Module 1428 / 33

Common mistakes

  • Splitting into services for theoretical scale. Instead, keep one unit until a measured constraint requires a split.
  • Implementing first. Instead, write the adopter's call site, then build behind it.
  • Retrying an operation whose outcome is unknown. Instead, check, then retry.
  • Believing a "saved" indicator. Instead, read back and compare.
  • Trimming a pipeline by intuition. Instead, measure billed minutes and learn what each check protects.
  • Counting a backup as a restore. Instead, perform a restore and time it.
Module 1429 / 33

Laboratory 9: what is due

  • A one-page architecture note justifying one deployable unit or more.
  • The interface, designed call-site first.
  • The server with no inbound ports open.
  • A spending cap and a billing alert, each shown in a screenshot.
  • A release script that validates before deploying, and a restore performed and timed.
  • The 14.5 items that apply, each marked handled, deferred or not applicable.
Module 1430 / 33

Assignment 6 · A search engine served securely: what is due

  • An index and query engine over your own files, scored on 20 queries.
  • One page and one search endpoint, bound to the local machine.
  • Outside access by tunnel or private network, with no inbound port opened.
  • Limits on query length and request rate.
  • A results page that works with scripts disabled, and a stable address per result.
  • Hand in the prompts, the session log, the verifying commands and the list of what was not checked.
Module 1431 / 33

Section problems: what is due

  • Problem 5, the usage ledger: tokens per task, grouped by model.
  • Write the examples for its helper before the helper.
  • Print the three most expensive pairs and move one task to a cheaper model.
  • Problem 6, the fastest one-off: the lowest latency no other call shares.
  • State the result for an empty log before writing code.
  • Hand in the prompts, the session log, the verifying commands and the list of what was not checked.
Module 1432 / 33

Recap

  • Keep one deployable unit until a measured constraint requires a split.
  • Design the call site first and keep the public surface small.
  • Model the cost per session before building.
  • Owning a product means owning the eight headings of 14.5.
  • Check before retrying, read back after writing, prove each state separately.
  • Validate offline before upload, and measure billed minutes before cutting a check.
Module 1433 / 33

Before next class

  • Begin Laboratory 9 with the one-page architecture note.
  • Perform one restore from backup and write down how long it took.
  • Start section problem 5, the usage ledger.
  • Read Module 15, sections 15.1 to 15.3.
  • Find your machine's memory size and bring the number.