Software Engineering with Language Models
Module 1 · Making a music video with an agent
- Class time
- 75 minutes
- Slides
- 39, each with teacher’s notes
- Goes with
- Problem set 1; the twenty-second music video project; Assignment 2 (Part IV); Section problems (Part IV)
Where this module sits
- This is the first class of Part I, where we define the terms of the course.
- The course overview and its three rules come before this class.
- This class describes the model as a component, one term at a time.
- Each term is shown at a moment in the making of C0der Bøi, a music video made with an agent.
- Each term is paired with the engineering constraint it imposes.
- Next class is Module 2 on the harness, which is the program that wraps the model.
The music video
- C0der Bøi is a pop-punk music video of 2 minutes 50 seconds, made with Claude Code on 14 and 15 October 2026.
- Claudita, a cartoon personification of Claude, sings an original song in the style of a 2002 pop-punk single.
- One prompt gave 55 numbered song takes, and take 55 is in the video.
- The video has 143 shots of about 1.2 seconds each, and 15 of them are lip-synced.
- Video generation on fal cost about $56, and rendering on a Mac cost nothing.
The models and tools behind the video
- Claude Opus 5.5 in Claude Code wrote the concept, the lyrics, the edit and every script.
- Google Lyria 3.5 made the music and the vocals, and Gemini 3 Pro Image made the character sheets and the stills.
- Kling v3 turbo on fal made the clips, Kling Lip Sync made Claudita's lip-sync and Seedance 2.0 made the performance shots.
- Gemini 2.5 Pro was the listening judge that scored the song takes.
- Whisper measured the word timings and librosa the beat grid, and ffmpeg and a canvas page in headless Chrome rendered the film.
Objectives
- Make a twenty-second music video with an agent and name the tool, the cost and the check for each step.
- Define a large language model, its weights, inference and the training cutoff.
- Define eight component terms and state the constraint each one imposes.
- Explain why the context window is a hard, shared budget.
- Choose between ordinary code, a chat model, a reasoning model and a decision model.
- Rank prompt, retrieval and fine-tuning by cost and by ease of undoing.
Agenda
- 4 min: the first 30 seconds of C0der Bøi, and the numbers behind it.
- 4 min: where we are and the objectives.
- 14 min: 1.1 the object and its eight terms.
- 10 min: 1.2 the context budget.
- 12 min: 1.3 three kinds of model.
- 10 min: 1.4 adapting a model.
- 11 min: 1.5 the pipeline and 1.6 cost and rights.
- 10 min: common mistakes, work due and recap.
1.1 The large language model
- "Large language model (LLM)": a neural network, almost always a Transformer.
- It is trained to assign probabilities to the next token of a sequence.
- Weights, also called parameters: the numbers fixed by training.
- Inference: running the model.
- Training cutoff: the date after which the model saw no data.
Diagram of one inference
request text weights
(the prompt) (fixed by training)
| |
v v
+---------------------+
| model: inference |
+---------------------+
|
v
a probability for every next token
No state between calls
- Nothing carries over from one call to the next.
- A model's knowledge of a task has two sources: the weights and the text sent with the request.
- The weights were fixed at the training cutoff.
- For example, when a chat seems to remember, the application is resending the conversation.
- As a counter-example, typing a correction leaves the weights unchanged.
Tokens
- Token: the unit a model reads and writes.
- A token is a word piece from a fixed vocabulary of roughly 50,000 to 250,000 entries.
- English prose runs near 0.75 words per token.
- The constraint is that cost, speed and limits are all counted in tokens.
- For example, 1,000 words of prose is roughly 1,333 tokens.
- As a counter-example, a file's character count does not give its token count.
Three terms for the input to an inference
| Term | Definition | Constraint |
|---|---|---|
| Context window | Maximum tokens in one request, input and output together | A hard, shared budget |
| Prompt | Everything sent to the model for one inference | The only channel for task-specific information |
| System prompt | The part of the prompt the application sets before the user speaks | Establishes role and rules. It is text; enforcement sits in the harness |
Sampling and temperature
Prompt: "The kettle is"
(made-up probabilities)
token p
on 0.55 ###########
hot 0.25 #####
boiling 0.15 ###
purple 0.01
other 0.04 #
Sampling draws one token from these.
Training cutoff and hallucination
- Training cutoff: the date after which the model saw no data.
- The constraint is that anything newer must be supplied by retrieval or documentation.
- Hallucination: fluent output that is false or unsupported.
- The cause is that the model is optimised for plausible continuation.
- Plausibility and truth are different properties.
Calibration
- Calibration: agreement between stated confidence and observed accuracy.
- In a calibrated made-up case, about 90 of 100 items given 0.9 are right.
- In an uncalibrated made-up case, 60 of 100 items given 0.9 are right.
- The constraint is that an uncalibrated confidence figure cannot be used as a threshold.
- To check, compare stated confidence with accuracy on labelled items.
Ask the room: the term for each case
- A: one prompt run twice gives two different answers.
- B: confident prose describes a function that does not exist.
- C: the model knows nothing about a library released last week.
- D: a long file and its summary cannot both fit in one request.
- E: the model does not follow a rule the application set.
- For each case, name the term and the constraint it imposes.
1.2 The context budget and the resend
step input sent tokens
1 instructions+request 3,500
2 + result 1, reply 1 8,000
3 + result 2, reply 2 12,500
4 + result 3, reply 3 17,000
window after step 4: 21,500
input sent, steps 1-4: 41,000
Three consequences of the resend
- Long sessions get slower and more expensive with every step, because the input grows.
- Quality falls before the limit is reached.
- Irrelevant text in the window competes for attention with the relevant text.
- At the hard limit someone must choose what to drop.
- Matching the task to the window is the first design decision.
Worked example: a 4,096-token window
- A local model has a 4,096-token window for instructions, input and answer.
- Reserve 600 tokens for instructions and 500 for the answer (made-up allowances).
- That leaves 4,096 - 600 - 500 = 2,996 tokens for input.
- At 0.75 words per token, that is about 2,250 words of English prose.
- The model can summarise, extract from, classify or rewrite a page of text.
- The model cannot hold a source file and reason about it.
In-class exercise: filling the window
- The window is 60,000 tokens, the instruction file is 2,000 and the reserved answer is 4,000.
- Every step appends 5,000 tokens of tool output and a 1,000-token reply.
- Work in pairs on paper for four minutes.
- Find the step on which the window overflows.
- Find the input size sent on steps 1, 2 and 3.
- Name the first thing you would cut.
1.3 Three kinds of model
| Kind | Emits | How it was tuned | Use and cost |
|---|---|---|---|
| Chat | Prose | Pretrained, then tuned on human preference | General and easy; confidence poorly calibrated |
| Reasoning | Prose, after hidden deliberation | Further trained on rewards checked automatically | Multi-step work; slowest and dearest per call |
| Decision | A probability for each answer in a fixed set | Cross-entropy plus a calibration term | Classification, filtering, routing; fast and cheap |
Decision models in detail
- The input is a state and a question with a fixed answer set.
- The output is a probability for each answer, produced in a single forward pass.
- The question types are a yes/no proposition, a choice among described options and a score on an ordered rubric.
- A decision takes milliseconds to a few hundred milliseconds and costs a small fraction of a chat call.
- Published training recipes add a calibration term (a Brier loss) to the usual cross-entropy.
- The probability can then be used as a threshold.
A threshold and the guard after it
THRESHOLD = 0.90 # from measurement
p = decide(message, "Refund request?")
if p >= THRESHOLD:
if within_limit(amount): # guard
issue_refund()
else:
send_to_review()
else:
send_to_review()
The design rule and vendor figures
- Use ordinary code for control flow.
- Use a decision model for classification, filtering and routing.
- Use a reasoning model only where reasoning is needed.
- Put a deterministic guard in code after any model whose output triggers an action.
- Read vendor figures as marketing until you reproduce them on your own labelled set.
- A classifier that cannot invent text can still choose the wrong label.
Ask the room: the component for each task
- Task A: send each incoming support message to one of six queues.
- Task B: find why a test fails across four files and fix it.
- Task C: draft a friendly reply to a customer.
- Task D: retry a failed request three times and then stop.
- For each task, name the component: ordinary code, a decision model, a chat model or a reasoning model.
Worked example: a calibration check for a classifier
- A classifier labels 500 items and states 0.80 confidence on each (made-up figures).
- A hand check finds 300 of the 500 correct.
- Observed accuracy is 300 / 500 = 0.60.
- The gap is stated 0.80 minus observed 0.60 = 0.20, so the classifier is over-confident.
- A rule to "act at 0.80 or above" would act on items that are right 60% of the time.
- As policy, set the threshold from measured accuracy per confidence bin and escalate everything below it.
1.4 Four terms for adapting a model
- In-context learning: performing a new task from instructions and examples in the prompt, with no weight change.
- Retrieval-augmented generation (RAG): fetching relevant documents at request time and placing them in the context.
- Embedding: a vector for a text, placed so that similar meanings lie close together. It is the basis of semantic search.
- Fine-tuning: continuing training on your own data.
- Continued pretraining uses raw text, and supervised fine-tuning uses example conversations.
- Preference optimisation (DPO) uses pairs of better and worse answers.
Cheaper adaptation and the order of preference
- LoRA (low-rank adaptation): training a small set of added weights while the base model stays frozen.
- QLoRA: LoRA on a quantised base.
- Quantisation: storing weights at lower precision, commonly 4 or 8 bits.
- Distillation: training a small model to imitate a large one.
- The order of preference is prompt, then retrieval, then fine-tuning.
- Knowledge that was never published enters only through the context you supply.
Worked example: three adaptations of one task
- The made-up task is to answer staff questions about an internal expenses policy.
- For in-context learning, put a policy summary and three worked questions in the prompt.
- For retrieval, embed the policy sections and fetch the closest sections for each question.
- For fine-tuning, specify a dataset of reviewed question-and-answer conversations.
- Compare the three on two criteria: cost and how hard a change is to undo.
- Ship retrieval, because the policy is unpublished and too long to send whole.
Ask the room: the term for each moment in the video
- The agent rereads the 143-row edit list. What is it spending?
- One song prompt gives 55 different takes. Which term explains that?
- The image model gives Coder Boy a new face and a beard. Which term is that?
- A scoring script rates a country-sounding take highly. What has failed?
- The long session is compacted into a summary. Which budget ran out?
1.5 The pipeline from brief to stills
- Brief and concept: the agent turns a few messages about the song, the story and the look into the concept and the shot ideas.
- Lyrics: the agent drafts the lyrics, and two subagents review them for humour, craft and originality.
- Song: Lyria makes the takes, and
score_takes.pyand the listening judge injudge_avril.pyscore them. - Timing: librosa finds the beat grid, and Whisper finds the word timestamps.
- Characters and keyframes: the image model makes a character sheet for each person, then a still for each shot with that sheet as the reference.
The pipeline from clips to the final cut
- Clips:
fal_video.pysends each still and a motion prompt to Kling v3 turbo. - Lip-sync: a performance clip goes to Kling Lip Sync with the matching slice of the song.
- Three sync checks: audio cross-correlation, mouth opening against vocal loudness, and a Gemini judge watching with sound.
- Edit:
edl.pyholds one row for each shot, and the start times snap to the beat grid. - Compositor and render:
src/film.jsdraws each frame on a canvas, andrender.mjspipes the frames to ffmpeg.
1.6 Cost and rights
- Kling v3 turbo: $0.112 per second of 720p video, about $0.45 for a four-second clip.
- The whole video cost about $56 of fal credit and small Google and ElevenLabs charges, and rendering cost $0.
- Write an original melody and original lyrics. A parody may imitate an era's style, but it must not copy a recording.
- Do not depict a real person, and do not work around a provider's likeness check.
- Design objects that are clearly your own, and read each provider's terms before you sell the result.
The twenty-second music video project
- Make a music video between 19 and 21 seconds long with an agent.
- In the starter kit,
plan.pyruns the eight checks,beats.pyprints the beat grid andrender.pyrenders with ffmpeg. - The free path uses your own song and your own photos or drawings, and it costs nothing.
- On the generated path, five four-second Kling clips cost about $2.25 in October 2026.
- Hand in the video,
plan.json,credits.mdandnotes.mdwith every note you gave, word for word.
Common mistakes
- Do not count characters or words. Count tokens with a tokenizer.
- Do not treat the system prompt as enforcement. Guarantees go in the harness (Rule 2).
- Do not trust a single run. Output is sampled, so run important things more than once.
- Do not use stated confidence as a threshold. Measure calibration first.
- Do not expect recent or unpublished knowledge. Supply it by retrieval or documentation.
- Do not fine-tune first. Try the prompt, then retrieval, then fine-tuning.
Problem set 1: the three questions
- Q1: count the tokens in three of your own source files with a tokenizer library.
- Q1: state how many such files fit in a 200,000-token window.
- The window also holds a 5,000-token instruction file and a reserved 8,000-token answer.
- Q2: a classifier reports 0.9 confidence on 1,000 items and is right on 780. Say whether it is over- or under-confident and what threshold policy you would set.
- Q3: write one task as a request for in-context learning, for retrieval and as a fine-tuning dataset specification.
- Q3: say which you would ship and why.
Assignment 2 (Part IV): the warm-ups
- Put hand-worked answers in
by-hand.txtbefore you run anything. - Trace by hand: keep a running token total turn by turn and find the turn that exceeds the budget.
- Countdown: ask for summaries of 10 sentences down to 1 and count how often the instruction was followed.
- Ten runs: send one prompt ten times at temperature 1 and ten times at temperature 0 and count the distinct answers.
- Refuse the impossible:
calibration_gap(confidences, outcomes)rejects a confidence outside 0 to 1. - By hand, then by library: compute cosine similarity on paper and then on sentence embeddings.
Section problems (Part IV): strings
- These are short problems for a small group, done in under an hour, with expected results written before the code.
- Keep only: use
keep_only(text, allowed)to strip a reply to its digits, then list three replies where that fails. - Count the differences: line up two runs of one prompt at temperature 0 character by character.
- Number the lines: return
[index, line]pairs and say why tools number lines for a model. - Doubling: repeat every character, then compare character counts with token counts.
Recap
- A model assigns next-token probabilities. Its weights are fixed and it keeps no state between calls.
- Cost, speed and limits are counted in tokens, and the context window is one hard, shared budget.
- The prompt is the only channel for task information, and the system prompt is advisory text.
- Output is sampled, and a confidence figure needs calibration before it serves as a threshold.
- The design rule is code first, then a decision model, then a reasoning model, with a guard before any action.
- Adapt in this order: prompt, then retrieval, then fine-tuning.
Before next class
- Read Module 2, "The agent harness".
- Start Problem set 1 by counting the tokens in three of your own source files.
- Do warm-up 1 of Assignment 2 by hand and write it into
by-hand.txt. - Bring an instruction file you use. In Module 2's problem set you take five rules from it.
- Note one rule you rely on that exists only as a sentence in a prompt.