Syllabus Module 1 · Making a music video with an agent
NSW 3011 / 39
Software Engineering with Language Models

Module 1 · Making a music video with an agent

Class time
75 minutes
Slides
39, each with teacher’s notes
Goes with
Problem set 1; the twenty-second music video project; Assignment 2 (Part IV); Section problems (Part IV)
Module 12 / 39

Where this module sits

  • This is the first class of Part I, where we define the terms of the course.
  • The course overview and its three rules come before this class.
  • This class describes the model as a component, one term at a time.
  • Each term is shown at a moment in the making of C0der Bøi, a music video made with an agent.
  • Each term is paired with the engineering constraint it imposes.
  • Next class is Module 2 on the harness, which is the program that wraps the model.
Module 13 / 39

The music video

  • C0der Bøi is a pop-punk music video of 2 minutes 50 seconds, made with Claude Code on 14 and 15 October 2026.
  • Claudita, a cartoon personification of Claude, sings an original song in the style of a 2002 pop-punk single.
  • One prompt gave 55 numbered song takes, and take 55 is in the video.
  • The video has 143 shots of about 1.2 seconds each, and 15 of them are lip-synced.
  • Video generation on fal cost about $56, and rendering on a Mac cost nothing.
Module 14 / 39

The models and tools behind the video

  • Claude Opus 5.5 in Claude Code wrote the concept, the lyrics, the edit and every script.
  • Google Lyria 3.5 made the music and the vocals, and Gemini 3 Pro Image made the character sheets and the stills.
  • Kling v3 turbo on fal made the clips, Kling Lip Sync made Claudita's lip-sync and Seedance 2.0 made the performance shots.
  • Gemini 2.5 Pro was the listening judge that scored the song takes.
  • Whisper measured the word timings and librosa the beat grid, and ffmpeg and a canvas page in headless Chrome rendered the film.
Module 15 / 39

Objectives

  • Make a twenty-second music video with an agent and name the tool, the cost and the check for each step.
  • Define a large language model, its weights, inference and the training cutoff.
  • Define eight component terms and state the constraint each one imposes.
  • Explain why the context window is a hard, shared budget.
  • Choose between ordinary code, a chat model, a reasoning model and a decision model.
  • Rank prompt, retrieval and fine-tuning by cost and by ease of undoing.
Module 16 / 39

Agenda

  • 4 min: the first 30 seconds of C0der Bøi, and the numbers behind it.
  • 4 min: where we are and the objectives.
  • 14 min: 1.1 the object and its eight terms.
  • 10 min: 1.2 the context budget.
  • 12 min: 1.3 three kinds of model.
  • 10 min: 1.4 adapting a model.
  • 11 min: 1.5 the pipeline and 1.6 cost and rights.
  • 10 min: common mistakes, work due and recap.
Module 17 / 39

1.1 The large language model

  • "Large language model (LLM)": a neural network, almost always a Transformer.
  • It is trained to assign probabilities to the next token of a sequence.
  • Weights, also called parameters: the numbers fixed by training.
  • Inference: running the model.
  • Training cutoff: the date after which the model saw no data.
Module 18 / 39

Diagram of one inference

 request text        weights
 (the prompt)   (fixed by training)
       |               |
       v               v
   +---------------------+
   |  model: inference   |
   +---------------------+
              |
              v
 a probability for every next token
Module 19 / 39

No state between calls

  • Nothing carries over from one call to the next.
  • A model's knowledge of a task has two sources: the weights and the text sent with the request.
  • The weights were fixed at the training cutoff.
  • For example, when a chat seems to remember, the application is resending the conversation.
  • As a counter-example, typing a correction leaves the weights unchanged.
Module 110 / 39

Tokens

  • Token: the unit a model reads and writes.
  • A token is a word piece from a fixed vocabulary of roughly 50,000 to 250,000 entries.
  • English prose runs near 0.75 words per token.
  • The constraint is that cost, speed and limits are all counted in tokens.
  • For example, 1,000 words of prose is roughly 1,333 tokens.
  • As a counter-example, a file's character count does not give its token count.
Module 111 / 39

Three terms for the input to an inference

TermDefinitionConstraint
Context windowMaximum tokens in one request, input and output togetherA hard, shared budget
PromptEverything sent to the model for one inferenceThe only channel for task-specific information
System promptThe part of the prompt the application sets before the user speaksEstablishes role and rules. It is text; enforcement sits in the harness
Module 112 / 39

Sampling and temperature

Prompt: "The kettle is"
(made-up probabilities)
 token    p
 on       0.55  ###########
 hot      0.25  #####
 boiling  0.15  ###
 purple   0.01
 other    0.04  #
Sampling draws one token from these.
Module 113 / 39

Training cutoff and hallucination

  • Training cutoff: the date after which the model saw no data.
  • The constraint is that anything newer must be supplied by retrieval or documentation.
  • Hallucination: fluent output that is false or unsupported.
  • The cause is that the model is optimised for plausible continuation.
  • Plausibility and truth are different properties.
Module 114 / 39

Calibration

  • Calibration: agreement between stated confidence and observed accuracy.
  • In a calibrated made-up case, about 90 of 100 items given 0.9 are right.
  • In an uncalibrated made-up case, 60 of 100 items given 0.9 are right.
  • The constraint is that an uncalibrated confidence figure cannot be used as a threshold.
  • To check, compare stated confidence with accuracy on labelled items.
Module 115 / 39

Ask the room: the term for each case

  • A: one prompt run twice gives two different answers.
  • B: confident prose describes a function that does not exist.
  • C: the model knows nothing about a library released last week.
  • D: a long file and its summary cannot both fit in one request.
  • E: the model does not follow a rule the application set.
  • For each case, name the term and the constraint it imposes.
Module 116 / 39

1.2 The context budget and the resend

step  input sent            tokens
1     instructions+request   3,500
2     + result 1, reply 1    8,000
3     + result 2, reply 2   12,500
4     + result 3, reply 3   17,000

window after step 4:        21,500
input sent, steps 1-4:      41,000
Module 117 / 39

Three consequences of the resend

  • Long sessions get slower and more expensive with every step, because the input grows.
  • Quality falls before the limit is reached.
  • Irrelevant text in the window competes for attention with the relevant text.
  • At the hard limit someone must choose what to drop.
  • Matching the task to the window is the first design decision.
Module 118 / 39

Worked example: a 4,096-token window

  1. A local model has a 4,096-token window for instructions, input and answer.
  2. Reserve 600 tokens for instructions and 500 for the answer (made-up allowances).
  3. That leaves 4,096 - 600 - 500 = 2,996 tokens for input.
  4. At 0.75 words per token, that is about 2,250 words of English prose.
  5. The model can summarise, extract from, classify or rewrite a page of text.
  6. The model cannot hold a source file and reason about it.
Module 119 / 39

In-class exercise: filling the window

  • The window is 60,000 tokens, the instruction file is 2,000 and the reserved answer is 4,000.
  • Every step appends 5,000 tokens of tool output and a 1,000-token reply.
  • Work in pairs on paper for four minutes.
  • Find the step on which the window overflows.
  • Find the input size sent on steps 1, 2 and 3.
  • Name the first thing you would cut.
Module 120 / 39

1.3 Three kinds of model

KindEmitsHow it was tunedUse and cost
ChatProsePretrained, then tuned on human preferenceGeneral and easy; confidence poorly calibrated
ReasoningProse, after hidden deliberationFurther trained on rewards checked automaticallyMulti-step work; slowest and dearest per call
DecisionA probability for each answer in a fixed setCross-entropy plus a calibration termClassification, filtering, routing; fast and cheap
Module 121 / 39

Decision models in detail

  • The input is a state and a question with a fixed answer set.
  • The output is a probability for each answer, produced in a single forward pass.
  • The question types are a yes/no proposition, a choice among described options and a score on an ordered rubric.
  • A decision takes milliseconds to a few hundred milliseconds and costs a small fraction of a chat call.
  • Published training recipes add a calibration term (a Brier loss) to the usual cross-entropy.
  • The probability can then be used as a threshold.
Module 122 / 39

A threshold and the guard after it

THRESHOLD = 0.90  # from measurement

p = decide(message, "Refund request?")
if p >= THRESHOLD:
    if within_limit(amount):    # guard
        issue_refund()
    else:
        send_to_review()
else:
    send_to_review()
Module 123 / 39

The design rule and vendor figures

  • Use ordinary code for control flow.
  • Use a decision model for classification, filtering and routing.
  • Use a reasoning model only where reasoning is needed.
  • Put a deterministic guard in code after any model whose output triggers an action.
  • Read vendor figures as marketing until you reproduce them on your own labelled set.
  • A classifier that cannot invent text can still choose the wrong label.
Module 124 / 39

Ask the room: the component for each task

  • Task A: send each incoming support message to one of six queues.
  • Task B: find why a test fails across four files and fix it.
  • Task C: draft a friendly reply to a customer.
  • Task D: retry a failed request three times and then stop.
  • For each task, name the component: ordinary code, a decision model, a chat model or a reasoning model.
Module 125 / 39

Worked example: a calibration check for a classifier

  1. A classifier labels 500 items and states 0.80 confidence on each (made-up figures).
  2. A hand check finds 300 of the 500 correct.
  3. Observed accuracy is 300 / 500 = 0.60.
  4. The gap is stated 0.80 minus observed 0.60 = 0.20, so the classifier is over-confident.
  5. A rule to "act at 0.80 or above" would act on items that are right 60% of the time.
  6. As policy, set the threshold from measured accuracy per confidence bin and escalate everything below it.
Module 126 / 39

1.4 Four terms for adapting a model

  • In-context learning: performing a new task from instructions and examples in the prompt, with no weight change.
  • Retrieval-augmented generation (RAG): fetching relevant documents at request time and placing them in the context.
  • Embedding: a vector for a text, placed so that similar meanings lie close together. It is the basis of semantic search.
  • Fine-tuning: continuing training on your own data.
  • Continued pretraining uses raw text, and supervised fine-tuning uses example conversations.
  • Preference optimisation (DPO) uses pairs of better and worse answers.
Module 127 / 39

Cheaper adaptation and the order of preference

  • LoRA (low-rank adaptation): training a small set of added weights while the base model stays frozen.
  • QLoRA: LoRA on a quantised base.
  • Quantisation: storing weights at lower precision, commonly 4 or 8 bits.
  • Distillation: training a small model to imitate a large one.
  • The order of preference is prompt, then retrieval, then fine-tuning.
  • Knowledge that was never published enters only through the context you supply.
Module 128 / 39

Worked example: three adaptations of one task

  1. The made-up task is to answer staff questions about an internal expenses policy.
  2. For in-context learning, put a policy summary and three worked questions in the prompt.
  3. For retrieval, embed the policy sections and fetch the closest sections for each question.
  4. For fine-tuning, specify a dataset of reviewed question-and-answer conversations.
  5. Compare the three on two criteria: cost and how hard a change is to undo.
  6. Ship retrieval, because the policy is unpublished and too long to send whole.
Module 129 / 39

Ask the room: the term for each moment in the video

  • The agent rereads the 143-row edit list. What is it spending?
  • One song prompt gives 55 different takes. Which term explains that?
  • The image model gives Coder Boy a new face and a beard. Which term is that?
  • A scoring script rates a country-sounding take highly. What has failed?
  • The long session is compacted into a summary. Which budget ran out?
Module 130 / 39

1.5 The pipeline from brief to stills

  • Brief and concept: the agent turns a few messages about the song, the story and the look into the concept and the shot ideas.
  • Lyrics: the agent drafts the lyrics, and two subagents review them for humour, craft and originality.
  • Song: Lyria makes the takes, and score_takes.py and the listening judge in judge_avril.py score them.
  • Timing: librosa finds the beat grid, and Whisper finds the word timestamps.
  • Characters and keyframes: the image model makes a character sheet for each person, then a still for each shot with that sheet as the reference.
Module 131 / 39

The pipeline from clips to the final cut

  • Clips: fal_video.py sends each still and a motion prompt to Kling v3 turbo.
  • Lip-sync: a performance clip goes to Kling Lip Sync with the matching slice of the song.
  • Three sync checks: audio cross-correlation, mouth opening against vocal loudness, and a Gemini judge watching with sound.
  • Edit: edl.py holds one row for each shot, and the start times snap to the beat grid.
  • Compositor and render: src/film.js draws each frame on a canvas, and render.mjs pipes the frames to ffmpeg.
Module 132 / 39

1.6 Cost and rights

  • Kling v3 turbo: $0.112 per second of 720p video, about $0.45 for a four-second clip.
  • The whole video cost about $56 of fal credit and small Google and ElevenLabs charges, and rendering cost $0.
  • Write an original melody and original lyrics. A parody may imitate an era's style, but it must not copy a recording.
  • Do not depict a real person, and do not work around a provider's likeness check.
  • Design objects that are clearly your own, and read each provider's terms before you sell the result.
Module 133 / 39

The twenty-second music video project

  • Make a music video between 19 and 21 seconds long with an agent.
  • In the starter kit, plan.py runs the eight checks, beats.py prints the beat grid and render.py renders with ffmpeg.
  • The free path uses your own song and your own photos or drawings, and it costs nothing.
  • On the generated path, five four-second Kling clips cost about $2.25 in October 2026.
  • Hand in the video, plan.json, credits.md and notes.md with every note you gave, word for word.
Module 134 / 39

Common mistakes

  • Do not count characters or words. Count tokens with a tokenizer.
  • Do not treat the system prompt as enforcement. Guarantees go in the harness (Rule 2).
  • Do not trust a single run. Output is sampled, so run important things more than once.
  • Do not use stated confidence as a threshold. Measure calibration first.
  • Do not expect recent or unpublished knowledge. Supply it by retrieval or documentation.
  • Do not fine-tune first. Try the prompt, then retrieval, then fine-tuning.
Module 135 / 39

Problem set 1: the three questions

  • Q1: count the tokens in three of your own source files with a tokenizer library.
  • Q1: state how many such files fit in a 200,000-token window.
  • The window also holds a 5,000-token instruction file and a reserved 8,000-token answer.
  • Q2: a classifier reports 0.9 confidence on 1,000 items and is right on 780. Say whether it is over- or under-confident and what threshold policy you would set.
  • Q3: write one task as a request for in-context learning, for retrieval and as a fine-tuning dataset specification.
  • Q3: say which you would ship and why.
Module 136 / 39

Assignment 2 (Part IV): the warm-ups

  • Put hand-worked answers in by-hand.txt before you run anything.
  • Trace by hand: keep a running token total turn by turn and find the turn that exceeds the budget.
  • Countdown: ask for summaries of 10 sentences down to 1 and count how often the instruction was followed.
  • Ten runs: send one prompt ten times at temperature 1 and ten times at temperature 0 and count the distinct answers.
  • Refuse the impossible: calibration_gap(confidences, outcomes) rejects a confidence outside 0 to 1.
  • By hand, then by library: compute cosine similarity on paper and then on sentence embeddings.
Module 137 / 39

Section problems (Part IV): strings

  • These are short problems for a small group, done in under an hour, with expected results written before the code.
  • Keep only: use keep_only(text, allowed) to strip a reply to its digits, then list three replies where that fails.
  • Count the differences: line up two runs of one prompt at temperature 0 character by character.
  • Number the lines: return [index, line] pairs and say why tools number lines for a model.
  • Doubling: repeat every character, then compare character counts with token counts.
Module 138 / 39

Recap

  • A model assigns next-token probabilities. Its weights are fixed and it keeps no state between calls.
  • Cost, speed and limits are counted in tokens, and the context window is one hard, shared budget.
  • The prompt is the only channel for task information, and the system prompt is advisory text.
  • Output is sampled, and a confidence figure needs calibration before it serves as a threshold.
  • The design rule is code first, then a decision model, then a reasoning model, with a guard before any action.
  • Adapt in this order: prompt, then retrieval, then fine-tuning.
Module 139 / 39

Before next class

  • Read Module 2, "The agent harness".
  • Start Problem set 1 by counting the tokens in three of your own source files.
  • Do warm-up 1 of Assignment 2 by hand and write it into by-hand.txt.
  • Bring an instruction file you use. In Module 2's problem set you take five rules from it.
  • Note one rule you rely on that exists only as a sentence in a prompt.