Software Engineering with Language Models
Module 5 · The Transformer and the scaling era (2013–2026)
- Class time
- 60 minutes
- Slides
- 33, each with teacher’s notes
- Goes with
- Problem set; Assignment 2 (Part IV)
Where this module sits
- Module 4 ended in 2012, when data, compute and technique were available together.
- The n-gram model still had two limits: a short context and words treated as unrelated symbols.
- Bengio and colleagues (2003) had replaced discrete symbols with learned vectors.
- This class runs from word vectors in 2013 to coding agents in 2026.
- We teach each step as a mechanical change and name its paper.
- Part III then takes up the harness, the program that wraps the model.
Objectives
- Trace the route from word vectors to agents that use tools.
- Name the paper that introduced each step.
- Explain what changed mechanically at each step.
- Reproduce the Transformer's five numbered mechanics on a whiteboard.
- Compute attention weights for a small case by hand.
- Derive from the cost of attention why context length is a budget.
Agenda
- 9 min: word vectors, sequence-to-sequence, attention.
- 20 min: the Transformer, with two calculations by hand.
- 8 min: pretraining, scale and the prompt.
- 10 min: alignment, then eight steps from chat to agent.
- 5 min: parallel lines and what the history predicts.
- 8 min: mistakes, problem set, Assignment 2, recap.
5.1 Word vectors from word2vec and GloVe
- word2vec (Mikolov et al., 2013) learned word embeddings from billions of words.
- Its network was shallow.
- Directions in the vector space carry meaning.
- The step from "man" to "woman" is close to the step from "king" to "queen".
- GloVe (2014) arrived at similar vectors by counting which words occur together.
Worked example: the king and queen offset
man = (1, 1)
woman = (1, 3)
king = (5, 1)
woman - man = (0, 2)
king + (0, 2) = (5, 3)
word nearest (5, 3): queen
5.1 Sequence-to-sequence and its bottleneck
source words
|
[encoder LSTM]
|
one fixed vector <- bottleneck
|
[decoder LSTM]
|
target words, one at a time
5.2 Attention
- Bahdanau, Cho and Bengio (2014) removed the fixed-vector bottleneck.
- At each step the decoder computes a weighted average over all encoder states.
- The decoder learns to assign the weights according to relevance.
- The model learns to align source and target words without being told how.
- Attention began as an add-on to recurrent networks.
5.3 Self-attention and position in the Transformer
- Vaswani et al. (2017) dropped the recurrent network entirely in "Attention Is All You Need".
- Self-attention: every position attends directly to every other position in the same layer.
- Step 1: map each token to a vector.
- Add positional information to that vector.
- Attention by itself is order-blind.
5.3 Query-key-value attention (step 2)
score(i, j) = q_i . k_j / sqrt(d)
w(i, .) = softmax of score(i, .)
out_i = sum over j of w(i, j) v_j
Worked example: attention weights by hand
d = 4; divide scores by sqrt(4) = 2
q3 = (2, 0, 2, 0)
k1 = (1, 0, 1, 0) q3.k1 = 4 -> 2
k2 = (0, 1, 0, 1) q3.k2 = 0 -> 0
k3 = (1, 1, 0, 0) q3.k3 = 2 -> 1
softmax(2, 0, 1)
= (0.665, 0.090, 0.245)
v1 = 10, v2 = 20, v3 = 30
out3 = 6.65 + 1.80 + 7.35 = 15.8
5.3 Heads and the feed-forward block (steps 3 and 4)
- Step 3: several attention heads run in parallel.
- Different heads can track different relations at once.
- Examples of such relations are syntax, coreference and position.
- Step 4: a position-wise feed-forward network follows each attention block.
- Residual connections wrap the attention block and the feed-forward network.
- Layer normalisation also wraps the attention block and the feed-forward network.
5.3 The causal mask (step 5)
may the row attend to the column?
t1 t2 t3 t4
t1 yes no no no
t2 yes yes no no
t3 yes yes yes no
t4 yes yes yes yes
In-class exercise: the causal mask
- The setup is three tokens, one head and one decoder layer.
- The scaled scores for the query at position 2 are 0, 0 and 4.
- Compute position 2's weights with the causal mask applied.
- Replace token 3 with a different token and compute the weights again.
- Write the assertion that a test of the mask must make.
5.3 Two consequences and one cost
- Training parallelises across all positions of a sequence.
- The Transformer therefore uses GPUs far more efficiently than a recurrent network does.
- Any two tokens are joined by a path of length one.
- Long-range dependencies are therefore easy to learn.
- The cost is that attention is quadratic in sequence length.
- Context length is a budget for that reason (Module 1).
Ask the room: the cost of a longer context
- A sequence has 1,000 tokens.
- Count the query and key pairs that one attention head scores.
- The sequence grows to 10,000 tokens.
- Count the pairs again.
- State what the two counts imply for filling a large context window.
5.4 Pretraining in GPT and BERT
| Property | GPT | BERT |
|---|---|---|
| Paper | Radford et al., 2018 | Devlin et al., 2018 |
| Transformer part | Decoder | Encoder |
| Pretraining task | Predict the next token | Fill in masked tokens |
| Context used | Tokens before | Both sides |
| Afterwards | Fine-tuned per task | Fine-tuned per task |
5.4 Self-supervision and subword tokens
- The pretraining objective is self-supervised.
- The text supplies its own labels.
- The supply of training data is therefore the written record.
- Byte-pair encoding (Sennrich et al., 2016) is a method of subword tokenisation.
- Its vocabulary is fixed at tens of thousands of pieces.
- Any string can be spelled from those pieces, which ended the out-of-vocabulary problem.
5.5 Scale and the prompt (GPT-2 and GPT-3)
- GPT-2 (2019) had 1.5 billion parameters and wrote coherent paragraphs.
- GPT-3 (2020) had 175 billion parameters.
- Given a description and a few examples, a large enough model performs a new task.
- The description and the examples go in the input, and no weight is updated.
- This is called in-context learning.
- The prompt replaced fine-tuning as the interface.
5.5 Scaling laws
- Kaplan et al. (2020) measured that loss falls as a smooth power law.
- The law holds in parameters, in data and in compute.
- Laboratories could then plan budgets around capability.
- Hoffmann et al. (2022) corrected the recipe.
- Large models before 2022 had too little training for their size.
- Parameters and training tokens should grow together.
5.6 Alignment to instructions
- Straight from pretraining, a model continues text and does not follow requests.
- Instruction tuning is fine-tuning on instructions paired with the responses wanted.
- Reinforcement learning from human feedback comes from Christiano et al. (2017) and Ouyang et al. (2022).
- With written principles (Bai et al., 2022) the model is trained against a document of stated values and behaviours.
- ChatGPT (November 2022) combined a capable model, this tuning and a chat interface.
- ChatGPT reached a hundred million users in about two months.
5.6 Reinforcement learning from human feedback
people compare pairs of outputs
|
preferences (A over B)
|
train a reward model
|
optimise the language model
against the reward model
5.7 From chat to agent (2020 to 2023)
| Year | Step | Mechanical change |
|---|---|---|
| 2020 | Retrieval-augmented generation (Lewis et al.) | Documents are fetched at query time and placed in the context |
| 2022 | Chain-of-thought prompting (Wei et al.) | A request for intermediate reasoning raises multi-step accuracy |
| 2022 | ReAct (Yao et al.) | Reasoning text, actions and the results observed alternate in a single loop |
| 2023 | Function calling in commercial APIs | The model's output is a structured request, which the surrounding program runs and answers |
5.7 From chat to agent (2023 to 2026)
| Year | Step | Mechanical change |
|---|---|---|
| 2023 to 2024 | Long context, multimodal input | Windows of hundreds of thousands of tokens; image, audio and document input |
| 2024 | Model Context Protocol; computer use | Tools and data sources get a standard wire format; a model operates a screen |
| 2024 to 2025 | Reasoning models | Training to deliberate out of sight, on a variable budget, before answering |
| 2025 to 2026 | Coding agents and harnesses | A model looped with a shell, a file system and version control, for hours on one task |
Worked example: a trace of one agent turn
user : make test_cart.py pass
model : request: run the tests
harness : runs them, appends output
model : reasoning text, then
request: edit cart.py
harness : edits, reruns, appends
model : "the tests now pass"
5.8 Parallel lines worth knowing
- Llama, Mistral, Qwen, DeepSeek and others release open weights that anyone can run locally.
- Quantisation stores weights in 4 to 8 bits instead of 16.
- With efficient runtimes, capable models run on a laptop (Module 15).
- Diffusion models (Ho et al., 2020) learn to reverse a gradual noising process.
- Diffusion models took over image, audio and video generation.
- Interpretability is work on reading the internal computations of trained networks.
5.9 Predictions from the history
- Capabilities that needed careful prompting become default behaviour a generation later.
- Scaffolding that compensates for a weakness does not last long.
- A fixed level of capability costs less by a large factor each year.
- Designs that are too expensive now are worth prototyping.
- Evaluation lags capability: a widely reported benchmark is close to saturated.
- Private evaluation on your own task is the kind that stays informative (Module 8).
Ask the room: the claim "it only predicts the next token"
- The claim is that a model only predicts the next token, so it cannot plan.
- State what is correct in the claim, using the mechanism from 5.3.
- State what the claim omits, using the table in 5.7.
- Work in pairs for two minutes.
- One pair reports on each half.
Common mistakes
- Treating a large context window as free. Fix: count tokens, because attention cost is quadratic.
- Crediting the 2017 paper with inventing attention. Fix: attention came from Bahdanau, Cho and Bengio in 2014.
- Confusing GPT with BERT. Fix: ask which pretraining task was used, next token or masked token.
- Believing in-context learning updates weights. Fix: nothing is trained, so resend the examples.
- Saying the model ran a command. Fix: the model requested it and the harness ran it.
- Trusting a widely quoted benchmark. Fix: write a private evaluation on your own task.
Problem set
- Problem 1: implement one head of scaled dot-product attention in fewer than twenty lines of NumPy.
- Hand in the code and the causal-mask verification.
- Problem 2: train a bigram model over characters and a one-layer Transformer on the same small corpus.
- Hand in the held-out loss for each model and explain the gap by context length.
- Problem 3: take one row of the 5.7 table and find the earliest tool you use that depends on it.
- Problem 4: write a one-page response to the claim "it only predicts the next token".
Assignment 2 (Part IV): the label drill and the pyramid
- There are five warm-ups, and the hand-worked answers go in by-hand.txt before anything runs.
- The label drill stops at CORRECT_IN_A_ROW = 3 or MAX_ITEMS = 30.
- In the pyramid you build the three models of the problem set, bottom row first.
- The corpus path, the context length and the data split are constants shared by all rows.
- Change the context length, rebuild all three rows and report what moved.
- Hand in the prompts, the session log, the verifying commands and the list of what was not checked.
Recap
- Word vectors turned relations between words into directions (2013).
- Attention removed the fixed-vector bottleneck (2014), and the Transformer removed recurrence (2017).
- One layer has five mechanics: position, query-key-value attention, heads, a feed-forward network and a causal mask.
- Parallel training and one-hop paths enabled scale, and quadratic attention is the cost.
- Pretraining and scale made the prompt the interface in place of fine-tuning.
- The model still predicts tokens, and the harness decides what those tokens may do.
Before next class
- Do problem 1 first, because its mask check is reused in row 2 of the pyramid.
- Work the Assignment 2 warm-ups by hand and fill in by-hand.txt.
- Read Module 6, sections 6.1 to 6.7.
- Choose a repository you own for Laboratory 2.
- Bring one rule from an instruction file you use.