Syllabus Module 5 · The Transformer and the scaling era (2013–2026)
NSW 3011 / 33
Software Engineering with Language Models

Module 5 · The Transformer and the scaling era (2013–2026)

Class time
60 minutes
Slides
33, each with teacher’s notes
Goes with
Problem set; Assignment 2 (Part IV)
Module 52 / 33

Where this module sits

  • Module 4 ended in 2012, when data, compute and technique were available together.
  • The n-gram model still had two limits: a short context and words treated as unrelated symbols.
  • Bengio and colleagues (2003) had replaced discrete symbols with learned vectors.
  • This class runs from word vectors in 2013 to coding agents in 2026.
  • We teach each step as a mechanical change and name its paper.
  • Part III then takes up the harness, the program that wraps the model.
Module 53 / 33

Objectives

  • Trace the route from word vectors to agents that use tools.
  • Name the paper that introduced each step.
  • Explain what changed mechanically at each step.
  • Reproduce the Transformer's five numbered mechanics on a whiteboard.
  • Compute attention weights for a small case by hand.
  • Derive from the cost of attention why context length is a budget.
Module 54 / 33

Agenda

  • 9 min: word vectors, sequence-to-sequence, attention.
  • 20 min: the Transformer, with two calculations by hand.
  • 8 min: pretraining, scale and the prompt.
  • 10 min: alignment, then eight steps from chat to agent.
  • 5 min: parallel lines and what the history predicts.
  • 8 min: mistakes, problem set, Assignment 2, recap.
Module 55 / 33

5.1 Word vectors from word2vec and GloVe

  • word2vec (Mikolov et al., 2013) learned word embeddings from billions of words.
  • Its network was shallow.
  • Directions in the vector space carry meaning.
  • The step from "man" to "woman" is close to the step from "king" to "queen".
  • GloVe (2014) arrived at similar vectors by counting which words occur together.
Module 56 / 33

Worked example: the king and queen offset

man   = (1, 1)
woman = (1, 3)
king  = (5, 1)

woman - man   = (0, 2)
king + (0, 2) = (5, 3)

word nearest (5, 3): queen
Module 57 / 33

5.1 Sequence-to-sequence and its bottleneck

source words
     |
[encoder LSTM]
     |
one fixed vector   <- bottleneck
     |
[decoder LSTM]
     |
target words, one at a time
Module 58 / 33

5.2 Attention

  • Bahdanau, Cho and Bengio (2014) removed the fixed-vector bottleneck.
  • At each step the decoder computes a weighted average over all encoder states.
  • The decoder learns to assign the weights according to relevance.
  • The model learns to align source and target words without being told how.
  • Attention began as an add-on to recurrent networks.
Module 59 / 33

5.3 Self-attention and position in the Transformer

  • Vaswani et al. (2017) dropped the recurrent network entirely in "Attention Is All You Need".
  • Self-attention: every position attends directly to every other position in the same layer.
  • Step 1: map each token to a vector.
  • Add positional information to that vector.
  • Attention by itself is order-blind.
Module 510 / 33

5.3 Query-key-value attention (step 2)

score(i, j) = q_i . k_j / sqrt(d)
w(i, .)     = softmax of score(i, .)
out_i       = sum over j of w(i, j) v_j
Module 511 / 33

Worked example: attention weights by hand

d = 4; divide scores by sqrt(4) = 2
q3 = (2, 0, 2, 0)
k1 = (1, 0, 1, 0)  q3.k1 = 4 -> 2
k2 = (0, 1, 0, 1)  q3.k2 = 0 -> 0
k3 = (1, 1, 0, 0)  q3.k3 = 2 -> 1
softmax(2, 0, 1)
  = (0.665, 0.090, 0.245)
v1 = 10, v2 = 20, v3 = 30
out3 = 6.65 + 1.80 + 7.35 = 15.8
Module 512 / 33

5.3 Heads and the feed-forward block (steps 3 and 4)

  • Step 3: several attention heads run in parallel.
  • Different heads can track different relations at once.
  • Examples of such relations are syntax, coreference and position.
  • Step 4: a position-wise feed-forward network follows each attention block.
  • Residual connections wrap the attention block and the feed-forward network.
  • Layer normalisation also wraps the attention block and the feed-forward network.
Module 513 / 33

5.3 The causal mask (step 5)

may the row attend to the column?

      t1   t2   t3   t4
t1    yes  no   no   no
t2    yes  yes  no   no
t3    yes  yes  yes  no
t4    yes  yes  yes  yes
Module 514 / 33

In-class exercise: the causal mask

  • The setup is three tokens, one head and one decoder layer.
  • The scaled scores for the query at position 2 are 0, 0 and 4.
  • Compute position 2's weights with the causal mask applied.
  • Replace token 3 with a different token and compute the weights again.
  • Write the assertion that a test of the mask must make.
Module 515 / 33

5.3 Two consequences and one cost

  • Training parallelises across all positions of a sequence.
  • The Transformer therefore uses GPUs far more efficiently than a recurrent network does.
  • Any two tokens are joined by a path of length one.
  • Long-range dependencies are therefore easy to learn.
  • The cost is that attention is quadratic in sequence length.
  • Context length is a budget for that reason (Module 1).
Module 516 / 33

Ask the room: the cost of a longer context

  • A sequence has 1,000 tokens.
  • Count the query and key pairs that one attention head scores.
  • The sequence grows to 10,000 tokens.
  • Count the pairs again.
  • State what the two counts imply for filling a large context window.
Module 517 / 33

5.4 Pretraining in GPT and BERT

PropertyGPTBERT
PaperRadford et al., 2018Devlin et al., 2018
Transformer partDecoderEncoder
Pretraining taskPredict the next tokenFill in masked tokens
Context usedTokens beforeBoth sides
AfterwardsFine-tuned per taskFine-tuned per task
Module 518 / 33

5.4 Self-supervision and subword tokens

  • The pretraining objective is self-supervised.
  • The text supplies its own labels.
  • The supply of training data is therefore the written record.
  • Byte-pair encoding (Sennrich et al., 2016) is a method of subword tokenisation.
  • Its vocabulary is fixed at tens of thousands of pieces.
  • Any string can be spelled from those pieces, which ended the out-of-vocabulary problem.
Module 519 / 33

5.5 Scale and the prompt (GPT-2 and GPT-3)

  • GPT-2 (2019) had 1.5 billion parameters and wrote coherent paragraphs.
  • GPT-3 (2020) had 175 billion parameters.
  • Given a description and a few examples, a large enough model performs a new task.
  • The description and the examples go in the input, and no weight is updated.
  • This is called in-context learning.
  • The prompt replaced fine-tuning as the interface.
Module 520 / 33

5.5 Scaling laws

  • Kaplan et al. (2020) measured that loss falls as a smooth power law.
  • The law holds in parameters, in data and in compute.
  • Laboratories could then plan budgets around capability.
  • Hoffmann et al. (2022) corrected the recipe.
  • Large models before 2022 had too little training for their size.
  • Parameters and training tokens should grow together.
Module 521 / 33

5.6 Alignment to instructions

  • Straight from pretraining, a model continues text and does not follow requests.
  • Instruction tuning is fine-tuning on instructions paired with the responses wanted.
  • Reinforcement learning from human feedback comes from Christiano et al. (2017) and Ouyang et al. (2022).
  • With written principles (Bai et al., 2022) the model is trained against a document of stated values and behaviours.
  • ChatGPT (November 2022) combined a capable model, this tuning and a chat interface.
  • ChatGPT reached a hundred million users in about two months.
Module 522 / 33

5.6 Reinforcement learning from human feedback

people compare pairs of outputs
            |
   preferences (A over B)
            |
   train a reward model
            |
optimise the language model
  against the reward model
Module 523 / 33

5.7 From chat to agent (2020 to 2023)

YearStepMechanical change
2020Retrieval-augmented generation (Lewis et al.)Documents are fetched at query time and placed in the context
2022Chain-of-thought prompting (Wei et al.)A request for intermediate reasoning raises multi-step accuracy
2022ReAct (Yao et al.)Reasoning text, actions and the results observed alternate in a single loop
2023Function calling in commercial APIsThe model's output is a structured request, which the surrounding program runs and answers
Module 524 / 33

5.7 From chat to agent (2023 to 2026)

YearStepMechanical change
2023 to 2024Long context, multimodal inputWindows of hundreds of thousands of tokens; image, audio and document input
2024Model Context Protocol; computer useTools and data sources get a standard wire format; a model operates a screen
2024 to 2025Reasoning modelsTraining to deliberate out of sight, on a variable budget, before answering
2025 to 2026Coding agents and harnessesA model looped with a shell, a file system and version control, for hours on one task
Module 525 / 33

Worked example: a trace of one agent turn

user    : make test_cart.py pass
model   : request: run the tests
harness : runs them, appends output
model   : reasoning text, then
          request: edit cart.py
harness : edits, reruns, appends
model   : "the tests now pass"
Module 526 / 33

5.8 Parallel lines worth knowing

  • Llama, Mistral, Qwen, DeepSeek and others release open weights that anyone can run locally.
  • Quantisation stores weights in 4 to 8 bits instead of 16.
  • With efficient runtimes, capable models run on a laptop (Module 15).
  • Diffusion models (Ho et al., 2020) learn to reverse a gradual noising process.
  • Diffusion models took over image, audio and video generation.
  • Interpretability is work on reading the internal computations of trained networks.
Module 527 / 33

5.9 Predictions from the history

  • Capabilities that needed careful prompting become default behaviour a generation later.
  • Scaffolding that compensates for a weakness does not last long.
  • A fixed level of capability costs less by a large factor each year.
  • Designs that are too expensive now are worth prototyping.
  • Evaluation lags capability: a widely reported benchmark is close to saturated.
  • Private evaluation on your own task is the kind that stays informative (Module 8).
Module 528 / 33

Ask the room: the claim "it only predicts the next token"

  • The claim is that a model only predicts the next token, so it cannot plan.
  • State what is correct in the claim, using the mechanism from 5.3.
  • State what the claim omits, using the table in 5.7.
  • Work in pairs for two minutes.
  • One pair reports on each half.
Module 529 / 33

Common mistakes

  • Treating a large context window as free. Fix: count tokens, because attention cost is quadratic.
  • Crediting the 2017 paper with inventing attention. Fix: attention came from Bahdanau, Cho and Bengio in 2014.
  • Confusing GPT with BERT. Fix: ask which pretraining task was used, next token or masked token.
  • Believing in-context learning updates weights. Fix: nothing is trained, so resend the examples.
  • Saying the model ran a command. Fix: the model requested it and the harness ran it.
  • Trusting a widely quoted benchmark. Fix: write a private evaluation on your own task.
Module 530 / 33

Problem set

  • Problem 1: implement one head of scaled dot-product attention in fewer than twenty lines of NumPy.
  • Hand in the code and the causal-mask verification.
  • Problem 2: train a bigram model over characters and a one-layer Transformer on the same small corpus.
  • Hand in the held-out loss for each model and explain the gap by context length.
  • Problem 3: take one row of the 5.7 table and find the earliest tool you use that depends on it.
  • Problem 4: write a one-page response to the claim "it only predicts the next token".
Module 531 / 33

Assignment 2 (Part IV): the label drill and the pyramid

  • There are five warm-ups, and the hand-worked answers go in by-hand.txt before anything runs.
  • The label drill stops at CORRECT_IN_A_ROW = 3 or MAX_ITEMS = 30.
  • In the pyramid you build the three models of the problem set, bottom row first.
  • The corpus path, the context length and the data split are constants shared by all rows.
  • Change the context length, rebuild all three rows and report what moved.
  • Hand in the prompts, the session log, the verifying commands and the list of what was not checked.
Module 532 / 33

Recap

  • Word vectors turned relations between words into directions (2013).
  • Attention removed the fixed-vector bottleneck (2014), and the Transformer removed recurrence (2017).
  • One layer has five mechanics: position, query-key-value attention, heads, a feed-forward network and a causal mask.
  • Parallel training and one-hop paths enabled scale, and quadratic attention is the cost.
  • Pretraining and scale made the prompt the interface in place of fine-tuning.
  • The model still predicts tokens, and the harness decides what those tokens may do.
Module 533 / 33

Before next class

  • Do problem 1 first, because its mask check is reused in row 2 of the pyramid.
  • Work the Assignment 2 warm-ups by hand and fill in by-hand.txt.
  • Read Module 6, sections 6.1 to 6.7.
  • Choose a repository you own for Laboratory 2.
  • Bring one rule from an instruction file you use.