Software Engineering with Language Models
Module 15 · Local models and fine-tuning on your own hardware
- Class time
- 60 minutes
- Slides
- 33, each with teacher’s notes
- Goes with
- Laboratory 10
Where this module sits
- Module 1 defined quantisation, fine-tuning and decision models.
- Module 14 gave the cost of hosted designs per session.
- In this class you decide when a model should run on hardware you control.
- You size the model to the machine, adapt it and measure the result.
- The capstone asks for the running cost per user per month in its cost model.
Objectives
- Give four reasons to run a model locally and say what stands against them.
- Compute a memory budget from weights, cache and overhead.
- Test an advertised local configuration with arithmetic.
- Match a model tier to a machine and a small specialised model to a narrow job.
- Describe the phases, objectives and guardrails of a fine-tuning run.
- Score hosted, local and tuned models on a blind-graded test.
Agenda
- 4 min: where this module sits and the objectives.
- 4 min: reasons to run locally.
- 14 min: the memory budget, two worked examples and speculative decoding.
- 12 min: sizing to the machine and small specialised models.
- 13 min: the phases, objectives and guardrails of fine-tuning and a worked run.
- 13 min: interpretability, common mistakes, Laboratory 10 and the recap.
Four reasons to run a model locally
- Confidentiality: some data must not leave the premises, by contract or by choice.
- Cost: a local model has no charge per token once the hardware exists.
- Latency and availability: a local model needs no network round trip and works offline.
- Control: a local model is fixed and does not change under you.
- A capability gap to the best hosted models stands against these reasons.
- The cost of owning hardware and operations also stands against them.
Ask the room: local or hosted
- Case 1: a contract says the client notes may not leave the premises.
- Case 2: a one-off analysis needs the strongest model available.
- Case 3: a field tool is used where there is no network.
- Case 4: a grader must give the same verdicts next year.
- For each case, name the reason that decides it and say what is given up.
The memory budget
- Memory required = weights + key-value cache for the context + runtime overhead.
- Weights at 4 bits per parameter take about half a gigabyte per billion parameters.
- Context length is a memory cost, because the cache grows with every token held.
- The first question for any local setup is how much model fits.
- The second question is how much context can be kept.
- The third question is how much computation the runtime can avoid.
The memory budget as a diagram
+------------------------------+
| weights |
| set by parameters x bits |
+------------------------------+
| key-value cache |
| grows with tokens in context |
+------------------------------+
| runtime overhead |
+------------------------------+
the total must fit the machine
Worked example: 27 billion parameters at 4 bits
parameters 27 billion
bits each 4
bytes each 0.5
weights 27 x 0.5 = 13.5 GB
card 16 GB
remaining 16 - 13.5 = 2.5 GB
must hold cache + overhead
Worked example: testing an advertised claim
claim 27B model, "4-bit quality"
8.65 GB file
256,000-token context
16 GB card
check 8.65 GB x 8 / 27 billion
= about 2.5 bits per weight
cache for 256,000 tokens:
memory not accounted for
Bits traded for context and throughput
- The figures come from the published cards for one large open-weight model.
- The 2.9-bit build is about 90 GB smaller than the 4-bit build.
- The smaller build holds nearly three times the cached context.
- Its throughput under heavy concurrency is about half that of the larger build.
- All builds score within a point of the original on three standard test sets.
Speculative decoding
- A cheap drafter proposes several next tokens.
- The main model verifies them in one batched pass.
- Accepted tokens cost no separate forward pass.
- Output is unchanged when verification is exact.
- The gain depends on how often the draft is accepted.
One round of speculative decoding
drafter proposes t1 t2 t3 t4
main model checks all four, one pass
result ok ok ok no
kept t1 t2 t3
cost 1 pass, 3 tokens
then main model's own
token replaces t4
Sizing a model to the machine
| Machine memory | Model class | Score | Download |
|---|---|---|---|
| 128 GB and above | Largest open model | 21/22 | 80 GB |
| 32 to 96 GB | 27B class | 21/22 | 18 GB |
| 16 to 31 GB | 8B class | 14/22 | 6 GB |
Lessons from the reference assistant
- At launch the application checks memory and downloads the model that fits.
- The test has 22 questions from the task, graded blind against official guidance.
- The application draws about 100 watts while answering.
- It takes 5 to 40 seconds per question.
- It gives 100 to 300 answers per battery charge.
- Every answer lists its articles and opens the source at the passage.
Small specialised models
- The most useful local models are often not chat models.
- Decision classifiers have a few hundred million parameters, answer in milliseconds and need under a gigabyte.
- Forecasting models have 5 to 209 million parameters.
- A forecaster takes a history of values and returns future quantiles.
- A forecaster answers in 10 to 140 milliseconds on a processor.
- In vision, open recorders detect objects in ordinary camera feeds on local hardware.
A voice agent on a phone
- Speech recognition runs on the device.
- A small language model runs on the device.
- A 32-million-parameter encoder decides which character the user addressed.
- A speech synthesiser of 31 million parameters runs at 8 bits on the processor.
- The agent generates the next line while the current line is being spoken.
- Claims for native speech-to-speech models are not yet backed by weights or benchmarks.
The model built into the operating system
- One desktop system is reported to ship a terminal command for its on-device model.
- The command supports piping.
- It can return schema-constrained JSON.
- It runs a local server that speaks the common chat interface.
- Its window is 4,096 tokens.
- That window limits it to summarising, extracting, classifying and rewriting.
Ask the room: the recogniser and the still mouth
- A lip-reading dictation tool sent video of a still mouth to its recogniser.
- The recogniser returned fluent sentences with confidence.
- No text model was involved.
- Propose a fix that leaves the model unchanged.
The claim for fine-tuning and its evidence
- A small open model, tuned on good proprietary labels, can beat a frontier generalist.
- The win is on one narrow task.
- The tuned model is cheaper and faster.
- This is reported for fraud review with a 9-billion-parameter model, and no figures were published.
- The pattern is plausible when labels are plentiful and the task is fixed.
Phases 1 to 4 of a fine-tuning run
- Goal: it is questioned until concrete.
- Compute: the agent detects local hardware or prepares a rented machine.
- Base model: it fits the goal, the hardware and the licence.
- Data: find, import, write or generate it, then preview and clean it.
Phases 5 to 8 of a fine-tuning run
- Recipe: hyperparameters stay within published guidelines, and memory, time and cost are estimated first.
- Training: it runs detached and shows a live loss curve.
- Taste test: the same prompts go to the tuned and base models.
- Export: the result is an adapter or merged weights, with model card and licence files.
Chains of training objectives
| Chain | Use |
|---|---|
| Supervised fine-tuning alone | An instruction-following base |
| Continued pretraining, then supervised tuning | Teach a domain, then a dialogue form |
| Supervised tuning, then preference optimisation | Sharpen style |
Guardrails worth copying
- The agent asks for confirmation before any action that costs money.
- It asks for confirmation before installing software, sending data out or publishing.
- The model cannot bypass the confirmation.
- The first step is a real GPU operation, which exposes a broken driver.
- No secret passes through the conversation.
Worked example: two documented fine-tuning runs
- Run 1 tuned a 2-billion-parameter model to a writing voice.
- The first run used 826 examples and trained for about two epochs.
- Run 2 trained an image adapter on 160 captioned photographs.
- In the second run the step-500 checkpoint was preferred to the final checkpoint, by inspection.
- Roughly an hour on a rented 24 GB card cost about forty cents.
The vocabulary of interpretability
- Interpretability asks what individual parts of a network compute.
- Polysemantic neuron: one that appears to do several unrelated jobs.
- Superposition: the hypothesis that a network stores more features than neurons, by overlapping them.
- Sparse autoencoder: a tool for separating such features.
A contested claim and its status
- A long informal essay this season argues the opposite of the prevailing view.
- It claims that, with better tools, most neurons in modern networks are legible one at a time.
- It claims that earlier illegibility came from architecture and tooling.
- It claims that what suppresses a neuron often carries the distinction that matters.
- Its status is a working hypothesis: the percentages depend on thresholds, and key results await a paper.
In-class exercise: tests of a unit's label
- Look at strong and weak activations.
- Look at what suppresses the unit, and at its connections.
- Turn it up and read what the model writes.
- Test the explanation on held-out cases.
- Remove the unit before trusting the label.
- For a unit labelled "dates", write what would disprove the label at each step.
Ask the room: recall or derivation
- The design trains a model only on text from before a cutoff date.
- It then tests whether the model derives what was discovered after that date.
- The design separates recall from derivation.
- Say why an ordinary benchmark cannot make that separation.
Common mistakes
- Budgeting memory for weights only. Instead, add the cache and the overhead.
- Accepting "4-bit quality" from a label. Instead, divide the file size by the parameter count.
- Scoring on a general benchmark. Instead, use a handful of task questions, graded blind.
- Sending every input to a recogniser. Instead, gate the input on evidence of something to recognise.
- Taking the final checkpoint by default. Instead, compare checkpoints by inspection.
- Trusting a neuron's label. Instead, test it on held-out cases, then remove the unit.
Laboratory 10: what is due
- The task is narrow and has 500 or more labelled examples.
- (a) A memory budget for two candidate local models, checked against observed use.
- (b) A 20-question blind-graded test.
- (c) Scores for a hosted model, a local model, and the local model tuned or with retrieval.
- (d) Accuracy, latency at the 50th and 95th percentile, cost per thousand items, energy.
- (e) A statement of which system you would deploy.
Recap
- The reasons to run locally are confidentiality, cost, latency and availability, and control. The capability gap and operations stand against them.
- Memory = weights + key-value cache + overhead, and 4 bits is half a gigabyte per billion parameters.
- Check claims by arithmetic: 8.65 GB for 27 billion parameters is about 2.5 bits per weight.
- Small specialised models answer in milliseconds. Gate a recogniser's input on evidence.
- Fine-tuning has eight phases and confirmations the model cannot bypass.
- Measure with a small blind test drawn from the real task.
Before next class
- Start Laboratory 10 with part (a): two memory budgets, then observed use.
- Write five of the twenty test questions and their expected answers.
- Read Module 16 on the project, the submission and the grading table.
- Bring the name of one real person who would use your capstone project.
- List which of the laboratory modules 9 to 15 the project could draw on.