Syllabus Module 15 · Local models and fine-tuning on your own hardware
NSW 3011 / 33
Software Engineering with Language Models

Module 15 · Local models and fine-tuning on your own hardware

Class time
60 minutes
Slides
33, each with teacher’s notes
Goes with
Laboratory 10
Module 152 / 33

Where this module sits

  • Module 1 defined quantisation, fine-tuning and decision models.
  • Module 14 gave the cost of hosted designs per session.
  • In this class you decide when a model should run on hardware you control.
  • You size the model to the machine, adapt it and measure the result.
  • The capstone asks for the running cost per user per month in its cost model.
Module 153 / 33

Objectives

  • Give four reasons to run a model locally and say what stands against them.
  • Compute a memory budget from weights, cache and overhead.
  • Test an advertised local configuration with arithmetic.
  • Match a model tier to a machine and a small specialised model to a narrow job.
  • Describe the phases, objectives and guardrails of a fine-tuning run.
  • Score hosted, local and tuned models on a blind-graded test.
Module 154 / 33

Agenda

  • 4 min: where this module sits and the objectives.
  • 4 min: reasons to run locally.
  • 14 min: the memory budget, two worked examples and speculative decoding.
  • 12 min: sizing to the machine and small specialised models.
  • 13 min: the phases, objectives and guardrails of fine-tuning and a worked run.
  • 13 min: interpretability, common mistakes, Laboratory 10 and the recap.
Module 155 / 33

Four reasons to run a model locally

  • Confidentiality: some data must not leave the premises, by contract or by choice.
  • Cost: a local model has no charge per token once the hardware exists.
  • Latency and availability: a local model needs no network round trip and works offline.
  • Control: a local model is fixed and does not change under you.
  • A capability gap to the best hosted models stands against these reasons.
  • The cost of owning hardware and operations also stands against them.
Module 156 / 33

Ask the room: local or hosted

  • Case 1: a contract says the client notes may not leave the premises.
  • Case 2: a one-off analysis needs the strongest model available.
  • Case 3: a field tool is used where there is no network.
  • Case 4: a grader must give the same verdicts next year.
  • For each case, name the reason that decides it and say what is given up.
Module 157 / 33

The memory budget

  • Memory required = weights + key-value cache for the context + runtime overhead.
  • Weights at 4 bits per parameter take about half a gigabyte per billion parameters.
  • Context length is a memory cost, because the cache grows with every token held.
  • The first question for any local setup is how much model fits.
  • The second question is how much context can be kept.
  • The third question is how much computation the runtime can avoid.
Module 158 / 33

The memory budget as a diagram

+------------------------------+
| weights                      |
| set by parameters x bits     |
+------------------------------+
| key-value cache              |
| grows with tokens in context |
+------------------------------+
| runtime overhead             |
+------------------------------+
  the total must fit the machine
Module 159 / 33

Worked example: 27 billion parameters at 4 bits

parameters    27 billion
bits each     4
bytes each    0.5
weights       27 x 0.5 = 13.5 GB
card          16 GB
remaining     16 - 13.5 = 2.5 GB
must hold     cache + overhead
Module 1510 / 33

Worked example: testing an advertised claim

claim  27B model, "4-bit quality"
       8.65 GB file
       256,000-token context
       16 GB card
check  8.65 GB x 8 / 27 billion
       = about 2.5 bits per weight
cache  for 256,000 tokens:
       memory not accounted for
Module 1511 / 33

Bits traded for context and throughput

  • The figures come from the published cards for one large open-weight model.
  • The 2.9-bit build is about 90 GB smaller than the 4-bit build.
  • The smaller build holds nearly three times the cached context.
  • Its throughput under heavy concurrency is about half that of the larger build.
  • All builds score within a point of the original on three standard test sets.
Module 1512 / 33

Speculative decoding

  • A cheap drafter proposes several next tokens.
  • The main model verifies them in one batched pass.
  • Accepted tokens cost no separate forward pass.
  • Output is unchanged when verification is exact.
  • The gain depends on how often the draft is accepted.
Module 1513 / 33

One round of speculative decoding

drafter proposes   t1  t2  t3  t4
main model checks  all four, one pass
result             ok  ok  ok  no
kept               t1  t2  t3
cost               1 pass, 3 tokens
then               main model's own
                   token replaces t4
Module 1514 / 33

Sizing a model to the machine

Machine memoryModel classScoreDownload
128 GB and aboveLargest open model21/2280 GB
32 to 96 GB27B class21/2218 GB
16 to 31 GB8B class14/226 GB
Module 1515 / 33

Lessons from the reference assistant

  • At launch the application checks memory and downloads the model that fits.
  • The test has 22 questions from the task, graded blind against official guidance.
  • The application draws about 100 watts while answering.
  • It takes 5 to 40 seconds per question.
  • It gives 100 to 300 answers per battery charge.
  • Every answer lists its articles and opens the source at the passage.
Module 1516 / 33

Small specialised models

  • The most useful local models are often not chat models.
  • Decision classifiers have a few hundred million parameters, answer in milliseconds and need under a gigabyte.
  • Forecasting models have 5 to 209 million parameters.
  • A forecaster takes a history of values and returns future quantiles.
  • A forecaster answers in 10 to 140 milliseconds on a processor.
  • In vision, open recorders detect objects in ordinary camera feeds on local hardware.
Module 1517 / 33

A voice agent on a phone

  • Speech recognition runs on the device.
  • A small language model runs on the device.
  • A 32-million-parameter encoder decides which character the user addressed.
  • A speech synthesiser of 31 million parameters runs at 8 bits on the processor.
  • The agent generates the next line while the current line is being spoken.
  • Claims for native speech-to-speech models are not yet backed by weights or benchmarks.
Module 1518 / 33

The model built into the operating system

  • One desktop system is reported to ship a terminal command for its on-device model.
  • The command supports piping.
  • It can return schema-constrained JSON.
  • It runs a local server that speaks the common chat interface.
  • Its window is 4,096 tokens.
  • That window limits it to summarising, extracting, classifying and rewriting.
Module 1519 / 33

Ask the room: the recogniser and the still mouth

  • A lip-reading dictation tool sent video of a still mouth to its recogniser.
  • The recogniser returned fluent sentences with confidence.
  • No text model was involved.
  • Propose a fix that leaves the model unchanged.
Module 1520 / 33

The claim for fine-tuning and its evidence

  • A small open model, tuned on good proprietary labels, can beat a frontier generalist.
  • The win is on one narrow task.
  • The tuned model is cheaper and faster.
  • This is reported for fraud review with a 9-billion-parameter model, and no figures were published.
  • The pattern is plausible when labels are plentiful and the task is fixed.
Module 1521 / 33

Phases 1 to 4 of a fine-tuning run

  1. Goal: it is questioned until concrete.
  2. Compute: the agent detects local hardware or prepares a rented machine.
  3. Base model: it fits the goal, the hardware and the licence.
  4. Data: find, import, write or generate it, then preview and clean it.
Module 1522 / 33

Phases 5 to 8 of a fine-tuning run

  1. Recipe: hyperparameters stay within published guidelines, and memory, time and cost are estimated first.
  2. Training: it runs detached and shows a live loss curve.
  3. Taste test: the same prompts go to the tuned and base models.
  4. Export: the result is an adapter or merged weights, with model card and licence files.
Module 1523 / 33

Chains of training objectives

ChainUse
Supervised fine-tuning aloneAn instruction-following base
Continued pretraining, then supervised tuningTeach a domain, then a dialogue form
Supervised tuning, then preference optimisationSharpen style
Module 1524 / 33

Guardrails worth copying

  • The agent asks for confirmation before any action that costs money.
  • It asks for confirmation before installing software, sending data out or publishing.
  • The model cannot bypass the confirmation.
  • The first step is a real GPU operation, which exposes a broken driver.
  • No secret passes through the conversation.
Module 1525 / 33

Worked example: two documented fine-tuning runs

  • Run 1 tuned a 2-billion-parameter model to a writing voice.
  • The first run used 826 examples and trained for about two epochs.
  • Run 2 trained an image adapter on 160 captioned photographs.
  • In the second run the step-500 checkpoint was preferred to the final checkpoint, by inspection.
  • Roughly an hour on a rented 24 GB card cost about forty cents.
Module 1526 / 33

The vocabulary of interpretability

  • Interpretability asks what individual parts of a network compute.
  • Polysemantic neuron: one that appears to do several unrelated jobs.
  • Superposition: the hypothesis that a network stores more features than neurons, by overlapping them.
  • Sparse autoencoder: a tool for separating such features.
Module 1527 / 33

A contested claim and its status

  • A long informal essay this season argues the opposite of the prevailing view.
  • It claims that, with better tools, most neurons in modern networks are legible one at a time.
  • It claims that earlier illegibility came from architecture and tooling.
  • It claims that what suppresses a neuron often carries the distinction that matters.
  • Its status is a working hypothesis: the percentages depend on thresholds, and key results await a paper.
Module 1528 / 33

In-class exercise: tests of a unit's label

  • Look at strong and weak activations.
  • Look at what suppresses the unit, and at its connections.
  • Turn it up and read what the model writes.
  • Test the explanation on held-out cases.
  • Remove the unit before trusting the label.
  • For a unit labelled "dates", write what would disprove the label at each step.
Module 1529 / 33

Ask the room: recall or derivation

  • The design trains a model only on text from before a cutoff date.
  • It then tests whether the model derives what was discovered after that date.
  • The design separates recall from derivation.
  • Say why an ordinary benchmark cannot make that separation.
Module 1530 / 33

Common mistakes

  • Budgeting memory for weights only. Instead, add the cache and the overhead.
  • Accepting "4-bit quality" from a label. Instead, divide the file size by the parameter count.
  • Scoring on a general benchmark. Instead, use a handful of task questions, graded blind.
  • Sending every input to a recogniser. Instead, gate the input on evidence of something to recognise.
  • Taking the final checkpoint by default. Instead, compare checkpoints by inspection.
  • Trusting a neuron's label. Instead, test it on held-out cases, then remove the unit.
Module 1531 / 33

Laboratory 10: what is due

  • The task is narrow and has 500 or more labelled examples.
  • (a) A memory budget for two candidate local models, checked against observed use.
  • (b) A 20-question blind-graded test.
  • (c) Scores for a hosted model, a local model, and the local model tuned or with retrieval.
  • (d) Accuracy, latency at the 50th and 95th percentile, cost per thousand items, energy.
  • (e) A statement of which system you would deploy.
Module 1532 / 33

Recap

  • The reasons to run locally are confidentiality, cost, latency and availability, and control. The capability gap and operations stand against them.
  • Memory = weights + key-value cache + overhead, and 4 bits is half a gigabyte per billion parameters.
  • Check claims by arithmetic: 8.65 GB for 27 billion parameters is about 2.5 bits per weight.
  • Small specialised models answer in milliseconds. Gate a recogniser's input on evidence.
  • Fine-tuning has eight phases and confirmations the model cannot bypass.
  • Measure with a small blind test drawn from the real task.
Module 1533 / 33

Before next class

  • Start Laboratory 10 with part (a): two memory budgets, then observed use.
  • Write five of the twenty test questions and their expected answers.
  • Read Module 16 on the project, the submission and the grading table.
  • Bring the name of one real person who would use your capstone project.
  • List which of the laboratory modules 9 to 15 the project could draw on.