Software Engineering with Language Models
Module 3 · Semantic labeling and the data behind models
- Class time
- 60 minutes
- Slides
- 32, each with teacher’s notes
- Goes with
- Laboratory 1; Assignment 2 (Part IV); Assignment 4 (Part IV)
Where this module sits
- Module 1 covered the model; Module 2 covered the harness around it.
- Module 3 covers the data behind models, starting from the definition of semantic labeling.
- Module 1 defined calibration; here you measure it.
- Laboratory 1 is the first laboratory of the course and is graded on evidence.
- Module 3 is the last module of Part I; Part II begins with Module 4 on history.
Objectives
- Define semantic labeling and its special cases.
- Name four uses of labels in a system.
- Design a closed label schema using the five rules of section 3.2.
- Choose among four labeling pipelines and budget the chosen pipeline per item.
- Measure label quality with agreement, a gold set, a calibration check and a confusion matrix.
- Identify the labelled examples a product already produces.
Agenda
- 4 min: frame and objectives.
- 7 min: 3.1 definition and uses.
- 16 min: 3.2 schema design, five rules.
- 12 min: 3.3 four pipelines.
- 14 min: 3.4 measuring quality; 3.5 the supply chain.
- 7 min: common mistakes, work due, recap.
3.1 Definition of semantic labeling
- "Semantic labeling": assigning to each item of data a label that states what it means.
- The meaning is stated in terms of a schema chosen in advance.
- An item can be a sentence, an image region, a video clip, a transaction or a span of code.
- Example: a support message labelled "billing".
- Counter-example: the length of a message in characters, which says nothing about what the message means.
The family of terms
- Annotation: the general activity.
- Semantic segmentation: the special case of labeling every pixel.
- Named-entity labeling: a text case.
- Intent labeling: a text case.
- Each of these activities assigns meaning from a schema fixed beforehand.
Four uses of labels
- Training: supervised training consumes labels.
- Evaluation: a model's accuracy is defined against labelled examples.
- Retrieval: a labelled corpus can be filtered before it is searched.
- Operations: routing a support ticket, categorising a transaction and flagging a message that needs a reply are labeling tasks.
3.2 Schema design and the closed label set
- The schema determines the ceiling on quality more than the labeler does.
- Schema rule 1: close the label set.
- A closed set is a fixed list of mutually exclusive categories.
- Examples are a tax code list and a ticket queue list.
- A closed set makes labeling a choice among listed categories.
- Open-ended labels drift.
A test for each label (schema rule 2)
- Each label is defined by a test.
- Each definition includes a positive example.
- Each definition includes a negative example.
- If two trained people disagree on an item, the definition is incomplete.
Worked example: a label defined by a test
label: billing
test: asks about a charge, an
invoice or a refund
positive:
"I was charged twice in March."
negative:
"The invoice page will not load."
-> bug
An explicit escape label (schema rule 3)
- The schema includes "other" or "out of scope" as a label.
- Without that label, a labeler must put every item into some category.
- In published benchmarks, small classifiers score well on in-scope intents.
- The same classifiers collapse on out-of-scope detection.
- Failing to detect out-of-scope items is the failure that matters in production.
Observations and inferred states (schema rule 4)
observed (labelers record these):
person = box("person")
weapon = box("weapon")
inferred (a rule, kept in code):
armed = overlaps(weapon, person)
Ask the room: observed and inferred statements
- "A person is holding an object."
- "The person is armed."
- "The message contains the word refund."
- "The customer is unhappy."
- "Two boxes overlap."
- Sort each statement as observed or inferred, and say where the inferred statements are computed.
Provenance for every label (schema rule 5)
item: msg-0412
label: billing
by: model, run 3
schema: v2
confidence: 0.84
In-class exercise: breaking a draft schema
- Draft schema for commit messages (made-up): fix, feature, refactor, docs, other.
- The teacher shows ten commit messages.
- Label all ten messages alone in two minutes, without talking.
- Compare your labels with a neighbour's and mark each disagreement.
- Rewrite one disputed label as a test with a positive and a negative example.
3.3 The detect-then-classify pipeline
image or video frame
|
v
[ small local detector ]
| finds and crops objects
v
[ general classifier ]
| one call per crop
v
probability per class
Worked example: a budget per item
- The job (made-up volume) is 40,000 images with one crop each.
- Published figures per crop: about 270 ms and 280 input tokens.
- Time, one crop at a time: 40,000 x 0.27 s = 10,800 s, or three hours.
- Input tokens: 40,000 x 280 = 11.2 million.
- Cost at roughly three cents per thousand images: about $1.20.
- Before you commit to the classifier, measure the same three numbers on 100 of your own crops.
The detect-track-count pipeline
frame detected track count
1 car #7 0
2 car #7 0
3 car@line #7 1
4 car #7 1
No tracker: each frame's car
is a new object. Count: 4.
Model as labeler and human as auditor
- A cheap multimodal classifier with a fixed schema labels in bulk.
- One practitioner reports tagging every goal of a hockey season for under two dollars.
- The report includes no accuracy check.
- Step 1: label a random sample by hand.
- Step 2: measure agreement between the hand labels and the model's labels.
- Step 3: only then trust the bulk run.
Human only for the novel
- A scheduled pipeline categorises bank transactions into tax categories.
- The pipeline stores every decision.
- The pipeline asks the person only about merchants it has not seen.
- The pipeline exports a deterministic report.
- The closed label set keeps the model's job simple.
- The stored decisions make the pipeline consistent.
Ask the room: the pipeline for each job
- Job A: count lorries passing a gate camera.
- Job B: sort product photographs by visible defect type.
- Job C: tag a year of support messages by topic.
- Job D: file each month's card transactions under expense categories.
- Name the pipeline for each job and its first quality check.
3.4 Measuring label quality
| Measure | Method | Use |
|---|---|---|
| Inter-annotator agreement | Two labelers, same items; report Cohen's kappa | Below about 0.6 the schema needs work |
| Gold set | A few hundred items; careful labels, adjudicated | Scores every labeler; never used for training |
| Calibration check | Bin predictions by stated confidence; compare with observed accuracy per bin | Shows whether confidence can be a threshold |
| Confusion matrix | Count which pairs of labels are confused | Shows which definitions to rewrite |
Worked example: computing kappa
- Two annotators label 100 items "bug" or "other" (made-up counts).
- Both annotators say "bug" on 40 items and "other" on 30 items, so observed agreement is 0.70.
- Annotator A says "bug" 60 times and annotator B says "bug" 50 times.
- Agreement expected by chance: 0.6 x 0.5 + 0.4 x 0.5 = 0.50.
- Kappa = (0.70 - 0.50) / (1 - 0.50) = 0.40.
- 0.40 is below 0.6, so revise the schema before labeling more.
A calibration check by bin
stated items right observed
0.5-0.6 20 11 0.55
0.6-0.7 30 20 0.67
0.7-0.8 50 36 0.72
0.8-0.9 60 45 0.75
0.9-1.0 40 30 0.75
Top two bins: over-confident.
Worked example: reading a confusion matrix
| Gold label | Model: bug | Model: billing | Model: other |
|---|---|---|---|
| bug | 30 | 2 | 8 |
| billing | 1 | 27 | 2 |
| other | 9 | 1 | 20 |
3.5 The supply chain for labels
- Frontier training depends on purchased human expertise.
- One purchase is specialist annotators.
- Another purchase is graded reasoning traces.
- A third purchase is constructed environments in which a model's actions can be scored automatically.
- Public job postings and procurement notices signal which capabilities are being built next.
- Every accepted correction in your own product is a labelled example you already own.
Common mistakes
- Open-ended labels. Fix: close the set and add an escape label.
- A label name with no test. Fix: a test, a positive example and a negative example.
- Labeling an inferred state directly. Fix: label observations; derive the state in code.
- Trusting a bulk model run. Fix: hand-label a random sample and measure agreement first.
- Reporting raw agreement. Fix: report Cohen's kappa.
- Training on the gold set. Fix: keep it for scoring only.
Laboratory 1: work due
- A corpus of at least 500 items you care about.
- A closed schema of five to ten labels, with definitions and examples.
- 100 items labelled by you and by a second person, and kappa computed.
- The schema revised and the round repeated until kappa exceeds 0.7.
- The remainder labelled with a model and scored against your 100.
- A report of accuracy, the confusion matrix, cost per item and the cases you could not decide.
Assignment 2 (Part IV): the label drill
- Annotators pass the drill before they label for Laboratory 1.
- The answer key is your schema and the first 40 items you labelled.
- The program shows a random key item, takes a label and says whether the label matched.
- On a miss the program shows the expected label with its definition and resets the streak to zero.
- The program stops at
CORRECT_IN_A_ROW = 3or afterMAX_ITEMS = 30, and reports which limit it reached. - Report how many items each annotator needed and which labels caused resets.
Assignment 4 (Part IV): the fire exercise
- Take 200 items with a numeric score, such as the confidence of the model's labels from Laboratory 1.
THRESHOLDis a named constant.- Items that score above the threshold are highlighted for a person; items that score below it pass without review.
- Label 50 items by hand.
- At three threshold values, count the items highlighted, the highlighted items that deserved it, and the deserving items that were missed.
- Choose one value and defend it by what a miss costs and what a false alarm costs.
Recap
- A label states what an item means in terms of a schema chosen in advance.
- Labels are used in training, evaluation, retrieval and operations.
- The five schema rules cover a closed set, tests with examples, an escape label, observed apart from inferred, and provenance.
- There are four pipelines; budget latency, tokens and cost per item before choosing a classifier.
- The four measures are kappa, a gold set, a calibration check and a confusion matrix.
- Accepted corrections in a product are labelled examples its owner already holds.
Before next class
- Read Module 4, "From symbols to statistics (1943–2012)".
- Choose your Laboratory 1 corpus and draft the schema, including the escape label.
- Label your first 40 items; those items become the answer key for the label drill.
- Arrange the second annotator for the 100 shared items.
- Reread Module 1's definition of a large language model.