Syllabus Module 3 · Semantic labeling and the data behind models
NSW 3011 / 32
Software Engineering with Language Models

Module 3 · Semantic labeling and the data behind models

Class time
60 minutes
Slides
32, each with teacher’s notes
Goes with
Laboratory 1; Assignment 2 (Part IV); Assignment 4 (Part IV)
Module 32 / 32

Where this module sits

  • Module 1 covered the model; Module 2 covered the harness around it.
  • Module 3 covers the data behind models, starting from the definition of semantic labeling.
  • Module 1 defined calibration; here you measure it.
  • Laboratory 1 is the first laboratory of the course and is graded on evidence.
  • Module 3 is the last module of Part I; Part II begins with Module 4 on history.
Module 33 / 32

Objectives

  • Define semantic labeling and its special cases.
  • Name four uses of labels in a system.
  • Design a closed label schema using the five rules of section 3.2.
  • Choose among four labeling pipelines and budget the chosen pipeline per item.
  • Measure label quality with agreement, a gold set, a calibration check and a confusion matrix.
  • Identify the labelled examples a product already produces.
Module 34 / 32

Agenda

  • 4 min: frame and objectives.
  • 7 min: 3.1 definition and uses.
  • 16 min: 3.2 schema design, five rules.
  • 12 min: 3.3 four pipelines.
  • 14 min: 3.4 measuring quality; 3.5 the supply chain.
  • 7 min: common mistakes, work due, recap.
Module 35 / 32

3.1 Definition of semantic labeling

  • "Semantic labeling": assigning to each item of data a label that states what it means.
  • The meaning is stated in terms of a schema chosen in advance.
  • An item can be a sentence, an image region, a video clip, a transaction or a span of code.
  • Example: a support message labelled "billing".
  • Counter-example: the length of a message in characters, which says nothing about what the message means.
Module 36 / 32

The family of terms

  • Annotation: the general activity.
  • Semantic segmentation: the special case of labeling every pixel.
  • Named-entity labeling: a text case.
  • Intent labeling: a text case.
  • Each of these activities assigns meaning from a schema fixed beforehand.
Module 37 / 32

Four uses of labels

  • Training: supervised training consumes labels.
  • Evaluation: a model's accuracy is defined against labelled examples.
  • Retrieval: a labelled corpus can be filtered before it is searched.
  • Operations: routing a support ticket, categorising a transaction and flagging a message that needs a reply are labeling tasks.
Module 38 / 32

3.2 Schema design and the closed label set

  • The schema determines the ceiling on quality more than the labeler does.
  • Schema rule 1: close the label set.
  • A closed set is a fixed list of mutually exclusive categories.
  • Examples are a tax code list and a ticket queue list.
  • A closed set makes labeling a choice among listed categories.
  • Open-ended labels drift.
Module 39 / 32

A test for each label (schema rule 2)

  • Each label is defined by a test.
  • Each definition includes a positive example.
  • Each definition includes a negative example.
  • If two trained people disagree on an item, the definition is incomplete.
Module 310 / 32

Worked example: a label defined by a test

label: billing
test: asks about a charge, an
  invoice or a refund
positive:
  "I was charged twice in March."
negative:
  "The invoice page will not load."
  -> bug
Module 311 / 32

An explicit escape label (schema rule 3)

  • The schema includes "other" or "out of scope" as a label.
  • Without that label, a labeler must put every item into some category.
  • In published benchmarks, small classifiers score well on in-scope intents.
  • The same classifiers collapse on out-of-scope detection.
  • Failing to detect out-of-scope items is the failure that matters in production.
Module 312 / 32

Observations and inferred states (schema rule 4)

observed (labelers record these):
  person = box("person")
  weapon = box("weapon")

inferred (a rule, kept in code):
  armed = overlaps(weapon, person)
Module 313 / 32

Ask the room: observed and inferred statements

  • "A person is holding an object."
  • "The person is armed."
  • "The message contains the word refund."
  • "The customer is unhappy."
  • "Two boxes overlap."
  • Sort each statement as observed or inferred, and say where the inferred statements are computed.
Module 314 / 32

Provenance for every label (schema rule 5)

item:       msg-0412
label:      billing
by:         model, run 3
schema:     v2
confidence: 0.84
Module 315 / 32

In-class exercise: breaking a draft schema

  • Draft schema for commit messages (made-up): fix, feature, refactor, docs, other.
  • The teacher shows ten commit messages.
  • Label all ten messages alone in two minutes, without talking.
  • Compare your labels with a neighbour's and mark each disagreement.
  • Rewrite one disputed label as a test with a positive and a negative example.
Module 316 / 32

3.3 The detect-then-classify pipeline

image or video frame
        |
        v
[ small local detector ]
        |  finds and crops objects
        v
[ general classifier ]
        |  one call per crop
        v
probability per class
Module 317 / 32

Worked example: a budget per item

  1. The job (made-up volume) is 40,000 images with one crop each.
  2. Published figures per crop: about 270 ms and 280 input tokens.
  3. Time, one crop at a time: 40,000 x 0.27 s = 10,800 s, or three hours.
  4. Input tokens: 40,000 x 280 = 11.2 million.
  5. Cost at roughly three cents per thousand images: about $1.20.
  6. Before you commit to the classifier, measure the same three numbers on 100 of your own crops.
Module 318 / 32

The detect-track-count pipeline

frame  detected   track  count
1      car        #7     0
2      car        #7     0
3      car@line   #7     1
4      car        #7     1

No tracker: each frame's car
is a new object. Count: 4.
Module 319 / 32

Model as labeler and human as auditor

  • A cheap multimodal classifier with a fixed schema labels in bulk.
  • One practitioner reports tagging every goal of a hockey season for under two dollars.
  • The report includes no accuracy check.
  • Step 1: label a random sample by hand.
  • Step 2: measure agreement between the hand labels and the model's labels.
  • Step 3: only then trust the bulk run.
Module 320 / 32

Human only for the novel

  • A scheduled pipeline categorises bank transactions into tax categories.
  • The pipeline stores every decision.
  • The pipeline asks the person only about merchants it has not seen.
  • The pipeline exports a deterministic report.
  • The closed label set keeps the model's job simple.
  • The stored decisions make the pipeline consistent.
Module 321 / 32

Ask the room: the pipeline for each job

  • Job A: count lorries passing a gate camera.
  • Job B: sort product photographs by visible defect type.
  • Job C: tag a year of support messages by topic.
  • Job D: file each month's card transactions under expense categories.
  • Name the pipeline for each job and its first quality check.
Module 322 / 32

3.4 Measuring label quality

MeasureMethodUse
Inter-annotator agreementTwo labelers, same items; report Cohen's kappaBelow about 0.6 the schema needs work
Gold setA few hundred items; careful labels, adjudicatedScores every labeler; never used for training
Calibration checkBin predictions by stated confidence; compare with observed accuracy per binShows whether confidence can be a threshold
Confusion matrixCount which pairs of labels are confusedShows which definitions to rewrite
Module 323 / 32

Worked example: computing kappa

  1. Two annotators label 100 items "bug" or "other" (made-up counts).
  2. Both annotators say "bug" on 40 items and "other" on 30 items, so observed agreement is 0.70.
  3. Annotator A says "bug" 60 times and annotator B says "bug" 50 times.
  4. Agreement expected by chance: 0.6 x 0.5 + 0.4 x 0.5 = 0.50.
  5. Kappa = (0.70 - 0.50) / (1 - 0.50) = 0.40.
  6. 0.40 is below 0.6, so revise the schema before labeling more.
Module 324 / 32

A calibration check by bin

stated     items  right  observed
0.5-0.6      20     11     0.55
0.6-0.7      30     20     0.67
0.7-0.8      50     36     0.72
0.8-0.9      60     45     0.75
0.9-1.0      40     30     0.75

Top two bins: over-confident.
Module 325 / 32

Worked example: reading a confusion matrix

Gold labelModel: bugModel: billingModel: other
bug3028
billing1272
other9120
Module 326 / 32

3.5 The supply chain for labels

  • Frontier training depends on purchased human expertise.
  • One purchase is specialist annotators.
  • Another purchase is graded reasoning traces.
  • A third purchase is constructed environments in which a model's actions can be scored automatically.
  • Public job postings and procurement notices signal which capabilities are being built next.
  • Every accepted correction in your own product is a labelled example you already own.
Module 327 / 32

Common mistakes

  • Open-ended labels. Fix: close the set and add an escape label.
  • A label name with no test. Fix: a test, a positive example and a negative example.
  • Labeling an inferred state directly. Fix: label observations; derive the state in code.
  • Trusting a bulk model run. Fix: hand-label a random sample and measure agreement first.
  • Reporting raw agreement. Fix: report Cohen's kappa.
  • Training on the gold set. Fix: keep it for scoring only.
Module 328 / 32

Laboratory 1: work due

  • A corpus of at least 500 items you care about.
  • A closed schema of five to ten labels, with definitions and examples.
  • 100 items labelled by you and by a second person, and kappa computed.
  • The schema revised and the round repeated until kappa exceeds 0.7.
  • The remainder labelled with a model and scored against your 100.
  • A report of accuracy, the confusion matrix, cost per item and the cases you could not decide.
Module 329 / 32

Assignment 2 (Part IV): the label drill

  • Annotators pass the drill before they label for Laboratory 1.
  • The answer key is your schema and the first 40 items you labelled.
  • The program shows a random key item, takes a label and says whether the label matched.
  • On a miss the program shows the expected label with its definition and resets the streak to zero.
  • The program stops at CORRECT_IN_A_ROW = 3 or after MAX_ITEMS = 30, and reports which limit it reached.
  • Report how many items each annotator needed and which labels caused resets.
Module 330 / 32

Assignment 4 (Part IV): the fire exercise

  • Take 200 items with a numeric score, such as the confidence of the model's labels from Laboratory 1.
  • THRESHOLD is a named constant.
  • Items that score above the threshold are highlighted for a person; items that score below it pass without review.
  • Label 50 items by hand.
  • At three threshold values, count the items highlighted, the highlighted items that deserved it, and the deserving items that were missed.
  • Choose one value and defend it by what a miss costs and what a false alarm costs.
Module 331 / 32

Recap

  • A label states what an item means in terms of a schema chosen in advance.
  • Labels are used in training, evaluation, retrieval and operations.
  • The five schema rules cover a closed set, tests with examples, an escape label, observed apart from inferred, and provenance.
  • There are four pipelines; budget latency, tokens and cost per item before choosing a classifier.
  • The four measures are kappa, a gold set, a calibration check and a confusion matrix.
  • Accepted corrections in a product are labelled examples its owner already holds.
Module 332 / 32

Before next class

  • Read Module 4, "From symbols to statistics (1943–2012)".
  • Choose your Laboratory 1 corpus and draft the schema, including the escape label.
  • Label your first 40 items; those items become the answer key for the label drill.
  • Arrange the second annotator for the 100 shared items.
  • Reread Module 1's definition of a large language model.