Label Generation

Programmatic labeling, label noise, weak supervision, distant supervision

You want to predict which support tickets will need escalation. You have 500,000 old tickets — plenty of data. Then the label reality hits: "escalation" meant something different before 2022, 60% of tickets have no recorded outcome at all, and the outcomes that exist only showed up 14 days after the ticket closed. This is harder than picking a model. It is harder than building features. In most real production ML, *getting the labels right is the actual bottleneck.*

Four ways to get labels, each with its own cost, delay, and noise

*Natural labels* come free from user behavior — clicks, purchases, chargebacks. The label already exists; the only catch is delay (clicks in seconds, chargebacks in days), so you design the pipeline around that wait.

*Human annotation* has experts or crowd workers label examples directly — Scale AI or Surge for hard tasks, MTurk for easy ones, running from roughly fifty cents to five dollars an example. You *must* measure quality: inter-annotator agreement (Cohen's κ) above 0.7 is acceptable, 0.8 is good, and below 0.6 means the task itself is ambiguous — fix the guidelines before spending on more labels.

*Weak supervision* (Snorkel) writes many *labeling functions* — small heuristics such as keyword matches, regexes, existing classifiers, or crowd rules-of-thumb — each of which votes a label or abstains on every example. A small set of *gold-labeled examples* (a few hundred, hand-labeled) is held out to estimate each labeling function's accuracy and how often functions agree with each other; Snorkel's label model uses those estimates to weight and combine the noisy votes into one probabilistic label per example. Worked example: five labeling functions, each roughly 65-75% accurate alone, weighted and combined against the gold set, can produce an aggregate label model above 90% accuracy — because their errors aren't perfectly correlated with each other, so they cancel out when combined — all without collecting a single additional human label beyond the gold set.

*Active learning* trains a model, finds the examples it's most unsure about, and sends only those to humans — often reaching the same accuracy as random labeling with 5–10× fewer labels.

The trap almost everyone hits: label delay

If the truth for an event at time T only arrives at T+7 days, you must drop the last 7 days from training. Skip that, and your recent examples are full of positives that simply haven't been labeled yet. The model sees a near-zero positive rate in the latest window and "learns" that recent traffic is safe. That's not a pattern — it's an artifact of how you built the dataset.

A second common trap: labels that are quietly biased by subgroup

Aggregate accuracy can look fine — 92% overall — while one demographic group's error rate is far higher, because that group is a small enough share of the test set to hide inside the average. Two checks catch this: per-group accuracy metrics (compute accuracy separately for each subgroup — never trust the blended number alone), and *confident learning* — a technique, implemented in the open-source `cleanlab` library, that flags likely-mislabeled examples by finding where the model's predicted-probability distribution disagrees with the label it was actually given. Run a confident-learning audit on the affected subgroup specifically: if it surfaces a disproportionate share of likely-mislabeled examples there, the disparity is a labeling problem, not just a sampling artifact — the fix is auditing and relabeling those examples plus reweighting the minority class, not just collecting a bigger test set.

And the myth to kill: "we have tons of data, we don't need labels." Data is not labeled data. Ten million unlabeled rows teach a supervised model nothing. Label quality is the ceiling on model quality — a model can't learn the right thing from the wrong signal, no matter how much you feed it. Spend on annotation *quality* before annotation *quantity.*

Key points

• Invest in annotation quality infrastructure before annotation quantity — a more accurate labeling interface with inter-annotator tracking — even if it produces fewer total labels — tends to beat a noisier interface pumping out a much larger volume of labels. Label quality sets the ceiling on model quality. A model trained on systematically biased labels learns the bias as signal and reproduces it at inference. No amount of additional data fixes systematic noise. Measure Cohen's κ on a 200-example sample before scaling. If κ < 0.6, the labeling task is too ambiguous — refine the guidelines before committing budget to 10,000 more noisy labels.
• Trap: ignoring label delay in training data construction is the most common subtle bug in production ML pipelines. If ground truth for event at time T arrives at T+7 days, exclude the last 7 days from training to avoid future leakage. Training pipelines that use "the last 30 days of data" without respecting label delay will have systematically under-labeled positive examples in the most recent window. The model learns that recent traffic is low-converting — a temporal artifact of the data construction, not a real pattern.
• Diagnostic: compute inter-annotator agreement on a 200-example sample before scaling annotation. If κ < 0.6, the labeling task is too ambiguous to scale. The agreement number tells you whether the problem is the guidelines (fixable), the task definition (refine), or genuine boundary ambiguity (accept and use label smoothing). Collecting 10,000 more labels with κ = 0.5 does not improve the model — it amplifies the inconsistency.

Recap

• Getting labels right is usually the real bottleneck — harder than picking a model or building features.
• Natural labels = free from behavior (clicks, purchases, chargebacks); the catch is delay.
• Human annotation = \$0.50–\$5/example; measure Cohen's κ — >0.7 acceptable, >0.8 good, <0.6 = ambiguous task, fix guidelines first.
• Weak supervision (Snorkel): many *labeling functions* (heuristics) vote or abstain on each example; a small gold-labeled set calibrates each function's accuracy, then the label model combines the noisy votes into one probabilistic label. Active learning: label only uncertain examples, 5–10× fewer labels.
• Label-delay trap: truth at T+7d → drop the last 7 days, or the model "learns" recent traffic is safe.
• Kill the myth "we have tons of data, we don't need labels": unlabeled rows teach a supervised model nothing.
• Spend on label quality before quantity — quality is the ceiling; more labels from a biased process amplify the bias.

Check your understanding