Defining Metrics: North Star, Guardrails, and Why Your Metric Is Probably Wrong

The hardest part of ML in industry is not the model — it is deciding what to optimise. The north star metric encodes your theory of user value. Proxy metrics are what you actually measure. Guardrail metrics prevent your optimisation from breaking things you care about. Getting this framework wrong means building the right model for the wrong problem — shipping something that looks successful and actually degrades the product.

Every ML project starts with a metric — the quantity the model is trained to optimise and the quantity by which success is measured. In many teams, this metric is chosen quickly, rarely questioned, and silently drives thousands of engineering hours in the wrong direction. The framework for choosing metrics correctly is one of the most impactful things a staff DS brings to a team.

The north star metric: what you ultimately care about

The north star is the single metric that best captures the product's value creation. For a social network: Daily Active Users. For a subscription: annual revenue retention. For a marketplace: gross merchandise volume. The north star is not what you directly optimise — it is too slow to move and too aggregate to guide individual decisions. But it is the ultimate arbiter of whether a change was good.

North stars that break: maximising the wrong proxy is Goodhart's Law in action. If a social network optimises for daily active users, they may inflate DAU by sending push notifications that bring users to the app but damage engagement quality. If a video platform optimises for watch time (YouTube's 2012-2016 approach), recommendation algorithms may serve longer but lower-quality or more extreme content. The north star must be validated against a theory of sustainable user value, not just engagement.

Proxy metrics: what you actually measure in experiments

Because the north star is too slow-moving for experiments (you would need weeks to detect a change in retention), experiments use faster-moving proxy metrics: session engagement rate, feature adoption, pages per session, user satisfaction scores (thumbs up/down). The proxy metric is what you optimise in the model objective and what you measure in A/B tests.

Proxy validity: does improvement in the proxy reliably predict improvement in the north star? This should be validated empirically across past experiments: do experiments that moved the proxy also move the north star? If the correlation is weak, the proxy is not valid. Proxy-north-star misalignment is the root cause of most "we shipped, metrics looked great, nothing changed in the business" failures.

Guardrail metrics: what you will not sacrifice

Guardrail metrics are metrics that must not decrease, regardless of improvements to the primary metric. Examples: page load time (optimising recommendation quality must not slow the page), ad revenue (adding features must not decrease monetisation), customer support contacts (improvements must not create confusion that spikes support volume). A feature that increases session engagement by 10% but decreases ad revenue by 5% is a net loss at most companies, even if the primary metric moved in the right direction.

The HEART framework (Google)

Happiness (user satisfaction scores, NPS), Engagement (depth and frequency of feature usage), Adoption (new users using the feature), Retention (returning users), Task Success (completion rate, error rate). HEART provides a structured way to think about which dimensions of user experience an experiment might affect. For each experiment, choose 1-2 primary metrics and 3-5 guardrails across the HEART dimensions most relevant to the change.

Metric decomposition: where did the number come from?

When a metric moves, knowing why it moved requires decomposition. Revenue = DAU × sessions per user × events per session × revenue per event. If revenue drops, decompose each factor: is DAU down (acquisition/retention problem)? Is revenue per event down (monetisation problem)? Decomposition prevents premature conclusions and directs investigation to the right part of the funnel. Standardise this decomposition before experiments run so that post-hoc analysis does not require data archaeology.

Sensitivity and minimum detectable effect

A good experiment metric is sensitive enough to detect meaningful effects in a reasonable experiment duration. To evaluate: compute the metric's standard deviation across users, estimate the sample size needed to detect a 1% lift (your expected effect size) at 80% power. If the required sample size is 3 months of traffic, the metric is too noisy and you need a surrogate with lower variance. Session-level metrics have lower variance than user-level metrics; short-window metrics are more sensitive than long-window metrics.

Try on Colab: simulate a product change that increases click rate but decreases click-to-purchase conversion (users are enticed by clickbait but disappointed by the content). Model three metrics: CTR, conversion rate, and revenue per user. Show that optimising for CTR produces a model with higher CTR, lower conversion, and ambiguous revenue. Then formalise the correct combined metric (revenue per impression) and show it correctly penalises clickbait.