The Recsys Feedback Loop You Can't Escape
Every recommendation system creates its own training data. The users who see item A click on it; item A gets more impressions; it gets more clicks; it ranks higher; it gets even more impressions. Within months, your model has learned to recommend what it already recommended — not what users actually want.
What the Feedback Loop Is
A recommendation system decides what users see. Users click on what they see. Those clicks become the next training batch. The model learns from its own outputs.
This is not a design flaw — it is the fundamental architecture of every production recommendation system. But it creates a compounding dynamic that, without explicit intervention, systematically degrades catalog coverage, user experience, and long-run relevance. Items the model never shows get zero clicks and never improve their position. Items it shows frequently accumulate clicks and rank higher in every subsequent training cycle.
The loop is self-reinforcing by construction.
Failure Mode 1: Popularity Spiral
Top-10 items capture 60%+ of traffic within 90 days of deployment. Catalog coverage collapses. Long-tail items — which may have high latent interest among specific user segments — never surface because they never accumulate the click signal that would make the model show them.
The distribution of item impressions follows a power law that steepens over time. At launch, the top 10% of items may capture 40% of impressions. After six months of feedback-loop training, the same 10% capture 70%. The rest of the catalog effectively does not exist for the model.
The business cost: catalog that was expensive to acquire, produce, or license generates zero engagement not because users don't want it, but because it was never shown.
Failure Mode 2: Exploration Starvation
Without explicit exploration, the model converges on a local optimum. New items released after training cutoff never break through because they have no historical signal. A model trained on six months of data assigns near-zero probability to items launched in month seven.
This creates a structural disadvantage for new content. It also means that shifts in user preference — new trends, seasonal changes, emerging interests — are absorbed slowly or not at all. The model's prior on item quality is dominated by historical click rates, which are themselves a product of historical exposure, not historical preference.
Failure Mode 3: Demographic Homogenisation
If one demographic clicks more than others, the model skews toward their preferences. Other segments see progressively less relevant content. They churn. The surviving audience becomes more homogeneous. The model optimises harder for the dominant demographic's signal.
This is a feedback loop operating on user population composition, not just item distribution. The outcome is a system that serves a narrowing segment extremely well and a growing segment not at all — and whose offline metrics look fine throughout the process because they are measured on the surviving, homogeneous user population.
Why Offline Evaluation Hides This
Your offline AUC or NDCG looks fine because you evaluate on historical clicks — which were themselves generated by the feedback loop. The held-out test set reflects the distribution of items the model already showed, not the distribution of items users would have clicked if they'd been shown different content.
Offline metrics don't capture what users would have clicked if they'd been shown items outside the model's historical top-K. This is the exposure bias problem in recommendation system evaluation. A model that perfectly predicts historically shown items is not necessarily a good recommender — it may be an excellent memorizer of its own previous decisions.
Four Interventions
*Forced exploration budget:* Reserve 5–10% of recommendation slots for items outside the model's top-K. Log results. Feed them back into training. Implementation: ε-greedy assigns a fixed fraction of slots to random-in-catalog items. Thompson sampling estimates per-item uncertainty and allocates exploration budget proportional to uncertainty. The exploration budget is not a concession to users — it is an investment in training signal diversity.
*Inverse propensity scoring (IPS):* Reweight training examples by the inverse probability of them being shown. Items shown more frequently are downweighted. Items shown rarely are upweighted. If item A was shown with probability 0.8 and was clicked, its contribution to the training gradient is scaled by 1/0.8 = 1.25. If item B was shown with probability 0.05 and was clicked, its contribution is scaled by 1/0.05 = 20. IPS corrects for the exposure bias in the training signal and gives the model a less distorted view of item quality.
*Popularity debiasing:* Subtract a popularity prior from item scores at inference time. Score_debiased = Score_model - α × log(impression_count). The α hyperparameter controls the strength of debiasing. A high α strongly suppresses popular items; a low α leaves the model's ranking largely intact. The correct α is empirical — tune it on a held-out diversity metric, not on offline AUC.
*Diversity constraints:* At serving time, enforce minimum category or genre diversity in the top-K results. Maximal marginal relevance reranks items by a combination of relevance and novelty relative to already-selected items. Determinantal point processes provide a probabilistic framework for diverse subset selection. Both prevent the top-K from collapsing to a single content type while maintaining relevance.
The Production Checkpoint
Before any recsys model ships, audit three things:
Catalog coverage over the last 30 days: if fewer than 10% of catalog items appeared in recommendations, the feedback loop is already dominating the system before you've even added this model.
Top-10 item share: if more than 50% of clicks go to 10 items, diversity is broken. The model is serving a popularity leaderboard with extra steps.
New-item CTR vs average CTR: if items launched in the last 30 days have less than 30% of average CTR, exploration is broken. New content cannot break through the historical signal barrier.
These three checks take 20 minutes to run. They will tell you more about the health of your recommendation system than any offline evaluation metric.