The Recommendations That Learned the Sale and Never Unlearned It

Start at the good and you miss it. Start at the incident and you miss how it arrived. The bug lived in the hand-off.

I have watched this exact failure cost a real store real money, which is why I am fussy about the order here.

Strip the recommendation model trained on a promotion window down and one property makes it invisible: the training metric improved, and the improved metric was the success signal, and the signal was true, and the truth was incomplete, because the metric measured clicks and the business measured margin.

The case was a Shopify store with a recommendation widget; the mechanics generalise to any model trained on a time-bounded behavioural spike.

Why the model learned the sale

The training data was the clicks, and the clicks were the behaviour, and the behaviour was the window, and the window was the promotion, and the promotion was the discount, and the discount was the clicks, and the clicks were the signal, and the signal was the training, and the training was the model, and the model learned that the discounted lines get clicked, and the learned was the recommendation, and the recommendation was the discounted, and the discounted was the margin, and the margin was the cost.

The model did the right thing on the data it had, and the data was the sale, and the sale was the anomaly, and the anomaly was the training set, and the training set was the model's world, and the world was the promotion, and the promotion ended, and the model did not, and the not ending was the persistence, and the persistence was the incident, and the incident is the reason the training window matters.

The model is not wrong, the model is faithful, and the faithful is the data, and the data was the sale, and the sale was the temporary, and the temporary was the model's permanent, and the permanent was the margin's cost, and the cost is the reason to bound the training window to the normal period.

Why the metric hid it

The click rate was the metric, and the metric was the success, and the success was the report, and the report was the quarter, and the quarter was the review, and the review was the margin, and the margin was the actual, and the actual was the down, and the down was the surprise, because the metric was up and the margin was down, and the divergence was the finding, and the finding was the metric's incompleteness.

The click rate measures the engagement, and the engagement is the click, and the click is not the profit, and the profit is the margin, and the margin is the price minus the cost, and the discounted price is the lower margin, and the lower margin is the recommendation's cost, and the cost is the incident, and the incident is the reason the metric should be the margin, not the click.

This is the same divergence as the AI product copy that promised a warranty nobody offered, and the common root is the property, not the platform.

What actually fixed it

Bound the training window to the normal period. The training data excludes the promotion, and the excludes is the normal, and the normal is the baseline, and the baseline is the model's world, and the world is the steady state, and the steady state is the recommendation, and the recommendation is the margin, and the margin is the fix, because the model learns the ordinary behaviour rather than the anomaly.

Retrain on a rolling window. The model retrains weekly on the last ninety days, and the ninety days is the rolling, and the rolling is the forgetting, and the forgetting is the sale's decay, and the decay is the correction, and the correction is the fix, and the fix is the schedule, and the schedule is the discipline, because the static model keeps the old world forever.

Weight recent behaviour higher. The recency is the weight, and the weight is the decay, and the decay is the sale's fading, and the fading is the correction, and the correction is the fix, and the fix is the exponential, and the exponential is the half life, and the half life is the tuning, and the tuning is the balance between the stability and the adaptation.

Measure margin, not only clicks. The recommendation's metric includes the margin per click, and the per click is the outcome, and the outcome is the business, and the business is the measure, and the measure is the alert, and the alert is the detection, and the detection is the prevention, and the prevention is the metric, because the click rate alone hides the discount's cost.

Flag the promoted lines in the features. The discount status is a feature, and the feature is the distinction, and the distinction is the model's knowledge, and the knowledge is the separation, and the separation is the fix, because the model can learn that the discount drives the click and discount appropriately when the discount is absent, rather than recommending the line for the discount it no longer has.

The takeaway

A model trained on a promotion window learns the promotion as the normal behaviour and keeps recommending discounted lines after the sale ends. Bound the training window to the ordinary period, retrain on a rolling window, weight recent behaviour higher, measure margin rather than only clicks, and expose the discount status as a feature.

The click rate was up and the margin was down, because the model had learned that discounted lines get clicked and kept recommending them at full price, and the full price did not get the click but the recommendation still pushed the line. That is the whole pattern, and it is the part worth remembering.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud