The Golden Dataset That Caught a Quiet Model Regression

I would like to say the incident was the problem, because that would be a shorter post, but it started at the better.

This is the anatomy of the model regression, and of the specific tool that catches it: the golden dataset, which is a frozen set of inputs with expected behaviours, and the frozen set is the baseline, and the baseline is what turns the upgrade from a hope into a measurement, and the measurement is the difference, and the difference is the regression, and the regression is the finding.

This applies to any model behind a feature, and the mechanism is the same regardless of the vendor.

Why the upgrade is a hope without a baseline

An upgrade without a baseline is a hope, because the evidence is the vendor's, and the vendor's evidence is the benchmark the vendor chose, and the chosen benchmark is the one the vendor wins, and the winning is the marketing, and the marketing is not the measurement of your feature, because your feature is your inputs, and your inputs are not the benchmark, and the not benchmark is the gap, and the gap is the regression, and the regression is the cost of the hope.

The hope is reasonable, because the vendor's claim is usually true on average, and the average hides the tails, and the tails are your cases, and your cases are your traffic, and your traffic is your revenue. The regression is invisible without the baseline, because the invisible is the unmeasured.

The baseline is the fix, because the baseline is your inputs, and your inputs are your feature, and the feature is what the upgrade should not break, and the not breaking is the requirement, and the requirement is the measurement, and the measurement is the golden dataset, and the dataset is the tool.

What the golden dataset is

The golden dataset is a set of inputs, and each input has an expected behaviour, and the expected behaviour is the specification, and the specification is what the feature should do, and the should is the human's, and the human's specification is the baseline, and the baseline is frozen, and the frozen is the stability, and the stability is what makes the comparison meaningful, because the comparison is the same inputs on the old model and the new, and the same inputs mean the difference is the model, and the model is the variable, and the variable is the measurement.

The dataset is small, and the small is the point, because the small set is curated, and the curated set is the cases that matter, and the cases that matter are the ones the feature is for, and the feature is the specification, and the specification is the golden set, and the golden set is the fifty or two hundred inputs that define the feature, and the definition is the baseline, and the baseline is the measurement.

The dataset includes the edges, and the edges are the cases the distribution omits, and the vendor's benchmark omits them too, and that mutual omission is where the regression hides. The hiding place is why the golden set must be yours: your edges are your traffic, and your traffic is the regression's cost.

How it caught the regression

The upgrade was run against the golden set, and the results were compared to the baseline, and the comparison showed a drop, and the drop was on a specific case, and the case was an input with a negation, and the new model handled the negation worse, and the worse was the regression, and the regression was the finding, and the finding was the measurement, and the measurement was the golden set, and the golden set was the fifty inputs, and the fifty inputs included the negation, because the negation was in the traffic, and the traffic was the reason the input was in the set, and the set was the catch.

Without the set, the upgrade would have shipped, and the worse handling of negations would have surfaced as user reports, then an investigation, then a rollback. That sequence is the price of the missing baseline, and the baseline is fifty inputs, and fifty inputs are cheap, and the cheapness is the argument.

How to build it

Start from the traffic. The inputs are the real ones, and the real ones are the distribution, and the distribution is the feature's actual use, and the actual use is what the model must handle, and the must is the specification, and the specification is the dataset, and the dataset is the sample, and the sample is the fifty, and the fifty are the baseline.

Label the expected behaviour. The label is the human's, and the human's label is the specification, and the specification is the intent, and the intent is the correct output, and the correct output is the assertion, and the assertion is the test, and the test is the golden case, and the case is the dataset's entry, and the entry is the baseline's unit.

Include the edges and the failures. The edges are the thin cases, and the failures are the cases the model got wrong before, and the wrong before is the regression's history, and the history is the dataset's growth, and the growth is the incident becoming a case, and the case is the prevention, and the prevention is the dataset's purpose.

Freeze it. The freeze is what makes the comparison meaningful, because the same inputs on two models means the difference is the model, and the difference is the measurement.

Run it on every model change. The run is the gate, and the gate is the check, and the check is the comparison, and the comparison is the regression, and the regression is the block, and the block is the prevention, and the prevention is the gate's purpose, and the gate is the CI, and the CI is the automation, and the automation is the discipline.

The rule

A model upgrade without a baseline is a hope, and the hope is the vendor's benchmark, and the benchmark is not your traffic, so the regression hides in the gap. Build a frozen golden dataset from real traffic, label the expected behaviour, include the edges and past failures, and run it as a gate on every model change.

The upgrade was better on average and worse on a negation, and the average was the marketing and the negation was the traffic, and the traffic is the revenue. I now check for that property first, because it is where the failure actually lives.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud