The Canary That Received All the Traffic

It ended at the incident. It started at the failed. Every hand-off in between did exactly what it was told, and this is the one that lied.

Strip the canary that is not a canary down and one property makes it a false signal: the small deployment receives the whole traffic, so it fails on capacity, and the capacity failure is read as a code failure, and the reading is the wrong rollback.

This was Kubernetes with an ingress weight annotation, and the mechanics are the same for any traffic split applied at the wrong layer.

Why the canary took everything

The weight was the annotation, and the annotation was the route's, and the route's was the ingress's rule, and the rule was the backend's selection, and the selection was the service, and the service was the selector, and the selector was the label, and the label was the both deployments, and the both was the not the split, and the not split was the round robin, and the round robin was the pod count's proportion, and the proportion was the one against the ten, and the ten was the wrong direction.

The intent was the five percent, and the five percent was the traffic's fraction, and the fraction was the not the pod's, and the not pod's was the configuration's error, and the error was the layer, and the layer was the route versus the service, and the versus was the confusion, and the confusion was the incident.

The canary pod received the proportional share by the pod count, and the count was the one of the eleven, and the eleven was the nine percent, and the nine percent was the close to the intent, and the close was the misleading, because the errors were the five percent and the five percent was the capacity, and the capacity was the one pod's limit, and the limit was the saturation.

Why the errors looked like the code

The errors were the canary's, and the canary's was the new version, and the new version was the suspect, and the suspect was the rollback, and the rollback was the conclusion, and the conclusion was the wrong, because the errors were the 503 and the timeouts, and the timeouts were the saturation, and the saturation was the load, and the load was the capacity, and the capacity was the not the code.

The distinction was the error's type, and the type was the read, and the read was the absent, and the absent was the assumption, and the assumption was the code, and the code was the rollback, and the rollback was the lost learning, and the learning was the real bug's absence, and the absence was the false alarm, and the alarm was the cost, and the cost is the reason the canary's errors need the classification before the rollback.

This is the same misattribution as the HPA that scaled on a lagging metric, same mechanism, different costume.

What actually fixed it

Set the weight at the layer that splits the traffic. The weight was the ingress's or the mesh's, and the mesh's was the request's fraction, and the fraction was the five percent, and the five percent was the fix, and the fix was the configuration's layer, and the layer was the discipline, because the service's selector is the not the split.

Verified the actual traffic split before trusting the canary. The metric was the per-version request count, and the count was the fraction's proof, and the proof was the verification, and the verification was the fix, and the fix was the dashboard, and the dashboard was the discipline, because the intended split is the not the actual split.

Gave the canary the proportional resources or the proportional traffic. The one pod was the one eleventh of the traffic, and the eleventh was the capacity's match, and the match was the fix, and the fix was the sizing, and the sizing was the discipline, because the canary's capacity should match its share.

Classified the canary's errors before rolling back. The classification was the 5xx versus the timeout versus the business error, and the three were the different diagnoses, and the diagnoses were the capacity or the code, and the two were the different actions, and the actions were the fix, and the fix was the reading.

Alerted on the canary's saturation, not only its error rate. The saturation was the CPU and the queue, and the queue was the capacity's signal, and the signal was the distinction, and the distinction was the fix, and the fix was the metric, and the metric was the discipline.

The rule

A canary only tests the code if it receives the intended fraction of the traffic and has the capacity to serve it. Set the weight at the layer that actually splits requests, verify the real split from the metrics, size the canary to its share, and classify its errors before concluding anything.

The canary showed a five percent error rate and the team rolled back a release that was fine, because the weight was configured on the route while the service selector sent traffic to both deployments, so the single canary pod took its share by pod count and saturated. The lesson I keep is the property, not the incident.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud