The Alert That Fires Every Day Is Not an Alert

The fired behaved. The incident behaved. The chain between them is where working as designed stopped meaning working.

This is the thesis: an alert that fires predictably and is resolved without action is not an alert, it is a scheduled notification, and the cost is not the noise itself but the trained response that carries over to the alert that matters.

This is an opinion, and the threshold is genuinely hard to set, and the rule of thumb at the end is where I land.

What the daily alert actually is

The alert is the threshold, and the threshold is the crossed, and the crossed is the daily, and the daily is the predictable, and the predictable is the not the actionable, and the not actionable is the noise, and the noise is the acknowledged, and the acknowledged is the decision, and the decision is the nothing is wrong, and the nothing is the six weeks' evidence.

The six weeks are the data, and the data is the alert's uselessness, and the uselessness is the removal's case, and the case is the not made, and the not made is the inertia, and the inertia is the threshold's ownership, and the ownership is the absent, and the absent is the reason the alert survives its own evidence.

The alert is also the signal's decay, and the decay is the attention's finite, and the finite is the allocation, and the allocation is the every page, and the every page is the budget's consumption, and the consumption is the real alert's starvation, and the starvation is the incident, and the incident is the cost.

Why the trained response is the real cost

The response is the muscle memory, and the muscle memory is the dismiss, and the dismiss is the fast, and the fast is the efficient for the noise, and the noise is the ninety nine percent, and the ninety nine percent is the training, and the training is the one percent's misclassification, and the misclassification is the incident, and the incident is the hour.

The hour is the not the alert's failure, and the not failure is the response's, and the response's is the conditioning, and the conditioning is the system's product, and the product is the organisation's, and the organisation's is the fixable, and the fixable is the alert's removal, and the removal is the discipline.

The noise is also the new engineer's training, and the training is the wrong lesson, and the wrong lesson is the alerts are the ignored, and the ignored is the culture, and the culture is the incident's cause, and the cause is the deeper than the threshold, and the deeper is the fix's target.

This is the same product framing as the logs that are a product nobody owns, and the shared property is the one worth fixing.

Where the alert is genuinely right

The alert is right when the action is the immediate, and the immediate is the page's justification, and the justification is the human's necessity, and the necessity is the keep, and the keep is the discipline.

The alert is right when the silence is the worse, and the worse is the missed incident, and the missed is the cost, and the cost is the noise's justification, and the justification is the honest, and the honest is the threshold's setting, and the setting is the trade.

The alert is wrong when the resolution is the acknowledge, and the acknowledge is the no change, and the no change is the six weeks, and the six weeks is the evidence, and the evidence is the removal, and the removal is the fix, and the fix is the decision.

The thumb rule I use

Every alert should have an owner and a runbook action, and an alert resolved by acknowledgement alone for more than a few occurrences should be deleted, retuned or demoted to a ticket, because the trained response is the cost that outlives the noise.

Count the resolutions without action. The count is the evidence, and the evidence is the removal's case, and the case is the review, and the review is the monthly, and the monthly is the discipline, and the discipline is the fix, because the uncounted alert is the permanent alert.

Delete or demote rather than retune by default. The retune is the often wrong, and the wrong is the still firing, and the still firing is the removal, and the removal is the honest, and the honest is the fix, and the fix is the decision, and the decision is the discipline.

Make the page require an action. The action is the runbook's step, and the step is the justification, and the justification is the keep, and the keep is the discipline, and the discipline is the fix, because the page without the action is the notification.

Route the non-urgent to a ticket. The ticket is the queue, and the queue is the not the interruption, and the not interruption is the budget's preservation, and the preservation is the fix, and the fix is the routing, and the routing is the discipline.

Review the alert list with the incident history. The history is the which alerts fired during the real incidents, and the fired is the useful, and the useful is the keep, and the keep is the fix, and the fix is the evidence, and the evidence is the discipline, because the alert that never helped is the cost.

Where I land

The daily alert is not a small annoyance, it is a six-week training programme in ignoring pages, and the training works, which is why the real incident was dismissed for an hour. The fix is not a better threshold, it is the willingness to delete the alert that its own history has already proved useless.

The alert fired every morning for six weeks and was acknowledged every morning without anybody changing anything, and when the real outage arrived it made the same sound. Get that property wrong and every layer above it lies to you politely.

Published on debugly.dev by Rohit Bhadani | Founder of Krova Cloud