Observability Platforms Sold You a Very Expensive Search Box

We spent a serious amount of money on an observability platform. It was genuinely impressive. Dashboards loaded fast, the query language was pleasant, and the demo showed an engineer finding a needle in a haystack in seconds.

Then we had a real incident, and the thing we needed was the one thing the platform had not been told to keep. The haystack was enormous and the needle had been dropped on the floor to save ingest costs. That gap, between what the demo promises and what the retention policy allows, is where the money goes.

Here is the thesis. Observability as sold is mostly a search engine over telemetry. Search is useful, but search is not observability, and conflating the two has led teams to buy volume when what they needed was the ability to ask new questions.

The definition that matters

There is a precise, useful definition from control theory that the industry borrowed and then forgot. A system is observable if you can infer its internal state from its external outputs, without having to ship new code to answer a new question.

The operational part of that definition is the last clause. Observability means you can ask a question you did not anticipate, using data you already have, without redeploying. High cardinality, high dimensionality, wide events. The ability to slice by any combination of attributes after the fact.

A search box over logs does not give you that if the logs were pre shaped. If someone decided at write time which fields to emit and which to drop, then any question that needs a dropped field is unanswerable, and you will discover that precisely during the incident that needed it.

What you actually bought

To be fair, the platforms are not useless. They are very good at specific things, and I want to name them so the argument is not a straw man.

They give you aggregation and alerting over known questions. Error rates, latency percentiles, saturation. The dashboard you build on day one answers the questions you knew on day one.

They give you a shared, fast place to look, which removes the "which of our four tools has this log" tax. That is a real productivity win, even if it is consolidation rather than observability.

And they give you retention and scale that you would not run yourself. Storing and indexing petabytes is genuinely hard, and renting it is reasonable.

None of those is the ability to ask a new question. They are the ability to ask old questions faster. That is worth money, but it is a different purchase, and it should be priced and evaluated as one.

The cost model punishes the right behaviour

Here is the perverse part. The behaviours that make a system more observable, emitting wide events with many attributes, sampling less, keeping high cardinality dimensions, are exactly the behaviours that raise the bill. So teams rationally do the opposite. They drop fields, they sample aggressively, they collapse cardinality, to keep costs down.

The result is a system that is cheaper and less observable, optimised for the invoice rather than for the incident. I have watched teams delete the user ID from spans to save money, then spend an incident unable to join the traces for one affected user. The saving was visible monthly. The cost was visible once, and it was larger.

This is the same dynamic as code coverage as a marketing metric: the number that gets managed is the one that is measured, and it is not the one that matters.

What would make it observability

A few things separate a search box from an observable system, and they are mostly choices, not purchases.

Wide events over pre shaped logs. Emit one rich event per request with every dimension you might slice by, rather than a dozen thin logs decided in advance. The question you ask later is a filter over the event, not a wish that someone logged the right thing.

Retain the tail, not just the average. Sample the boring middle aggressively if you must, but keep every error and every slow request unsampled. Incidents live in the tail, and sampling the tail away is sampling away the signal you pay for.

Make the question cheap to ask. The test of observability is whether a new question is a query or a deploy. If answering "which users on plan X in region Y hit the slow path" requires shipping code, you have instrumentation, not observability.

The counterargument, fairly

There is a real defence of the search box. Most incidents are not novel. They are the same five failures wearing new clothes, and a fast search over comprehensive logs resolves most of them quickly. For a small team, comprehensive logs plus a good search is often enough, and the elaborate wide event machinery is overhead they do not need.

I accept that. The argument is not that search is worthless. It is that search is a floor, not a ceiling, and that buying it should not be mistaken for buying observability. The teams that get bitten are the ones whose systems are complex enough to produce novel failures but whose telemetry was bought as if it were simple.

The rule of thumb

When evaluating the bill, ask one question: name the last incident where you needed a field or a dimension you had not planned for. If the answer is that you had it and could query it, your money bought observability. If the answer is that you discovered the gap mid incident, you bought a search box, and the next purchase should be aimed at the gap, not at more volume.

Volume is easy to sell and easy to bill. The ability to ask a new question is harder to build and harder to demo, and it is the whole point.

If your dashboards are green during the incident, that is the same lesson from another angle, covered in three metrics that catch incidents. And if the telemetry you need only exists in someone's head, no platform fixes that, which is the argument of nobody reads your dashboards.