Every Data Observability Stack Has the Same Blind Spot

For about two and a half years, Uber calculated its own take by deducting its cut from the gross fare, including taxes and fees, rather than the net. The data flowed on schedule. The schemas were valid. The row counts were normal. Every automated check a modern data team runs would have shown green throughout, because nothing about the data was structurally wrong. It was just wrong about the world, to the tune of tens of millions of dollars, and no freshness monitor on earth was ever going to catch it.

This is the blind spot at the center of the data observability boom, and it’s worth being precise about, because the category is genuinely useful and that usefulness is exactly what makes the gap dangerous.

What Observability Actually Watches

Data observability, as the tools define it, rests on five pillars: freshness, volume, distribution, schema, and lineage. Is the data on time? Is there the right amount of it? Do the values fall in expected statistical ranges? Did the structure change? Where did it come from?

Every one of those is a signal about the plumbing. They tell you whether data moved correctly through your pipelines, and they’re good at it. When a table stops updating, when a load drops half its rows, when a column changes type, observability fires, and it should. Those failures are real and common and worth catching automatically.

But notice what every pillar has in common: they all evaluate the data’s structure and statistics, never its meaning. They watch whether the pipes are flowing. They are constitutionally blind to whether what’s flowing through them still means what your dashboards assume it means.

The Failures That Pass Every Check

Sort real data disasters into two buckets and the gap becomes obvious.

In the first bucket are the structural failures observability was built for. A gaming company ingested malformed customer data and took a nine-figure hit; a volume-and-distribution monitor would flag that. A public health agency lost nearly 16,000 case records to a spreadsheet row limit; a volume check catches that. A fat-fingered trade that swaps units creates a distribution spike a monitor can see. These are the demos in every observability vendor’s deck, and the tools genuinely help.

In the second bucket are the failures that burn you precisely because they look perfect. Uber’s gross-versus-net error. A credit bureau that shipped miscalculated scores, some off by 20-plus points, through a logic error that produced entirely plausible-looking numbers. A finance team that quietly redefines “active user” upstream, so the metric keeps computing, keeps arriving on time, and now means something different than it did last quarter. In each case the data is fresh, complete, correctly typed, and statistically unremarkable. And in each case it is business-catastrophically wrong. No pillar covers this bucket, because there is no structural anomaly to detect. The anomaly is in the meaning, and meaning isn’t in the schema.

Why the Tools Can’t See It, Even in Principle

This isn’t a maturity gap that the next release closes. It’s structural, and the observability literature quietly concedes as much. One practitioner framing on this very publication notes that observability “largely addresses metadata errors” and does not address semantic correctness, business logic validation, or cross-system reconciliation. The tool learns your data’s normal statistical shape and alerts on deviation from it. But semantically-wrong data has a perfectly normal shape, by definition, and here’s the sharper problem: When the wrong data arrives steadily, the adaptive baseline learns the error as the new normal. The anomaly detector doesn’t just miss the semantic failure. Given enough time, it ratifies it.

The mechanism that produces these incidents even has a name: semantic drift, or metric definition drift. An upstream business process changes, a definition shifts, a source system starts capturing something subtly different, and the column that used to mean one thing now means another, with no structural fingerprint at the moment of change. Freshness: fine. Volume: fine. Meaning: silently, expensively different.

What Actually Guards the Meaning

The point isn’t to abandon observability, which does its real job well. The point is to stop believing it covers a job it was never built for, and to add the layer that guards meaning rather than movement.

Three practices do that. First, validate against business truth, not just statistical baselines: reconciliation checks that compare your data to an independent source of ground truth, the finance system, the source-of-record, a manual control total, so that “the numbers match reality” becomes a monitored condition rather than an assumption. Second, make metric definitions explicit and governed through a semantic layer, so that when “active user” changes, the change is a reviewed event with an owner, not a silent drift someone discovers a quarter later in a board meeting. Third, add business-rule assertions that encode what the data is supposed to mean, the invariants a human analyst would notice were violated, which no freshness check will ever encode for you.

Of those, reconciliation against an external source of truth is the one I would implement first, because it is the only check that can catch a failure the rest of your stack has already declared healthy.

The Stakes Just Went Up

There was always a human backstop for this: an analyst who looked at the number, frowned, and said “that can’t be right.” That instinct caught a lot of semantically-valid-but-wrong data before it did damage, precisely because humans check meaning by default and machines check structure by default.

Automation is removing that backstop. As agentic systems begin acting on data autonomously, there is no longer a person between the metric and the decision to frown at a number that computes cleanly but describes a world that no longer exists. The observability blind spot stops being a reporting inconvenience and becomes an operational one, where a fresh, complete, schema-valid, business-wrong number drives an action at machine speed.

Your observability dashboard is green. That tells you the plumbing works. It tells you nothing about whether the water is safe to drink, and the incident that eventually burns you will arrive perfectly on time, in exactly the right volume, meaning something you never checked for.