From Data Complexity to Data Decisions: Rethinking the Vs

The V list has a strange history. It started as a compact way to explain why some data problems demanded different technology. Then it kept growing. Volume, velocity, and variety became five, seven, 10, and more, depending on the source.

That expansion is usually treated as a taxonomy problem. Which Vs belong on the list? I think the more useful question is simpler. What does each dimension help us diagnose?

The NIST Big Data Interoperability Framework centers volume, velocity, variety, and variability as fundamental drivers, while later frameworks and practitioner models add dimensions such as veracity, value, validity, volatility, viscosity, vocabulary, and others. There is no single canonical count, and the further the list grows, the less useful the count becomes.,

That is where I think the framework still earns its place. It can help us decide what problem we are actually dealing with.

The List Is Less Important Than the Constraint

A billion clean, structured, stable rows can be easier to manage than a much smaller environment spread across conflicting systems, inconsistent definitions, and changing business rules.

That distinction matters because the visible symptom often points teams toward the wrong investment. A large dataset suggests infrastructure. A slow report suggests performance. A disputed number suggests data quality. The problem is that the symptom and the constraint are not always the same thing.

The NIST framework makes this distinction useful because the characteristics point to different technical pressures. Volume changes storage and processing needs. Velocity changes what the architecture must do when information needs to be processed quickly. Variety introduces different repositories, domains, types, and semantics. Variability deals with changes in the data itself or in the way it arrives and behaves. NIST frames these characteristics as part of the problem definition, not as a scorecard for maturity.

For me, the practical test is simple. If the diagnosis is right, it should change the decision.

Volume may lead to partitioning, distributed processing, storage design, or cost management. Velocity moves the discussion toward freshness, latency, event processing, and service levels. Variety and vocabulary can move it toward metadata, semantic alignment, integration patterns, and data contracts. More compute will not reconcile two definitions that mean different things.

The same environment can have several of these problems at once. It usually does. The point is to identify which one is actually blocking the outcome that matters.

A Technology Problem Can Be a Definition Problem

I have seen a metric become commercially important enough that a small gap in its definition could create a material billing dispute even though every technical layer was doing what it had been told to do.

The metric was an active user, and the rule depended on qualifying actions across different time windows. On paper it looked precise, but basic questions were still open. What happens when a user misses a threshold by one day? Does a later action reactivate the user? When a new cycle begins, does the previous activity still count? Can extra activity in one period carry forward? How long can someone go without a qualifying action and still remain active?

One version of the rule looked specific enough at first. A user became active after a defined activation action. During the first 90 days, the user had to receive defined engagement material such as email, notifications, or other content. By six months, at least one deeper qualifying action had to occur. Across the first 12 months, two of those deeper actions were required.

Then the questions started. If the first 90-day requirement was missed but a deeper action happened on day 91, did that later action override the missed requirement? If nothing happened in the first six months but four qualifying actions happened in the second six, was that user less active than someone who completed two actions spread across the year? If the 12-month requirement was missed, when could the user re-enter? What had to happen on renewal? What reset? What counted as cancellation or exclusion from the service?

Those are not implementation details. They determine the metric. They also expose a common gap between a rule written in a contract and the way product, operations, systems, and data have to make that rule executable. If those groups do not resolve the transitions between the windows, the words can look precise while the behavior remains ambiguous.

Those questions are not edge cases once the metric drives money. Under one interpretation, a user can complete qualifying actions early in one cycle and the next required action late in a later cycle. The result can be a long period, close to two years in an extreme case, with no qualifying action while the user still appears active.

The usual investigation starts somewhere else. People check the dashboard, the refresh, the filters, the query, the transformation scripts, and the source record. Then they ask whether the operational action was recorded correctly. All of that can be worth checking. But all of it can also be correct.

At that point, lineage and reconciliation have done something useful by eliminating technical explanations. The source system can have recorded exactly what it was asked to record. The pipeline can have transformed the data correctly. The dashboard can have implemented the written logic correctly. The number can still be wrong for the decision. No pipeline can decide what active should mean when the business rule never settled it.

The diagnosis now moves into vocabulary, validity, governance, and value. The definition needs an owner, a grain, clear inclusion and exclusion rules, effective dates, cycle behavior, reactivation logic, and a decision about what the metric is actually meant to represent. If payment depends on that count, ambiguity stops being a reporting inconvenience. It becomes revenue, billing, and relationship risk.

At scale, the difference does not need to be subtle. If one side plans the operation around 20,000 active users and the other validates only 15,000 under a different interpretation of the same rule, that is a 25 percent shortfall against the operating plan. Depending on the economics of the service, that can change staffing, margin, billing, and whether the operation is viable.

Successful Jobs Can Still Consume Incomplete Data

A different failure can happen with a perfectly clear business definition.

Consider a nightly pipeline with dozens of jobs and tables. Some jobs are independent. Others depend on several upstream outputs. A consolidated table near the end may need all of them before it can be trusted. The schedule is tight because the business expects the analytical environment to be ready early in the morning.

Now one upstream job takes a little longer than usual. Nothing crashes. It finishes successfully. The downstream job also finishes successfully. The problem is timing. The consumer started seconds or a few minutes before the source it depended on had finished.

At large scale, this can be difficult to notice. If millions of records are processed, the total may look close enough to normal. One record, one transaction, or a small group can still be missing. In one context that difference is immaterial. In another it changes a metric, a customer action, a payment, or another decision that matters much more than the row count suggests.

A green status tells me that a job ran. It does not tell me that every dependency was ready when the job started.

This is a velocity problem, but not only a performance problem. The checks have to include freshness, completion time, dependency order, expected row movement, and whether the required upstream assets were complete before a downstream consumer began. Veracity and validity cannot be inferred from a successful execution status.

When the Framework Expands, the Questions Get Better

The familiar list is only the beginning. Later variants add validity, vulnerability, volatility, viscosity, venue, vocabulary, vagueness, and other labels. Some lists go further still.

I do not think the goal is to build the longest possible list. There is no prize for collecting more Vs. As the count grows, provenance gets less consistent and several labels overlap. I still find many of the additional Vs useful for one reason. They force us to ask better questions.

Validity asks whether data is fit for the specific purpose, which is different from asking whether the source record is accurate. Vulnerability moves the discussion toward security and privacy exposure. Viscosity asks how difficult data is to move or integrate. Vocabulary exposes semantic disagreement, while vagueness raises another issue. Even when the data is technically correct, do we know what it actually represents?

An additional V is useful only if it reveals a genuinely different kind of problem.

Visualization and value show why the list stops behaving like a clean taxonomy. Visualization is not simply a property of a dataset. It asks whether people can see the pattern, exception, or change that matters. Value sits even further downstream. It asks whether any of the technical work improves a decision, reduces risk, changes performance, or opens an opportunity.

I am comfortable with that inconsistency. Some Vs describe the data, some describe how it changes, some describe how people interpret it, and others describe the outcome we expect from using it. They sit at different layers, which is exactly why they can expose different kinds of friction.

Modern enterprise environments rarely fail inside one technical layer. The architecture can work while the meaning fails. The data can be accurate while the use case is invalid. The pipeline can be fast while a dependency is late. The dashboard can be clear while the underlying metric is wrong.

Some Complexity Changes While the Pipeline Keeps Running

Variability is easy to underestimate because change does not always look like a failure.

Variability means that a dataset, its structure, its semantics, its flow, or the relationships inside it can change. In machine learning, the related problem of concept drift is well established. The relationship between inputs and the target can move over time, which means a model that worked well before may gradually become less representative of current reality.

A schema-flexible source is a good example. Imagine an embedded object that originally contains a cancellation date. The transformation is made tolerant on purpose. If a malformed value or unexpected type arrives, the pipeline does not stop. It takes the usable value, handles the exception, and keeps the daily process moving.

In a document database with a flexible schema, the data team can still build a careful transformation around the structure it knows. If that embedded object contains a history of dates, the script may flatten it, select the first usable date, validate the type, and continue. Those defensive rules exist for a reason. A daily pipeline that fails every time one record arrives differently is not useful.

That resilience is useful until the meaning changes.

Later, the source begins storing another date in the same structure. It may represent reactivation, a second service state, or another event. Nothing has to break technically because the new value still satisfies the existing type and structure.

Now make the change a little more concrete. The object that used to represent cancellation starts recording activation, pause, reactivation, or another state change for a specific service. The transformation still finds a valid date, but the first date in the object no longer necessarily means cancellation, and the data team may not know that the semantic contract changed.

A defensive rule such as taking the first usable value can now treat a reactivation date as a cancellation date. Depending on the process, that can change an entitlement, a payment, a status, or another downstream action. The code did what it was designed to do. The structure changed around it.

The downstream consequence depends on the business. In a financial application, a wrong state can affect whether an account action is allowed. In a travel application, it can affect whether a booking or service is treated as active, paused, or cancelled. The technical difference can be one field. The business consequence can be much larger.

This is where variety, variability, validity, and vocabulary overlap. Schema and type monitoring can show that something changed. Semantic lineage and data contracts can make the expected meaning explicit. Exception monitoring can keep defensive handling from making bad data invisible. But profiling only tells us what the data is doing. It does not prove that the new behavior is correct.

The same problem can happen even when the schema does not change. Imagine a travel platform with a field intended to store the category of service purchased, such as flight, hotel, or car rental. Later, another team starts writing workflow events into the same field, such as check-in, change request, or cancellation. Every new value is still a valid string. The pipeline keeps running. The dashboard simply gains a few new categories.

But the field no longer answers one business question. It now mixes what the customer bought with what happened to that service afterward. A report that used to answer which services customers were buying becomes polluted by operational events. No null spike is required. No type error is required. No job has to fail. The technical contract can remain intact while the semantic contract has changed.

Volatility asks a related question. How long does the data remain relevant?

The NIST Big Data Reference Architecture treats data lifecycle, processing, and management as architecture concerns because different applications have different requirements. Some decisions depend on the newest available signal. Others require years of history. Keeping everything forever is not automatically a sign of maturity. It can add cost, risk, and noise without improving the decision.

Data can remain historically accurate and still become less useful for a current purpose.

Trust cannot be a one-time certification when the environment keeps changing.

Turn the Vs Into Checks

Once the Vs are treated as diagnostic questions, they can also help decide what deserves monitoring. That is the practical use I care about most.

A simple version looks like this:

• Watch row counts, growth, processing time, and cost.
• Watch freshness, duration, lag, and dependency order.
• Watch schema changes, type drift, and unexpected structures.
• Veracity and validity. Reconcile outputs and test nulls, duplicates, integrity, and business rules.
• Track definition ownership, grain, inclusions, exclusions, version, and effective dates.
• Watch distribution shifts, changing behavior, and new patterns in the data.
• Track relevance, retention, and expiry.
• Watch access, exposure, and handling controls.
• Keep the business test outside the technical score.

This does not mean building one dashboard with every V and calling the problem solved. A data quality dashboard or scorecard is a diagnostic control. It becomes much more useful when it sits on top of lineage, data observability, data contracts, and enough data context to explain what each asset is supposed to mean and how it is supposed to behave.

Then an alert can say more than failed or passed. It can show what changed, what was expected, where the first divergence appeared, which downstream assets are affected, and which rule needs attention.

That matters because teams often make pipelines resilient for good reasons. A malformed text value in a numeric field may be converted to null so one bad record does not stop an entire daily load. That can be the right operational choice. The mistake is allowing the exception to disappear. Keep the pipeline moving, but keep the evidence too.

Stronger technical monitoring does not remove the need for business judgment. It gives that judgment more room. If lineage, observability, contracts, and quality checks can quickly show that the technical path is healthy, the data team can spend less time reopening working scripts and more time with the people who understand the product, the process, the source system, and the business rule.

That is usually the harder part. It is also where many of the problems that matter are hidden.

Value Keeps the Framework Honest

Too many data conversations begin with a solution already chosen. A new platform, a catalog, better dashboards, a governance program, real-time data.

Sometimes those conclusions are right, but the order matters.

Start with the constraint. What is making the data hard to operate? What is making it hard to trust? What is making it hard to interpret? What is changing over time? Which business decision, process, or analytical outcome is being blocked?

Then decide what capability actually needs to change.

The Big Data label has aged, but the questions behind it have not. Organizations still deal with scale, speed, heterogeneity, trust, changing behavior, semantics, security, relevance, and value.

The useful question is simpler. Which characteristic is creating the constraint right now? What evidence supports that diagnosis? What decision should change because of it?

If the answer does not change the decision, the framework is just vocabulary.

That is the point where data complexity becomes a data decision.