Generating Rules Is the Easy Part
Point a profiling engine at a few thousand columns, and it will hand you a catalog of null checks, range checks, uniqueness constraints, and distribution rules before lunch. It is an impressive demo. The harder part starts afterward, when someone has to decide which of those rules deserve to be enforced.
That decision is where programs succeed or fail in production. A rule can be statistically defensible and still wrong for the business. It flags a legitimate transaction, burns hours of an analyst’s time, and slowly teaches the team to stop paying attention. The reverse is just as dangerous. A dataset can pass every generated check and still violate a requirement that accounting, compliance, or a customer-facing decision depends on.
The industry has spent years making rule discovery faster and broader. The next problem is less glamorous: Does this proposed rule protect a real business requirement?
The Real Cost of Alerts Nobody Trusts
We usually talk about alert fatigue as a monitoring problem. In data quality, it is closer to a control problem. When teams are asked to investigate too many low-value findings, they triage hard, mute checks, or start treating exceptions as routine. The failures that matter then have to compete for attention with everything else.
Monte Carlo’s State of Data Quality for 2026 report puts numbers on the pattern. Once a notification channel passes 50 alerts a week, update and response rates drop by about 15%. Past 100 a week, engagement falls by roughly another 20%. Those figures measure how people engage with alerts, not how accurate automated rule discovery is. They still show what noise costs an operating team.
I recall a conversation with a practitioner at a large U.S. property-and-casualty insurer that makes the point better than any statistic. Their summary was: “Data quality was super-easy to establish for a large number of sources – but we need an army to chase and research false alerts.” The company eventually turned off alerting and removed most of its checks, keeping little beyond record-volume monitoring. This is my first-hand recollection, not an audited case study, and I am not naming the organization.
I understand why they did it, even though the outcome is troubling. A monitoring program that is too expensive to run can end up protecting less than a smaller, carefully chosen set of controls. The answer is not to abandon statistical checks. It is to make sure the people receiving the alerts can explain why each one matters.
Three Ways Profiling Can Mislead You
An observed pattern is not a business policy
Say a banking dataset shows that 99.9% of transaction amounts are positive. A profiling engine will happily recommend that every amount be greater than zero. That sounds sensible until refunds, reversals, chargebacks, and accounting adjustments show up.
The rule the business actually needs separates transaction types. Completed purchases require a positive amount. Approved reversals may be negative, provided they reference a valid original transaction. That check depends on what the records mean and how they relate to each other. No amount of staring at one column’s distribution will produce it.
Historical data can normalize historical mistakes
Consider a loan portfolio where the debt-to-income ratio has been calculated incorrectly for several years. A profiler will learn that distribution perfectly. It may flag a sudden departure from the norm and still miss the long-standing calculation error underneath.
This is not just a theoretical weakness. The Deequ project’s documentation on automatic constraint suggestion says plainly that its heuristics assume the observed data is static and correct, and it advises reviewing suggestions before production use. Consistency is useful evidence. It is not proof of correctness.
Some critical controls never show up in a column profile
A fully paid loan should carry an outstanding principal balance consistent with the institution’s payoff and adjustment policies. A charged-off loan should follow the applicable accounting and servicing rules. A transaction should reference a valid, appropriately authorized account. A loan approval must satisfy the relevant product and risk policies.
None of that can be established reliably from a single column’s distribution. These requirements depend on definitions, relationships, reference data, documented exceptions, and sometimes regulatory obligations. This is the quieter danger of automated discovery: false confidence. The dashboard shows every generated check passing while a serious business violation goes undetected.
Putting Business Context to Work
Take a loan origination dataset. A profiler can tell you that loan amounts are numeric, interest rates sit in familiar ranges, and borrower identifiers are usually present. All useful. But none of it says whether a loan was approved under the right policy, or whether its balance is consistent with its payment history.
A context-aware process draws on the definitions of fields, approval and servicing policies, relevant regulatory requirements, the relationships among accounts and transactions, and documented operational exceptions. It also asks which errors reach the financial statements, the credit-risk models, or a customer-facing action. What comes out should be a set of candidate controls tied to identifiable business purposes, not a longer list of statistical curiosities.
There is a catch, and I would be doing you a disservice to skip it. Enterprise documentation is often outdated, incomplete, or contradictory. Natural-language requirements can be ambiguous. An AI model can misread an exception or invent a relationship that does not exist. Context improves the inputs to rule generation. It does not remove the need for review, traceability, and testing.
Same Data, Different Recommendations
The difference is easy to see with transactions. Profiling notices that almost every amount is positive and proposes a blanket positivity rule. Business context surfaces approved reversals and the requirement that they reference an original transaction, which leads to a conditional check.
| Approach | What it sees | Recommended rule |
| Profiling alone | Nearly all amounts are positive | Every amount must be greater than zero |
| Profiling plus business context | Reversals exist and must trace back to an original transaction | Purchases must be positive; approved reversals may be negative if they reference a valid original transaction |
Illustrative rule recommendations from the same banking data. This is an example, not a measured vendor benchmark, and both approaches still need validation against business requirements.
Measure the Rules That Matter
Counting generated rules is a poor proxy for effectiveness. A system that recommends 300 checks may be worth less than one that proposes 40, if most of the 300 are redundant, irrelevant or expensive to investigate. A serious evaluation needs at least three measures:
- Business-rule precision: what share of the generated rules are valid, meaningful, and fit for production
- Business-rule recall: what share of the essential rules the business needs were actually discovered
- Alert relevance: what proportion of the resulting alerts point to genuinely actionable defects
Precision and recall need an independently reviewed reference catalog of business requirements. Alert relevance needs execution results and incident-level ground truth. These are harder to collect than a rule count, but they tell you whether automation is reducing work and improving coverage, or just moving the effort from rule authoring to alert investigation.
The trade-offs deserve attention too. An organization may tolerate a lower-priority statistical warning in an exploratory pipeline while demanding much stronger assurance for a financial reconciliation. The right balance depends on the cost of missed defects, the cost of false alarms, and the consequences of acting on incorrect data.
AI Agents Raise the Stakes
This distinction matters even more once software can act on the data it reads. A conventional dashboard may show a wrong number, and an analyst has time to question it. An AI agent may use that same number to recommend a collection action, draft a customer response, or kick off a workflow before anyone looks at the underlying records.
For a loan-servicing agent, the question is not only whether balances and payment histories look statistically normal. It is whether they are mutually consistent and valid for the specific action under consideration. Context-aware checks can add a control at exactly that point. They are no substitute for authorization, application safeguards, or human oversight, but they can keep a technically plausible data error from turning into a business decision.
The Next Advance Is Judgment, Not Volume
Statistical profiling still has an essential role. It is very good at finding structural defects, unfamiliar values, and unexpected changes at scale. AI-assisted generation can help translate written requirements into executable checks. Neither should be discarded.
But the standard for success has to change. The question is not how many rules a system can discover in an afternoon. It is how many important failures those rules can catch, how many critical requirements they miss, and how much investigation they impose on the people expected to respond.
The next generation of automation should produce rules that have a reason to exist, and make that reason clear enough for a business owner to defend.
Developing a Data Quality Program
Learn how to navigate critical components of designing and launching a data quality program with precision.
