by Jay Zaidi
Do enterprises treat data as a strategic asset? Are organizations making the right investments with respect to defining data quality requirements, proactively monitoring the quality of business critical data and managing its quality throughout the corporate information supply chain? Such questions must be asked by modern enterprises; they run on data, data is their lifeblood, data is perhaps the most important factor in corporate decision-making and financial reporting. Disclosures depend on data and risk is constantly monitored and managed using data. Therefore, it is clear that data is a crucial strategic asset and must be treated as such.
The Central Challenge
The use of passive tests that require the examination of documentary evidence related to internal controls and risk management are not the only on-going tests within the modern enterprise. Regulators and internal audit teams are also mandating more active tests, focused on transparency and accountability. These tests are integrated into a multitude of business processes and data management procedures, with metrics that address the information supply chain from all ends of the spectrum. New legislation and oversight bodies have been instituted to monitor and address systemic issues related to data quality, particularly with multinationals and government regulated firms in the Banking and Financial Services, Health Care, Pharmaceuticals, Consumer Packaged Goods, Retail, Government, Transportation, Real Estate, and Electronics industries to name just a few.
To ensure that consumers of corporate data have the highest level of confidence in such data, proper data quality business practices are essential for prospering in the 21st century. Enterprises need to consider the significant human and capital resources they expend in reactively addressing data quality-related issues (usually in silos), such issues include: redundant checks built into the enterprise’s systems and processes, the lack of visibility into the quality of data as it flows through various systems, the significant legal and reputational risk exposure caused by poor quality data, the impact of low quality data on decision-making and the impact of poor quality data on time-to-value. While these are only a short list of the many issues that organizations must address with their data quality, they highlight the central problem – poor data quality is a systemic issue that affects all levels of an organization and must be dealt with as such.
The Solution
Data quality has emerged as a crucial field within data management and data governance, especially due to a number of high-profile incidents resulting from poor data quality management. The focus must now be on addressing those systemic issues discussed above, which typically result in exposing companies to major risks. This requires a shift from reactive to proactive approaches regarding data quality management and the implementation of the Holistic Data Quality (HDQ) framework.
HDQ is a term that I coined to highlight a paradigm shift in data quality management. Quality should not be evaluated or managed in vertical business silos, but in a holistic integrated (cross-silo) approach, based on the HDQ framework. Such an approach incorporates consistent quality measures, exception-based reporting and robust analytics. Implementing HDQ at the enterprise level results in higher quality data, lowers costs for the remediation of quality issues, provides transparency into hot spots and outliers, and significantly reduces the costs and resources expended on supporting internal and external audits. There are significant regulatory compliance benefits as well.
Given the data deluge of the past few years and the increasing complexity of the data landscape (e.g. Cloud and ASP deployments), there are no quick fixes. However, taking a strategic approach and systematically implementing HDQ and associated shared service capabilities will enable firms to overcome these challenges.
The foundational components of a HDQ solution are the following:
- Data quality dimensional framework
- Solution Architecture
- Data quality software
- Business Process Management (BPM) and Web Services orchestration capability
- Metadata repository
- Enterprise data quality mart
- Business Intelligence (BI) software
- Services Oriented Architecture (SOA) platform
- Issues Management System
The dimensional framework enables consistent data quality requirements and metric definition. The solution architecture is driven by enterprise data quality business use cases and data processing patterns. The data quality software and BPM capabilities are utilized to implement business solutions and data governance processes. The Metadata repository hosts data dictionaries, business glossaries, data lineage, data interface definitions and other metadata that is necessary for managing data. The data quality mart and BI platforms capture and provide enterprise reporting and intelligence to show quality hot spots, data patterns, data anomalies, outliers and the overall health of business critical data. The SOA platform provides the ability to vend enterprise class data quality services. The issues management system is a central repository of data issues and typically has workflow capability to route and manage issues, along with an enterprise level reporting component.
Implementing an enterprise level HDQ strategy requires strong program management, systems integration, services oriented architecture, data warehousing and business intelligence expertise.
Proven Results
In the past, I have developed and successfully deployed the HDQ framework, associated methodologies, best practices, design patterns and disruptive tools to address data and information quality challenges. This is a “must have” component of every firm’s Enterprise Data Management program. One must embark on it with a clear vision, sponsorship at the Board and C-Level, a commitment for long-term financial and political support from senior management and an appetite for change.
Jay,
You raise an interesting concept with “HDQ”, and I absolutely agree that a paradigm shift has taken place. Companies that realize it, specifically in regards to risk, will excel and so many business decisions can be easily enforced with the right data. In your last “results” section, you didn’t say specifically what results you saw with this systematic application: what sorts of metrics did you utilize to define success?
Another article you may be interested in is this blog post written on crisis and risk management, based off the book “Dirty Rotten Strategies” by Mr. Ian Mitroff:
http://info.ibs-us.com/blog/bid/39996/Crisis-and-Enterprise-Risk-Management-What-Do-You-Have
Definitely worth checking out, and a great read!
Great post,
Ashley
Ashley:
Thanks for your comments. We’ve seen a significant interest in data quality across the enterprise – at all levels – once we provided transparency into data quality metrics across the information supply chain. Business and Operations teams are now actively monitoring the metrics and addressing the root causes of data anomalies and outliers. We have also noticed a healthy dialog between IT, business and operations of the producer and consumer organizations – to address data issues found downstream (in data marts and warehouses).
The primary measure of success is an improvement of data quality over time and our ability to proactively manage data quality – rather than doing it as an afterthought – when operational and audit issues were encountered.
-Jay
I really like this approach, Jay. What’s not yet clear is whether this is another silo–a separate DQ architecture atop or underneath or ringing the business performance architecture–or is it integrated?
Paul Erb
Virginia, USA
Paul:
Great question and thanks for your comment.
The siloed approach to data management won’t work any more, due to the increased regulatory pressures and governance challenges that many firms are facing. In addition to this, there is a need for more transparency into state of enterprise data and it’s quality – end-to-end, not just in individual silos.
What I am proposing with the new HDQ paradigm is gaining a holistic view of business critical data and it’s quality across a firm’s Information Supply Chain. The individual lines of business (horizontal silos) will continue to monitor and manage the line-of-business critical data. However, the team at the enterprise level that has a holistic view of data will be able to feed information to the individual silos – based on patterns, data anomalies and downstream or upstream issues that it finds (something that the horizontal silos will never be able to get, since it’s view is limited). The two layers are integrated – Enterprise and LOB’s have the ability to share rules, scorecards and other artifacts, assuming they use the same tool set and standards.
Hope this helps! If not, please respond and I’ll try to expand on the above.
-Jay
Paul:
For a practical implementation of an Enterprise Data Quality Dashboard based on the HDQ methodology, please refer to the presentation I delivered at the 2011 Winter Data Governance Conference – http://www.jayzaidi.com/#!articles/vstc1=quality-dashboard.
Thanks!
-Jay
Hi Jay, great article and vision. Definitely moves us beyond reactive governance and management. We are seeing growing recognition of data as critical business assets. That suggests we will eventually see data as merely another class of asset to be governed like any other.
So can we begin to manage risk arising from data quality like we already manage risk for other assets? Can the question of acceptable levels of quality for a given data domain or vertical business process be stated as a calculation of risk?
I believe this may be occurring in at least two banks that I have presented to in the past six months. In both cases, data governance resides or is largely driven by the Chief Risk Officer. In one of the two cases, data management as a whole has been placed under risk management.
I am wondering if you are seeing the same trend. And could this be the path to finally bringing business partners to the table with data quality issues framed within the context business value?
Max:
I truly believe that business has to take ownership of data governance and quality, if such programs are going to succeed. However, this is easier said than done, because of a lack of education and awareness regarding governance and quality on the part of business teams, largely because they’ve traditionally been managed and owned by IT.
Does this mean the current situation can not or will not change? I don’t think so. All it means is that this is a major paradigm shift and it will take longer that one would like to gain traction.
Personally speaking, I believe every firm (especially the large ones) must designate a Chief Data Officer (CDO) who should not just facilitate better management and governance of data, but allows educate, bring awareness to data related issues, arbitrate data-related issues and drive program adoption across the firm. Another reason why a CDO role is important is because data is political – since the siloed organizational models have created data fiefdoms – which would be hard to break through without a C-level point person with enough clout.
I am seeing a positive trend toward business ownership, but the progress isn’t as fast as I’d expect.
-Jay
[...] Holistic Data Quality – A New Paradigm in Data Quality Management: http://www.dataversity.net/holistic-data-quality-a-new-paradigm-in-enterprise-data-quality-managemen… Proactive and Reactive Techniques To Address Information Quality Challenges Head On: [...]
[...] Please click on this link to see the entire article – http://www.dataversity.net/archives/7403 [...]