by Jay Zaidi
Do enterprises treat data as a strategic asset? Are organizations making the right investments with respect to defining data quality requirements, proactively monitoring the quality of business critical data and managing its quality throughout the corporate information supply chain? Such questions must be asked by modern enterprises; they run on data, data is their lifeblood, data is perhaps the most important factor in corporate decision-making and financial reporting. Disclosures depend on data and risk is constantly monitored and managed using data. Therefore, it is clear that data is a crucial strategic asset and must be treated as such.
The Central Challenge
The use of passive tests that require the examination of documentary evidence related to internal controls and risk management are not the only on-going tests within the modern enterprise. Regulators and internal audit teams are also mandating more active tests, focused on transparency and accountability. These tests are integrated into a multitude of business processes and data management procedures, with metrics that address the information supply chain from all ends of the spectrum. New legislation and oversight bodies have been instituted to monitor and address systemic issues related to data quality, particularly with multinationals and government regulated firms in the Banking and Financial Services, Health Care, Pharmaceuticals, Consumer Packaged Goods, Retail, Government, Transportation, Real Estate, and Electronics industries to name just a few.
To ensure that consumers of corporate data have the highest level of confidence in such data, proper data quality business practices are essential for prospering in the 21st century. Enterprises need to consider the significant human and capital resources they expend in reactively addressing data quality-related issues (usually in silos), such issues include: redundant checks built into the enterprise’s systems and processes, the lack of visibility into the quality of data as it flows through various systems, the significant legal and reputational risk exposure caused by poor quality data, the impact of low quality data on decision-making and the impact of poor quality data on time-to-value. While these are only a short list of the many issues that organizations must address with their data quality, they highlight the central problem – poor data quality is a systemic issue that affects all levels of an organization and must be dealt with as such.
The Solution
Data quality has emerged as a crucial field within data management and data governance, especially due to a number of high-profile incidents resulting from poor data quality management. The focus must now be on addressing those systemic issues discussed above, which typically result in exposing companies to major risks. This requires a shift from reactive to proactive approaches regarding data quality management and the implementation of the Holistic Data Quality (HDQ) framework.
HDQ is a term that I coined to highlight a paradigm shift in data quality management. Quality should not be evaluated or managed in vertical business silos, but in a holistic integrated (cross-silo) approach, based on the HDQ framework. Such an approach incorporates consistent quality measures, exception-based reporting and robust analytics. Implementing HDQ at the enterprise level results in higher quality data, lowers costs for the remediation of quality issues, provides transparency into hot spots and outliers, and significantly reduces the costs and resources expended on supporting internal and external audits. There are significant regulatory compliance benefits as well.
Given the data deluge of the past few years and the increasing complexity of the data landscape (e.g. Cloud and ASP deployments), there are no quick fixes. However, taking a strategic approach and systematically implementing HDQ and associated shared service capabilities will enable firms to overcome these challenges.
The foundational components of a HDQ solution are the following:
- Data quality dimensional framework
- Solution Architecture
- Data quality software
- Business Process Management (BPM) and Web Services orchestration capability
- Metadata repository
- Enterprise data quality mart
- Business Intelligence (BI) software
- Services Oriented Architecture (SOA) platform
- Issues Management System
The dimensional framework enables consistent data quality requirements and metric definition. The solution architecture is driven by enterprise data quality business use cases and data processing patterns. The data quality software and BPM capabilities are utilized to implement business solutions and data governance processes. The Metadata repository hosts data dictionaries, business glossaries, data lineage, data interface definitions and other metadata that is necessary for managing data. The data quality mart and BI platforms capture and provide enterprise reporting and intelligence to show quality hot spots, data patterns, data anomalies, outliers and the overall health of business critical data. The SOA platform provides the ability to vend enterprise class data quality services. The issues management system is a central repository of data issues and typically has workflow capability to route and manage issues, along with an enterprise level reporting component.
Implementing an enterprise level HDQ strategy requires strong program management, systems integration, services oriented architecture, data warehousing and business intelligence expertise.
Proven Results
In the past, I have developed and successfully deployed the HDQ framework, associated methodologies, best practices, design patterns and disruptive tools to address data and information quality challenges. This is a “must have” component of every firm’s Enterprise Data Management program. One must embark on it with a clear vision, sponsorship at the Board and C-Level, a commitment for long-term financial and political support from senior management and an appetite for change.
