Introduction to Rails

The Rails development framework continues to garner popularity for its ease-of-use in the rapid implementation of web-based applications that use a database.  Paired with the Ruby object-oriented scripting language, Rails brings a certain elegance to software engineering, leading to an almost unheard of artifact in the industry – programmer happiness.

From the ground up, Rails embraces the model-view-controller (MVC) software architecture pattern, differentiating itself from other frameworks like Microsoft’s ASP.NET, where MVC only saw a relatively recent introduction.  Database migrations, test-driven development, Agile methodologies and even often maligned documentation are all added into the Rails stack.

Rails’ general ease-of-use extends to the deployment and maintenance cycles as well.  The framework naturally handles typical programming chores, leaving software engineers the time to focus on providing solutions to their customers.

Due to their shared open-source origins, MySQL tends to be the standard database used in Rails applications. However, database adapters do exist for most of the popular relational databases, including Oracle, Sybase, SQL Server, and DB2.  These are typically installed using RubyGems, the packaged libraries used by both Ruby and Rails, although sometimes extra compiling work is required if only the driver source code is available.

Interacting with a database is pretty seamless throughout a Rails application’s lifecycle; this is especially useful when a DBA is not available for the project.  In fact, Rails even makes it simple to use a different database engine for development, test, and production.  Looking at a few of the mechanisms used by Rails for this interaction truly reveals the power and elegance of the framework.

Rails, Data, and the Development Process 

Rails allows the software engineer to perform many typical development tasks in rapid manner, which lends itself nicely with the iterative process typical of Agile development methodologies.

During a project’s initial stages, as the first data models are defined, the Rails developer uses the [scaffold] command. Given a few field names and data types as arguments, it actually creates the model classes, html views for the CRUD functionality to maintain the data, controllers to navigate to those views, and the migration logic to maintain the database’s physical structure.  It even stubs out a few unit test classes for good measure. Accomplishing this much work in one simple command line entry only hints at the full power of the Rails framework.  It only takes one more command [rake db:migrate] to apply all the model changes to the database.  Rails automatically timestamps newly created migration tasks, giving the developer a measure of version control.

This abstraction of migrations allows for easy model changes as a development project goes through its various iterative stages.  When freshly scaffolding a new table, the first migration is created automatically.  On the other hand, additional migrations for table changes are explicitly created by the developer.  These migration classes include separate Ruby methods to both perform and rollback the migration. For instance, the migration class would contain logic to add and drop a column in the two methods, if the migration involved a new column on a database table.  Executing a simple [rake db:migrate] command performs the migration.  If it’s an older version, specifying a specific migration version number allows for a rollback.

In typical Rails migrations, the model classes automatically change in addition to the physical database, but the developer needs to make the view changes manually.  Still, it’s obvious from these simple examples to see how well the Rails framework allows the rapid iterative development process typical of Agile to shine.

Rails, Active Record, and Relational Databases

Active Record serves as Rails’ ORM layer.  It encompasses a wide array of functionality: database connections, primary keys, relationships, aggregation, searches, callbacks, validation and others, including the previously discussed CRUD, models, and migrations.

When defining models in code, the developer uses the [belongs_to] and [has_one] declarations to define one-to-one relationships between tables.  One-to-many relationships use the [has_many] declaration on the parent table, while the [has_and_belongs_to_many] declaration is used to define many-to-many relationships. Code declarations are used for database validation as well.  Required fields use the [validates_presence_of] declaration, while numeric fields use [validates_numericality_of].  Explicit methods are written to handle more complex business rule validation, which get called by the model class before a record is saved.

Rails uses callbacks to execute logic during various points in a model’s lifecycle.  Active Record makes twenty callbacks available to the developer; most of these being before/after pairs on different operations to an object, like [before_save] and [after_save], or [before_destroy] and [after_destroy].  These callbacks provide hooks to insert business logic and it’s also possible for a callback to execute any number of handler methods in order.

Rails automatically handles single-table inheritance; true inheritance is possible through the use of abstract classes.  If a class method called [abstract_class?] defined in a model returns true, Active Record doesn’t attempt to manage a physical database table for that model.  Active Record evens provides support for tree data structures using the [acts_as_tree] declaration when the class is defined.  The [acts_as_list] declaration is used when modeling child lists.

Transaction support is also included in Active Record, as well more esoteric features like Facade Columns which are accomplished by overriding the attribute accessor code in the model class.  In short, Active Record is a robust ORM layer that allows the developer to accomplish database interaction in a rapid manner.

Taking Rails beyond MySQL

As mentioned previously, MySQL tends to be the database engine of choice in Rails applications, although SQLite remains popular for development.  Remember, Rails allows for three separate database engines to be used for development, test, and production. Every Rails project contains a database configuration file called database.yml.  This file contains the connection information for all the databases used in a project.  There are three separate sections in this file for test, development, and production.  The adapter setting in each section specifies the database engine to be used.

Rails comes with SQLite and MySQL support built-in, with other adapters available for many popular database engines.  There is also an ODBC adapter supporting ODBC-enabled databases.  Installing these adapters is usually as simple as installing its RubyGem, but occasionally the driver source code needs to be compiled manually.

New adapters get introduced regularly, so keep an eye on the active Rails community at RubyOnRails.org to stay abreast of the latest developments in Rails database support.

Rails 3 and the Future

Released in early 2011, Rails 3 introduced Active Model which extracts much of the shareable Active Record functionality like validation and i18n into a separate framework, allowing it to be used by models not needing relational database persistence.

One benefactor of Active Model is Mongoid, an object-document-mapping database written in Ruby.  Other NoSQL databases, like DataMapper, also leverage this functionality that formerly resided in Active Record.  It also facilitates support for Ruby-developed relational stores like Sequel.

Considering the vibrant Rails community, there is little doubt the framework will continue to evolve in its support for newer data persistence initiatives.  The refactoring of Active Record creating a separate Active Model framework proves that support for non-traditional NoSQL databases.  In some quarters, there is some question if Rails sports the horsepower to handle big data initiatives; it’s definitely a point worth consideration when evaluating the framework.

Rails has ushered in a revolution in rapid web application development that almost rivals Microsoft’s early 1990s introduction of Visual Basic for desktop applications.  While it lessens the need for traditional database roles on a development project, especially the DBA, the benefits achieved through rapid iterative development go a long way in providing customer solutions in a quick and cost-effective manner.  And if happy programmers are an extra result of using Rails, where’s the harm in that?

Anyone interested in learning more about Rails should check out the previously mentioned online community at RubyOnRails.org.   The book Agile Web Development with Rails is also highly recommended; it was written by members of the core Rails development team.

As we entered 2011, organizations grappled with how they could manage and benefit from the large amounts of data created daily, exceeding 2.5 quintillion bytes. This data came from internal and external sources such as sensors, social media, digital images, videos, transaction records, and cell phone GPS.

To tackle this challenge, CIOs and CTOs partnered with IT vendors to extend their information platforms and build Big Data analytics solutions. Big Data solutions were infused into new and existing IT infrastructures with services that help organizations assess their cloud computing and data center needs, as well as server, storage and security requirements. Additionally, Big Data technology allowed enterprises to analyze any up to tens of petabytes of data without altering its native format.

Big Data solutions were deployed across a wide set of industries this year, enabling new actionable insights which resulted in:

• Better profiling of customer preferences and habits;
• Focused loyalty programs based on deriving customer insights;
• Leveraged buyer intent to give customers more tailored products and services based on their wants and needs;
• Improved operational efficiency and utilization of infrastructure; and
• Better risk management and predictive abilities.

In 2012, I predict Big Data solutions will expand beyond analytics and into transactional usage. Organizations will institute a comprehensive data discovery practice to help enterprises embrace Big Data to derive such insights. Over the next year, it will be critical for organizations to craft their Big Data solutions to not only manage and analyze the given information, but to also transform it into actionable goals that will benefit their businesses.

Other trends I will follow in 2012 include:

• Enterprises will facilitate multiple types of discovery that will feed into transactional and analytic applications, enabling real-time business intelligence and action;
• Customer adoption of Big Data technologies will increase to tackle the challenges associated with analyzing the large volumes of data;
• The need to track and verify the veracity of the derived information and understand the lineage will be critical; and
• Interpreting consumer sentiment with a high degree of accuracy will require understanding sarcasm, positive and negative expressions.

Finally, the focus of data will also shift from simply gathering the data available, to taking the insights learned by the data and using it to better understand the customer. Organizations will be able to provide their customers with exactly what they want and need, and increase their customer loyalty. The combination of advanced analytics and associated governance of Big Data will allow businesses to make well-informed decisions for corporate and competitive advantage.

About the Presentation

Big Data problems are quickly presenting themselves in almost every area of computing from Social Network Analysis to File Processing. Many technologies, such as those in the NoSQL space were developed in response to the limitations of current storage systems as an effective mechanism to deal with these mountains of data. And much of that data is interconnected in ways that, when organized properly, gives interesting and often valuable information. InfiniteGraph, the distributed and scalable graph database, was designed specifically to traverse connections and provide the framework for a new set of products built to provide real-time business decision support and relationship analytics. This presentation examines the technology behind InfiniteGraph and explores a couple of common use cases involving very large scale graph processing.

About the Speaker

Darren Wood is the Architect and Lead Developer of InfiniteGraph, a distributed graph-oriented data management solution from Objectivity Inc. Darren has spent the majority of his career architecting and building distributed systems with an emphasis on elastic scalability and data management. Prior to joining Objectivity in 2007, Darren held positions as a Senior Consultant with IONA Technologies and a Development Team Lead for Citect Australia. Darren holds a First Class Honors Degree in Computer Systems Engineering from the University of Technology in Sydney, Australia.

Part 1 of “Assessing Data Management Maturity”discussed the primary elements of the Data Management Maturity Assessment (DMMA) conducted by April Reeve, an advisory consultant at EMC Consulting, for a mortgage bank. The primary elements of this maturity assessment were:

• The Capability Maturity Model (CMM) developed by Carnegie Mellon University as the inspiration
• The DAMA Data Management Book of Knowledge (DMBOK) as the guide used for the creation of the Key Practice Areas (KPA) that would be assessed, especially the best practice activities and technologies
• The incorporation of ISACA best practice activities for the Data Security Management KPA and TOGAF Version 9.0 best practice activities for the Data Architecture Management KPA.
• The addition of a Data Integration Management (DIM) as a tenth KPA since the DMBOK has only nine and Data Integration is not included in it as a separate area but incorporated in the other nine.

The beginning of the DMMA process includes determining which business lines to assess. The mortgage bank decided to include their primary and reverse mortgage businesses, retail banking business and cross-business functions as the business lines o assess. A key part of the maturity assessment process is determining the people to be interviewed.  This assessment was done with a very large number, but usually a much smaller number of people are included.  It is important that those interviewed include people from both the business and technology areas involved with each business lines to be assesses as well as from both senior level and operational level people involved with the day to day processing of the organization data.  Once these decisions were completed then a Current State Analysis was conducted, which included the completion of a series of interviews with around 100 employees in various departments of the bank, about 15 interviews with people who were judged to have the best perspective on current data management maturity in the organization, and then interviews of the interviewers by Ms. Reeve. The results of the interviews were collated and an assessment was created based on the 5 levels of the CMM framework:

• Immature (Initial)
• Repeatable (Repeatable)
• Managed (Defined)
• Monitored (Managed)
• Continuous Improvement (Optimizing)

Part 2 of this article will discuss the results of the maturity assessment, how the bank interpreted it, the bank’s primary decisions for the Future State Analysis roadmap and how they planned to move forward with the given improvements.

The Assessment Results – Calculating and Presenting

Once the CSA was completed all the results had to be collated, with comments included to add more meaning to the numbers generated from within the CMM framework. In the case of the mortgage bank, everyone interviewed were weighted equally – but this may not be the case for other organizations. It is possible to add different weighting calculations to certain individuals or departments depending on which KPAs are focused on. All the activities, tools and lines of business were also weighted equally, thus no mathematical skewing of the results was employed. The calculated values, with rating from 1 to 5 as per the CMM levels, are important, but the comments are equally important as they point out specific gaps and opportunities with each of the KPAs. Graphic Three shows the final results of the Current State Analysis for the mortgage bank: the left column lists each area( KPA) and the top row shows the four main business lines assessed.

Graphic Three

The results are color coded (with the order of success from least to most being red, orange, yellow and green) and given a numerical value (with a higher number being a better result). The areas that received the highest scores for the mortgage bank were Data Security Management (DSM) and Document and Content Management (DCM).  However, the fact that the Data Security Management values are primarily green is misleading since data security is critical to a bank, thus an average of 2.58 was deemed not acceptable; they wanted a minimum score in Data Security of 4 or a Monitored Level. Their DCM score was averaged at 2.82 though they also wanted to improve their Document Management practices to at least a 3 in that category, with a 4 being their goal. Their overall goal was to have all 3’s, but with no single average at 3 for the entire maturity assessment, the mortgage bank had to focus on their most pressing needs.

The question then remains: “How does an organization improve their scores to acceptable or goal levels?” The third stage of the DMMA process is the Future State Analysis (FSA) which includes an assessment of the CSA, where the organization wants to go concerning the various KPAs and priority action steps to reach their goals.

Understanding the Assessment Results

The numerical values of Graphic Three don’t provide much actionable information. The mortgage bank had a CSA score of 2.58 in DSM, but wanted a 4, so there is a gap of 1.42. But what did that mean? The mortgage bank had a number of areas in data security that needed improvement including, but not limited to:

• Improved user access management and tracking, and auditing of user activities were required.  This included private client information.
• Additional data security controls were required for the use of production data within application development environments.
• Data security management activities needed to be enhanced to CMM level 4 to meet the requirements of the bank’s executives.
• MS Access environment (primarily supporting Data Consumers) needed to be upgraded to a technology which would adequately support Data Security Management controls or managed with compensating controls that would meet the bank’s requirements:
• The bank was not adequately tracking who had access to what data, who extracted the data from the MS access databases and what they did with that data.

The FSA included a list of benefits for each of the KPAs so that the stakeholders involved in the DMMA had a better understanding of “why” they should make such changes. The possible benefits of the bank following the recommendations in the Data Security Management KPA included:

Ÿ  Reduced risk of reputational, customer relationship and financial damage due to security breaches.

Ÿ  Enhancement in the organization’s ability to comply with regulatory requirements for security and privacy.

Ÿ  Enhancement of the organization’s ability to control track and audit Data access.

The completed FSA for the mortgage bank included a comprehensive assessment for all 10 of the KPAs and detailed the findings, opportunities/recommendations, risks and benefits for each of them. This information provided them with a roadmap they could follow into the future and had prioritized items for each KPA, though some large questions remained – notably concerning costs.

Using the Assessment Results to plan improvement

Any organization that undertakes a DMMA needs to be aware that once the results are finalized, a specific roadmap must be developed and followed or the entire process will likely fail. The mortgage bank could not implement every change identified in the maturity assessment: The improvements in data security management alone would take considerable time and an investment in training, upgraded software, better hardware and a range of new processes throughout the organization. The goal of having every KPA at level 3 was unrealistic in the immediate future, especially when none of their current numerical values even reached 3. Thus, they had to weigh the relative importance of each KPA and move forward on those most important.

A Data Integration Management score of 1.7 where they wanted a 3, meant having to develop an enterprise Data Integration Strategy and Architecture, select a standard tool set, then install, train and integrate that tool set throughout the organization, while establishing a plan to streamline the integration environment over time through phased project execution. They decided that improvement in that particular KPA needed to wait – the roadmap centered primarily on DSM, DCM and a few other higher priority levels. All organizations who undertake such an assessment must come to an agreement on their priorities.

Most organizations complete a three year roadmap, but even with such a map stabilized and under implementation, it is also necessary for a yearly reassessment to make sure the roadmap is still being followed, how well it is working and what changes need to occur. Creating a roadmap from the entire DMMA process and then not following it is not only a waste of time and resources, but also costly. Graphic Four shows the basic roadmap that was created after the maturity assessment (a fully functional one is much more detailed) for the mortgage bank in 2011. It illustrates the enabling activities the bank needed to employ to move the roadmap forward, the immediate priorities and gave a colored graphical illustration of what areas were being worked on with a timeline for the completion of certain projects.

Graphic Four

The DAMA DMBOK was not written for the purpose of implementing it into a capability maturity framework for the assessment of an organization’s data management abilities, but since it lists best practices for the entire field of Data Management, it lends itself well as a framework for such an assessment. Any organization that desires an assessment is advised to work with a consulting firm with expertise in performing data management maturity assessments, since the experience and expertise of the consulting firm will probably make the efficiency of performing the assessment process and documenting the results and next steps more cost effective to the organization than trying to perform it internally. The creation, implementation, assessment and collation of the results are something any organization – no matter its size – could use, but how to complete a successful DMMA requires an intimate understanding of all the steps throughout the process.

Larry Hardesty reports, “There are a host of mathematical tools for finding possible relationships among data, but most of them require some prior knowledge about what those relationships might be. The problem becomes much harder if you start with a blank slate, and harder still if the datasets are large. But that’s exactly the problem that researchers at MIT, Harvard University and the Broad Institute tackle in a paper appearing this week in the journal Science.”

He continues, “David Reshef, a joint MD-PhD student in the Harvard-MIT Division of Health Sciences and Technology (HST) — who, along with his brother, Yakir, is lead author on the new paper — says his team’s approach to data mining tries to maximize two properties that are often in conflict; he calls these generality and equitability. The graph of the relationship between two variables in a dataset could take any shape: For a company’s hourly employees, the graph of hours worked to wages would approximate a straight line. A graph of flu incidence versus time, however, might undulate up and down, representing familiar seasonal outbreaks, whereas adoption of a new technology versus time might follow a convex curve, starting off slowly and ramping up as the technology proves itself. An algorithm for mining large datasets needs to be able to recognize any such relationship; that’s what Reshef means by generality.”

Hardesty goes on, “Equitability is a little more subtle. If you actually tried to graph workers’ hours against wages, you probably wouldn’t get a perfectly straight line. There might be some overtime hours at higher wages that throw things off slightly, or Christmas bonuses, or reimbursement for expenses. In engineers’ parlance, there could be noise in the signal. Most data-mining algorithms score possible relationships between variables according to their noisiness; the noisier the relationship, the less likely that it represents a real-world dependency. But linear relationships, undulating relationships or curved relationships with the same amount of noise should all score equally well. That’s equitability.”

Read more here.

photo credit: MIT

New survey findings out of KPMG show that “Corporate executives have high expectations for the next generation of IT systems, seeking advanced analytical capabilities that can cut through the complexity of company and market data to identify the most important information needed to make better competitive decisions.” Stephen G. Hasty of KPMG commented, “The priority companies are now putting on information intelligence came through loud and clear in our survey, which found that data needs are driving company investment decisions regarding the upgrade and advancement of ERP and IT architectures… In the past, the focus was usually on connecting processes or transactions from siloed or disparate systems, often creating a hodge-podge of unfocused information.”

The article states, “The KPMG survey, taken during the 2011 Oracle Open World conference in San Francisco in October, found that 41 percent of the 336 respondents said their organizations planned an ERP upgrade, with 26 percent planning implementation in the next 12 months. Nearly half (49 percent) of the survey respondents acknowledged their organization was overwhelmed with data, making decision-making difficult. They also said consideration of a cloud strategy will present future data-management challenges (59 percent), although 56 percent of the respondents acknowledged that cloud can help them become more agile and competitive.”

Read more here.

photo credit: KPMG

Hovhannes Avoyan has provided an overview of the popular NoSQL solution, Cassandra. Avoyan writes, “Cassandra is a complex, highly scalable NoSQL database system that was initially developed by Facebook and is now under the purview of the Apache Foundation.  Apache Casandra has risen as a welcome alternative to HBase, another highly scalable, and available key value store. Cassandra is different from most Relational Databases in that it stores data by column instead of by row.  This makes aggregating column values super fast.  It also makes dealing with the database somewhat different as well.  Instead of creating tables, you create column families.”

He continues, “Cassandra is able to scale read/write throughput linearly with machine count based on its unique architecture.  Each Cassandra node does exactly the same thing.  Cassandra nodes together are called rings.   To make all this scaling happen seamlessly, Cassandra nodes employ a gossip protocol which enables the nodes to talk to each other and figure out where to send reads and writes in the ring. Cassandra like many highly scalable database systems uses data partitioning to achieve some of its incredible speed characteristics.  Cassandra’s default behavior is to partition your data randomly across your ring.  Incredibly, even with your data partitioned, you can still add and remove new nodes from your Cassandra ring without compromising availability!”

Read more here.

photo credit: Apache

Johnson & Johnson is looking for an IT Manager of Business Intelligence/Data Warehousing in Jacksonville, FL; Raritan, NJ; or Horsham, PA. According to the post, this position “is responsible for the overall technical delivery of all initiatives, project or operational, related to SAP BW as well as other assigned areas of accountability within the BI/DW space.  The I/T Manager BI/DW will actively develop their teams to deliver high quality solutions, on time and on budget; will ensure appropriate priorities are set within their organizations, and will ensure that services are delivered per agreed SOW’s with the Customer.  The I/T Manager BI/DW will collaborate with GS, ITS, and BU/IT departments, as well as functional management, to develop and deliver the overall GS strategy.  They will be champions of good systems development and management processes as well as actively promote awareness and alignment with all J&J compliance requirements.”

Qualifications include: “A minimum of a B.A./B.S. degree in Information Technology, Computer Science or technical field is required.  A minimum of 8 years overall I/T experience with SAP BW or other BI/DW experience is required.  Strong BI project delivery experience – managing global, cross-functional, multi-disciplined projects and service delivery is required.  Ability to drive resolution for critical issues by bringing teams together – both J&J internal and external vendors is required.  A global mindset – ability to work across multiple countries and cultures is required.  Strength in interdependent partnering and collaboration skills to build strong relationships with internal and external customers, suppliers and business partners is required.”

Learn more and apply here.

photo credit: Johnson & Johnson

Brightlight Consulting is looking for a Hadoop Solution Architect in Redmond, WA. This position “will be responsible for implementation of complete Hadoop solutions, including data acquisition, storage, transformation, and analysis and a solid understanding of infrastructure planning, scaling, and administration considerations that are unique to Hadoop processing for Business Intelligence. We are looking for candidates who have a broad set of technology skills across these areas and who can demonstrate an ability to apply Hadoop solutions to big data problems and learn quickly as these solutions evolve and client demands grow.”

Qualifications include: “Minimum of 1 year of hand-on experience programming on a high-scale or distributed system. Minimum of 4 years systems development and implementation experience. Hands-on experience with the Hadoop stack (e.g. MapReduce, Sqoop, Pig, Hive, Hbase, Flume). Hands-on experience with related/complementary open source software platforms and languages (e.g. Java, Linux, Apache, Perl/Python/PHP, Chef). Hands-on experience with BI tools and reporting software (e.g. MicroStrategy, Cognos). Hands-on experience with analytical tools, languages, or libraries (e.g. SAS, SPSS, R, Mahout). Hands-on experience with ‘productionalizing’ Hadoop applications (e.g. administration, configuration management, monitoring, debugging, and performance tuning). Knowledge of NoSQL platforms (e.g. key-value stores, graph databases, RDF triple stores)”

Learn more and apply here.

photo credit: Brightlight Consulting

Do enterprises treat data as a strategic asset? Are organizations making the right investments with respect to defining data quality requirements, proactively monitoring the quality of business critical data and managing its quality throughout the corporate information supply chain? Such questions must be asked by modern enterprises; they run on data, data is their lifeblood, data is perhaps the most important factor in corporate decision-making and financial reporting. Disclosures depend on data and risk is constantly monitored and managed using data. Therefore, it is clear that data is a crucial strategic asset and must be treated as such.

The Central Challenge

The use of passive tests that require the examination of documentary evidence related to internal controls and risk management are not the only on-going tests within the modern enterprise. Regulators and internal audit teams are also mandating more active tests, focused on transparency and accountability. These tests are integrated into a multitude of business processes and data management procedures, with metrics that address the information supply chain from all ends of the spectrum. New legislation and oversight bodies have been instituted to monitor and address systemic issues related to data quality, particularly with multinationals and government regulated firms in the Banking and Financial Services, Health Care, Pharmaceuticals, Consumer Packaged Goods, Retail, Government, Transportation, Real Estate, and Electronics industries to name just a few.

To ensure that consumers of corporate data have the highest level of confidence in such data, proper data quality business practices are essential for prospering in the 21^st century. Enterprises need to consider the significant human and capital resources they expend in reactively addressing data quality-related issues (usually in silos), such issues include: redundant checks built into the enterprise’s systems and processes, the lack of visibility into the quality of data as it flows through various systems, the significant legal and reputational risk exposure caused by poor quality data, the impact of low quality data on decision-making and the impact of poor quality data on time-to-value. While these are only a short list of the many issues that organizations must address with their data quality, they highlight the central problem – poor data quality is a systemic issue that affects all levels of an organization and must be dealt with as such.

The Solution

Data quality has emerged as a crucial field within data management and data governance, especially due to a number of high-profile incidents resulting from poor data quality management. The focus must now be on addressing those systemic issues discussed above, which typically result in exposing companies to major risks. This requires a shift from reactive to proactive approaches regarding data quality management and the implementation of the Holistic Data Quality (HDQ) framework.

HDQ is a term that I coined to highlight a paradigm shift in data quality management. Quality should not be evaluated or managed in vertical business silos, but in a holistic integrated (cross-silo) approach, based on the HDQ framework. Such an approach incorporates consistent quality measures, exception-based reporting and robust analytics. Implementing HDQ at the enterprise level results in higher quality data, lowers costs for the remediation of quality issues, provides transparency into hot spots and outliers, and significantly reduces the costs and resources expended on supporting internal and external audits. There are significant regulatory compliance benefits as well.

Given the data deluge of the past few years and the increasing complexity of the data landscape (e.g. Cloud and ASP deployments), there are no quick fixes. However, taking a strategic approach and systematically implementing HDQ and associated shared service capabilities will enable firms to overcome these challenges.

The foundational components of a HDQ solution are the following:

• Data quality dimensional framework
• Solution Architecture
• Data quality software
• Business Process Management (BPM) and Web Services orchestration capability
• Metadata repository
• Enterprise data quality mart
• Business Intelligence (BI) software
• Services Oriented Architecture (SOA) platform
• Issues Management System

The dimensional framework enables consistent data quality requirements and metric definition. The solution architecture is driven by enterprise data quality business use cases and data processing patterns. The data quality software and BPM capabilities are utilized to implement business solutions and data governance processes. The Metadata repository hosts data dictionaries, business glossaries, data lineage, data interface definitions and other metadata that is necessary for managing data. The data quality mart and BI platforms capture and provide enterprise reporting and intelligence to show quality hot spots, data patterns, data anomalies, outliers and the overall health of business critical data. The SOA platform provides the ability to vend enterprise class data quality services. The issues management system is a central repository of data issues and typically has workflow capability to route and manage issues, along with an enterprise level reporting component.

Implementing an enterprise level HDQ strategy requires strong program management, systems integration, services oriented architecture, data warehousing and business intelligence expertise.

Proven Results

In the past, I have developed and successfully deployed the HDQ framework, associated methodologies, best practices, design patterns and disruptive tools to address data and information quality challenges.  This is a “must have” component of every firm’s Enterprise Data Management program. One must embark on it with a clear vision, sponsorship at the Board and C-Level, a commitment for long-term financial and political support from senior management and an appetite for change.