Big Data, Data, And Governance

by Ian Rowlands

I’m wrestling with what started out as a simple question for me, but which is becoming more complex daily as I talk to people about it. As the Big Data phenomenon has more and more of an influence on the shape and direction of the way Information Technology supports business, what does that mean for Data Governance?

I thought I had this clear in my mind, and was getting some kind of consensus from people I was talking to. Recently, however, two themes have started to emerge that at least are making me think a bit more about it. I thought I’d share the issues and see what the Dataversity community might have to say about them.

Where I started was with the notion that once the dust has settled Data Governance would still be a single discipline, with the fundamentals not much disturbed by the arrival of the fascinating new classes of data parked under the “Big Data” umbrella. (Actually, “single discipline” might better be expressed as “meta discipline” incorporating data stewardship, issue management, lifecycle management, privacy and security management, metadata management, master data management, data quality management, data integration and business process management.)

The first theme to disrupt this comfortable view is the discovery that a lot of the Big Data specialists I’ve been talking to couldn’t care less about Data Governance. The argument seems to run along the lines that “we trust the algorithms that we run the data through, and so we trust the conclusions – so who needs Governance?” The counter proposition, of course, is that knowing where the data comes from and how authentic it is, how it’s going to be used and the decisions it drives, and what data was used to drive decision, will be critical. But what do you think? Does Big Data not need Governance? Will “data” increasingly be processed with “Big Data” technologies, and will the convergence of these two issues eliminate Data Governance altogether?

The second theme is about the recognition that “Big Data” is not just bigger, but qualitatively different from “data”. Big data is stored in its primal state, uncleansed and untransformed. The notion of “data quality” meaning “data accuracy” shifts more explicitly to a notion of “data fitness for purpose”. That means that data quality is much more multi-dimensional than it used to be. (Actually a better analogy might be between scalar and vector quantities). Perhaps that has a beneficial impact on Data Governance, pushing it towards being more relevant to business users, and less a technical ghetto?

To make it clear, I don’t buy the death of Governance, and I think implementing Big Data solutions without Governance is likely to lead to trouble … And none of this changes (admittedly self-interested) perspective that says Data Governance is impossible without metadata management. But what do you think?

Related Posts Plugin for WordPress, Blogger...

Ian Rowlands

Ian Rowlands is ASG’s Vice President of Product Management. He heads product management for Metadata and Application Management and is also tasked with providing input across ASG’s entire portfolio. Ian has also served as Vice President of ASG’s repository development organization. Prior to joining ASG, Ian served as Director of Indirect Channels for Viasoft, a leading Enterprise Application Management vendor that was later acquired by ASG where he was responsible for relationships with Viasoft’s distributor partners outside North America. He has worked extensively in metadata management and IT systems and financial management, and presented at conferences world-wide, including DAMA and CMG. 

  1 comment for “Big Data, Data, And Governance

  1. August 6, 2014 at 2:16 am

    Hi Ian.

    Thanks for your post

    I agree with the view that data governance and successful big data and have written a number of blog posts on this subject. In http://blog.masterdata.co.za/2013/09/16/big-data-quality-matters/ I compared the results of surveys in the US and Europe that showed a link between the two.

    In South Africa we are in a very early stage of adoption of big data analytics – companies have been holding off trying to understand the impact and to see where they will find value, and to understand how they can use big data without contravening privacy laws etc.

    It looks like we will go straight from “no big data” to “formal big data” without going through the “big data is an experiment and anything goes” phase. Interestingly I have had clients asking how they can include big data lineage and other metadata into their governance frameworks – there is definitely some awareness that this is important for big data to succeed.

Leave a Reply

Your email address will not be published. Required fields are marked *