by Ian Rowlands
I'm returning to a theme that was one of the motivations for starting blogging with Dataversity. How come so many metadata programs get off the ground, do well for a while and then wither on the vine?
I've been focused recently on the issues surrounding what I think of as projects on the "bleeding edge". (Warning: shameless self-promotion. I'm speaking on the topic at EDW, so I've been thinking about the projects I and my colleagues have been involved in recently.) What came to the surface surprised me, and drove me back to the blogosphere.
What I was expecting about the challenges around “Big Metadata” projects was that there would be characteristically special technical issues. To be sure there were a few, but they weren't really pervasive -- but there was something that came up consistently.
I suppose I should say what I mean by "Big Metadata” projects. I haven't really formalized it, but you can look at it by taking the "Big Data" characteristics -- volume, variety and velocity, and applying them to the world of metadata. Of course one part of the model breaks down. A key characteristic of "Big Data" is that the v/v/v challenges force an adoption of distinctive -- "Not Only SQL" -- technologies, with Hadoop to the fore. The metadata world has no de-facto standard technology, and so I suppose it's not startling that no "new generation" metadata technology is emerging as preeminent. Some techniques are bubbling up -- especially federation -- but as I hinted, technology isn't really the marker.
So what is really making the "bleeding edge" a tough place to be? The answer is, in retrospect, so painfully predictable that I've been kicking myself for not predicting it! It's the amplifying effect that "Bigness" has on discipline and cultural issues.
Now don't mistake me. Of course there are challenges when the amount of metadata expands beyond the window open to collect it, the variety increases the complexity of lineage projects beyond the scope of conventional graphics and the number of users grows too fast to be able to handle all their requests for help -- but all of those things point to the real issue. When the challenges grow, it's time to shift from project mode to program mode. It's a whole new level of management challenge.
The other thing that seems to have been common to most of the recent "Big Metadata" projects is that they've involved the bridge from departmental to corporate implementation, and it's the politics that start the blood flowing!
So there you have it. Sure "Big Metadata" projects have special technical challenges – but if our experience is typical, it's the lack of discipline and the cross-discipline cultural confusion that will give you the bloody nose!