by Angela Guess
In a recent article Mike Lynch discusses the en vogue terms in Silicon Valley and what they really mean: “To be cool at the Valley party these days, you have to talk about big data. The problem is, no one can quite tell you what exactly that is. It’s not data warehousing or Business Intelligence — because that is so boring. In-memory databases come up a lot and are often ascribed magical properties, however all I can seem to learn about this is they are faster due to the avoidance of going to disk. One can get the same answers for unexpected unoptimised queries as one could for boring databases, but quicker. So just as dumb, but quick dumb. But they are usually mentioned by software marketing people with a twinkle in the eye and an implication that somehow you will also get better answers, but no one can explain why.”
Lynch continues, “So, to be cool and feature in the next Silicon Valley movie, try knocking ’em dead with the following big data gambits: NoSQL and Hadoop. For those of you that wish to casually toss this word in, Wikipedia will tell you Hadoop is a data-intensive framework that works with thousands of nodes and petabytes. It was originally developed to support distribution for Nutch. If you are no wiser at that point, then bluff. While Hadoop and NoSQL have perfectly respectable academic, dev and nerd communities behind them, you don’t have to look far to see what happens when marketing departments get their hands on the big data idea.”
He goes on, “We’re in for a few months of chaos and crazy statements before the terminology is cleaned up. We have already seen one large vendor of in-memory databases launch with a set of crazy claims, only to back-pedal as they realised marketing had not only confused the customers, but also themselves. Big data is a generic term and will no doubt have benefits, but it needs clarity of definitions and claims before it can truly assert itself as a revolution rather than an evolution of some rather tired ideas. My mind is open and I await that clarity when the hype dies down.”
His opinions are quite interesting given Autonomy’s new Aurasma product is based on a no-sql solution in couchdb. Reliably informed they use this for storage of all document data over their “intelligent” IDOL platform. Does Mr lynch consider these to be competing products? or is IDOL just really a search engine and not an enterprise document storage platform?