Conversations about Big Data – Why Do We Need Big Data?

by David Schlesinger CISSP

With the arrival of the Internet everything changed.  Not just the introduction of email, SPAM and selling ice-cream makers via websites; not even eBay™ and massive multi-player games; no, the thing that changed was that all along the broadband connection there sits a word full of people.

Once upon a time all we had to worry about was an infinite number of monkeys typing Shakespeare on an infinite number of typewriters.  Now we have teenagers in Tampa and geeks in Greece who are sending us data to somehow manage.

And, if it were only a few teens and geeks we could handle it; but a typical web-fronted enterprise counts its customers in the millions,, and the Marketing Dept. will segment them into almost uncountable configurations.  Well, they would be uncountable but banks of servers now capture and count every bit and byte then store it on an ever-expanding array of spinning disk drives.  Does this sound like your place?

Traditional database management systems were designed to handle data generated by business processes in place during the time they were invented.  That certainly makes sense, but just as dirt roads evolved into super highways when the car replaced the horse, the Internet replaced folks standing behind cash registers making sales one at a time with sleepless web servers capable of thousands of transactions a second.  Sales data now exceeds many legacy data systems’ ability to calculate marketing analysis before conditions change and make the marketing responses redundant.

To solve this some very bright people arrived with some sparkling logic and new data handling ideas to cope with this huge data flow.  They believed that by leveraging the low cost of commodity computers and then playing fast and loose with rigidly formatted data, they could improve data processing throughput two orders of magnitude. (That’s 100X for you non-Star-Trek-watchers.)

Yes, these are the same computers that you use to read email at home and play Solitaire during meetings at work.  Turns out they move fast when stacked real high using an architecture that asks many small computers to do lots of short and simple jobs. (Just like my grandfather used to say, “Many CPUs make light work.”)

Breaking up may be hard to do, but that’s precisely what Big Data systems do to your data. Each processor chips in a few cycles and information flows quickly.  But, there’s a catch.

Getting answers quickly is more than just about speed; just like a banana-split is more than just a split banana.  Big Data is about speed utilized to make prompt business decisions. You need to first reflect on your business process and determine what decisions are pivotal for success and which data you require to improve them. Getting responses from your sales efforts in near-real-time and being able to tie your inventory to manufacturing without faulty long-term estimates gives your business a competitive advantage.  .

What you still need, after that technology expansion into Big Data, is a sharp mind, an innovative business eye, and working to provide your customers with the best experience you can provide.  That has not changed because people have not changed. We are still all people, even if we’re now sitting, cybernetically, in a cloud.

 

Related Posts Plugin for WordPress, Blogger...

David Schlesinger

David Schlesinger has 25 years of experience in information technology and data security management. His book, The Hidden Corporation, is a “business novel” that describes data protection issues in a large corporation and follows a dedicated employee as she uncovers the root causes of improper data exposure and discovers solutions. David is CISSP certified in cybersecurity and is on the Board of Directors of the Phoenix ISSA, a security professional association. He has authored two US Patents for data governance methods that use Metadata classifications to audit and automate user rights and regulatory compliance, and speaks widely at data management and security conferences. David is a Senior Security Architect and currently consults with commercial and government organizations on information protection involving enhanced Metadata, self-aware data architecture, data classification practices, and information regulatory compliance. 

Leave a Reply

Your email address will not be published. Required fields are marked *

Add video comment