The Law of Diminishing Returns: How Much Data is Too Much?

By on

excessby Angela Guess

William Terdoslavich recently wrote in InformationWeek, “At the heart of big data is the search for “insight” — some correlation or finding that eludes the seeker until he or she adds another terabyte or 10 of data, just in case it is lurking there. At a certain point, the law of diminishing returns has to kick in. Adding another 100TB becomes redundant. Vendors exercise their right to remain silent when asked, ‘How much data is too much?’ Big data skeptics don’t have a precise answer, either. But they are more likely to speak of the limitations of Big data than to shout out its promise. They act as the agnostics questioning the IT theology of the Hadoop evangelists.”

He continues, “For Cathy O’Neil, who holds a doctorate in mathematics from Harvard University and has worked in academia and the private sector, the issue is less about the law of diminishing returns and more about people not understanding the data. The technology is ‘encouraging people to use algorithms they don’t understand,’ O’Neil said in an interview. ‘You don’t need a lot of data [to not] know what you are doing.’ O’Neil’s skepticism is well grounded in her experiences, since her career wound its way through academia, to Wall Street, and then to the New York City startup scene. She has seen the plain gap between the technologists who craft the algorithms and the business people who rely on them. Data is just a way of codifying information, O’Neil explained. Any data gathered should be relevant to a problem, otherwise useless data clouds the results of a query.”

Read more here.

photo credit: Flickr/ slsch1

We use technologies such as cookies to understand how you use our site and to provide a better user experience. This includes personalizing content, using analytics and improving site operations. We may share your information about your use of our site with third parties in accordance with our Privacy Policy. You can change your cookie settings as described here at any time, but parts of our site may not function correctly without them. By continuing to use our site, you agree that we can save cookies on your device, unless you have disabled cookies.
I Accept