Loading...
You are here:  Home  >  Data Education  >  Big Data News, Articles, & Education  >  Big Data News  >  Current Article

The Law of Diminishing Returns: How Much Data is Too Much?

By   /  December 1, 2015  /  No Comments

excessby Angela Guess

William Terdoslavich recently wrote in InformationWeek, “At the heart of big data is the search for “insight” — some correlation or finding that eludes the seeker until he or she adds another terabyte or 10 of data, just in case it is lurking there. At a certain point, the law of diminishing returns has to kick in. Adding another 100TB becomes redundant. Vendors exercise their right to remain silent when asked, ‘How much data is too much?’ Big data skeptics don’t have a precise answer, either. But they are more likely to speak of the limitations of Big data than to shout out its promise. They act as the agnostics questioning the IT theology of the Hadoop evangelists.”

He continues, “For Cathy O’Neil, who holds a doctorate in mathematics from Harvard University and has worked in academia and the private sector, the issue is less about the law of diminishing returns and more about people not understanding the data. The technology is ‘encouraging people to use algorithms they don’t understand,’ O’Neil said in an interview. ‘You don’t need a lot of data [to not] know what you are doing.’ O’Neil’s skepticism is well grounded in her experiences, since her career wound its way through academia, to Wall Street, and then to the New York City startup scene. She has seen the plain gap between the technologists who craft the algorithms and the business people who rely on them. Data is just a way of codifying information, O’Neil explained. Any data gathered should be relevant to a problem, otherwise useless data clouds the results of a query.”

Read more here.

photo credit: Flickr/ slsch1

You might also like...

Benchmarking the Full AI Hardware/Software Stack

Read More →