Paul Miller recently responded to an assertion that Big Data tools trump the knowledge and experience of Data Scientists. He writes, "Data scientists are an increasingly capable bunch, and the tools at their disposal sometimes appear almost magical in their capability to derive insight. Competitions such as those run by Kaggle (more on them in a moment) clearly show that an aptitude for numbers and analysis can deliver some remarkable results, even when that analysis is being undertaken by individuals who lack specific domain expertise. But to suggest that simply 'letting the numbers speak for themselves' is an effective way to make real decisions is, quite simply, bonkers."
Miller continues, "Data is merely one input to an effective decision making process. Prior knowledge, policy considerations, and an awareness of experimental bias, sampling error, and quaint notions such as ground truth continue to play a fundamental part. Data scientists undeniably bring a wealth of skills to the table, but so do domain specialists. The domain specialists would be unwise to presume that they can continue to keep pace with exploding data volumes without judicious application of data science. But for data scientists to presume, even for a moment, that they and their algorithms can replace domain expertise is laughable."
Image: Courtesy Flickr/ NASA Goddard Space Flight Center