The Future of Big Data: Smart Data Innovations and Challenges

By on

kf_futsd_102516Big Data with Advanced Analytics is now being used extensively to find patterns and trends, and to predict the results of certain activities, such as movements in the stock market through advanced processing techniques packaged together into the term Smart Data – or data with intelligence.

Data Analytics, along with Data Science techniques, have become quite popular as an approach to processing Big Data. Advanced Data Analytics uses cost-effective, tried-and-tested methods, while Data Science is often more experimental, and focused on developing and exploring new algorithms (and paradigms) in the search for trending information.

Data Analytics takes an eclectic approach, adopting and absorbing the latest Data Science innovations to turn Big Data into Smart Data. These tools, once accessible only to the experts creating them, are now available for use by non-tech businesses and other organizations. However, there are limits and restrictions, because the algorithms have not been tailored to the specific needs of the organization, and may not be a good fit.

The Evolution of Services

Prediction: Smaller organizations will hire Data Analytics service contractors to organize and process their Big Data. These contractors will have a wide variety of resources and algorithms available. Large corporations will continue to use Data Science experiments in an effort to maintain a leading edge and bring multiple approaches together to create Smart Data products.

 While new software and hardware is created on a weekly basis, many companies continue to use systems they have used for years. Their hesitation is based on costs, and the possibility things can go horribly wrong. Some established companies have offered Data Analytics-as-a-Service, in part to maximize profits on Data Science programs they had developed for themselves. Additionally, startups with a focus on offering a broad range of Data Analytics services are becoming a reality.

Blazent, a startup based Michigan, uses a variety of Big Data technologies like Machine Learning Library, Cassandra, Spark, and Hadoop.  They also have the capacity to store and retain the history of data sets, and provide near real-time analytics. Their platform integrates and reconciles data from over 230 different sources in an effort to provide the most accurate Data Intelligence possible. Blazent has also developed a reputation for helping their clients with the accuracy and consistency of their data.

Algorithms- The Illusion of Objectivity

 Prediction: Laws and ethics will become a standard part of Big Data algorithms.

It is generally believed algorithms are objective, but in fact, they are designed by human beings.

While some choices may have been made with good intentions, algorithms include human prejudices and misunderstandings. These biases are being built into automatic systems which manage our lives, and the decisions are often beyond dispute, because the “computer” made the decision.

Algorithms may include human subjectivity and questionable paradigms. As a consequence, a number of race and class biases have surfaced in the use of Big Data. For example, the Hudson City Savings Bank was fined $33 million after using data about zip codes to redline potential borrowers. In the United States and the United Kingdom, banks were penalized for data-based decisions discriminating against insurance or credit applicants by race. Amazon has recently been criticized for using algorithms limiting its Prime delivery service to poorer neighborhoods (they claim to be correcting the situation).

Cathy O’Neil, author of Weapons of Math Destruction, a book tracking the effects of data discrimination stated, “The discriminatory and even predatory way in which algorithms are being used in everything from our school system to the criminal justice system is really a silent financial crisis.” Algorithms are now being used in politics, to decide how much an individual pays for loans or insurance, and to try controlling smog in China (best of luck with that one).

The Big Data “Churn” model (Churn rate measures customer or employee attrition), which is popular with HR departments, presents another path for politically incorrect behavior. Xerox has used this model, and examined job applicants who lived farther from the job, finding they were more likely to have problems or quit prematurely. The consensus opinion was long commutes are difficult. But then Xerox’s managers noted another correlation. Many people driving long distances were coming from poor neighborhoods. Xerox made the decision to sacrifice a little efficiency, and avoided punishing people for traveling from a distant, poor neighborhood by not hiring them.

Artificial Intelligence and Smart Data

Prediction: Artificial Intelligence (AI) will become a basic component in the processing of Big Data into Smart Data.

 AI is becoming pervasive throughout the world of business. Every industry is having its decision making process fundamentally altered by Artificial Intelligence. Automated decision making is becoming more and more popular. The desire for faster, smarter decisions, and the hope of a competitive edge, is driving the trend of merging of AI and Big Data.

Cognitive Business Intelligence (CBI) incorporates Artificial Intelligence, and has been described as the next evolutionary step in processing unstructured data, human language, images, and video. Nikhilesh Tiwari, of Helical Insight (the first “open source” framework for Business Intelligence) said, “CBI will make the entire data analysis and decision making instantaneous.” He added, “AI can take the place of Data Scientists to generate reports and other visualizations. With search driven analytics, tools like Helical Insight can empower anyone to be a data scientist.” One very useful benefit of AI-powered Business Intelligence is its ability to translate real-time data into written reports, allowing organizations to make decisions with data that is just seconds old.

Gartner, Inc. has predicted the global revenue in the analytics market and Business Intelligence (BI) world will reach $16.9 billion in 2016. Ian Bertram, Managing Vice President of Gartner stated,

“The shift to the modern BI and analytics platform has now reached a tipping point. Organizations must transition to easy-to-use, fast and agile modern BI platforms to create business value from deeper insights into diverse data sources.”

Smart Data and the Internet of Things

Prediction: The Internet of Things (IoT) will continue to grow, invading people’s privacy until laws are put in place providing privacy protection for the individual. Some smaller businesses will choose to avoid the IoT, using a “We respect your privacy,” advertising base.

The potential of the Internet of Things (IoT) is just beginning to be understood. While it has innumerable possibilities for positive impacts in industries across-the-board, it also allows companies to access aspects of people’s lives that are raising security concerns. Consider the air purifier on your desk (purchased on impulse), which collects data on the smoke, air pollution, and dust in the air. After collecting this information, the manufacturer now has data that it can use for a range of reasons, some positive like improving air quality, others more problematic such as selling the information to insurance companies. Smart Data Analytics with Big Data assets allows for the processing of such (seemingly innocuous) data sets in effectively infinite ways.

The Internet of Things is a continuously expanding industry. Gartner has predicted there will be over 20 billion devices used by consumers and businesses in 2020. The sheer volume of information that will be gathered creates the problem storing and processing it. And then there are the ethical issues of how to use, and not use, the information gathered. Corporations and governments will have information allowing them to manipulate the data to their benefit. Transparency, social media, and accurate information will be the best defense against corporate and governmental manipulation tactics.

Big Data, Security, and Privacy

Prediction: Artificial Intelligence will become instrumental in eliminating hacking and providing website and internet security. New algorithms will continue to be developed in a constant effort to block hackers. Small businesses and individuals will take steps to maintain their privacy and free time by installing AI security programs.

There have been a number of very prominent cyberattacks over the last few years. Unfortunately, the number keeps growing, and includes the copy and theft of data on millions of Target shoppers during the holiday season in 2015, the hacking of the U.S. Office of Personnel Management which released personal information on over 21 million people, and invasion of the Democratic National Committee’s computers. Banks, grocery stores, and large retailers are often the targets of hackers looking for identity theft information. Even Yahoo has been hacked.

The need for significantly improved website and internet security is real. The Defense Advanced Research Projects Agency (DARPA), created a contest where seven autonomous AI bots were released into a system. The AI bots found several hidden security weaknesses tucked away in the coding. The following day, the most successful AI bot was pitted against some of the best human hackers on the planet. There were mixed results, with the AI bot beating some of the human teams, but not all.

Because Artificial Intelligence is fueled by data, more data about an individual means better predictions about their habits and needs. The service provider of an AI will have access to a huge amount personal data, which could be collected and turned into Smart Data. Honest businesses will need to be transparent in how they gather and use people’s personal information. To use AI effectively over the long haul, businesses will need to make customer security and privacy a basic part of their business philosophy.

Leave a Reply