The concept of Big Data has been around for more than a decade – but while its potential to transform the effectiveness, efficiency, and profitability of virtually any enterprise has been well documented, the means to effectively leverage Big Data and realize its promised benefits still eludes some organizations. Ultimately, there are two main hurdles to tackle when it comes to realizing these benefits.
The first is realizing that the real purpose of leveraging Big Data is to take action – to make more accurate decisions and to do so quickly. We call this situational awareness. Regardless of industry or environment, situational awareness means having an understanding of what you need to know, what you have control of, and conducting analysis in real-time to identify anomalies in normal patterns or behaviors that can affect the outcome of a business or process. If you have these things, making the right decision within the right amount of time in any context becomes much easier.
Defining these parameters for any industry is not simple, and thus surmounting Big Data’s other remaining challenge of creating new approaches to data management and analysis is also no small feat. Achieving situational awareness used to be much easier because data volumes were smaller, and new data was created at a slower rate, which meant our world was defined by a much smaller amount of information. But new data is now created at a hugely exponential rate, and therefore any data management and analysis system that is built to provide situational awareness today must also be able to do so tomorrow. So, the imperative for any enterprise is not to just create systems that manage Big Data and provide situational awareness, but to build systems that provide scalable situational awareness.
Take, for instance, the utilities industry. This space is in particular need of scalable situational awareness so that they can realize benefits for a wide range of important functions critical for enabling Smart Grid paradigms. A properly-functioning power grid network shifts power around to where it is needed. Scalable situational awareness for utilities then means knowing where power is needed, and where it can be taken from, to keep the grid stable. When power flow is not well understood its direction will start changing rapidly, moving energy around like a power hurricane.
As with any hurricane, at the middle there is an eye that is totally quiet and dark (a fitting, although ironic, analogy considering the goal of awareness). This is what happened in 2003, during one of the worst blackouts to hit the Northeast. The various power companies involved were quickly analyzing all of the information but, despite the fact that they were all communicating, they didn’t know what exactly to do to alleviate the drastic shift in power flow and, thus, ended up making the wrong decisions that resulted in the blackout.
If situational awareness had been present, the blackout could have been prevented. This seems especially relevant given the recent blackout in India, and begs the question: is the U.S. aware enough of the potential dangers that we have taken steps to enable our smart grid to respond in the correct way to avoid such outages?
Utilities in the U.S. and beyond can learn much about how to achieve scalable situational awareness from other industries, most notably building management and telecommunications, which have learned to deal with Big Data’s complexity and scale well. For industries like utilities to achieve scalable situational awareness, it requires building standards-based, interoperable, and scalable data management systems.
I’m actually going to be discussing this in more depth at the upcoming NoSQL Now! conference, during the Hadoop track on Wednesday. This is especially exciting as we’re doing some groundbreaking work with situational awareness in conjunction with companies like the Electric Power Research Institute (EPRI). Will you be attending the show? Do you want to learn more about the value of situational awareness in the world of Big Data?