The concept of Big Data has been around for more than a decade – but while its potential to transform the effectiveness, efficiency, and profitability of virtually any enterprise has been well documented, the means to effectively leverage Big Data and realize its promised benefits still eludes some organizations. Ultimately, there are two main hurdles to tackle when it comes to realizing these benefits.

The first is realizing that the real purpose of leveraging Big Data is to take action – to make more accurate decisions and to do so quickly. We call this situational awareness. Regardless of industry or environment, situational awareness means having an understanding of what you need to know, what you have control of, and conducting analysis in real-time to identify anomalies in normal patterns or behaviors that can affect the outcome of a business or process. If you have these things, making the right decision within the right amount of time in any context becomes much easier.

Defining these parameters for any industry is not simple, and thus surmounting Big Data’s other remaining challenge of creating new approaches to data management and analysis is also no small feat. Achieving situational awareness used to be much easier because data volumes were smaller, and new data was created at a slower rate, which meant our world was defined by a much smaller amount of information. But new data is now created at a hugely exponential rate, and therefore any data management and analysis system that is built to provide situational awareness today must also be able to do so tomorrow. So, the imperative for any enterprise is not to just create systems that manage Big Data and provide situational awareness, but to build systems that provide scalable situational awareness.

Take, for instance, the utilities industry. This space is in particular need of scalable situational awareness so that they can realize benefits for a wide range of important functions critical for enabling Smart Grid paradigms. A properly-functioning power grid network shifts power around to where it is needed. Scalable situational awareness for utilities then means knowing where power is needed, and where it can be taken from, to keep the grid stable. When power flow is not well understood its direction will start changing rapidly, moving energy around like a power hurricane.

As with any hurricane, at the middle there is an eye that is totally quiet and dark (a fitting, although ironic, analogy considering the goal of awareness). This is what happened in 2003, during one of the worst blackouts to hit the Northeast. The various power companies involved were quickly analyzing all of the information but, despite the fact that they were all communicating, they didn’t know what exactly to do to alleviate the drastic shift in power flow and, thus, ended up making the wrong decisions that resulted in the blackout.

If situational awareness had been present, the blackout could have been prevented.  This seems especially relevant given the recent blackout in India, and begs the question: is the U.S. aware enough of the potential dangers that we have taken steps to enable our smart grid to respond in the correct way to avoid such outages?

Utilities in the U.S. and beyond can learn much about how to achieve scalable situational awareness from other industries, most notably building management and telecommunications, which have learned to deal with Big Data’s complexity and scale well. For industries like utilities to achieve scalable situational awareness, it requires building standards-based, interoperable, and scalable data management systems.

I’m actually going to be discussing this in more depth at the upcoming NoSQL Now! conference, during the Hadoop track on Wednesday. This is especially exciting as we’re doing some groundbreaking work with situational awareness in conjunction with companies like the Electric Power Research Institute (EPRI).  Will you be attending the show? Do you want to learn more about the value of situational awareness in the world of Big Data?

Wall Street’s getting all sentimental on us. Not in the sense that they are all of a sudden pining away for some form of the past. No, what I mean is that new trading strategies that look beyond just ticker quantitative financial data to incorporate information from sources like RSS feeds, breaking news, and “unstructured” Twitter feed data are increasingly the Wall Street norm.

But these new strategies quickly run headlong in to some serious high performance computing challenges, which is exactly why Wall Street is taking a shine to NoSQL.

Essentially, what financial institutions are trying to do is add elements of sentiment analysis to existing trading strategies and algorithms. The thought is that if they can correlate information from things like news feeds with current data about the market value of a position or asset, they can create a “network of networks” data model that lets them see how changes in one system affect another. So, for example, rising geopolitical concerns near the Strait of Hormuz can be correlated with the risk of trading oil-related assets, and automatically adjust investment strategies in real time.

Doing this is not trivial, and is why I’ve heard a lot lately – and will hear more tomorrow at the High Performance Computing on Wall Street event – about how it is changing the analytics and database technologies they use to build such applications.  Many attempts to construct these “network of networks” data models have tried to stuff these new information streams in to the model using the existing mathematical algorithms to make predictions. To put it bluntly, that doesn’t really work for Wall Street’s needs. These older algorithms are based purely on sampling and statistical analysis of lots of quantitative data.

But trying to take the qualitative sentiment among whole populations, transform it in to something that can be quantitatively measured, and then correlate that to lots of financial data requires a completely new class of algorithms that don’t just use sampled sets of data. It requires using the full sets, and, because this is Wall Street after all, processing all that data in a meaningful timeframe (read: very-near-real time).

This is of course the type of situation where, as I’ve mentioned before, NoSQL versus more traditional information management systems, like relational databases, start to look extremely appealing. But there is still a word of warning for many larger enterprises, especially like those on Wall Street, that need to think about not just large, simple volumes of data and performance at speed, but in addition very rich data models and things like concurrency of the system, too.

In the last five years or so, a great deal has been learned about how these “network of networks” data models actually function. In particular, it’s now known from examining things like Google’s Web Spider algorithms that the structure is almost exactly the same for an extremely high percentage of all networks, and armed with that structural knowledge,  when an analytic question is asked, a large percentage of the data in these networks is immediately identified as completely irrelevant to analysis and can be ignored to get quick answers for things like what Wall Street wants to do. This causes a completely new set of programming and information management system requirements that makes many of the first generation NoSQL technologies unsuitable for these advanced, complex analytics applications in an industry where both speed and accuracy are of equal, and intense significance. Wall Street should be discerning in how it approaches NoSQL. Stay tuned for feedback on how NoSQL is received at next week’s event.

Because of the wealth of information presented by Big Data, businesses are asking more difficult questions of their data instead of searching for simple answers in tables that are found in traditional relational databases (RDBs). When presented with the amount of scale and complexity found in Big Data, RDBs often simply fall short.

What businesses need now are databases and applications that can link richer data sets and provide more intelligence faster – extending beyond the simple keys, values and relational JOIN operations.

New NoSQL solutions provide a great alternative to RDBs, especially in terms of handling complex data and scaling out, but the reality is that a move to NoSQL technologies may actually create more headaches as businesses will have to invest more money for employee training on new coding languages.

Right now, the trend seems to be that NoSQL vendors are each building their solution around a unique language in the hopes of cornering the market, similar to what Oracle did with SQL. One solution is to instead utilize more standardized and common coding languages like Java to leverage these NoSQL technologies. Using a Java API (JPA) allows developers to use existing coding skills to process previously unmanageable Big Data sets without learning another exclusive programming language. By using JPA, businesses can skip employee training on NoSQL programming and move straight to implementing Big Data applications quickly and easily, using skills most developers learn on day one of computer science class.

One market that can benefit tremendously from this is the utilities sector, particularly as smart girds flood their information systems with large amounts of rich data. The result is that smart grid data and information complexities, just as with other big data scenarios, quickly become unmanageable in an RDB. So as utilities look to meet the global need for clean, economic energy, they must address these challenges in order to better manage speed, reliability, effectiveness, and accessibility.

Another real consideration and benefit of using JPA for Big Data is that it limits risk. For example, if utilities find that NoSQL is not living up to its claims, then there has been no lost effort – companies can simply replace the JPA driver and point it at their favorite relational database and still use the same skill set.  It’s a great way to test if these cutting edge information management technologies are right for them without betting the farm.

The answer for utilities increasingly seems to lie with NoSQL architectures that can support graph-like data structures, handle complex event processing and act as a near-real-time analytics systems. But again, cost and adoption barriers come in to play. Smart grids cost enough without having to train an entire staff on new programming languages. For quick and easy integration, JPA must begin to play a key role in NoSQL’s rollout evolution. In doing so, the utilities industry will start seeing those clean energy benefits faster, more easily and without negatively impacting the bottom line.

At the recent Super Computing conference in Seattle, the major theme was Big Data and technologies that enable Big Data management. This was certainly an interesting departure from the typical computational issues these types of events tend to focus on like the CPU versus GPU, high speed network interconnects and storage. But the focus on Big Data certainly makes sense given that the many attendees are really feeling the strain of the data explosion – taking on issues involving time and space dependent data sets at the terabyte and even petabyte range.  As such, much of the discussion was related to software instead of just the bits and bytes of the next evolution in the hardware stack.

Through these discussions a clear theme emerged: software systems over the next decade will need to be rewritten.  In particular, the existing software solutions are not designed to leverage multi-core processors nor solid-state storage, both of which are undeniable trends which will dominate the hardware architectures of the data center and the enterprise within the next five years.

This is not to say that existing software systems are not designed for these technological evolutions, but essentially software design decisions of the past were made based on bottlenecks that simply do not exist in the newer hardware.  For example, databases that have heavy I/O components were designed for block based I/O, which meant moving data into memory in big chunks because spinning disk seek time was the limiting factor in throughput. With this, it was better to seek a small number of times and read more data.

However, with SSD in Flash/DRAM drives, seek time is negligible. So, software can be changed to become more granular in how it deals with storage, and by becoming more granular it can then accommodate higher levels of concurrency and throughput.  Similarly, many software systems were designed using largely single threaded, exclusive access data structures and those systems must be redesigned with multi-threading and concurrent access data structures in mind to achieve the parallelism and concurrency demanded by the next generation of real-time business solutions.

The task of redesigning software components based on decades of code for this new paradigm in hardware, while maintaining backwards compatibility with solutions already deployed in the enterprise, is an impossible effort.  It means change will come to all and, out of necessity, organizations will re-evaluate and choose new software components to carry them into the next decade and beyond.  Else, they will fall behind as competition moves to these new technologies which will enable their real-time business in this increasingly competitive business landscape.

One thing NoSQL 1.0 is bringing to the table is a shift in architecture. The benefit here is the elimination of critical dependency on the JOIN operation and the ability to scale through distribution. Done correctly, this allows for algorithmic processing as opposed to strictly query-based processing – but the way query is being introduced in NoSQL 1.0 solutions has many people approaching NoSQL like they have the traditional relational database (RDB).  This simply leads back to a place where one can more easily ask SQL-type questions like “what are my top 10 sales performers for the quarter?”

However, the questions which have the most impact are more profound. These questions can be something like, “how does this user’s pattern of usage compare to others, and based on the similarity, what can I offer that user?” or “match the three-dimensional structure of a certain protein against a database of ten million other proteins to find a potential new drug candidate match.” These kinds of questions are algorithmic and not naturally set-based logic operations.

The reality is that, today, database systems need to be able to handle the complexity of storing useful models (e.g. predictive models) that can be coupled into the business system to create a proactive business as opposed to one that is simply reactive. With that, simple indexed queries are a first and important step to handling end user needs, as these are fundamental building blocks to a useful solution.

In the end, NoSQL 1.0 vendors should be aiming to enable existing enterprise developer skill sets and lower the risk to adoption. Query is only one important aspect of what people need.  First, they need an architectural shift and the ability to do algorithmic based processing. Then, they need query to round out the solution and allow other adjunct requirements found in an enterprise deployment.

The enterprise needs a query capability that integrates with their existing skill sets and the trick is how to combine that query ability with the NoSQL shift in architecture.

I recently wrote about my observations at the recent NoSQL Now! Conference and am happy to bring you the second and final part of my observations.

Hardware: Scale Up Before Scaling Out

Surprisingly, hardware came up several times. This has been largely ignored in the past but is now coming to the forefront as a hardware revolution is emerging at the same time the industry embraces the database revolution. The reality is that a commodity hardware node is a multi-core node, which will be stunningly true within the next 18-24 months. To remain efficient, databases need to scale up before they scale out or the new generation of hardware will be wasted, creating unnecessary operational costs for production deployments.

Some industry leaders like Michael Stonebreaker argued that the cost of computing in databases is tied to the problems of dealing with concurrency and how implementations leverage such things as locking, latching and buffer management to overcome design issues.  In short, the argument is that if you move to a single thread and only scale out, you can somehow scale as needed by removing those aspects of implementation. But it isn’t a reasonable expectation for a machine with eight or more cores, limited to eight or more threads, to consume all the cycles of those cores. Simply put, the behavioral patterns of computation are just not uniform enough such that you can queue them up and have them exit the front of the queue right when needed. There are not hundreds but thousands of concurrent users, and lining them up behind eight single core threads isn’t the right solution, except for perhaps a narrow use case. What do you think?

Defining NoSQL

At the conference there were a number of companies with different NoSQL database products that had different implementation designs and operational characteristics. With all these varying database products, it seems difficult to distinguish what exactly is classified as NoSQL. The answer could lie in soft schema, which is a major theme I observed in the vast majority of products. This shift toward soft schema likely represents what NoSQL is coming to mean.

The shift toward soft schema is really a movement of architectural responsibility for the management of relations between data from database data-models to application models. This means moving away from database joins to application linking. By moving to soft-schema, all of these products are able to speed the access to related data through links, which amounts to a high speed B-tree style lookup for any relation. Further, since the responsibility for resolution is moved up a layer in the software stack, distribution can be implemented to deliver scale-out. It is the architectural shift to soft schema and data linking, delivering the ability to perform at scale that defines a product as NoSQL. How would you define NoSQL?

I just returned from the tremendous NoSQL Now! Conference in San Jose. The sessions were ripe with promising ideas and the floor was buzzing with discussion. As such, there were a few key themes over the three days.

Fitting NoSQL into the Enterprise:

First, I couldn’t help but notice there were a surprising number of enterprises in attendance. This is certainly indicative of a larger trend that NoSQL will play an important role in enterprise business solutions. However, in order to make adoption in the enterprise work, early generation NoSQL technology needs to evolve in order to be consistent with the needs of enterprise development and deployment while not sacrificing core NoSQL value delivered in the shift to a soft-schema. To that end, it was clear that transactions, standard-based API’s and robust production tooling are increasingly recognized as the needed features for next generation NoSQL products. Ultimately, solutions that deliver the core architecture shift ushered in by NoSQL, and that deliver enterprise-class tooling and API’s will be the likely winners in the inevitable consolidation within the space.

A Benchmarking Debate

Second, benchmarking was a hot topic. Analyst Robin Bloor summed it up nicely when he stated that, “the database is no longer a commodity. Do your own proof of concept to make sure the choice is the right one for your problem.” This statement rings true especially given that standard database benchmark tests are in some sense out of touch with reality. Existing industry benchmarks essentially only look at the cost of a piece of data moving in and out of a relational RPC, something that represents only a small percentage of the overall unit of work. However, the true cost of a unit of work in today’s computational environment involves several components, including programming language, object mapping, population, and storage in and out of the database. The cost of that operation is significantly impacted by the type of database itself and the inevitable work necessary to move raw data into the behavioral context. For example, if you are trying to tell the world that you are 40 times faster on something that only represents 10 percent of the total operational cost, then you have only really improved the system by some four percent under the very best conditions. That’s simply not good enough.

Because of this, new benchmarks are needed to reflect the true cost of computing, touching all areas impacted by the choice of database. This leads to the conclusion that some databases are more suitable than others for certain types of application requirements. For example, if it is pure hierarchical XML content without interlinking – moving straight from a webpage form to database – then quite likely an XML database or something similar would be the right choice. On the other hand, if you are dealing with richly linked, networked enterprise domain models then something like an object-oriented database is a better choice. Of course, there are other factors and other application profiles, but the key is to make sure you’re using the right tool for the job.

Stay tuned for part two of my NoSQL Now! observations.

Join the Forum discussion on this post