This fall has been an exciting time with an extended European tour. The trip began with the Gartner ITxpo in Barcelona followed by visits with customers in London and Europe. It was a pleasure to see the warm reception we received everywhere we went. The topic on everyone’s mind was NoSQL.
I’ve attached a photo I took at a show in Spain with my Italian colleague Franco. My kids are always talking about taking “selfies.” I’d rather take pictures of them. They didn’t accompany me on this trip, so we have to make do with this photo of two “distinguished” IT guys.
After returning from abroad, I hardly had time to unpack before leaving for Vegas and the Gartner Application Architecture, Development & Integration Summit 2014, December 8-10. (Stop by our booth and we can take a selfie!)
It’s hard to imagine, but the year will be nearly over by the time I return home. As we reach the end of the year, discussions inevitably turn to the future. Each year I like to think about upcoming trends in the database industry. Here are my thoughts about 2015:
NoSQL will continue to shake up the old-school, SQL-only mentality
The amount of interest in NoSQL is quite eye-opening. You can see why the big, traditional relational companies (you know who you are!) are scrambling to add NoSQL offerings to their product lines. Nobody expects this trend to subside; certainly not the people who talked to me at shows.
NoSQL is being used on multiple fronts, primarily to allow applications to handle the huge amount of data that is being streamed into backend enterprise systems like business intelligence and data warehouses. Its versatility in dealing with schema-less data is a perfect match for one of the Big Data Vs: Variety (the other Vs include volume, velocity, and possibly a few more depending on who you ask). The Internet-of-things is already a reality, and more and more companies are looking for ways to monetize this information, which would be almost impossible without NoSQL technologies.
SQL will do just fine
It’s easy to forget that most of the IT world is built around SQL-based databases. It would be foolish to count them out just because NoSQL is getting so much press. That doesn’t mean there is room for complacency in the SQL world. In 2014 we saw stories about a Bitcoin exchange having problems that traced back to the eventually-consistent nature of their NoSQL database. Eventual consistency is not a good idea in this era when currencies and commodities can be bought and sold in micro-seconds.
I think there is another lesson to be learned: When dealing with massive amounts of data, today’s generation of programmers look to new technologies, such as NoSQL. They have little regard for the decades that their predecessors have been chanting the SQL mantra. Any company that clings too tightly to any one data model is in for a tough time in today’s world.
SQL and NoSQL will become best friends
Another lesson form the Bitcoin story is that some transactions require more data integrity than others. If you’re trying to use statistics to infer what deals a customer may be interested in (“Customers who bought this also were interested in that”), there is really no harm in using slightly out-of-date data. If you’re trying to tell someone their current account balance, you better have accurate data.
We are starting to see some of the NoSQL players attempt to offer more robust transaction processing. In 2015, we will see more companies incorporating both SQL and NoSQL elements in their databases as NoSQL becomes part of a mission-critical focus within the enterprise. I certainly hope this will be a “Best Practice” for any database that handles transactions involving my hard-earned money.
Big Data will get real
Big Data is turning from hype into real implementations. In the coming year, it will evolve to be more specific and targeted to company needs. As more companies adopt Big Data solutions in 2015, product offerings will become more mature and best practices will start to emerge. What may have been seen as hype in the past will soon become a reality.
The combination of technologies such as Hadoop and NoSQL will bring both the scalability and performance that BigData needs into the mainstream. Companies are seeing the need to expand beyond a single data warehouse server to a “logical DW” where multiple analytics systems can extract the information they need regardless of its source. The streaming of data from different sources, combined with the huge throughput of information that is created in real time, will lead to a convergence of two so-far separate worlds: real-time transactions and batch analytical processes. A key to turning this into a reality is in-memory computing—an old friend from the database industry recreated to provide the power that this combination of forces will demand.
Smart Data Recovery
Because of the Internet-of-things, the variety and velocity of data that needs to be collected and analyzed in real time is unimaginable. This creates a tremendous potential for new business opportunities, and therefore multiple startups are popping up every day with new solutions and new ideas on how to handle this problem.
A trend in this arena is called Smart Data Recovery, which can be translated as automatic tools that can sniff, collect, and categorize sources of data that have potential information to be processed and analyzed. Such data can be provided by sensors, real-time systems, or an application on a smartphone of a customer who just stepped into a store. You can see how traditional ETL tools need to evolve to a much more elastic way of identifying and connecting to such data sources. This area is still taking its first baby steps, but it’s easy to see its potential.
So my predictions for 2015 can be summed up as “NoSQL will thrive; SQL with survive; and Big Data—aided by Smart Data Recovery—will come alive.”