The Big Data Executive Survey released by NewVantage Partners earlier this year shows that close to three-quarters of the respondents surveyed believe they are getting measurable value from their Big Data and Artificial Intelligence projects, which the survey considers as extensions of Analytics capabilities. That number, NewVantage reports, is half again higher than in the 2017 survey, which suggests that more value is being achieved as companies grow familiar with the technologies.
Such findings would come as no surprise to Luke Roquet, worldwide field operations VP at Unravel Data. Roquet previously worked at Big Data Management platform vendor Hortonworks. “Every business at some level is a data business,” Roquet says. “They’re generating top-line revenues with the new Analytics capabilities these platforms allow.”
Roquet has seen the momentum grow in the space, as major companies – particularly in the tech sector – invest in the technology and talent to drive successful results.
That said, a significant portion of companies still struggle, he points out, as many companies can’t compete for the Big Data talent that’s attracted to working at big-name tech companies. These lesser-known companies find it “hard to operationalize their platforms and build apps on top of them without the top-tier talent that big companies have,” Roquet explains.
That conundrum is compounded by an issue highlighted in The Big Data Executive Survey, which notes the difficulties companies have at making the shift to a data-driven culture. Virtually all companies say they are trying, but only about a third have succeeded, the survey reports. Not surprisingly, under such circumstances nearly 75 percent of respondents share a concern over the threat of disruption and displacement.
“Executives,” the survey records, “perceive growing threats from data-driven, highly agile competitors, including the big Tech Giants – Amazon, Google, Apple, and Facebook – as well as those competitors within their own industry who are demonstrating the ability to compete on data and analytics, especially those who have forged data cultures which give them agility and speed.”
A Better Handle on Big Data Concerns
The problem, therefore, becomes how to get the most out of Big Data platforms without necessarily having the talent on hand and culture in place to do it. Specifically, how can smaller companies not only to build strong apps on top of Big Data stacks but maximize the time to value of the new apps they build?
The answer has a couple of parts. The first one is not to assume that you can build a Data Lake and “they will come,” Roquet says. Too often the Data Lake is just where things end, creating disillusionment over platforms and resentment over costs.
Instead, pick an issue that is something you know will deliver meaningful value to the business once solved – that is, something it can’t do today — and which can also be accomplished via a strong Big Data app effort in about six months. Then go for it. Upon completion, it should be possible to plant a flag as a standard-bearer for Big Data’s success.
The next level, he says, comes with helping to operationalize Big Data apps at scale that are threaded together across many different systems – Big Data ETL, Spark, MapReduce, Hive, and so on – maximizing application performance.
The approach that Unravel Data takes goes hand-in-hand with the Agile/DevOps approach to development. It’s about “putting power in the hands of application developers to quickly build apps and get real-time feedback of the performance of their apps on a Big Data stack,” he says. That’s in contrast to the historical approach of building apps without developers having a view into what happens with them on Big Data platforms. In such cases, the feedback loop is broken between Hadoop platform operators managing complex multi-tenant environments that see the activity impacting integrity of the platform and app developers who need a lens for insight to improve the efficiency of their apps.
The end result of Unravel Data’s approach is companies that may not have a large amount of Big Data talent can be more efficient with the talent they do have. “We need to make it more efficient to have a lean, mean, agile team rather than to have to hire a huge staff to operationalize Big Data,” Roquet says. “We make organizations more efficient with their talent when the supply is not there.”
Building Up from Starfish
Unravel Data has its roots in Project Starfish, a Machine Learning platform project at Duke University to help with application performance management on the Hadoop stack. Company co-founder and CTO Shivnath Babu’s research team built it there. Unravel Data keeps up with the pace of the Big Data world by expanding to provide application performance management for the full Big Data ecosystem, from ingestion through analysis, accommodating engines including Hive and Impala and streaming platforms like Kafka and Spark.
Unravel Data has spent the last year and a half going to market with the product and landing core enterprise customers in major industries. Customers include healthcare providers such as Kaiser Permanente, some of the world’s largest financial institutions, and data-driven technology companies such as Autodesk, YP, and Leidos. While the focus has been on major large-scale enterprise users of Big Data that are keenly aware of its challenges and opportunities, Roquet says the company does have a parallel track of mid-market investment.
“We want to be with them from day one and we are also investing now with these customers as they begin the journey to lay the groundwork successfully,” he says.
This is a new era, he says, where every company will adopt Big Data technologies as a core component of becoming a digital business.
“Big Data is inevitable, just as e-commerce or a web presence was, and APM [application performance management] helped with that,” Roquet says. “APM will be a key component to be successful with Big Data, too.”
Among the possible use cases is the unpredictability of meeting business app service level agreements (SLAs) in large multitenant Big Data platform environments. APM solutions give a lens into what happens so that, when a job doesn’t live up to its expectations, IT can understand the issue and be proactive in addressing it. The solution might require developers to add more resources, for instance, or change configuration settings.
Another use case is helping customers move to a hybrid or cloud-first world from on-premise Big Data, with Unravel Data APM providing insights into the what, where, when, and how apps are running to identify what workloads and data sets are appropriate for moving to the Cloud in a successful transition.
A third use case could be helping operators understand what issues may be degrading the entire platform, such as a poorly written query that consumes massive resources, so they can make changes to protect the integrity of the environment.
A fourth scenario might be coming to the aid of companies that are starting to do chargeback for on-premises and cloud apps, providing details on what it really costs apps to run with lingering issues such as resource consumption. “You can show that to the customer and give them insights to improve the efficiency of their app for a real cost benefit,” he says.
Expansion is on Unravel Data’s mind, as shown by its move to cover streaming analytic apps and continuing efforts to move into areas like NoSQL.
“It’s fundamentally the same problem set, where you need to give operators a view to understand what happens in the environment to keep the integrity and quality of the platform in place and also to give developers and application folks insight into what is going on in the apps, what they want to change and why,” Roquet says.
Photo Credit: whiteMocca/Shutterstock.com