The doorbell rings and it’s a big box with the words “HelloFresh” on the side in lime green letters. Inside, surrounded by biodegradable ice packs, are fresh ingredients for three different meals for you and your spouse, chosen from a varied slate of innovative entrees. All the vegies and meats in the box are portioned, all the sauces and spices are measured to the teaspoon, and all you’ll have to do for dinner that night is cook it. The enclosed recipe cards tell you how much prep and cooking time you’ll need, and include nutritional information. The Business Intelligence behind that box to your door came from a partnership between meal delivery service HelloFresh and converged Big Data platform service provider, MapR.
The Need for a Big Data Solution
HelloFresh is a subscription-based meal delivery service based in Germany, delivering over 7.5 million meals per month to more than 800,000 subscribers in nine global regions. Starting out with their own Business Intelligence solution in 2011, HelloFresh quickly discovered that with their rapid growth, they would soon outgrow their current system. “As we grew, our Data Analysis needs became more demanding. We knew our home-brewed solution would not be able to handle it,” says Nuno Simaria, HelloFresh Chief Technology Officer. “It’s not that things were breaking, but we knew if we wanted to stick to our roadmap that we would quickly hit a limit.”
Simaria says the company gathers data from a wide range of feedback points and analysis of this was previously done manually – an incredibly cumbersome exercise. “What we had just wasn’t scalable, especially as our needs became more demanding.” Processing two weeks’ worth of data for analysis took nine hours, he says.
According to Dale Kim, Senior Director of Industry Solutions for MapR in a recent DATAVERSITY® interview, HelloFresh has a large volume of data from a variety of different sources as well as “a very aggressive vision about what else they’d like to be doing” with that data. Kim says that HelloFresh collects information on product quality, delivery, usefulness of the recipes, and efficiency of the entire process.
“Part of it had to do with the volume, but a lot of it also had to do with the many different data formats, as well as with the speed at which data was coming in, so they had to be able to react quickly to a lot of different changes in the marketplace,” Kim says. “This is a classic Big Data environment where HelloFresh is using that data to make better decisions and offer better corporate products,” he says.
The Implementation
Since they were looking for a scalable, lower cost technology, Simaria says, “We decided to search for a Hadoop-based solution.” Even so, Simaria says the organization had some concerns about whether they’d be able to find the talent to support the new technology.
“It’s very hard to find data engineers in the market,” said Simaria in Hello Fresh Moves Analytics to Hadoop. The company wondered if that problem could be solved with existing staff. Simaria continued during that previous interview: “We empower our tech team to do what we call ‘Figure it out,'” a process that he says allows a small team the time and support to explore a possible solution. Two members of the HelloFresh engineering team spent six weeks learning how to setup and operate a Hadoop cluster:
“We’ll give you the budget, and we’ll give you the time,” said Simaria. “This is something we’ve done with other technologies as well. If it is not easy for us to access talent in the market in the short term, we will empower our developers and our engineers who are interested in problem solving, and we will let them discover the complexities of that technology.”
Kim agrees that there can be challenges with newer technologies like Hadoop, Spark, and NoSQL “just building the talent” but with the rapid growth in the market, he believes there will be “a tremendous talent pool coming about soon as a result of the proliferation of these newer technologies.”
HelloFresh decided to deploy their Hadoop solution based on the MapR Platform. They chose MapR over other alternatives for several reasons.
“A constant complaint about Hadoop in the industry has to do with the complexity of managing the file system,” says Simaria. “The way MapR has solved it is different. The file system is quite powerful. Also MapR has been recommended to us by other companies.”
MapR was able to offer NFS for faster Data Management, disaster recovery features like Snapshots, and MapR support services. Other technologies used in the HelloFresh solution include Tableau, Impala, and a few third-party services hosted on AWS.
“It took us six months to build our home-brewed platform and that was done in a technology that we are fluent in,” says Simaria. “It took us less than two months to build the same platform and do the data migration with MapR technology.” Kim credits some of the features of MapR’s converged platform for the rapid development timeline. High availability, disaster recovery, and security administration are built into the platform so that HelloFresh, “could focus more on the higher-value type things, rather than the lower level details.”
The Results
The speed of the MapR Platform translates into a more agile business. Simaria says:
“The biggest advantage of the MapR ecosystem is how quickly we can process data compared to before. It used to take nine hours to process two weeks of data. Now it takes 20 minutes. This fast performance makes us more agile, and allows us to try things, make mistakes and correct them. You can experiment a lot more,” he continues. “Now we can process tens of millions of rows in a few minutes. With MapR, we get results quickly, and can assess and iterate in minutes. We get more valuable business insights and can act on them quickly.”
The new platform also enables HelloFresh to do deeper Data Analysis with greater flexibility. With their previous system, it was difficult to offer non-technical business people views into the data, and management had to ask IT for specified reports. Now it’s more self-service: “It gives management the capacity to explore the data and find things they are not expecting to find. It’s a more flexible way to do analysis,” Simaria says.
They can also more easily analyze data across multiple data sources like their corporate database, website, distribution centers, customer care tickets, and application usage. “We need to analyze data in different ways. It’s not just customer acquisition and retention, we gather and derive insights from a lot of feedback points,” he explains. “In the past, it was cumbersome and a very manual exercise. We can now automate that process and get insights to decision makers much faster.”
Since they can process faster and add more data to the mix, they can see how data relates to each other and recognize more advanced relationships. “We can get new insights into our offerings,” see what’s working “and decide where to invest,” he says.
So while you’re serving the Lasagna-Baked Fusilli with Kale and Mozzarella you made from the box on the porch, just remember: dinner was brought to you by HelloFresh – and Big Data.