I spoke with all the leading data analytics and business intelligence (BI) analysts last year from Gartner and Forrester and it was unanimous, they all hated the term “Big Data”.  The reason is that the name implies only that this class of problems is about high data volume, but that’s not the case.  Most Big Data projects were being done on smaller volumes of data, but involved data that was typically not stored in a traditional data warehouse.  So while the Big Data challenge, solution, or situation can be about big data volume, it’s even more likely to involve a great variety of data types, data that is rapidly changing and/or moving at high velocity, or some combination of these three traits (volume, variety, velocity).

In fact, most of our work with clients across industries like telecommunications, energy, financial auditing, and manufacturing, leads us to believe a more appropriate trifecta of traits is volume, variety, and volatility, because changing data is such a common occurrence.  In our experience, which was confirmed by input from those Gartner and Forrester analysts, most Big Data projects today focus on  solving the variety and volatility problems and less so on the volume dimension.

It’s not to say that volume is a simple thing to deal with, especially when trying to process vast amounts of data for analytics and true real-time insights. But, the reality is that greater volume has been addressed, to a large degree, already and more of a key concern today is getting making sure analytic models can account for business complexity and the rate at which data and business situations change.

First, let’s look at variety, which can be interpreted a couple different ways. For energy utilities, for example, the sheer number of different types of data is very high because there are so many different types of devices and sensors that make up the grid. But an even bigger challenge when it comes to Big Data environments like this is that it is extremely hard to join all of these different data types together in one unified analytic. Not only are meters pumping out large amounts of data, but there are many different meter types, each with their own characteristics that complicate the analytics.  Also, data sources are quite often separated by different systems in different departments that have their own processes, and unifying fractured data stores can compound Big Data complexity problems immensely. No matter where you look, we live in a heterogeneous data world.  There is no escape.

Now let’s consider volatility. I think volatility is a more appropriate trait to describe the challenges of Big Data analytics because it puts the emphasis on how quickly the data changes.  In Big Data environments, it is common for data to change almost constantly, and if this is not accounted for, any analytic results may be invalid the moment they are produced. This type of situation is especially true in industries where real-time intelligence is key, such as the stock market, or even for a telecom company where call data records only remain relevant for one day.

Solving both the variety and volatility challenges requires faster, agile analytics/BI approaches, such as those represented by today’s data discovery tools. The path to greater flexibility can’t go exclusively through a data warehouse where the procedures and processes used to manage the data are slow moving and costly.  That makes it too costly to integrate dissimilar data.  And of course it’s even worse if that data changes rapidly.  By the time you could integrate it, the data would be too stale to matter.  Data analysis software, including data visualization software , though, have been helping companies make data exploration and unification a much faster process by being completely agnostic to data format and location.  Ultimately, taming variety and volatility are key to making the most of Big Data analytics.

Irfan Khan of IT World recently shared a Big Data visualization that is both beautiful and a major asset to energy companies. Khan writes, “In his fascinating and comprehensive 1984 study Heaven’s Breath: A Natural History of the Wind, Lyall Watson observed, ‘Wind is invisible.’ And, as a result, ‘There are no photographs of the wind.’ But he wrote that before Big Data blew into town. Now we have a moving picture of the wind.”

He continues, “Data visualization experts Fernanda Viégas and Martin Wattenberg have given us this beautiful near-real time view of wind in motion throughout the continental United States. The striking live image uses the massive National Digital Forecast Database maintained by the National Weather Service. By clicking on the map you can drill down and see the wind blowing in your area of interest; or simply become mesmerized by flowing patterns the wind makes.”

Khan adds, “In addition to being a thing of beauty itself, the visualization also offers practical insight for energy companies in deploying wind farms to produce electricity. It can also be a tool for firefighters in the American West in their annual battle with forest fires. And even farmers can use the visualization to help them locate the best places to establish effective wind breaks to fight wind erosion, a major hindrance to agricultural productivity.”

See the visualization here.

by Karen Lopez @datachick

I posted that Tweet last week while attending the NASA 2013 Fiscal Budget Briefing at NASA Headquarters. It did well, being retweeted 200+ times and had the prospect of reaching 1.9 million people. I say prospect because not everyone who follows someone reads all their tweets. But 1.9 million isn’t such a bad reach when you have a message to get out. Most of this reach was due to various NASA-related accounts retweeting it, but it was helped by regular Twitter users doing their normal thing on Twitter: sharing information with their followers.

One of the trade offs of having such a huge outlier in my data is that the charts on my Twitter data analytics are nearly useless for all the other thousands of tweets I did last week (yes, I Tweet…a lot.)

That red circle in the upper right represents the number of replies (the size of the circle) and the number of retweets and impressions (the X and Y axis). Looks good, until you see the blob of blue in the lower left. The fact that this outlier in my data was so far out there makes the other pieces of data look almost zero on both axes . I think I do pretty well with my social media outreach, but this chart would so fuddle duddle the data that it hides important information about my “normal” performance on Twitter. In fact, it almost makes it look like all my Tweets perform equally as well…or poorly.

So what could you do to make this chart more meaningful:

• Remove the outlier from the chart and or the data, completely
• Create two charts, one with and one without the outlier
• Make the graph 1000 times taller and wider
• “Break” the Y Axis so that there’s a gap between 100k and 1.8 million
• Use other techniques such as a logarithmic scale to show the data ratios instead of quantities
• Use statistical methods to massage the data even more
• Make better data (in this case, send Tweets that fill the gap to make my outlier look more normal)

I could also try to include a longer time period, such as including all my Tweets, not just the ones from this past week.

So a few more Tweets that had more retweets, but the impressions still look almost zero. So that doesn’t really help show how the rest of my Tweets did.

In business data, I’ve seen people opt to remove the outlier in additional charts, but sometimes they mask or delete them with no indication that they’ve been removed. Sure, my almost 2 million impression Tweet is messing with the display of other data, but if my performance bonus was based on that sort of thing, I wouldn’t want the data to vanish like the $38 million that was cut from the NASA STEM outreach budget. Your data needs may be different, though. So it’s important to find out how business users want outlier date dealt with. The reference links below talk about other more advanced methods for dealing with outliers. All I know is that my “all” chart isn’t going to help me much as long as that outlier is in the data set.

I’d recommend that whatever technique you use, you ensure that the reader understands what has been done. Remember that the goal of all charts should be to reveal more about the data than just looking at the raw data. If your chart doesn’t do that (Congrats again, Klout), maybe you need to rethink how you are presenting the data.

Other References:

• Data006: Outliers can make or break you. (data-literacy.com)
• Liars and Outliers Update (schneier.com)

A recent article reports, “Oxford University is using data visualization software to gain a better view of its student administration, finance and procurement information. It has also discovered benefits in being able to react rapidly when the university is in the media spotlight and in communicating with potential and current students. Andy Cotgreave, senior data analyst in the university’s student administration team, chose Tableau four years ago and set up the first UK user group.”

Cotgreave said that Oxford has “mapped the trend in BI [business intelligence] as analysts often describe it. We started with a small business-led acquisition of a tool just to get a job done quickly, and we are looking to move enterprisewide.” The article adds, “Reporting on students has become an increasing burden for the 140 university departments and 38 colleges. ‘Bigger demand, insufficient tools,’ he said.”

The article continues, “Four years ago, he installed Tableau and ‘in an afternoon, I could see that it would give the repeatable analysis required.’ The annual programme statistics now take a day to produce; previously, they had taken weeks. This frees his team up to do more analysis, respond to ad hoc requests and sit down with divisions applying for funding to work contemporaneously… Over time, hidden patterns have emerged. Applying the data visualization software to annual programme statistics pulled from the Oracle system, ‘has enabled us to see things where previously we would have relied on intuition. So, now we know that XYZ department has a problem with its D.Phil. thesis submission rates.’”

Read more here.

photo credit: Tim Norvell

A new article reports, “The California Independent System Operator Corporation has installed an 80-foot by 6.5-foot screen in its control room to display real-time power-grid data from thousands of endpoints. Its new system is powered by software from Space-Time Insight, whose software melds real-time geospatial data with Google Maps to give ISO employees access to the data they need in a format that’s very useful. It might be the most-cutting edge ISO control room in the world.”

It continues, “The problem that required such an extreme solution is the massive scale at which everything takes place in California. The ISO, for example, manages about 85 percent of California’s power load, totaling more than 286 billion kWh of energy per year across more than 25,000 miles of line. Employees are inundated with data, said Jim McIntosh, an executive director at California ISO, so they needed something that would present it to them in a useful manner.”

The article goes on, “What Space-Time Insights’ software does is present data however it will be most beneficial for that customer’s particular needs. It maps data — from wherever and whenever it’s from — in-memory for fast access, organized geospatially as well as by time. The result might be the popular map view, or it might be any number of heat maps, correlation, trending or any other number of more-classical data views. Additionally, Space-Time’s Steve Erlich told me, customers can customize the application how they see fit to create their own visualizations, functions, data workflows or other capabilities.”

photo credit: ISO

A recent article discusses how Tableau, a Seattle-based data visualization company is making data more accessible and interesting to end users through visualization. The article notes, “As more and more data gets uploaded to the Web, it needs to be analyzed and made sense of. Data visualization is one of the most effective ways of doing this and Tableau is a leading company in this field… Tableau’s vision is to ‘see and understand the world’s data.’ The company has carved out a particular niche with media organizations, who use Tableau software to enable their readers to play with data in interactive way – for example customizing charts.”

It continues, “Wall St Journal, CNN Money and Seattle Times are a few of the media publications that use Tableau. One of the more impressive examples was a data visualization on CNN Money, showing where mortgage foreclosures were happening in Q1 2010. Here it is embedded below. You can drag your mouse over a certain part of the country, for example California, to view just the data from that area. Further analysis is possible using drag and drop controls and the small menu at the bottom of the chart. The idea is that the reader becomes more engaged with the story and dives deeper into data if they wish. It’s another part of the evolution of the read/write Web – nowadays people can interact with data, not just read it.”

The article goes on, “Tableau’s goal is that anyone who cares about data should be able to work with it. But often there has already been a lot of work done on the data from so-called ‘Data Journalists’ – who have been adopting Tableau’s software with glee. There is a term for this type of reporting: Computer Assisted Reporting (CAR). CAR has been around for a while, but Tableau makes the process faster and saves journalists from doing custom programming. According to Tableau’s Ellie Fields, ‘fast authoring has been a big driver of uptake’ among data journalists.”

Read more here.

photo credit: Tableau

Quentin Hardy recently reported on the Allosphere, “a tool for new ways of seeing enormous amounts of data – and quite possibly, ourselves. The machine is egg-shaped, three stories tall, and located on the campus of the University of California, Santa Barbara. Scientists stand on a catwalk that runs through its middle, and wearing 3D glasses look at enormous representations of human brains, molecular bonds, economic data, even the invisible manifestations of quantum physics. The visuals stretch around vision’s periphery to simulate immersion in an object, and sound from banks of speakers provide other pathways to information.”

He continues, “On a recent visit, I stood in the middle of a giant human brain, constructed from 256 MRI images. We moved over folds and through color-coded lobes, the density of blood flow around us reflected in the pitch of a background thrum – higher pitches for higher densities. I explored the bonds of 2000 zinc, hydrogen and oxygen atoms, an experimental lattice created to explore new solar cells that had taken five supercomputers six months to create. This time, the orchestra was the emission spectrum of electrons jumping to different orbits from the hydrogen. You could look at data like this through columns of numbers, of course, but the use of color, sound and time makes for a vastly quicker and more comprehensive understanding. As we pile up information of all sorts, new tools for understanding like the Allosphere will become commonplace.”

Hardy notes, “Companies like EMC, IBM and HP are meantime creating even more possibilities, through the capture and storage of all sorts of sensor, supply chain, and organizational and human behavioral data. There is an undeniable larger result here. Humans build tools, and in turn tools remake people. With the microscope we became aware of a world of protozoa, airplanes made us creatures that fly, and phones connected our voices around the world. The Web is clearly connecting us in new ways – so are the tools for understanding all the information the networked world brings us. New visualizations will, in time, help us see the world’s patterns and processes, and our relationship to them, entirely anew.”

Read more here.

photo credit: Jon Olav