In two of my recent posts, I discussed the potential impact big data can have on energy management and business, explaining how customer and device data yield results that can provide an organization with a strategic advantage, either in service or offerings, over its competitors. As we now know, big data is the newest and most valuable natural resource to any organization, no matter which industry. This is particularly true in healthcare, where patient data analysis can very likely be a matter of life and death.

Not unlike typical business applications, where the ultimate goal is to provide improved customer service, big data technology is playing a large role in healthcare institutions to analyze enormous volumes of patient data and ensure a higher level of personalized care. Through the use of analytics tools that collect, synthesize and analyze historical and real-time data, physicians and healthcare providers not only obtain a more holistic view of patient health, but are also given the opportunity to monitor patient condition more closely, and conduct in depth research into diseases and drugs more efficiently.

For instance, using big data technologies, providers can now consider a number of factors, including test results, past visits and environmental elements, when addressing specific patient needs and determining a course of care. The Premier healthcare alliance is using a big data platform to gain insights on, measure and improve patient health and safety while also reducing the overuse of procedures, readmissions, unnecessary ER visits and hospital-acquired conditions. Through this work, patients have a greater certainty they will receive the most effective treatment possible and that their care will follow nationwide best practices.

At institutions like UCLA Ronald Reagan Medical Center doctors are using real-time sensor data to prevent and react to sudden changes in condition for patients suffering from traumatic brain injuries (TBIs).  The technology targets rapid rises in brain pressure to predict potentially dangerous changes and alert caregivers. Similarly, at the University of Ontario Institute of Technology in conjunction with Toronto’s Hospital for Sick Children, neonatal intensive care specialists are using data from monitoring equipment to track the condition of premature babies. Doctors and nurses are now, for the first time, able to spot and prevent potentially life-threatening infections up to 24 hours earlier.

Going further, there is also an increase in the use of big data within the medical research community, as it provides researchers and scientists with more insights into unstructured and structured data they wouldn’t have normally been able to access and analyze. This in turn allows them to generate a better understanding of what causes diseases and what can treat or cure them. An example of this is the work currently being undertaken at the State University of New York (SUNY) at Buffalo. Using analytics technology, researchers at SUNY are studying the more than 2,000 genetic and environmental factors that may contribute to multiple sclerosis, a chronic neurological disease that affects approximately 400,000 people in the US. Through big data, the scientists are aggregating and analyzing medical records, lab results, MRI scans and patient surveys to develop algorithms containing genomic datasets to uncover critical factors that speed up the disease’s progression in patients. The insights gained will help doctors better target individual treatments.

Regardless of your organization’s role – energy management, finance, marketing, healthcare – understanding and making the most of available data is essential to success, both on a business level and a personal level.

Data: the plural of datum; individual facts, statistics, or items of information.

Big data: a collection of large, complex data sets varying in type, velocity and veracity.

By now, we’ve all heard the statistics: data will experience a 29 fold increase in volume, reaching 35,000 exabytes, by 2020 (IDC); enterprise data growth over the next five years is estimated to increase by more than 650 percent (Gartner); and data use is expected to grow by as much as 44 times, amounting to some 35.2 zettabytes globally (IDC). This blog post itself represents thousands of bytes of data, adding to the near 2.5 quintillion bytes created everyday by online purchases, text messages, traffic cameras and an endless array of other activities and devices that produce data.

In this new era of “data, data, everywhere,” the key to business success is the ability to make sense of Big Data. But businesses are struggling to better manage and analyze this data and intelligently use those insights to support specific business goals. For example, in a study conducted by Forrester and IBM, 25 percent of IT projects are over budget and 34 percent behind schedule. As data grows more and more complex, traditional data analysis simply can’t keep up – these “one size fits all” systems run more slowly and can lead to possible errors or biased conclusions. A simpler, more integrated approach to managing IT is needed.

Enter expert integrated systems. Driven by the need for a more comprehensive, optimized big data platform, expert integrated systems fundamentally change the IT lifecycle by performing complex analytics on big data quickly and efficiently, thereby reducing costs, saving time and resources and speeding innovation for the enterprise. These systems also provide increased flexibility, integrity, availability and scalability for any transactional workload – from managing large financial sets to seamlessly detecting fraud in real-time. With expert integrated systems, businesses can tackle traditional IT pains by making components work together as one system, combining the flexibility of a general purpose system, the elasticity of the cloud and the simplicity of an appliance tuned to the workload.

But don’t let this description fool you – expert integrated systems are far more than “just an appliance.” These systems can incorporate several major components designed to allow businesses to reduce the high costs and increasing complexity associated with managing information technology:

• “Scale-In” System Design: a new concept in system design that integrates the server, storage, and networking into a highly automated, simple-to-manage machine. Scale-in design provides for increased density at all layers of processing needs, namely compute, storage, and network bandwidth, while providing for ease of administration, upgrades, and maintenance. It provides a way to allow applications to maximize the data used in analysis as opposed to worrying about data placement and transformations.
• Patterns of Expertise: proven best practices and experiences that are captured into repeatable, automated form, to allow organizations to automatically deploy, manage and optimize systems by configuring, mixing and matching IT resources.
• Cloud Ready integration: built for the cloud, enabling corporations to quickly create private, self-service cloud offerings that can elastically scale based on demand, and help optimize resource usage, as needed.
• Clean-slate designs for optimal performance: allow organizations to optimize and innovate in the internal design of each new integrated solution, improving performance, scalability and resiliency, and eliminating the dependence on and use of antiquated architecture.
• Integrated management for maximum administrator productivity: incorporated unified management tooling and expertise patterns to enable low lifecycle cost of ownership and high administrator productivity. As workload-optimized systems, these solutions embed integrated expertise patterns that help automate and optimize the work of human administrators.

Keeping these principles in mind, managing big data and deploying the IT infrastructure to support it, should no longer be a cumbersome process. In this new era, simple, easy-to-use tools and platforms can help organizations make sense of the ever growing data deluge. For instance, The New York Stock Exchange (NYSE) is applying the expert integrated system approach to store and analyze seven years of historical trading data systems and identify and investigate trading anomalies faster and easier, translating to nearly one terabyte of data per day. As a result, the company has been able to improve simplicity and performance, in turn cutting data analysis time by eight hours.

Organizations today have sophisticated big data challenges that require a systems approach that can provide specific big data analytic workloads depending on the business requirement. Alternative “one-size-fits-all” approaches of applying the same analytics to different challenges is a flawed strategy that will not yield the level of insight needed to make better, faster, more accurate decisions to gain a competitive edge, and sustain success.  Expert integrated systems are the future of IT, helping integrate operations and strive towards one common business goal.

In 2012, big data came to a head as its value in improving business operations were realized, and a shift began with applying an increased use of analytics to understand, interact with and serve customers.

Yet, this is a new era – one that’s completely changing the type of information businesses collect and analyze to get closer to customers and impact business outcomes. We’re gathering data from everywhere, from purchase transaction records to cell phone GPS signals. Companies are even using analytics on social media sites in an effort to interpret human behavior, such as the presence of sarcasm in people’s posts, as well as positive and negative expressions.

Additionally, big data is driving new opportunities. There is a growing demand for people with the skills and knowledge to manage and understand new data. In 2012, new big data analytics university curriculums launched and students began taking courses to learn data management and analytics in order to prepare for careers in these capacities.  Data scientists and executives began re-thinking how they interpret information and draw on new insights.

In 2013, big data will continue to grow in relevance and expand across organizations in literally every industry. IBM’s 2012 Tech Trends Survey of over 1,200 IT and business decision makers found that 54 percent of respondents have adopted business analytics technologies with 55 percent planning to further increase analytics spending to enhance their competitiveness.

Based on the rise of big data in 2012, I have a few projections for 2013:

Big data will span uncharted territories

We will begin to see expanded use of big data in industries such as healthcare, retail, and energy. The healthcare industry will see a significant increase in big data usage to meet the needs of personalized medicine. In the retail space, big data, especially from social media sources, will play a huge part in understanding consumer buying habits and sales insights.  And the energy industry will continue to launch smart grids and use new big data and analytics technologies to create more efficient renewable energy strategies.

Big data will extend beyond IT departments

We know there’s enough data to go around; in fact, IDC estimates the big data market will reach $50.7 billion by 2016. Because of this projected growth, the IT department alone cannot tackle big data. Everyone inside an organization will lend a hand in interpreting it.

Just as marketing departments began to employ big data to better connect with customers in 2012, big data will begin to expand into new lines of business in 2013. We expect to see it in spaces such as procurement, where analytics will be used to gain insights on supplier risk and performance, as well as invoice data compared to contract terms. We will also use big data in finance, to more efficiently collect and cross-reference financial data sets against analyst reports, economic market data, financial reports, news stories, board notes and company balance sheet. And the human resources team will use big data as a way to gain predictive insights on marketplace developments and employee preferences, as well as employee retention rates and risk management.

Investments in IT will shift

As the use of big data expands across all industries and departments, more specialized analytics technologies will emerge, such as the appliance approach for specialized analytics workloads. This strategy will allow business managers to more quickly identify business value from big data, which will help alleviate the siloed approach that has existed too long in businesses. This will allow lines of business departments to share and compare data and identify additional insights not previously known.

The relational database will shift

The relational database is now being put into appliances, mobile devices and the cloud. In fact, Gartner estimates that, by 2016, 50 percent of data will be stored in the cloud.  Traditional database technologies are evolving to perform large-scale transactions previously only possible in a mainframe environment. Operational data management is also being performed in new environments such as connected networks and smart grids.

The ‘data scientist’ will play many roles

As big data continues to expand in 2013, the role of the data scientist will continue to evolve as well. Our next generation workers have already begun taking classes and experimenting with big data and analytics in 2012.

In 2013, the industry will continue to push for data scientists that are trained to work across different disciplines. The new data scientist may stem from this next generation of workers being trained for the new cross-discipline approach in college. Or, today’s current employees (an application developer or a database administrators, someone who already possess mathematics, statistical, and computer science skills) may partake in training to better understand various business departments, for example marketing, or procurement. The new data scientist will have a keen eye and ear to link IT and business to help the organization better understand and tackle big data.

Looking forward a few years, big data is certainly on a track to continue its growth.  In fact, the U.S. Bureau of Labor Statistics predicts there will be a 24 percent increase in demand for professionals with management analysis skills over the next eight years. The continued development and growth of big data will prove to be a great contribution to businesses world-wide. Technology advancements are creating new job opportunities for more creative and forward thinking data scientists, and moving forward, organizations will continue to leverage the insights found from big data to deliver more satisfying customer experiences.

The volume, velocity and variety of data generated by our smarter planet has created a data deluge for organizations in every industry. IBM estimates that, every day, we create 2.5 exabytes of data — so much that 90 percent of the data in the world today has been created in the last two years alone.

The energy and utilities industry had been relatively stable for the past 100 years, operating in a predictable, linear way with reliable service, in spite of population growth and geographical expansion.  Yet, these organizations are facing increasing pressure from consumers, businesses, and governments to provide new ways to increase energy efficiency.

Because of the change in the status quo, many energy and utility companies have realized they cannot continue to operate in the same fashion, and have rapidly adopted new technologies to help meet these changing demands.

The new technologies, including smart meters and smart grids, are providing companies with new capabilities, including the ability to detect power failures and blown transformers and automatically shut off other transformers to limit further damage. The list of benefits are astounding as well, including the ability to forecast demand, determine customer usage patterns, optimize unit commitment and more.

According to a study done by IBM in 2011, the next generation of smart grid technology is poised to grow from $4.9 billion in 2011 to $43.3 billion in 2020. And by 2015, it is expected that more than 300 million smart meters will be deployed worldwide.

While smart grids are poised for tremendous growth, these technologies are generating unprecedented volume, speed and complexity of data, capturing data from billions of data measurement points, including networking devices, transmission sensors, power lines and generation plants, causing many utilities to drown in the data deluge.

Harnessing the Big Data wave
Let’s consider how energy and utility companies are becoming more data driven and how building a solid foundation to manage, analyze, and use this information will pay off for the utility companies, their customers and communities.

Data gathered from smart meters can provide better understanding of customer segmentation, behavior and how pricing influences usage—if companies have the capability to use that data. For example, time-of-use pricing encourages cost-savvy retail customers to run their washing machines, dryers and dishwashers at off-peak times. These customers not only save money but also require less generation capacity from their energy company, which means lower capital outlay for new generation and overall greater operational efficiency for utilities.

But the possibilities don’t end there. With the additional information available from smart meters and smart grids, it is possible to transform the network and dramatically improve the efficiency of electrical generation and scheduling.

For example, Oncor, the largest regulated electric distribution and transmission company in Texas, is working with IBM to deliver a smarter power grid to help ensure efficient electricity delivery and enable 3 million Texan households to play a more active role in helping conserve energy. The new intelligent system provides Oncor with pinpoint access and insight into billions of data measurement points, from smart meters and networking devices, to transmission sensors, power lines and generation plants.

To date, many Texan households have noted a 10 percent decrease in energy usage since the implementation.

Analytics Managed Data
Using the meter and grid data as a base, various types of analytics can be applied to better understand a number of things, including:

• How pricing changes affect changes in demand
• Which customer segments are most likely to respond to requests to reduce power
• Detecting when the power flowing through a substation doesn’t match the input from the meters, which is a likely indicator of energy theft or diversion
• Understanding which portions of the distribution system are being stressed beyond their design points and should require maintenance or upgrades
• Determining where new generation investments should be made

It’s clear how powerful data can be when it is strategically managed, analyzed and used to transform operations, plan infrastructure, or shape consumer usage patterns. To capitalize on these new data opportunities, energy and utility companies are beginning to transform to smarter energy systems that feature a two-way flow of energy and information.

Organizations today operate in a world of increased complexity and an abundance of available information. This makes data – and big data specifically — the newest and most valuable natural resource to organizations, no matter which industry.

The opportunities that big data presents address the highest-priority issues of C-suite decision makers. Every CEO is exploring how big data technologies can help better manage financial performance, improve customer relationships and boost revenues.  However, big data doesn’t only affect the CEO – big data, and the insights that can be gleaned from it, are impacting all departments that make up an organization.

Here are four lines of business that are most affected by big data and are able to identify insights that help the overall organization increase profits, move into new markets, and achieve a competitive advantage.

• Marketing:  In the last six to twelve months, the greatest number of big data use cases for the enterprise is in the marketing department. Chief Marketing Officers (CMOs) are actively gleaning the textual value coming out of streams of data generated by social media, trying to better understand what consumers are talking about. Then, they need to decide what to do with this information. By using real-time analytics, CMOs can actually predict the precise moments to engage customers with the right information or the right suggestion in a personalized, authentic way.

  It is becoming increasingly important for CMOs to become data scientists themselves.  In terms of observing and discovering, they are the subject matter experts and can see and identify patterns that can transform the organization. In fact, by 2017, the head of marketing, not the head of technology, will be the biggest buyer of technology at a typical organization according to Gartner. With big data, the CMO can shape everything from how brands interact with customers, to the products and services they offer, to the structure of the company itself. This influx of data is now holding CMOs accountable for business results tied to technology investments and long-term growth beyond marketing campaigns. Because of this, we’re beginning to see CMOs and CIOs forging new relationships to better reach business goals.

• Finance:  Financial information is growing at an astounding rate of 70 percent each year, according to IBM research. Intricate data sets have to be collected and cross-referenced against analyst reports, economic market data, financial reports, news stories, board notes and company balance sheets. Big data analytics allows for increased insight, visibility and control over financial performance with predictive capabilities applied to key metrics and data on past performance. The predictive analytics, combined with what-if analysis, and traditional business intelligence in an executive-style dashboard, guides users with root-cause analyses. Organizations can uncover relationships among performance metrics, anticipate performance gaps and assess alternatives with scenario planning.

• Human Resources:  Big data is becoming a vital tool to modern human resources departments. By embracing the potential of big data, HR professionals can benefit from predictive insights on marketplace developments, employee retention and risk management to enhance the strategic value and improve the overall efficiency of the HR department.  Knowing how to collect, analyze and interpret large volumes of semi structured and unstructured data allows for better decision-making, as well as helps HR directors stay informed of competitors’ global hiring and firing activities. It also allows them to anticipate and react to the competitive poaching of top talent.

  In addition, big data can allow the organization to assess the impact of its HR strategy by analyzing employee attrition rates, and benchmarking compensation levels with competitors, which is made possible by collecting and analyzing data from an unlimited number of web sources. The feedback from this data can also identify training needs to meet the current and future requirements of the organization.

• Sales: According to a recent report from Lattice Engines and CSO, 71 percent of organizations expect big data to have a significant impact on their sales, while only 16 percent have big data strategies in place for sales. In addition, more than 89 percent of executives believe that sales reps miss opportunities because they cannot keep up with the available information about customers and prospects.

  Big data represents a huge opportunity for sales departments. The amount of available information on prospects can provide the sales team with insights on which clients to target and how to maximize value for their customers. In addition, predictive analytics can offer insights into what customers will buy next, allowing the organization to always be one step ahead.

Big data permeates through an organization from the C-suite to all line of business.  Two main keys to success will be: 1) to take an integrated big data strategy approach to ensure a holistic view of data cross-departments, and 2) ensure line of business workers are able to derive insights from the data they are seeing. This will help uncover new opportunities and provide the ability to make informed decisions to fuel business success.

The onset of big data has created an evolution that is shifting how organizations manage their data. Where the relational database once ruled as the primary data management solution, the era of big data is changing the data that business look at, the people working with it, and the technology and skills needed to manage and understand it. Most importantly, it presents new opportunities for businesses that are able to gain valuable insights from this data to inform their decisions.

In today’s business environment, it is ’all hands on deck’ to drive business outcomes, therefore there is a growing need for everyone within an organization to understand and apply big data.  Students are currently taking data management and analytics courses to prepare for the work force, data scientists are learning which questions to ask of the data and chief marketing officers (CMOs) are learning to make strategic business decisions based on new insights.

Data that used to reside solely in the IT department is now being discovered by the C-suite as providing additional business possibilities, and this trend will continue as more companies look for new ways to transform within their industries and redefine their success metrics.

With this new era of big data management, it is becoming imperative for all organizations to develop a program to not only capture all of this available data, but pull out valuable insights that can reduce costs, manage risk, and better serve customers.

Here are five key steps businesses in any industry can take to get started with big data today:

• Step 1:  Define the challenge and understand the opportunity that big data presents for the organization. Enterprises have so much data at their fingertips – from online transactions to videos to social media data. Being able to comb through this data and pull out patterns to gain insights and make critical decisions will give them a true competitive advantage. Organizations need to define these opportunities, and establish the larger end goal – whether it is cost savings, increased ROI, or reduced risk – to first begin to establish a big data program.

• Step 2: Discover the data that needs to be analyzed and where it is located. Depending on the industry, different types of data are more critical to the organization than others. These various types of data need to be located and analyzed in order to obtain critical insights. For example, Pacific Northwest National Laboratory (PNNL) must analyze terabytes of smart grid and climate data, while simultaneously detecting error and cyber security threats. On the other hand, Trident Marketing is increasing retention by looking at an array of customer data as well as social media data to help CMOs get closer to customers. I recommend a federated approach to big data, taking the analytics where the data resides. It’s faster and more cost effective than stuffing all the data inside a data warehouse.

• Step 3: Obtain the business buy-in. As big data is no longer just an IT issue and affects the entire business as a whole, it is necessary to start with the C-suite and get them to apply big data to growth opportunities. Show them the larger implications that their data presents for the organization. Map out potential savings and increased ROI that will impact the bottom line and future growth of the business. This is the secret to getting buy-in.

• Step 4: Establish the right technology. A critical step in the development of a big data program is investing in a big data platform that is scalable and can take information from any data source. This technology should provide a single view of the data by taking disparate systems and bringing them together, simplifying access to trusted data and keeping information out of silos.  Companies need the core analytics capabilities to be able to ask the right questions of their data and build a strong foundation for their program. The platform can be reused for future growth.

• Step 5: Ensure the right team and skills are in place. Having people with the right skills is equally as important as having the right technology. Building out a data scientist role or data science team will foster collaboration among the organization by working directly with the CMO or CIO to advise them on how to derive the maximum business value from their organization’s data. Skills must be a mix of business (or domain) and technical expertise. Aim for a balance in these skills.

Once the big data program has been implemented, the organization can then leverage the program for new opportunities and look for ways to expand it. It is important to monitor progress consistently to examine areas of success, as well as areas that need improvement in order to remain competitive. In an era where data management is constantly evolving, setting the foundation for big data today will help ensure an organization will thrive and produce actionable results in the future.

Every day, we create 2.5 quintillion bytes of data–so much that 90% of the data in the world today has been created in the last two years alone. This data comes from both humans and machines: from sensors used to gather climate information, posts to social media sites, digital pictures and videos posted online, transaction records of online purchases, and from cell phone GPS signals to name a few.

Today, organizations are realizing that establishing a competitive advantage depends on acquiring the relevant data that your organization can harness plus reduce the time to gain meaningful insights. Data needs to be properly analyzed to uncover the trends and insights that can impact customer engagement and improve business performance.

Companies who recognize the opportunities big data presents are focused on hiring talented data scientists who can sift through the data available today, uncover the actionable information, and provide strategic advice on how best to use that insight.

Universities are increasingly launching much-needed curriculums rooted in big data to develop students with skills in big data technology across disciplines such as marketing and engineering in order to have the right mix of business and data science understanding. In the meantime, an immediate step organizations need to take is to start to look from within to take this cross-training approach in order to remain competitive.

While there are various departments and job types such as application developers, database administrators and IT managers who possess mathematics, statistical, and even computer science skills which can serve as a basis for expanding into data science, a logical first place to focus is the marketing department. CMOs and marketing professionals possess the right combination of analytical minds that are used to crunch data on consumer trends, demographics, and sales ROI. They also have the business understanding of the domain that they are analyzing, and can move beyond the numbers to help the business as a whole.

Additionally, over the past year, the greatest number of big data use cases are happening in the marketing department. They are gleaning business value coming out of streams of text generated by social media, and trying to find out what people are talking about, and what decisions they make with that information. By using analytics, marketers can actually predict the precise moments to engage customers with the right information or the right suggestion in a personalized, authentic way.

It is becoming increasingly important for those people who may be involved in business or marketing functions to become the data scientists themselves.  In terms of observing and discovering, they are the subject matter experts and can see and identify patterns that can transform the organization. In fact, by 2017, the head of marketing, not the head of technology, will be the biggest buyer of technology at a typical organization according to Gartner. With big data, the CMO can shape everything from how brands interact with customers, to the products and services they offer, to the structure of the company itself. This influx of data is now holding CMOs accountable for business results tied to technology investments and long-term growth beyond marketing campaigns.

It is becoming imperative for organizations to speed up the process of acquiring big data skills by looking within, particularly at their CMOs and marketing department. Those with a marketing background are naturally suited for a data science role based on increasingly connected customers driving businesses to rethink how they make decisions and engage with their customers.  If we can make the CMO a data scientist, then we have achieved what we needed to, and that’s when the potential of big data will actually be realized – getting the business outcome out of tapping into what’s hidden in this big data.

As big data continues to explode, the conversations around it explore how to turn this new natural resource into a significant opportunity. I’m very excited to be a part of a big data discussion panel at the upcoming Enterprise Data World in Atlanta, April 29^th – May 3^rd, where we’ll dig a bit deeper and take a look at the current big data landscape and what lies ahead.

In my conversations with executives, big data is on everyone’s mind – and not just CIOs, but CEOS, venture capitalists, CFOS, CMOs, and risk managers. They are all faced with an increasing need to see deeper within their organizations to gain the insights required to make difficult, yet effective business decisions. I see big data creating and impacting business opportunities everyday now, and know the impact will only continue to grow.

The panel at Enterprise Data World will discuss the big data facts, fantasies and understand why the topic is so relevant today. The session will also address the changes coming to enterprise data management and also the new skills needed to take advantage of this wave of opportunity that is big data, a topic I’m very passionate about. Here at IBM, we’re helping to define the role of the ‘Data Scientist’ – a role that is always growing and changing. We are working to ensure there are academic initiatives combining computer engineering, mathematics, statistics and analytics classes and also the right technology tools out there to compliment these programs.

Not only will it be important for new graduates moving into the big data space to help us uncover strategic information for better decision making, we believe they will also help us to build bigger and better solutions that will help us extract data and continue to build strategic big data resources that will help us create a Smarter Planet.

I’m looking forward to sitting down with Robin Bloor of The Bloor Group, Paul Pedersen of 10gen, Neil Raden of Constellation Research, and April Reeve of EMC Consulting to discuss some critical questions about big data including, how big will big data get? What are CEOs saying about big data? What can you do to future-proof your career in the big data era? What are the key technologies we need to educate ourselves about? And, what is the future of Little Data technologies?

For more information on our panel, please visit here.

I hope to see you there!

The 2.5 quintillion bytes of “Big Data” we produce every day is everywhere – in enterprises, SMBs, in the cloud, on desktops, on mobile phones, and more.  It’s also in every industry and touches all layers of a business, from the C-suite to entry-level. And the growth of this data isn’t slowing down. IDC predicts total data volume will reach 35,000 exabytes in 2020, compared to 1,200 exabytes in 2010, representing a 29 fold increase in the next ten years. The key to business success is having the skills and ability to capture, sift through, and analyze these vast amounts of data, presenting an enormous opportunity for companies to improve customer relationships, provide better services and generate insight to drive new business opportunities.

The Big Data Skills Need

Companies worldwide are realizing the enormous potential translating big data can provide, but business and governments alike are grappling with the challenge of making sense of this data deluge to turn it into better business decisions. In fact, Forrester estimates firms effectively utilize less than 5 percent of available data – and miss 95 percent of data rich with insightful information – mainly due to the lack of training and skills necessary for the type of information gathering and analysis needed to transform this big data. .

Recognizing this need, companies and educational institutions are charging full-speed ahead in the effort to teach big data skills. Analytics and data-handling degree programs have popped up at colleges and university across the country, including Northwestern, Yale and Fordham, and even globally. Also, companies such as IBM are offering online training courses through their Academic Initiative, BigDataUniversity and developerWorks. These educational opportunities will provide students with the analytics skills needed for their upcoming careers, current employees with the opportunity to advance their skills, and everyone with the tools to become the new species of data handler primed to tackle the data obstacle.

According to the U.S. Bureau of Labor Statistics, there will be a 24 percent increase in demand for professionals with management analysis skills over the next 8 years. By garnering the big data skills today, businesses will have the skilled employees they need to make better business decisions for the future.

As we entered 2011, organizations grappled with how they could manage and benefit from the large amounts of data created daily, exceeding 2.5 quintillion bytes. This data came from internal and external sources such as sensors, social media, digital images, videos, transaction records, and cell phone GPS.

To tackle this challenge, CIOs and CTOs partnered with IT vendors to extend their information platforms and build Big Data analytics solutions. Big Data solutions were infused into new and existing IT infrastructures with services that help organizations assess their cloud computing and data center needs, as well as server, storage and security requirements. Additionally, Big Data technology allowed enterprises to analyze any up to tens of petabytes of data without altering its native format.

Big Data solutions were deployed across a wide set of industries this year, enabling new actionable insights which resulted in:

• Better profiling of customer preferences and habits;
• Focused loyalty programs based on deriving customer insights;
• Leveraged buyer intent to give customers more tailored products and services based on their wants and needs;
• Improved operational efficiency and utilization of infrastructure; and
• Better risk management and predictive abilities.

In 2012, I predict Big Data solutions will expand beyond analytics and into transactional usage. Organizations will institute a comprehensive data discovery practice to help enterprises embrace Big Data to derive such insights. Over the next year, it will be critical for organizations to craft their Big Data solutions to not only manage and analyze the given information, but to also transform it into actionable goals that will benefit their businesses.

Other trends I will follow in 2012 include:

• Enterprises will facilitate multiple types of discovery that will feed into transactional and analytic applications, enabling real-time business intelligence and action;
• Customer adoption of Big Data technologies will increase to tackle the challenges associated with analyzing the large volumes of data;
• The need to track and verify the veracity of the derived information and understand the lineage will be critical; and
• Interpreting consumer sentiment with a high degree of accuracy will require understanding sarcasm, positive and negative expressions.

Finally, the focus of data will also shift from simply gathering the data available, to taking the insights learned by the data and using it to better understand the customer. Organizations will be able to provide their customers with exactly what they want and need, and increase their customer loyalty. The combination of advanced analytics and associated governance of Big Data will allow businesses to make well-informed decisions for corporate and competitive advantage.