Let’s face it: the whole NoSQL/Big Data thing has traditional data management professionals (in whose ranks I count myself) at best concerned, at worst terrified.  Several of our most closely held shibboleths are being sundered.  Where’s the data model?  Where are the keys, the constraints?  How do I enforce data quality? How does it conform to ACID?  Who are these people saying the relational database is dead?

I’ve been paying a modicum of attention to the NoSQL discussions over the last few years, trying to wrap my head around what it means to us.  My education continued at Enterprise Data World this year, starting with another excellent presentation by NoSQL evangelist and co-founder of the NoSQL Now! Conference, Dan McCreary (an older version of one of his PowerPoints can be found on his web site), followed by some other presentations and panels where things like Hadoop and Mongo and Cloudera got increasingly demystified for me (and, in one lightning session, hilariously lampooned by Karen Lopez, which she shared here).  After a few years of learning and thinking about this stuff, including multiple listenings to Dan’s expositions, I finally had my ah-ha moment — or moments.

We can all relax.  First of all, relational databases aren’t going away.  The fact is that systems built on relational databases and those on NoSQL are working in two different problem spaces and will likely coexist for a long time.  And as for relations and constraints and data quality, my really big ah-ha was that in the NoSQL/Big Data world it is all about quantity, not quality, and about speed.  The role of the NoSQL technologies is to absorb large quantities of data rapidly without any preconception as to its structure and without any notions as to its quality (since NoSQL systems are usually recipients of data originating elsewhere and merely reflect the quality of the originating source).  The reason quality doesn’t matter?  It’s because the data will be used to perform aggregate analysis, bunching data together to look, for example, at patterns of an individual (consumer, patient, web surfer, social networker) or a large cohort, in the context of which bad data can either be tossed aside (if you can detect that it looks suspicious) or will be statistically unimportant when included in large aggregations.  Volume, speed (both of data acquisition and retrieval, because the technologies are built for parallel processing and everything gets indexed), and schema neutrality (which I think is a better term than the misleading “schema-less”) are the operative concepts.

By contrast, we will continue to have systems in which you need to be very certain that a row of data is valid at the moment it is inserted into the database or else you will have highly negative consequences; one could conjure an almost infinite number of scenarios where this is important.  For such applications, the relational paradigm with its structure and constraints – its beneficial rigidity, we could say — will continue to be a feasible approach.

So with my newfound comfort that the NoSQL technologies are complementing, not necessarily supplanting, our traditional systems, it got me to thinking about applications of the technology within our enterprise.  One place with potentially huge benefits would be data warehousing, for a couple of reasons: integration of new data would not require that all the rigid schema and ETL be pre-specified for the warehouse, which is probably the biggest time and money sump in warehousing today (a point Geoff Malafsky, founder of Phasic Systems, has been hammering home at conferences and in online discussions), and because of their relatively easily scaled performance and, it should be noted, easy port to the cloud.

But NoSQL systems from what I’ve gleaned are not terribly efficient forms of data storage.  In a relational system we define the metadata once, in the DDL, and the rest is all data which the system understands to conform to the structure defined in the metadata.  In NoSQL systems, the metadata are described alongside almost each and every datum.  For systems storing large objects like photographs, documents, or videos, the volume of data greatly exceeds the metadata, but it is perfectly possible to have systems where the volume of metadata dwarfs the actual data.  In the era of cheap massive storage maybe this is not such an issue, but try telling that to the guys managing the storage arrays at my company.

The other gray area for me, especially in corporate data warehousing, is the important problem of how we capture and describe the business metadata of a NoSQL system.  This is something we’ve gotten pretty good at over the last 15 years when we start with our logical and physical model representations and attach definitions to each entity or attribute.  In a paradigm in which you effectively acquire the data first and describe it later, how will that work?  Please post a comment if you have thoughts on this.  It will be the next phase of my NoSQL education… now that I’ve gotten past the anxieties.

 To view the recording of this webinar, click HERE.

About this Presentation

If you have ever used passwords such as “123456” or simply “password” then you need to attend this webinar!  Our presentation provides you with an overview of the organizational data security management requirements that are necessary to meet industry benchmarks.  Participants will understand the requirements for planning, developing, and executing security policies and procedures to provide proper authentication, authorization, access, and auditing of data and information assets.  By the end of our session, you will understand how effective data security policies and procedures ensure that the right people can use and update data in the right way, as well as the importance of restricting inappropriate access.

Learning Objectives:

• Top data security concerns & requirements
• The cost of not having accurate security
• Data security statistics & examples of security breaches
• Passwords & policy examples
• Data security management framework
• Guiding principles & best practices

About the Speaker

Peter Aiken is an award-winning, internationally recognized thought leader in the area of organizational data management, architecture, and engineering. As a practicing data manager, consultant, author and researcher, he has been actively performing and studying these areas for more than 30 years. His sixth book is titled XML in Data Management. He has held leadership positions with the US Department of Defense and consulted with more than 50 organizations in 20 different counties. Dr. Aiken’s achievements have resulted in recognition in Outstanding Intellectuals of the 21st Century and bibliographic entries in Who’s Who of Emerging Leaders in America, Who’s Who in Science and Engineering, and other recognitions. His entertaining but clear and concise insights make him a sought after speaker, lecturer and consultant. He is an Associate Professor in Virginia Commonwealth University’s Information Systems Department and the Founding Director of datablueprint.com.

This presentation is brought to you in collaboration with:

Tony Baer recently wrote an article for ZDnet regarding the current shortage of Data Scientists and how the laws of supply and demand might soon kick in. He writes, “It’s no secret that rocket… err … data scientists are in short supply. The explosion of data and the corresponding explosion of tools, and the knock-on impacts of Moore’s and Metcalfe’s laws, is that there is more data, more connections, and more technology to process it than ever. At last year’s Hadoop World, there was a feeding frenzy for data scientists, which only barely dwarfed demand for the more technically oriented data architects.”

He continues, “Behind all the noise for Hadoop programmers, there’s a similar, but quieter desperate rush to recruit data scientists. While some data scientists call data scientist a buzzword, the need is real. However, data science will be a tougher number to crack. It’s all about connecting the dots, not as easy as it sounds. The V’s of big data – volume, variety, velocity, and value — require someone who discovers insights from data… The variety of Big Data – in form and in sources – introduces an element of the unknown. Deciphering Big Data requires a mix of investigative savvy, communications skills, creativity/artistry, and the ability to think counter-intuitively. And don’t forget it all comes atop a foundation of a solid statistical and machine learning background plus technical knowledge of the tools and programming languages of the trade. Sometimes it seems like we’re looking for Albert Einstein or somebody smarter.”

Read more here.

Kim Nash of Network World recently discussed the Big Data held by Equifax. She writes, “As one of three credit bureaus in the United States, Equifax keeps financial data on every adult in America, plus people in 16 other countries. But the company knows much more than just what goes into an old-fashioned credit score. It maintains information about people who share the same phone number or address, ‘non-obvious’ relationships between individuals, loans for dental work, magazine subscriptions, rental history, real estate assets, investment wealth, retail purchasing, the type of federal tax return someone files, marital status, employment, utility payments, cable TV accounts, criminal records, debt-to-income ratios, changes of address, motor vehicle files, post office boxes, inferences about someone’s capacity to pay bills, predictions about someone’s propensity to pay, links to past and potential fraud crimes–and more.”

Nash continues, “This pile of more than 800 billion records is sliced, diced, analyzed and indexed into 26 petabytes of data. That’s more data than the FBI’s Investigative Data Warehouse, said to be the single biggest repository at the agency, with its relatively measly 1 billion unique documents. In all, Equifax has data on 500 million consumers and 81 million businesses worldwide. Says Equifax CIO Dave Webb: ‘We know more about you than you would care for us to know.’ In his wry British way, Webb alludes to the power of information and his push to derive ever more lucrative products and services from Equifax’s vast stores of it. Webb says Equifax can make money off IT innovation–that is, his staff’s ability to manipulate massive amounts of data better and faster than competitors can.”

Read more here.

photo credit: Equifax

Ray Scott of Computerworld UK writes that companies are overlooking the important practice of Test Data Governance. He states, “Try searching some of the most popular career and job sites for Test Data Governance roles; I did and found just 15, none of which were directly associated with my search criteria. Upon closer inspection, these roles only scratched the surface of what is meant by test data governance. Why? It is because test data governance has been traditionally overlooked due to the assumption that test data will somehow appear in development when needed.”

He continues, “As a developer, our seed data has traditionally come from handcrafted master files, locally created transactions or by executing in a localised environment. As testers, data has often been prepared in spreadsheets or included in test case definitions. However, the emergence of additional tasks such as load and stress testing, security and on-demand automation, these methods have become inadequate; the need for data has grown.”

Read more here.

Macy’s is looking for a Marketing Lead of Predictive Analytics in San Francisco, CA. The post states, “As a lead for predictive analytics, you will work with a world-class multidisciplinary team to extract / transform Big Data and deploy scalable predictive reasoning and data mining algorithms. Provide technical expertise and business consultative skills to address supervised and unsupervised learning problems in an applied retail analytics environment. Recommend and deploy modern machine learning and statistical methods (product recommendations, CART and Random Forests, clustering and classification algorithms, Bayesian models, neural nets, SVM etc.) for finding patterns/models from retail web and omnichannel data.”

Qualifications include: “Graduate degree in Computer Science, Applied Statistics or related field with at least three years’ experience preferred. Experience working with large volumes of data,  preferably in retail or e-commerce industry; experience with POS and clickstream data is a plus. Expertise in data extraction & transformation, relational databases & SQL. Strong attention to detail to identify data relationships, trends, and anomalies. Ability to apply theoretical or experiential knowledge on machine learning to solve large scale retail problems. Substantial experience working in a Unix/Linux server environment and competency in shell scripting. Experience in Big Data sets, cloud based architectures, and deployment frameworks for machine learning algorithms (e.g. Hadoop, Hive, Hbase, Mahout, etc.).”

Learn more and apply here.

photo credit: Macy’s

Walt Disney is looking for a Manager of Data Governance in Burbank, CA. According to the post, “The Manager will develop and institutionalize data collection standards and policies, and work to ensure that data collection initiatives, both internal and external, are consistent with enterprise-wide customer information priorities, strategies and policies. With extensive experience managing data collection and CRM development initiatives within large-scale consolidated customer information environments, the manager will collaborate with internal businesses on their data collection objectives and plans, providing insights on data collection and data use best practices. The Manager will play a critical role in establishing data standards; coordinating with legal, technical and operations teams to develop and implement business processes that enable high-quality data collection, and managing these processes through hands-on engagement with CRM tools, tool administrators and users.”

Qualifications include the following: “5+ years experience in database marketing and CRM. Excellent relationship skills with demonstrated ability to work successfully across levels and functions in leadership role. Basic understanding of consumer privacy practices. Excellent communication skills with proven ability to craft and deliver presentations / communications that convey key findings to business constituents. Experience leading projects with cross-functional teams. Deep experience implementing database marketing, digital CRM or other direct-to-consumer marketing initiatives with strong reliance on customer data, with significant experience developing and implementing marketing reporting & performance tracking programs. CRM operations experience within complex, large scale, customer data environments, with specific experience defining and/or overseeing data governance practices / standards.”

Learn more and apply here.

photo credit: Walt Disney

As the Georgia spring turned from April into May, over 600 data professionals converged on Atlanta for the 2012 edition of Enterprise Data World. This year’s conference was highlighted by the keynote address of Internal Revenue Service CTO, Terry Milholland, as well as a celebration marking the 30th anniversary of John Zachman’s seminal framework for Enterprise Architecture.

Over five days, EDW 2012 featured a host of sessions led by industry professionals covering the latest topics of interest in the data professional world, including Big Data, NoSQL, Data Governance, Master Data Management, and Enterprise Information Architecture. All sessions belonged to one of ten different tracks, making it easy for attendees to focus on a particular area of interest.

The conference exhibit hall offered nearly 30 companies and organizations the opportunity to present their products (and swag!) to an interested audience. Hors d’oeurves, including the alluringly decadent meatloaf-on-a-stick, along with other libations and delicacies helped to make the exhibit hall reception a memorable part of EDW 2012.

What sets EDW apart from many other data management conferences attended by this reporter is the tangible sense of community among the participants.  DAMA President, Dr. Peter Aiken commented on this, “as a result of these gatherings we are ensuring the organizations will have access to the knowledge and talent required to meet their individual data challenges. Together we are elevating the data management practice, empowering individual data managers, and enabling organizations to make use of their most valuable organizational asset.”

DATAVERSITY™ founder and conference co-host, Tony Shaw, remarked on the attendees’ hopes for the conference:

“The top three reasons that attendees tell us they come to Enterprise Data World are, one, to stay current with the important topics in enterprise data management, two, to learn about and evaluate new trends and technologies, and three, to learn best practices from their peers.  With those objectives in mind, the hot topics this year were definitely Big Data, Data Governance, and the new non-relational (NoSQL) technologies.”

Atlanta, especially the area surrounding the EDW-hosting Omni CNN Center, provided a plethora of opportunities for leisure during what little downtime was available between conference activities. Attendees sampled the local cuisine, as well as nearby attractions like the Georgia Aquarium or the World of Coca-Cola.

EDW 2012′s Memorable Keynotes and Speeches

Terry Milholland’s decade-long career in IT management may end up being known for his current role as CTO for the Internal Revenue Service. His EDW 2012 opening keynote speech focused on his work for the IRS highlighted by two major accomplishments:

First, after decades ranking at Level 1, the bottom of the Capability Maturity Model scale (CMMI), the IRS is poised to reach Level 3 later in 2012.  The significance of this accomplishment cannot be understated. Additionally, after 50 years of using an archaic tape-based system for taxpayer records, the service looks to rollout an enterprise database model for their taxpayer system later this year.

Combine these accomplishments with a move from weekly to daily processing, and it is easy to see why the Government Accounting Office awarded the IRS with Best Practice awards in nine different success factors. Milholland’s keynote emphasized the data-centric nature of his successful approach and thus was perfect for Enterprise Data World.

Maureen Clarry’s closing keynote, “Influencing Successful Data Projects,” fit nicely with Milholland’s; in fact, she referenced the opening keynote multiple times in her speech. Clarry, an organizational transformation expert, focused her talk on giving EDW attendees methods for implementing what they learned at the conference in their own organizations.

Clarry feels data management success relies more on overcoming obstacles or “potholes on the journey,” as opposed to a mere reliance on quality ideas and plans. Her talk, rich in practical tactics and lessons, served as a relevant closing to the entire conference.

Sandwiched in between the two keynote addresses were a host of memorable talks, not including the celebration for the 30th anniversary of Zachman Framework, covered later.  Business technology thought leader, John Ladley gave a humorous preview of EDW’s exhibit reception, covering most of the companies with booths in the exhibit hall.

Enterprise Data World’s Lightning Talks deserve a special mention. Essentially a quick-hitting series of five minute speeches by data industry leaders with DATAVERSITY founder, Tony Shaw, serving as moderator, the Lightning Talks served to both entertain and enlighten.

The various talks ran the gamut from Master Data Management to writing and publishing one’s own book on data. Some speakers used the opportunity to demo their organization’s latest product, while others used it to provide a measure of humor and levity to the proceedings, especially InfoAdvisor’s Karen Lopez and her rumination on Big Data, “Size Doesn’t Matter.”

“The Big Panel on Big Data”

Wednesday afternoon at EDW 2012 featured “The Big Panel on Big Data,” a moderated discussion highlighted by a collection of heavy hitters in the data industry, including IBM’s Anjul Bhambhri and Informatica’s John Haddad among others. The panel featured the shock-radio style moderation of Eric Kavanagh. This sometimes humorous approach made sense, considering Eric’s role as a host for the Bloor Group’s DM Radio program.

Probing questions on the main role of Big Data in today’s IT world triggered a thoughtful response from Constellation Research’s Neil Raden. Neil hoped that the uses of Big Data would go beyond mere commerce and provide benefits in the world of medicine and health care. In short, Big Data should “do something good for humanity.” Anjul Bhambhri echoed this sentiment, remarking that the knowledge gleaned from Big Data has the potential for improving all of our lives.

Concerning the set of challenges involved in working with Big Data, many of the experts remarked on the need to grow relevant knowledge in both the business and technical roles.  10gen’s Paul Pederson commented how people are still using old models to solve new problems.  Robin Bloor of the Bloor Group summed up the panel as a whole, stating that Big Data’s biggest challenges are “the same problems we haven’t solved with Little Data.”

Session Highlights at EDW 2012

As mentioned, the sessions at this year’s Enterprise Data World each followed one of ten tracks, some of which were sponsored by EDW’s corporate supporters, including Informatica, Adaptive, and Global IDs.  Included with the normal sessions were a Monday filled with more in-depth tutorials, along with an afternoon of seminars on Thursday.

Master Data expert, David Loshin, gave an excellent introductory overview on Master Data Management, a rapidly maturing discipline in the world of data professionals. David emphasized the point that MDM is a “set of disciplines, not a shrink-wrapped package.” He also stressed the need for Data Governance as part of MDM, “If a standard falls in a forest and nobody follows it, it’s not a standard.”

Joseph Shaffner from Sybase/SAP talked about his company’s latest products in the world of Big Data. SAP Hana is an in-memory database currently sporting query results 355 times faster than disk-based products. Also, their SAP IQ product is a columnar database well suited for data warehouse and analytics applications.

Cambridge Semantics were a notable presence at EDW 2012. Company VP of Marketing, Lee Feigenbaum, demoed their Anzo line of semantic web products useful for combining information on any topic from a wide-array of web-based sources and formats. His baseball-themed demo did a fine job in illustrating Anzo’s capabilities in handling data from the highly-structured to the unstructured.

Cap Gemini’s Jojy Mathew covered his real-world experience with operationalizing Data Governance and Data Quality in a detailed walkthrough rich with tips and tricks. This session was sponsored by Adaptive, who let Jojy speak more on his project successes as opposed to an aggressive marketing push. This proved to be an excellent approach for this kind of session.

Maximizing the value of data using Data Management was the focus of a session by DataFlux’s Brett Dorr. His company is a provider of software and services that support DM. Brett feels that a lack of both leadership and ownership remains one of the reasons companies still need better Data Management practices.

The CLAPR framework for Enterprise Database Architecture was the subject of a session given by industry veterans James O’Brien and Jack Bilig, along with DAMA president, Peter Aiken. CLAPR is most definitely NOT for database modeling; it provides a process for integrating the many aspects of an enterprise to create an integrated set of models based on an extensible framework. The process generally involves iterating through a set of root models, gradually modeling the subtypes at a finer granularity.

The Architecture Tradeoff Analysis Method (ATAM), developed at the Software Engineering Institute (SEI) at Carnegie-Mellon University, helps to determine risk in the planning stages of any software project. Considering the importance of data in most software projects, knowledge of ATAM is useful for most data professionals. Data Management veteran, Robert Abate, profiled ATAM in an excellent session covering the basics of this valuable Enterprise Architecture methodology.

A Celebration for John Zachman

John Zachman is considered by many to be the father of Enterprise Architecture. The EDW special event celebrating the 30th anniversary of his Zachman Framework provided some of the most entertaining and heartwarming moments of the conference.

Peter Aiken shared the dais with John Zachman using a slideshow to present the history of the framework, peppered with insightful and sometimes humorous comments from the person who created it. Over the last 30 years, Zachman continued to refine the framework, and his company, Zachman International, currently provides training and certification for professionals eager to hone their information architecture skills.

Finally, champagne was passed out to all in attendance and a shared toast brought more smiles and even a few tears in celebration of one man’s important role in the world of data and Enterprise Architecture.

DAMA and DATAVERSITY Team Up

Enterprise Data World 2012 was jointly presented by DATAVERSITY and DAMA, the preeminent data management professional organization. DAMA had many officers present at EDW, along with their booth in the exhibit hall. The organization also hosted tests for attendees looking to earn the CDMP and other certifications at the conference.

DAMA’s Data Management Body of Knowledge (DMBOK) is considered to be the Bible of Data Management principles and practices. More than one EDW session covered DMBOK and its associated disciplines.

DAMA’s VP of Conference Services, Cathy Nolan summed the data management organization’s role at EDW 2012, “the partnership between DAMA and DATAVERSITY has resulted in an outstanding educational event, Enterprise Data World. Non-DAMA attendees learn about the benefits of belonging to the largest independent Data Management Organization, while DAMA members network to improve not only their personal skills but their chapters located in 27 countries around the world.”

Make Plans to Attend Enterprise Data World 2013

There is little doubt that Enterprise Data World remains the foremost conference for data professionals all over the world. In addition to the United States, attendees from nearly twenty other countries take part in EDW 2012.

Veteran data professional James O’Brien attended his first EDW this year in Atlanta; his comments nicely sum up this year’s Enterprise Data World, “As a first time attendee and presenter at EDW I was, to put it simply, stunned!  Not only did I get opportunities to discuss matters with such luminaries as John Zachman and David Hay, the quality of the knowledge on the part of the attendees was simply astounding.  Everyone I talked to had in-depth knowledge in their data management area.  Never have I had the opportunity to interact with such world class talent as I did at EDW 2012.  I thoroughly enjoyed myself and look forward to being a regular attendee and presenter at future conferences.”

Jody Westby of Forbes reports that company boards are still clueless when it comes to cybersecurity. She writes, “The Governance of Enterprise Security: CyLab 2012 Report, released today by Carnegie Mellon CyLab, examines how boards of directors and senior management are managing privacy and cyber risks.  Although two previous reports were conducted in 2008 and 2010, this is the first global survey on these issues and the first to compare responses by industry sector.  The cross-sector comparisons in the 2012 report provide a compelling picture that critical infrastructure companies need to put cybersecurity and privacy on their boards’ agendas and place greater emphasis at the executive level on protecting their organizations’ digital assets (data, software programs, and networks).”

She continues, “Seventy-five percent (75%) of the 2012 survey respondents were from critical infrastructure industry sectors, primarily the financial, energy/utilities, IT/telecom, and industrials sectors.  The survey probed whether senior executives and board members were undertaking basic cyber governance activities, such as reviewing privacy and security budgets and top-level policies, establishing key roles and responsibilities for privacy and security, and reviewing security program assessments.  It also asked whether the board was receiving information critical to the management of cyber risks, such as regular reports on breaches and the loss of data.”

Read more here.

Sean Madden of FastCo has written an article regarding how Big Data is improving customer service. Madden writes, “Last month, I talked to Amazon customer service about my malfunctioning Kindle, and it was great. Thirty seconds after putting in a service request on Amazon’s website, my phone rang, and the woman on the other end–let’s call her Barbara–greeted me by name and said, ‘I understand that you have a problem with your Kindle.’ We resolved my problem in under two minutes, we got to skip the part where I carefully spell out my last name and address, and she didn’t try to upsell me on anything.”

He goes on, “Barbara’s congenial but no-nonsense approach was part of what made this experience different, but more important, she had access to exactly the right data about me, and that made the favorable exchange possible. The fact is, Amazon has been collecting my information for years–not just addresses and payment information but the identity of everything I’ve ever bought or even looked at. And while dozens of other companies do that, too, Amazon’s doing something remarkable with theirs. They’re using that data to build our relationship.”

Read more here.