1. Learn about NoSQL database technologies

2. Learn how to correctly reverse engineer application databases

3. Get experience data modeling for integration projects

Not all data modeling results in a brand new database built from scratch.  One of the most common reasons I’ve been doing modeling is for building canonical models for integration projects. These are often implemented as XML schemas, but I start with a logical data model of requirements using very similar techniques as I do for a database design project.

4. Learn a pattern or universal data model

5. Learn more features of your data modeling tool

Like many productivity tools, data modeling tools have evolved a great deal over the last few years.  We architects benefit from a significant number of mature features in our tools.  And like office productivity users, many of us use only 10-25% of those features.  Automation is a key part of many of the leading tools, yet some modelers are reluctant to learn new coding and scripting skills needed to master them.  You should.  The payoff for productivity can be significant after a short learning curve.

6. Learn about the new features of your target DBMSs

7. Build a data modeling process that allows you to produce releases quickly

As little as 15 minutes a day can keep your skills sharp

I deemed last year The Year of Data.  Big Data, Data Analytics, Open Data, Data Breaches, etc. were topics of discussion everywhere, even outside the IT world.  Heck, data management even made it into the US State of the Union Address. I think all this added focus on data has given us a ripe opportunity to help our business organizations to leverage data modeling and other data management approaches even more than ever.  It has also led to my inbox and Twitter stream to be flooded with questions about data modeling.  I see this uptick in inquiries as a good indicator of job stability for data professionals.  At least I hope so.  If anything, I’m hoping it means that the pendulum has swung a bit back to center from “software is everything” mentalities. With all these questions about data model coming in, I thought I’d answer the most common ones here so that I can share them more widely.

Why is having a data model so important?

Ah, the big question. My reasons:

• Technologies used to move and persist data come in many forms, over time and at the same time. I support the creation of XML Documents, multiple DBMSs, multiple versions of the same DBMS. By having a single logical data model of data requirements, I can separate the rules and definitions of the requirements from their implementation.
• The same data exists in many platforms, in many locations.  I need to be able to map those sources to targets across platforms and systems.  Doing this mapping, or data lineage, with a data model is much easier.  It also helps me understand the implications of making a change in one system on another system.  I can’t do that by just looking at code.
• Writing stuff down is often a good way to impress people that have to provide requirements, compliance issues, security requirements, etc. They like not having to answer the same question over and over for different people and roles.  I can’t tell you how many times a business person has asked me if IT ever writes anything down.  Imagine your frustration if you had someone working on your house and ever tradesperson started their day with “Tell me about what sort of house you want? How many people will live here? Do you want to be able go get to the second floor? Do you need a bathroom?  How about a shower?”  That’s how many business professionals see us in IT.
• A data model is a great way of capturing rules, constraints, definitions in a method that is technology independent. I can capture those things once and reference them in many places.
• We can measure databases against our data models to assess fit. This can be done for application packages and custom development.
• Enterprise data is complex.  I work with a data model that has 32,000 objects (tables, columns, datatypes, constraints, etc.) in it.  There is no way I could professionally manage change by just trying to remember all this information.  Nor could anyone else.
• Modeling helps you ask the right questions before a bunch of time is spent coding, creating screens, reports, etc. This reduces costs and the number of bug fixes required.
• Like all models, the data model is a communication tool and is good for tying requirements directly to designs and implementations.  This is especially true when I generate data prototypes based on the model.
• Data governance can’t easily be done via reverse-engineered pictures of databases.
• Once you’ve worked with a great data model, you can’t go back.

Is it worth spending money on consultants to teach us the tool and help us build a model, or should we expect a newly hired data architect to be able to do that for us?

It depends…am I the consultant? If you hire the right kind of architect, he or she should be able to do that sort of training if you budget time for them to do that. Often when I’m hired as the sole architect, I’m swamped with real work and no one has time to be trained nor is there time allocated for me to train anyone.  It also depends on the consultants. If they specialize, really, in data modeling, it may be worth it. If they have casual experience, then maybe not. I want to warn you that hiring a data architect that really is one is difficult. Most interviewees will be DBAs or developers who have once seen a data model or always wanted to get into this line of work. I typically have to review 50 resumes to find a candidate that has real enterprise level experience. If you are going to have only one person filling the data architect role, they can’t be working at the apprentice level unless they have a mentor. Most good data architects actually aren’t good part time DBAs, either. You may be better off trying to find a combination BA/DA.  I wrote about this data architect hiring dilemma previously.

What can we do with a tool like [insert real data modeling tool here] that we can’t do other less expensive tools?

What you can do with real data modeling tools is…modeling. The iterative process of doing logical models, having multiple physical models based on them, making changes to them (either because you are in development or because you have a new requirement) is how projects work in the real world.  Many of the less expensive tools allow you to reverse engineer a database and forward engineering a full database.  But here in the real world of development we make changes to designs and models all day long.  We need to estimate the cost of change, measure the impact of that change and make changes. I need to be able to generate changes to databases without a lot of hand scripting and I can’t drop the database and recreate it on a regular basis.  I have to do that while supporting multiple versions of the same DBMS or multiple types of DBMSs.   The less expensive ones either don’t’ do any of that, or they don’t easily support the iterative process of real life modeling and design. I also need to be able to produce reports, images and interactive versions of the data models. I also need to be able to share the models in formats that can be easily consumed by dozens of other modeling and development tools.   Most of the lesser tools don’t have these shiny features. Many of the real data modeling tools have features geared toward collaborating with other team members.  These features make sharing, updating, commenting on and getting hands-on with the data models easier and faster. It really comes down to finding a tool that helps you reduce costs of development, get that development done faster and getting better quality designs.  The value proposition for that is enormous compared to costs savings of having a tool that only makes nice pictures of one type of database.

What is your question?

Do you have a question about data modeling?  Leave it in the comments or contact me directly @datachick or via the Contact Us menu item above and I’ll blog a response here.

@datachick

Last week at Enterprise Data World, I gave a lightning talk (a strictly-enforced time-limited five minute presentation).  Since I can barely manage to fit meaningful presentations in one hour given how much I talk, I decided to go for a rant.  That may not come as a surprise to you.  If it does, welcome new reader.

Last year I also gave a lightning talk on the Myths of Normalization, which I’ve turned into a blog series here, too.  That was also a rant.  I’m seeing a pattern here.  You should probably envision me holding a martini and my tablet while I ranted.  It will put you in the mood. I didn’t actually hold a martini in my hand.  It was on the table beside me.

Since I got good feedback, I decided to share my script for my talk here. By the way, those of you snickering about this rant, know that I’m working on a similar one for the RDBMS area as we speak.  Look for it at a NoSQL or Big Data event near you.

Size Doesn’t Matter.  Or Does It?

I’m @Datachick. I think a lot about data.  Today I’m on a rant.  I know.  SHOCKING!!!

I’m a huge fan of Big Data and NoSQL. Really.  A really, really big fan.  Get it?  BIG DATA.  Today I want to share with you some of my more snarky observations about BIG DATA.  By the way, every single one of these rants is totally unfair, cherry picked and irreverent.  I know. It’s shocking.

by Karen Lopez @datachick

I posted that Tweet last week while attending the NASA 2013 Fiscal Budget Briefing at NASA Headquarters. It did well, being retweeted 200+ times and had the prospect of reaching 1.9 million people. I say prospect because not everyone who follows someone reads all their tweets. But 1.9 million isn’t such a bad reach when you have a message to get out. Most of this reach was due to various NASA-related accounts retweeting it, but it was helped by regular Twitter users doing their normal thing on Twitter: sharing information with their followers.

One of the trade offs of having such a huge outlier in my data is that the charts on my Twitter data analytics are nearly useless for all the other thousands of tweets I did last week (yes, I Tweet…a lot.)

That red circle in the upper right represents the number of replies (the size of the circle) and the number of retweets and impressions (the X and Y axis). Looks good, until you see the blob of blue in the lower left. The fact that this outlier in my data was so far out there makes the other pieces of data look almost zero on both axes . I think I do pretty well with my social media outreach, but this chart would so fuddle duddle the data that it hides important information about my “normal” performance on Twitter. In fact, it almost makes it look like all my Tweets perform equally as well…or poorly.

So what could you do to make this chart more meaningful:

• Remove the outlier from the chart and or the data, completely
• Create two charts, one with and one without the outlier
• Make the graph 1000 times taller and wider
• “Break” the Y Axis so that there’s a gap between 100k and 1.8 million
• Use other techniques such as a logarithmic scale to show the data ratios instead of quantities
• Use statistical methods to massage the data even more
• Make better data (in this case, send Tweets that fill the gap to make my outlier look more normal)

I could also try to include a longer time period, such as including all my Tweets, not just the ones from this past week.

So a few more Tweets that had more retweets, but the impressions still look almost zero. So that doesn’t really help show how the rest of my Tweets did.

In business data, I’ve seen people opt to remove the outlier in additional charts, but sometimes they mask or delete them with no indication that they’ve been removed. Sure, my almost 2 million impression Tweet is messing with the display of other data, but if my performance bonus was based on that sort of thing, I wouldn’t want the data to vanish like the $38 million that was cut from the NASA STEM outreach budget. Your data needs may be different, though. So it’s important to find out how business users want outlier date dealt with. The reference links below talk about other more advanced methods for dealing with outliers. All I know is that my “all” chart isn’t going to help me much as long as that outlier is in the data set.

I’d recommend that whatever technique you use, you ensure that the reader understands what has been done. Remember that the goal of all charts should be to reveal more about the data than just looking at the raw data. If your chart doesn’t do that (Congrats again, Klout), maybe you need to rethink how you are presenting the data.

Other References:

• Data006: Outliers can make or break you. (data-literacy.com)
• Liars and Outliers Update (schneier.com)

Rather than write about my accomplishments over the last year, I’m going to write about all the things I should have been working on, but didn’t. I have lots of excuses: not enough hours, not enough days or too many other things to take care of.

These are the types of things that Stephen Covey wrote about in First Things First under the theme of “Big Rocks.“  Basically, if I’d spent just a few minutes every day adding these to my schedule, I would have accomplished more.

If I had just added them to my life, I’d be in a much better position sitting here at the end of 2011. I might even be richer, taller and thinner. Maybe even smarter.

1. Read more about Data

I had all these great intentions to read more about Big Data, Data Quality, Data Modeling, Data Visualizations, Data Science…well, you get the idea. I have a pile of such books next to my desk and filling my iPad. I also have subscriptions to digital libraries to many of them, too. So it wasn’t a lack of access that stopped me.

I also should have been more caught up on reading my RSS feeds of other data bloggers. What’s that you say? There aren’t really that many data bloggers? I agree. Why aren’t you blogging so that I can get better at catching up on reading your thoughts about data, life and the world?

2. Learned more French and Hindi

I have lots of resources right at my fingertips for learning French and not because I live in Canada. Here in Toronto French is way down the list of languages spoken by the general population. In fact, it’s 11th on the list after English, other European and several Asian languages.

Since almost all of the projects I’ve worked on in the last few years involved offshore teams, I’ve also been studying Hindi — not because I have to, but because it has been helpful. Did you know that Hindi uses the same word, kal, for yesterday and tomorrow? That explains a lot about some status updates that didn’t really pan out. I recommend you work this concept into your next status meeting.

I’d also like to learn some Mandarin. I’ve read in several places that if one learns Hindi and Mandarin (and sometimes French and Hindi), one can speak with half the world’s population.

Learning about languages helps me in internationalizing data designs. I know now that there are places in the world where people don’t have middle names, half the population has the same last name or where people only have one name. That’s just the tip of the iceberg. Studying other languages and cultures makes my data models better.  While we are on this topic, stop making Postal Code all numeric in your designs.  The rest of the world will thank you.

I should have worked on a Mayan language, given that they are going to bring about the end of the world in late 2012.

3. Run more

I’ve been an on-and-off runner since embarrassing my middle school track team as an 880 distance runner. Mostly off, I’d say. But a few years ago I decided that I was old enough to not care about how fast I ran and signed up for a few races. This year I ran my second and third half marathons. One of them was with a group of other data professionals, even. My goals for both of them were to just finish upright and smiling. I did, in both races. But I want to start having some time goals and that means training differently. It also means running more and cross training more.  Just like with IT training, I have to train for speed differently than training for completion.

Running has been a good stress-reliever and I certain know that the life of an architect in the fast-paced, agile/SCRUM/anything goes world of what they call system development methods these days leads to a lot of stress. Running is also one of those things that I can do anywhere, even while traveling. I just need a way to fit those giant running shoes in my carry-on bag and I’m set.

I love the gadgets I use when I run: my Garmin GPS watch, my Runmeter app that reads Tweets people are sending me while I run and my fitbit that collects and stores data bout how many steps I took in a day, how far I traveled and how many calories I burned. This helps me measure and monitor my progress and predict future performance. Larry English and his stopwatch would be proud, I hope.

4. Develop a Million Dollar App

It seems that everyone is doing this these days. Even teenagers. If I could just find a way to build an app that ties together data and space together in a way that everyone would want to hand over their hard-earned cash for, then I’d be set. Or I could just write one that lets cats play a game on my iPad. One of those would work and I’m afraid it’s only the cat one.

5. Read the Scriptures

I could probably do with reading all kinds of good books, but here I mean the writings of Codd, Chen, Date and Zachman. I’ve read all these in the past, but I think I may have read too much vendor documentation between then and now. Terminology has been twisted and refactored so much in practice that it pays to go back and read the original theory from time to time. Hear that Microsoft with your “Entity Framework” and Access “database”?

Most of these good works are available at the ACM Library and at Amazon. It’s a much better value to join ACM and sign up for the library subscription that it is to buy them one off at an online bookstore. Also, check out access via corporate subscription services, too.

6. Perfect cloning techniques

Even though I attended many events in 2011 (Enterprise Data World, SQLSaturdays, SQLRally, DAMA Days, PASS Summit, NIEM National Training Event and more), I still missed many of the key data-related conferences like the Semantic Technologies conference that would have exposed me to emerging technologies and best practices. If I could just clone myself I could have been a the many overlapping events that happened in 2011. I think that might have allowed me to complete more of the other items on the list.

7. Install more DBMSs and Technologies

I should have taken the time to play with research the newest versions of SQL Server, DB2, Oracle, MySQL, Postgress, Hadoop, MongoDB, SQL Server Azure and ….well all the 10 million other databases/non-databases out there. Sure, I can’t know them all, but getting hands-on with new tools and features is the best way to understand the Next Big Thing we’ll have to design for. That’s hands-on, working with real, non-trivial problem sets. These technologies are coming to a project near you soon, if they haven’t already. You don’t won’t to be the only one in the room, especially as the data professional, saying “Hadoop? Is that a character from Dr. Seus?”

8. Stalked Astronauts Bought Real Estate in Cocoa Beach

I actually did do some astronaut stalking this past year. I spent a lot of time in Cocoa Beach, Florida this year, not to mention Cologne, Germany. All these visits were about watching launches of humans and rockets into space…and meeting smart people collecting and using data from beyond Earth’s boundaries.  I think over the year I was fortunate enough to meet more than 25 former and active astronauts, not to mention key members of NASA, CSA, and ESA teams. I even attended, virtually, the NASA IT conference.

Did you know that data.nasa.gov offers more than 1200 data sets for you to use? One of my favorites it the Great Images in NASA (GRIN) data set. Perfect for playing with those new database features in item 5 above.

We are still sending people into space, just not from Florida right now.  Heck, NASA sent 3 more astronauts to the International Space Station just last week.  So while the space coast real estate market is good for buyers right now, there are still unmanned missions launching from there in 2012.  All for collecting massive amounts of data about our world.

 9. Interacted with machines more often, so they know me better when they take over

Sure, I’m known for my love of gadgets. One of my favorites is my Xbox with Kinect. I primarily use it for sports and exercise . And yes, it collects and stores data about my workouts and play.  What I’m most excited about Kinect, though, is the expansion of its controller-less interfaces to other applications.

Imagine zooming in or out or double-clicking to drill down into the metadata of a table without having to use anything but your own hands in front of a projected display of your model. Or laying out your data model by just standing in front of your model and moving the entities with your hands. That’s what I want. When we later find out that Kinect forms a union at Cyberdyne Systems and becomes Skynet, I’ll be ready.

10. Loved My Data More

I think I did a decent job loving the data I was supposed to, but I could have done more. I could have tested more restores of databases (because you don’t really need backups; what you need are restores). I could have done more data profiling to see what data was lurking inside those columns with a name like description or notes. I could have tested my designs more, to ensure that they performed as well as I thought they would and that they could store the data they should have.

11. Kicked my procrastination habit

I’m going to start on this one next. I promise.

Love Your Data

What do all these things have in common? DATA.

Let’s make sure that when we are together again, at the end of 2012 (assuming the Mayans were wrong) that we don’t have a similar long list of regrets. Love Your Data.

Last week I flew from Washington, DC to Toronto via Philadelphia. This was at the end of about 3 weeks of many flights, having done a DAMA speaking tour, a class and two SQLSaturdays. I’ve been known to Tweet my travel experiences along the way, but usually these are first world problems like not getting an upgrade, weird non-TSA baggage inspections or complaining about the terribly rude service I seem to experience on a regular basis. But this time my experience in Philly was different. My flight from Philly was delayed. It seems all my flights from Philly are delayed, as if there is a big time suck in that general area. I knew about this delay because I use TripIt to track my itineraries and TriptPro notified me that my flight had been delayed by about 40 minutes. But as usual for many airlines, the boards in the airport and the official websites showed the original time — no delay. I have been told that airlines do this so that you will get to the gate anyway and therefore give the airline a few more seconds to slice off their abysmal on-time performance stats…by having you wait in a crowded, no-place-to-sit-not-even-on-the-floor gate area.

Alternative Data Sources

Because USAir had chosen not to delay the flight on the boards, I made my way to the gate area near the original boarding time of 12:45. There was no plane at the gate. This is your number one sign that the flight is delayed. A missing plane can’t be deboarded, it can’t be cleaned and it can’t be boarded. So I found a spot to sit on the floor underneath a pay phone next to the door to the jet way. As soon as I sat down, a harried gate agent called off about 20 names to come to the podium. Normally this might mean upgrades, but in this case I was sure it was the sometimes required, sometimes not “document check”. Since this flight was headed to Canada, airlines sometimes want to see your proof of citizenship again before you get on the plane. This document check is done because airlines are required to return you at their cost if you aren’t able to enter another country. So I showed my passport as the gate agent scolded me for waiting until boarding time to show up to the gate. She was frazzled and irritated because so many other passengers still hadn’t had their documents checked. I went back to my seat on the floor.

Sitting next to the door were three women who had just been scolded by the gate agent for asking if the flight was delayed. It seemed that 80% of the passengers who approached the desk were asking the same question. Before we had mobile data devices we passengers were in the dark about flights, but now we have access to third-party data sets that can tell us when flight are delayed or cancelled. The airlines haven’t quite changed their data sharing processes to acknowledge that. They still assume that we have no other sources of data for flight information. Gate agents everywhere are distracted by passengers asking why the board data and their data services data is in conflict.  We should fix that.

But I digress. Data issues do that to me.

Distraction and Flying the Plane

As the three women and I were discussing the fact that it was 1:04 and our plane had not yet arrived for our 1:18 take off, a woman who had no badge or uniform walked passed us and into the propped open door of the jet way. Our friendly gate agent was busy reviewing documents, then using the PA to call for missing passengers to show their documents. She was consumed by two tasks: checking documents and convincing passengers that the flight was on time. Finally one of the three women (I’ll just call her Woman One) approached the desk and pointed out to the gate agent that someone who appeared to have no credentials or uniform had boarded the jetway without clearing with the gate agent. The agent paused, then said that she knew what she was doing and no one had entered the jetway. We stared in disbelief. Woman One said again that she had seen someone board without clearance.

Remember that the Department of Homeland Security has a See Something, Say Something campaign. Airports are covered with signage repeating this message. So this group of women had seen something and said something. Now they had reported it to an airline employee. What was the gate agent’s response? She repeatedly told Woman One to sit down an shut up. Woman One kept up with her reports, but the gate agent was focused on two things: getting those dang documents checked and ensuring that we all stayed at the gate area so that the flight could be boarded as fast as possible.

Do you see what’s happened here? The gate agent was so focused on these two tasks that she missed the whole point of her job: to be the agent that ensured only the right people got on that jetway. Sure, the document check is part of that, but she was presented with a much bigger threat than expired documents and she discarded that report in favor of her heads-down task of matching names on a document to names on a list. The airline industry has a name for this: forgetting to fly the plane. My friend Mike Walsh (@mike_walsh ) has blogged and presented about this several times. In See Something, Say Something he writes about a flight crew on United 173 being so distracted by a light bulb they flew an airplane into the ground. Other air disasters involve the same type of mistake: pilots forgetting that their job is to first fly the plane. It seems so obvious is hindsight, but it’s easy to get distracted by project goals, urgent tasks and personal goals that we forget to do the number one priority in our jobs.

I have to admit that I’ve gotten bogged down in trying to make a macro do some whiz-bang nifty thing while forgetting that my job is to get data requirements documented and turned into a database design. Some other things I’ve seen:

• A data architect that is so focused on getting a data model laid out with no crossing lines and following a no-dead-crows approach that she has forgotten more than 20 important data requirements
• A DBA so focused on applying surrogate keys to every table that he forgot to ensure that the alternate keys were applied, therefore leaving the data in an potentially harmful situation
• A developer so focused on squeezing performance out of a query he failed to return all the data that should have been returned in the query
• A project manager so dedicated to getting perfect Gantt chart she forgot to actually manage the tasks and people who needed a project manager.
• A team so focused on a two week sprint they forgot to verify a bunch of assumptions that turned out to be wrong, causing a great deal of rework and loss of confidence in the team.
• A DBA so focused on applying a “best practice” that he applied it in a situation that it harmed performance instead of enhancing it.

So what happened in Philadelphia? We don’t know. We never saw the mysterious woman deboard the plane. And when we boarded, she wasn’t there on the plane. Obviously either all four of us missed her coming back off the plane or she exited the jet way on to the tarmac. We’ll never know who she was or what she was doing on the plane or the tarmac. There has been no news of anything bad happening as a result of a failure to “fly the plane” that day. The opportunity certainly was there.

I’d love to hear about your examples of people who have forgotten to fly the dang plane.

It’s so easy to get bogged down in a short term task that we forget that our job is to do something more than that task. Make sure that you don’t lose sight of your job, no matter what it is. Every task has a purpose. Make sure you understand what the goal of a task so that you understand when you are distracted from flying the dang plane.

This is the third post in a series on Normalization Myths.  You’ll want to read the prior posts first.

Normalization Myths that Really Make Me Crazy – Introduction to a Rant
Myth 1 – Normalization: Friend, Foe, or Frenemy The Survey

In my survey of Twitter users on their feelings about normalization, the most common thought I received was:

I realize that this is one of those sayings used to help people remember the meaning of terms.  However, like other memory tricks, it fails when it comes to working in real-world situations.  Sure, I’d like to think that one cannot be too rich or too thin, but we know that tragically, both of those thoughts can go terribly wrong.

The hurts-works saying reinforces the normalization is evil concept.  It implies that normalization is harmful and must always be rolled back until it is doing less harm.  What this belief fails to recognize is that we database designers concern ourselves with normalization not just as an academic exercise, but to add value to the design by finding the right balance between data integrity and performance:

1. Database tables are normalized to minimize the impact of update (create, modify, and delete) anomalies.  Data anomalies mean data integrity suffers.  It can also mean worse performance as data quality issues grow over time.  When data integrity suffers, business suffers.
2. Good designers use cost, benefit and risk assessment to find the right normalization level within the context of the data, its use and its quality requirements.
3. If the data is not going to be updated after being created, then it’s really difficult for update anomalies to happen.  Therefore the need for higher normal forms in the data structures is less of a requirement.  This is why most data warehouse designs are highly denormalized.
4. When a designer “tunes” a database design to include denormalization, typically they are trading off the update performance or risk of data anomalies for better performance for querying the data.  This trade off may or may not be the right design for that context.
5. When one denormalizes a data structure for performance reasons, he is borrowing performance and data integrity to get that performance gain.   This gain does not come out of thin air.
6. There is no such thing as the “right” normal form for all tables or all data.
7. One of the least successful reasons to denormalize a structure I’ve experienced is for the sole reason of making a developer’s tasks easier.  Sure, there are project benefits for ensuring that development tasks can be completely faster, but rarely do the performance gains in developer time offset the cost to data quality and query performance that happens with these types of simplifications.  Optimizing developer time at the expense of data quality and performance is an optimization of the wrong subsystem in almost all cases.
8. Normalization requires the designer to understand the meaning of the data.  It is not possible to apply the normalization rules to data you don’t understand.  Therefore, the less one understands about the data, the less likely their database design will find the right trade off of cost, benefit and risk.

When someone says that we denormalize until it “works” what they really mean is they are denormalization until a query runs faster.  However, faster queries may or may not be the only goal of the database design.  We need to understand the objectives for the design in order to choose the right normalization level.  Context is everything in design.

I apply denormalizations to database designs on regular basis, even on transactional database designs.  I do this with the understanding of the trade-offs.  I ensure that compensating data integrity features are put in place to mitigate data anomalies.  Queries do need to perform well.  Sometimes it’s more important that the data be returned faster than it be correct.  Sometimes it’s much more important that the data be correct.  My job is to find that balance.  In order to do that, I need to understand the context of the project so that I know who my normalization friends and foes are.  I will address the overnormalization versus undernormalization issue in a future post.

Perhaps what the data profession needs is a series of top-up courses, to be reviewed every couple of years.  Maybe what we need is an intervention to help people understand the why normalization is even a topic in design.  I think this would make for a wonderful lunchtime presentation.  Perhaps your boss could even buy lunch.  This presentation wouldn’t be the how of normalization, but the whys and why nots.  Mastering the normal forms is fairly easy.  Understanding which one to use for a specific solution is the hard part.  The more your teammates understand the whys, the more likely they are to going to support your efforts.

Don’t let normalization become your demoralization.  Even if that dang auto-correct keeps trying to tell you it should be.

I recently blogged the introduction to this series:  Normalization Myths that Really Make Me Crazy – Introduction to a Rant.  You should check that introduction out for the background on this post.  It has caveats and warnings that you’ll need to keep handy while you read this series.

I’m starting with the basis for many of the myths that I’ll be covering in this series.   Just last week I learned that Normalization is Evil.  Actually, I’ve heard that every week for the last twenty some years.   The basis of this thought is that business users never ask for data quality and always ask for better speed.  Remember that stinking pile of poo from the introduction? I think it’s still nearby

Most business users don’t say “We need the data to be correct” because they expect that as a given.  Are we really going to stand before them and say “Is it okay if some, maybe more, of the sales tax remittances are wrong so that we can get better performance out of the system?” Or maybe we could wow them with “Is it okay if we calculate many of the customer bills incorrectly so that developers don’t have to code as much?”

Think that last one is a laugh?  I was once asked to design a database that gave each developer a single table to work with. This table would hold all the data assigned to be coded by the developer during a single sprint.  This design was seen as a great way of maximizing developer productivity.  Triggers, stored procedures and other code would take care of the data integrity I was assured.  Thankfully I was able to show just how much this design would harm performance.

To be fair, I work with data architects and database designers who think that all denormalizations are some sort of Sign of Beast — as if there is a higher power up there watching over their designs, ready to strike them down for thinking of performance tradeoffs.   In fact, my friend Michael Swart (blog | twitter)  has a great illustration of this concept:

The Friend or Foe myth is based on the idea that you have to be for or against normalization as a concept.  I’ve seen speakers start their presentation by asking people in the audience if they are pro-normalization or anti-normalization. When people raise their hands to pro-normalization, the speaker will half-jokingly ask them to leave. The odd thing about this this for-against mindset is that it seems to indicate that one can have a data structure that has no normalization to it at all or that one can design something without a normal form.  I suppose random meaningless numbers have no normal form, but we don’t need a design for that, right?

This got me thinking about people who are both for you and against you: the frenemy.

A “Frenemy” (alternately spelled “frienemy”) is a portmanteau of “friend” and “enemy” that can refer to either an enemy disguised as a friend or to a partner who is simultaneously a competitor and rival. [1]

Could it be that normalization has to be either a friend or a foe?  Is it something that you have to choose between Team Normal and Team Denormal?  Or is it a frenemy, one of those things that you can pretend to like but have to hate when it comes down to getting things done?  I asked the Twitterverse what they felt about normalization.  The following represent the range of responses I received:

 Not everyone was post their own beliefs; some were quoting what they have heard in the wild. You can see that I got a variety of opinions that ranged from Normalization is Evil to Normalization is Our Only Hope.  My sample is biased because these came primarily from those who have an interest in the same topics I am interested in.  I think if I’d asked the general IT population I would have received many more negative thoughts about normalization and people who believe in it.

It’s common for me to be questioned when I start a project by project managers and others about my friend/foe relationship with normalization.  I’ll get questions like:

PM: Do you believe in normalization?

Me: Yes.

I find this one hard to respond to without giggling.  It’s as if I’m being asked if I believe in Santa Claus. Or Ted Codd.

PM: [long pause] Okay…we’ll probably need to review your designs with the developers then.  They don’t take kindly to normalizers.

Me: That’s fine.  I love collaborating with the developers.  Together we can…

PM: [interrupting] How far do you go?

Me: [blushing] Um…what do you mean?

PM: What normal form? Third? Fifth?

Me: Oh. Well, it depends…

PM: We only go to third normal form here.  We are traditionalists.

This always makes me wonder if there is a Church of Normalization, Reformed.  This might also mean there’s a Church of Denormalization, Unformed.

Me: Okay.  What about First or Second Normal Form?

PM: Oh, we aren’t that radical.  Just Third.

I wonder if they might have some performance issues that could be easily rectified by their need to have every data structure have the same normalization level.  I make a note.

I can tell from these interview questions that the PM thinks that normalization is his frenemy: something that someone in the IT world thinks is a “good thing” for a design, but that everyone on the project thinks is evil.

Our job as data architects is to help team members understand that all data structures have a normal level, that normalization is an important part of meeting business needs and that the evil parts can sometimes be exorcised.  If you need to denormalize to meet a business goal, then do it.  It’s not evil incarnate.  It’s how design works: cost, benefit and risk.

I keep my normalizations friends close and my enemies closer.  As for frenemies, I don’t believe in them.

 [1]Wikipedia contributors, “Frenemy,” Wikipedia, The Free Encyclopedia, http://en.wikipedia.org/w/index.php?title=Frenemy&oldid=437999108 (accessed July 27, 2011).

At Enterprise Data World I gave a Lightning Talk, Karen’s List of the Most Irritating Normalization Myths.  This was a fast-paced, auto-advancing slides presentation with 10 slides covered in five minutes.  I covered the types of things I’ve heard in design reviews that either didn’t make sense or repeated some urban data legend.  I had only thirty seconds per slide, so it was a lot to cover at an extremely rapid pace.  The blogs here at Dataversity.net give me an opportunity to get my rants down in writing, something I may regret later.  This sort of risk never stopped me before, though.

Poo - by Kendra Little

What most people know about normalization they learned from word of mouth, much like most of us learned about sex — in the hallways of school, told as dirty little tales from school kid to school kid.   You probably remember just what sorts of data quality that information had.

I first learned about normalization in a real database class in university, where we learned the theory, then developed solutions that worked with both normalized and highly-denormalized data structures in order to best understand the costs, benefits and risks associated with design decisions.    I was shocked when I went into the “real” world and found out how many myths there are about what it is, how it works and how it is something to be avoided at all costs.  Normally (excuse the pun) I would not care about misinformation that my team members spot, but when their misunderstanding of the foundation theory of database design starts to impact my work, I have to call BS.    This blog series focuses on the myths and poorly worded complaints against normalization.

Then this week a prominent database technology expert whom I highly respect wrote a newsletter article about overnormalizing database designs.  I know for certain he knows a great deal about database design.  Most likely due to space constraints, he also perpetuated one of the myths I want to cover in this series.  I still highly respect him.  I’ve been known to make similar statements. I’d bet we all have.  Did I mention I highly respect him?

Writing about normalization is error prone because professionals want  to use precise terms.  A relation isn’t a table.  A set is not a table.  However, when I talk with team mates about normalization, it’s sometimes easier and more clear to them to make those sorts of analogies because people can visualize a relation as a table.  It’s still wrong, but it is more clear.  In this series, I will endeavor to be precise and still make analogies with real world artifacts such as tables and databases.  I therefore ask the professional purists to grant me some poetic license to do so.

Most people I’ve asked tell me they learned about normalization in one of these situations:

• In a 1-5 day course normalization was covered right at the start of the course.  It sort of made sense, but they didn’t really remember what the normal forms are and would have to go back to their notes to figure out what each normal form is.
• In a book they read about the normal forms.  They saw how a table was transformed from 1NF to 2NF, etc.  There was a brief explanation why normalization was important.

• In a formal education course, where there were 1 or two modules on normalization.  Students were required to show a data structure as it was normalized from 1NF to 5NF.  There was an exam section on the normal forms where they did a very similar exercise. They remember doing it, but have never taken a data structure through all the normal forms again.

• In a meeting they heard someone complain about a design that was or was not properly normalized.

• In a bar, where a co-worker griped because the database design was horrible because it was over-normalized and therefore had many tables.

Almost all the people I talk with pretty much equate normalization with something evil, as if data architects and database designers conspire to wedge as much normalization as they can in design just to work against developers and DBAs as much as possible.  Few (other than data professionals) understand that the main reason we are concerned about normalization in relational databases is to increase data integrity by reducing redundant data and mitigating update anomalies.  Notice how nothing in that statement speaks to query performance.  That’s because normalization is about updating data – creating, updating and deleting data.  You could, though, think of normalization as a method for increasing the performance of the data, not the code.

The Normal Forms

In this series I’m not going to cover normalization as a tutorial but I will share this description* from Wikipedia that covers the normal forms.

First normal form (1NF)
Reference: Two versions: E.F. Codd (1970), C.J. Date (2003)[11]
Definition: Table faithfully represents a relation and has no repeating groups

Second normal form (2NF)
E.F. Codd (1971)[2]
No non-prime attribute in the table is functionally dependent on a proper subset of a candidate key

Third normal form (3NF)
E.F. Codd (1971);[2] see +also Carlo Zaniolo’s equivalent but differently-expressed definition (1982)[12]
Every non-prime attribute is non-transitively dependent on every candidate key in the table

Boyce–Codd normal form (BCNF)
Raymond F. Boyce and E.F. Codd (1974)[13]
Every non-trivial functional dependency in the table is a dependency on a superkey

Fourth normal form (4NF)
Ronald Fagin (1977)[14]
Every non-trivial multivalued dependency in the table is a dependency on a superkey

Fifth normal form (5NF)
Ronald Fagin (1979)[15]

Every non-trivial join dependency in the table is implied by the superkeys

Domain/key normal form (DKNF)
Ronald Fagin (1981)[16]
Every constraint on the table is a logical consequence of the table’s domain constraints and key constraint

Sixth normal form (6NF)
C.J. Date, Hugh Darwen, and Nikos Lorentzos (2002)[4]
Table features no non-trivial join dependencies at all (with reference to generalized join operator)

I have modified the format of the table for compatibility with a variety of platforms and blog readers.

My first irritation, coming up in the next post, is about the love/hate relationship of normalization.  Your assignment is to reach deep down into your heart and identify your true feelings about normalization.   I’d love to hear about how you learned about normalization and how you came to your feelings about it.  Who says normalization is only math?

Your second assignment is to Tweet @kendra_little that you loved her poo art.   Thanks, Kendra.

* Wikipedia contributors, “Database normalization,” Wikipedia, The Free Encyclopedia,
http://en.wikipedia.org/w/index.php?title=Database_normalization&oldid=433633646 (accessed June 14, 2011).

I’m often asked to help clients fill IT professional positions – Database Administrators (DBAs), Data Architects, Data Administrators, etc.  Most of the time the job requirements are sufficient to help them find a good list of candidates worth interviewing.  However, in these times of limited budgets, I frequently see job descriptions that attempt to find the Wonder Candidate of the Century, a person so thoroughly talented that she is a master expert along an entire column of the Zachman Framework.

Wonder Candidates Mapped to Zachman Framework

It would seem to make sense that if you were hiring a data professional you’d design a position that fills in the Data column, right?  No?  It turns out, though, that most people don’t think and work along a column.  In my experience, people aren’t passionate about tasks that span columns from top to bottom.  They normally aren’t skilled along the whole column, either.  Referring to the Zachman Framework, what sorts of skills and passions would this candidate need: planning, architecting, designing, building systems, building parts, keeping the systems up and running.  Think about all the technologies, tools, methods and approaches these candidates would need to master to work at the professional level up and down an entire column: business strategy tools, tactical planning tools, database technologies, design tools, data modeling tools, code generation, query performance tuning, semantic technologies, legacy database systems, query development… Well, I’m getting tired thinking about it.

Roles Mapped to the Zachman Framework

I have this image when I read bad job postings of candidates being dragged by zombies up or down a column.  This dragging means we’d need to find people who have great analytical skills while at the same time having great detail-oriented building skills.  But having those skills isn’t enough; they’d also need to be passionate about every task along the column equally.  That’s the rub.  I’m not even certain I’ve met a person who is passionate about all the tasks from planning to keeping the enterprise functioning.  In fact, I think there’s a type of Mason-Dixon Line somewhere around the middle rows that separates the more analytical tasks from the building and maintaining tasks.

Secret Mason-Dixon Line on the Zachman Framework - Approx Location

“[T]he Mason–Dixon Line symbolizes a cultural boundary between the Northeastern United States and the Southern United States (Dixie).”[1]

The divide I’m referring to isn’t about politics, but more about where one feels most comfortable.  At a crude level, this is a difference between thinking and designing enterprise systems or building and running enterprise systems.  The line may be higher or lower in the Framework and there may be similar lines as we cross each row and column, but I see the most pronounced difference as we move from top to bottom.

I see job postings all the time that call for a Strategic Conceptual Enterprise Information Data Architect who has strong skills in modeling, strategic planning, query tuning, XML, data syndication, mirroring, SANs, debugging and < insert your favourite development languages here >.   Those people don’t really exist.  There may be people who can do a lot of those things, but in my experience they aren’t passionate about all of them. New hires won’t be happy and the organization will not realize the economies that they think they will.

I recommend that if organizations want to combine responsibilities that they do so across the columns in the same range of rows.  Combining positions where thought processes are similar (business and data analysts, DBAs and developers, etc.).  Analysts in general make for good analysts in other columns.  Operational people tend to think operationally, builders tend to think mostly of building, not planning well.  Let’s not drag people up or down the rows.

Go now and check your job postings.  Do they reflect the true nature of the job?  Or are they actually full of zombies ready to drag someone to an assignment that they don’t really want?

[1]Wikipedia contributors. “Mason–Dixon Line.” Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 2 Jan. 2011. Web. 3 Jan. 2011.