This webinar was given in collaboration with:

About the Webinar

Data is Yahoo!’s most strategic assets – from user engagement and insights data to revenue and billing data. Three years ago, Yahoo! invested in a Data Quality program.

By applying industry principles and techniques the Data Quality program has provided proactive and reactive system solutions to Audience data issues and root causes by addressing technical challenges of data quality at scale and engaging and leveraging the rest of the organization in the solution: from product teams all through the data stack (data sourcing, ETL, aggs and analytics) to analysts and sciences teams who consume the data. This methodology is now being scaled to the all data across Yahoo! including Search and Display Advertising.

This presentation covers:

• The solution methodology developed which builds in proactive and reactive DQ capabilities into Cloud-based products up-front and includes end-to-end data focus resulting in system improvements and fast issue resolution
• Solutions for technical challenges in the internet domain in Yahoo!’s massive data environment including end-to-end data monitoring and alerting, abuse and robot traffic detection, and latency vs. accuracy
• The DQ program approach to scale across all Yahoo! data that uses a central and embedded-in-the-businesses model with a strong focus on customer engagement

About the Speakers

Dan Defend & Aparna Vani

Dan Defend earned his Masters in Computer Science from University of Illinois. He has experience at Motorola as an Engineering Manager first in Unix OS and then in embedded cell phone software where he also led analysis and data-driven improvement using Digital Six Sigma methodology. He is currently leading the Data Quality program at Yahoo! where significant improvements are underway involving centralized monitoring and data cleansing across Yahoo!’s audience data pipeline and relying heavily on organizational leverage and distributed ownership and accountability.

Aparna Vani earned her Masters in ECE from University of Houston. She worked as Hardware Design engineer at Compaq and development lead at TVGuide. She has versatile experience as test designer and architecture at Motorola. Currently, she is working as Chief DQ Architect at Yahoo, responsible for data quality strategy and design, working on global organization wide projects like robot filtering and bcookie churn.

To view the recording of this webinar, click HERE.

This presentation was given in collaboration with:

About the Webinar

Data is Yahoo!’s most strategic assets – from user engagement and insights data to revenue and billing data. Three years ago, Yahoo! invested in a Data Quality program.

By applying industry principles and techniques the Data Quality program has provided proactive and reactive system solutions to Audience data issues and root causes by addressing technical challenges of data quality at scale and engaging and leveraging the rest of the organization in the solution: from product teams all through the data stack (data sourcing, ETL, aggs and analytics) to analysts and sciences teams who consume the data. This methodology is now being scaled to the all data across Yahoo! including Search and Display Advertising.

This presentation covers:

• The solution methodology developed which builds in proactive and reactive DQ capabilities into Cloud-based products up-front and includes end-to-end data focus resulting in system improvements and fast issue resolution
• Solutions for technical challenges in the internet domain in Yahoo!’s massive data environment including end-to-end data monitoring and alerting, abuse and robot traffic detection, and latency vs. accuracy
• The DQ program approach to scale across all Yahoo! data that uses a central and embedded-in-the-businesses model with a strong focus on customer engagement

About the Speakers

Dan Defend & Aparna Vani

Dan Defend earned his Masters in Computer Science from University of Illinois. He has experience at Motorola as an Engineering Manager first in Unix OS and then in embedded cell phone software where he also led analysis and data-driven improvement using Digital Six Sigma methodology. He is currently leading the Data Quality program at Yahoo! where significant improvements are underway involving centralized monitoring and data cleansing across Yahoo!’s audience data pipeline and relying heavily on organizational leverage and distributed ownership and accountability.

Aparna Vani earned her Masters in ECE from University of Houston. She worked as Hardware Design engineer at Compaq and development lead at TVGuide. She has versatile experience as test designer and architecture at Motorola. Currently, she is working as Chief DQ Architect at Yahoo, responsible for data quality strategy and design, working on global organization wide projects like robot filtering and bcookie churn.

A new article looks at how data governance efforts might benefit from a bit of flexibility. It begins, “Data governance requires the coordination of a complex combination of a myriad of factors, including executive sponsorship, funding, decision rights, arbitration of conflicting priorities… But sometimes this emphasis on enforcing policies makes data governance sound like it’s all about rules. In their book Practical Wisdom, Barry Schwartz and Kenneth Sharpe use the Nicomachean Ethics of Aristotle as a guide to explain that although rules are important, what is more important is ‘knowing the proper thing to aim at in any practice, wanting to aim at it, having the skill to figure out how to achieve it in a particular context, and then doing it.’”

The article adds, “Although there’s a tendency to ignore the existing practical wisdom of the organization, successful data governance is not about systematically applying rules or following rigid procedures, and precisely because the dynamic challenges faced, and overcome daily, by business analysts, data stewards, technical architects, and others, exemplify today’s constantly changing business world. But this doesn’t mean that effective data governance policies can’t be implemented.  It simply means that instead of focusing on who should lead the way (i.e., top-down or bottom-up), we should focus on what the rules of data governance are made of.”

Read more here.

photo credit: kladcat

It appears that Oracle is planning to become a player in both the Hadoop and NoSQL spaces very soon. The article reports, “There has been speculation for a while now that Oracle might someday make its foray into the Hadoop and NoSQL spaces, and next week looks like that time. With regard to Hadoop, CEO Larry Ellison made it clear during last week’s earnings call that the company is working on a connector that will let customers load unstructured data from Hadoop into their Oracle Exadata appliances. Now we have proof — and Oracle’s big data plans don’t stop with Hadoop.”

The author continues, “I’ve seen some Oracle-produced content highlighting the company’s plans for a big data platform, apparently slated for launch in the second half of 2012, that not only includes the Hadoop connector — called Oracle Loader for Hadoop — but also a NoSQL database. The goal, it seems, is to let customers acquire data from whatever sources they please and then feed it into an Oracle Exadata data warehouse system. Once there, data can be analyzed via number of means, including existing Oracle technologies such as in-database MapReduce, mining and statistical analysis with R.”

The article adds, “Reaffirming this information Thursday, Larry Dignan at ZDNet highlighted that the Oracle Loader for Hadoop is the topic of multiple sessions at next week’s Oracle OpenWorld conference as is ‘Oracle NoSQL Database.’ What remains to be seen, though, is how heavy Oracle — which has enabled Hadoop integration for some time, actually — will actually invest in Hadoop and NoSQL now that it appears interested in productizing them.”

Read more here.

photo credit: Oracle

Charles Babcock recently looked at the trends surrounding data centers and asked the question, “What do you get when you marry cloud data centers to a telecommunications company? My answer is: the beginnings of a cloud network, a chain of linked data centers that in some cases bring two or more data centers into a position of backing each other up. That’s something many enterprises would value as they move workloads into the cloud. The service freeze in Amazon Web Services’ northern Virginia center over the Easter weekend served as a reminder of the value of geographic distribution when it comes to backup and recovery.”

Babcock continues, “I hope Amazon Web Services is studying the trend. If you were an AWS customer in US East-1 (northern Va.), you couldn’t easily designate the AWS data center in Dublin as your preferred failover site. You couldn’t even select AWS’ US West data center, unless you constructed the network links to it yourself. You were stuck using a neighboring Amazon “availability zone,” which, it turns out April 22-24, was not necessarily in a separate data center and in some cases froze up at the same time as your primary zone did. Automated backup to a separate location, however, is a definite possibility, given linked chains in the cloud.”

Read more here.

photo credit: Double–M (formerly DoubleM2)

Hilton Worldwide is searching for a Vice President of Enterprise Data Management in Tysons Corner, VA. The post states, “Responsible for ensuring that the data assets of Hilton Worldwide are supported by a defined and communicated architecture supporting the organization in achieving its strategic goals.  The position requires senior-level expertise in all aspects of Data Management including Master Data Management (MDM), Business Intelligence (BI), Data Warehouse, Database Management and Integration.  He/she will set and govern enterprise data standards, including managing the overall data governance processes.”

It continues, “Key artifacts would include Enterprise-level Logical Data modeling, Physical Data modeling, development of a data strategy and associated polices, and selection of capabilities and systems to meet business information needs.  This Vice President is technically responsible for the management, planning, design and implementation of enterprise Data Architecture and Business Intelligence domains to ensure alignment of total Enterprise Architecture with business strategy – tools, guidelines, principles, policies and governance.”

Learn more and apply here.

photo credit: Hilton

Mimeo is looking for a Senior API Developer in New York City, Memphis, or Seattle. According to the post, “The successful candidate will have strong service-oriented design and development expertise as well as experience developing highly scalable, multi-tier web systems. He or she will work closely with other internal development teams, so strong interpersonal skills are important to build relationships with peers and other teams to achieve objectives, maintain objectivity, and give and welcome feedback on design/implementation decisions.”

Qualifications for the position include the following: “Deep experience in Microsoft technologies including C#, .NET BCL, LINQ, ASP.NET, WCF, SQL and MSMQ. At least 5 years of experience with web technologies such as JSON, XML, SOAP. Fluent in object oriented design and design patterns. Adept at SQL Server, including ability to understand schemas, write queries, and stored procedures. Experience with parallel computing topics and Microsoft’s PLINQ/TPL APIs. Experience with Unit Testing and Behavior Driven Design. Experience with PaaS or IaaS Platforms such as Microsoft Azure or Amazon Web Services and EC2. Familiarity with capabilities of other service platform frameworks such as PHP, Ruby, Java. Experience with NoSQL solutions.”

Learn more and apply here.

photo credit: Mimeo

Loraine Lawson recently reported on how good data management has been linked to better performance of data applications. She writes, “Primarily, when you think about data integration you think about, well, data – not the applications. In part, that’s because the way data is processed tends to separate out the two — you pull the data into a data warehouse, data mart, or whatever (hence, the ‘extract’ in ETL) — and then you do whatever it is you want to do and you load it. But it’s not really so cut and dried, is it? Applications are intimately tied to data creation and use. What’s more, a recent study by Ovum found that good data management may be critical to the performance of your applications.”

Lawson continues, “Ovum surveyed IT executives at 146 large enterprises in North America, Australia and the UK on application performance and management. The survey points to a strong connection between bad management of data and problems with applications. Specifically, 85 percent of the companies complained of application performance problems and the leading culprits trace back to bad data practices such as a lack of standardization, inadequate archiving practices and too many point interfaces. The study also found that 20-30 percent of data is duplicated across applications, which increases application maintenance costs and creates problems for data migration, synchronization and retention.”

Read more here.

photo credit: William Hook

Michael Stiffler recently shared some pointers to keep in mind while implementing a data governance program. First off, Stiffler notes, “Data governance is an iterative program.  You need to set the expectation that the program will evolve, especially as the company’s business requirements evolve. For instance, you may have to adjust your framework to include other areas that are now important to the business. Or perhaps you understand the corporate culture a little better now and need to adjust your approach to increase your chances of being successful.”

He adds, “Don’t do data governance just for the sake of doing data governance. You need to understand the key business drivers, and tie data governance to activities that will actually increase business value. What business pains can be mitigated through data governance? How should they be prioritized? These are questions you’ll need to answer.”

One question to ask is, “Are you able to measure success? How does the business define success? Dashboards and scorecards are one way of measuring and showing improvements to your data (which is one definition of success). This may also resonate better with the business folks, especially if you have gotten their input in defining ‘success.’”

Read more here.

photo credit: stevendepolo

In a recent interview David Loshin discussed emerging methods for handling Big Data security issues. Loshin said, “I had an interesting conversation a few weeks back about whether data in the cloud is better protected than data that’s sitting in your own systems. We drilled down into that in the context of big data, such as that managed by Hadoop and those types of frameworks; we discussed the fact that the people who are developing Hadoop applications or MapReduce applications are developers. They are presuming access to the data, and the data sitting out on a collection of nodes and in some massively parallel configurations — presumably, that’s data in an uncontrolled environment… and there is even greater opportunity for exposure.”

Loshin continued, “Essentially, the data needs to be moved over to the framework, be it Hadoop or whatever. It’s then exposed as the analysis is being done, and then the results are integrated back in, or reconnected back to, for example, some traditional data warehouse or business intelligence framework. When you’re looking at large collections of data, there’s the potential once again for lots of data being exposed. On the other hand, if you’re creating a controlled environment where there is no other means of getting access, one might say that there might be an opportunity for increasing the protection, if you’re instituting your development framework in the right way.”

Read more here.

photo credit: Rumalowa