Unexpected (and unwanted) data transformation problems can result from 50 (or more) issues that can be seen in the table that’s referenced in this blog post (see below). This post is an introduction to many causes of data transformation defects and how to avoid them. Data transformations are a process of altering data and data-related […]
It’s Essential – Verifying the Results of Data Transformations (Part 2)
Test cases, data, and validation procedures are crucial for data transformations, requiring an understanding of transformation requirements, scenarios, and specific techniques for accuracy and integrity. Data transformations require complex testing due to their sophisticated logic, computations, and dependency on real-time data streams. This necessitates extensive test case design, representative data, automation tools, and robust validation […]
It’s Essential – Verifying the Results of Data Transformations (Part 1)
Today’s data pipelines use transformations to convert raw data into meaningful insights. Yet, ensuring the accuracy and reliability of these transformations is no small feat – tools and methods to test the variety of data and transformation can be daunting. Transformations generally involve changing raw data that has been cleansed and validated for use by […]
Choosing Tools for Data Pipeline Test Automation (Part 2)
In part one of this blog post, we described why there are many challenges for developers of data pipeline testing tools (complexities of technologies, large variety of data structures and formats, and the need to support diverse CI/CD pipelines). More than 15 distinct categories of test tools that pipeline developers need were described. Part two delves […]
Choosing Tools for Data Pipeline Test Automation (Part 1)
Those who want to design universal data pipelines and ETL testing tools face a tough challenge because of the vastness and variety of technologies: Each data pipeline platform embodies a unique philosophy, architectural design, and set of operations. Some platforms are centered around batch processing, while others are centered around real-time streaming. While the nuances […]
Best Practices in Data Pipeline Test Automation
Data integration processes benefit from automated testing just like any other software. Yet finding a data pipeline project with a suitable set of automated tests is rare. Even when a project has many tests, they are often unstructured, do not communicate their purpose, and are hard to run. A characteristic of data pipeline development is the frequent […]
DataOps Highlights the Need for Automated ETL Testing (Part 2)
Click to learn more about author Wayne Yaddow. DataOps, which focuses on automated tools throughout the ETL development cycle, responds to a huge challenge for data integration and ETL projects in general. ETL projects are increasingly based on agile processes and automated testing. ETL (i.e., extract, transform, load) projects are often devoid of automated testing. The […]
DataOps Highlights the Need for Automated ETL Testing (Part 1)
Click to learn more about author Wayne Yaddow. DataOps, which focuses on automated tools throughout the ETL development cycle, responds to a huge challenge for data integration and ETL projects in general. ETL projects are increasingly based on agile processes and automated testing. ETL (i.e., extract, transform, load) projects are often devoid of automated testing. The […]
Avoid These Mistakes on Your Data Warehouse and BI Projects
Click to learn more about author Wayne Yaddow. The is the first part of a three part series. The links to Part 2 and Part 3 are at the bottom. Data warehousing (DW) and business intelligence (BI) projects are a high priority for many organizations who seek to empower more and better data-driven decisions and […]