Data integration tools are used to collect data from external (and internal) sources, and to reformat, cleanse, and organize the collected data. The ultimate goal of data integration tools is to combine data from a variety of different sources, and provide their users with a single, standardized flow of data. Use of these tools helps […]
10 Advantages of Real-Time Data Streaming in Commerce
While early science fiction shows like “Buck Rogers” (1939) and “The Fly” (1950) depicted teleportation technology, it was Star Trek’s transporter room that made real-time living matter transfer a classical sci-fi trope. While we haven’t built technology that enables real-time matter transfer yet, modern science is pursuing concepts like superposition and quantum teleportation to facilitate information transfer across any distance […]
What Are Data Products and Why Do They Matter?
Data products are software in the form of specialty tools and apps that are designed to support data used as a service. They may be as simple and straightforward as a program that converts a dataset into a visualization, or as complex as a machine learning system based on large language models (LLM), such as ChatGPT. […]
Building Data Pipelines with Kubernetes
Data pipelines are a set of processes that move data from one place to another, typically from the source of data to a storage system. These processes involve data extraction from various sources, transformation to fit business or technical needs, and loading into a final destination for analysis or reporting. The goal is to automate […]
Informatica Launches New Databricks-Validated Unity Catalog Integrations
According to a new press release, Informatica, a leading enterprise cloud data management company, has strengthened its strategic partnerships by launching enhanced Databricks-validated Unity Catalog integrations. These integrations enable no-code data ingestion and transformation pipelines to run natively on Databricks, providing a best-in-class solution for onboarding data from over 300 sources. The joint offering facilitates […]
Why Is Data Quality Still So Hard to Achieve?
We exist in a diversified era of data tools up and down the stack – from storage to algorithm testing to stunning business insights. In fact, it’s been more than three decades of innovation in this market, resulting in the development of thousands of data tools and a global data preparation tools market size that’s set […]
Leveraging Data Pipelines to Meet the Needs of the Business: Why the Speed of Data Matters
Gone are the days when customers would place an order and patiently wait for hours or even days for goods to be delivered, or when letters would travel through snail mail to reach their recipients. Today, businesses and individuals expect instant access to information and swift delivery of services. The same expectation applies to data, […]
Testing and Monitoring Data Pipelines: Part Two
In part one of this article, we discussed how data testing can specifically test a data object (e.g., table, column, metadata) at one particular point in the data pipeline. While this technique is practical for in-database verifications – as tests are embedded directly in their data modeling efforts – it is tedious and time-consuming when end-to-end data […]
Testing and Monitoring Data Pipelines: Part One
Suppose you’re in charge of maintaining a large set of data pipelines from cloud storage or streaming data into a data warehouse. How can you ensure that your data meets expectations after every transformation? That’s where data quality testing comes in. Data testing uses a set of rules to check if the data conforms to […]
Data Observability vs. Monitoring vs. Testing
Companies are spending a lot of money on data and analytics capabilities, creating more and more data products for people inside and outside the company. These products rely on a tangle of data pipelines, each a choreography of software executions transporting data from one place to another. As these pipelines become more complex, it’s important […]