Wayne Yaddow

MARCH 12, 2025

How GX helps data teams validate, test, and monitor complex data pipelines Introduction Data flows from diverse sources, and transformations are becoming increasingly complex. Great Expectations can enable a wide range of data transformations and conversion operations.

Data Transformation 52

Data Transformation Data Quality Testing Data Warehouse 52

datapine

SEPTEMBER 29, 2022

In this article, we will detail everything which is at stake when we talk about DQM: why it is essential, how to measure data quality, the pillars of good quality management, and some data quality control techniques. But first, let’s define what data quality actually is. 4 – Data Reporting. date, month, and year).

Data Quality 241

Data Quality Metrics Data-driven Management 241

Wayne Yaddow

FEBRUARY 4, 2025

Its EssentialVerifying Data Transformations (Part4) Uncovering the leading problems in data transformation workflowsand practical ways to detect and preventthem In Parts 13 of this series of blogs, categories of data transformations were identified as among the top causes of data quality defects in data pipeline workflows.

Data Transformation 52

Data Transformation Testing Data Quality Strategy 52

Wayne Yaddow

FEBRUARY 26, 2025

AI is transforming how senior data engineers and data scientists validate data transformations and conversions. Artificial intelligence-based verification approaches aid in the detection of anomalies, the enforcement of data integrity, and the optimization of pipelines for improved efficiency.

Data Transformation 52

Data Transformation Testing Data-driven Data Quality 52

DataKitchen

OCTOBER 5, 2021

The data organization wants to run the Value Pipeline as robustly as a six sigma factory, and it must be able to implement and deploy process improvements as rapidly as a Silicon Valley start-up. The data engineer builds data transformations. Their product is the data. Create tests. Run the factory.

Testing 152

Testing Dashboards Measurement Experimentation 152

DataKitchen

OCTOBER 5, 2022

DataOps Observability can help you ensure that your complex data pipelines and processes are accurate and that they deliver as designed. Observability also validates that your data transformations, models, and reports are performing as expected. to monitor your data operations. without replacing staff or systems?to

Testing 169

Testing Risk Data Processing Statistics 169

Jen Stirrup

SEPTEMBER 30, 2021

Before we dive in, let’s define strands of AI, Machine Learning and Data Science: Business intelligence (BI) leverages software and services to transform data into actionable insights that inform an organization’s strategic and tactical business decisions. Once the model has been trained, it will need to be tested.

Business Intelligence 135

Business Intelligence Data mining Machine Learning Testing 135

AWS Big Data

MAY 25, 2023

Data transformation plays a pivotal role in providing the necessary data insights for businesses in any organization, small and large. To gain these insights, customers often perform ETL (extract, transform, and load) jobs from their source systems and output an enriched dataset.

Reporting 77

Reporting Metrics Optimization Data Lake 77

DataKitchen

JULY 27, 2023

Azure Blob Storage serves as the data lake to store raw data. Azure Databricks, a big data analytics platform built on Apache Spark, performs the actual data transformations. Azure Machine Learning can then use this data to train, test, and deploy machine learning models. Azure Machine Learning).

Machine Learning 130

Machine Learning Cost-Benefit Data Transformation Testing 130

AWS Big Data

NOVEMBER 9, 2023

A modern data platform entails maintaining data across multiple layers, targeting diverse platform capabilities like high performance, ease of development, cost-effectiveness, and DataOps features such as CI/CD, lineage, and unit testing. It does this by helping teams handle the T in ETL (extract, transform, and load) processes.

Data Warehouse 119

Data Warehouse Testing Data Quality Reporting 119

CIO Business Intelligence

JUNE 7, 2022

Data analytics draws from a range of disciplines — including computer programming, mathematics, and statistics — to perform analysis on data in an effort to describe, predict, and improve performance. What are the four types of data analytics? Data analytics and data science are closely related.

Data Analytics 135

Data Analytics Diagnostic Analytics Management Analytics 135

DataKitchen

OCTOBER 25, 2022

The goal of DataOps Observability is to provide visibility of every journey that data takes from source to customer value across every tool, environment, data store, data and analytic team, and customer so that problems are detected, localized and raised immediately. That data then fills several database tables.

Testing 130

Testing Data-driven Visualization Dashboards 130

DataKitchen

OCTOBER 5, 2022

DataOps Observability can help you ensure that your complex data pipelines and processes are accurate and that they deliver as designed. Observability also validates that your data transformations, models, and reports are performing as expected. to monitor your data operations. without replacing staff or systems?to

Testing 130

Testing Risk Data Processing Statistics 130

CIO Business Intelligence

JULY 5, 2022

What is the difference between business analytics and data analytics? Business analytics is a subset of data analytics. Data analytics is used across disciplines to find trends and solve problems using data mining , data cleansing, data transformation, data modeling, and more.

Business Analytics 136

Business Analytics Prescriptive Analytics Data mining Diagnostic Analytics 136

Sisense

AUGUST 28, 2020

All this contributes to your overall data integrity profile. Logical data integrity is designed to guard against human error. We’ll explore this concept in detail in the testing section below. Data integrity: A process and a state. There are two means for ensuring data integrity: process and testing.

Data Integration 101

Data Integration Testing Data Quality Data-driven 101

AWS Big Data

AUGUST 1, 2024

However, you might face significant challenges when planning for a large-scale data warehouse migration. This will enable right-sizing the Redshift data warehouse to meet workload demands cost-effectively. This report shows how tables, views, and stored procedures rely on each other.

Data Warehouse 100

Data Warehouse KPI Optimization Cost-Benefit 100

Cloudera

OCTOBER 7, 2022

dbt allows data teams to produce trusted data sets for reporting, ML modeling, and operational workflows using SQL, with a simple workflow that follows software engineering best practices like modularity, portability, and continuous integration/continuous development (CI/CD). Introduction.

Data Warehouse 81

Data Warehouse Data Transformation Machine Learning Data Lake 81

Wayne Yaddow

MARCH 3, 2025

Photo by CDC on Unsplash Many data pipeline failures and quality issues that are detected by data observability tools in production could have been prevented earlier in the pipeline lifecycle with better pre-production testing strategies. Helps identify transformation errors, and data quality issues early, minimizing risks.

Data Quality 52

Data Quality Strategy ROI Testing 52

Cloudera

OCTOBER 19, 2021

Airflow has been adopted by many Cloudera Data Platform (CDP) customers in the public cloud as the next generation orchestration service to setup and operationalize complex data pipelines. The same Airflow job can now be used to generate different SQL reports. Looking forward.

Data Transformation 120

Data Transformation Interactive Machine Learning Testing 120

Cloudera

OCTOBER 13, 2020

At the heart of CDP is SDX , a unified context layer for governance and security, that makes it easy to create a secure data lake and run workloads that address all stages of your data lifecycle (collect, enrich, report, serve and predict). Enrich – Data Engineering (Apache Spark and Apache Hive). This is Now.

Data Warehouse 97

Data Warehouse Machine Learning Visualization Data Lake 97

AWS Big Data

JULY 18, 2024

The main driving factors include lower total cost of ownership, scalability, stability, improved ingestion connectors (such as Data Prepper , Fluent Bit, and OpenSearch Ingestion), elimination of external cluster managers like Zookeeper, enhanced reporting, and rich visualizations with OpenSearch Dashboards.

Dashboards 99

Dashboards Testing Data-driven Visualization 99

AWS Big Data

AUGUST 9, 2023

The advent of rapid adoption of serverless data lake architectures—with ever-growing datasets that need to be ingested from a variety of sources, followed by complex data transformation and machine learning (ML) pipelines—can present a challenge. Disable the rules after testing to avoid repeated messages.

Data Lake 94

Data Lake Metrics Testing Cost-Benefit 94

AWS Big Data

DECEMBER 13, 2023

In addition to using native managed AWS services that BMS didn’t need to worry about upgrading, BMS was looking to offer an ETL service to non-technical business users that could visually compose data transformation workflows and seamlessly run them on the AWS Glue Apache Spark-based serverless data integration engine.

Metadata 101

Metadata Data Lake Visualization Data Quality 101

AWS Big Data

DECEMBER 21, 2023

The data products used inside the company include insights from user journeys, operational reports, and marketing campaign results, among others. The data platform serves on average 60 thousand queries per day. The data volume is in double-digit TBs with steady growth as business and data sources evolve.

Data Lake 104

Data Lake Data Warehouse Data-driven B2B 104

AWS Big Data

JUNE 19, 2024

Amazon Redshift is a fast, scalable, secure, and fully managed cloud data warehouse that makes it simple and cost-effective to analyze all your data using standard SQL and your existing ETL (extract, transform, and load), business intelligence (BI), and reporting tools. All columns should masked for them.

Data Warehouse 102

Data Warehouse Testing Sales Structured Data 102

AWS Big Data

JANUARY 30, 2025

Our approach The migration initiative consisted of two main parts: building the new architecture and migrating data pipelines from the existing tool to the new architecture. Often, we would work on both in parallel, testing one component of the architecture while developing another at the same time.

Data Warehouse 52

Data Warehouse Data Architecture Machine Learning Data Transformation 52

CIO Business Intelligence

JUNE 13, 2022

According to Evanta’s 2022 CIO Leadership Perspectives study, CIOs’ second top priority within the IT function is around data and analytics, with CIOs seeing advancing organizational use of data as key to reaching enterprise objectives. To get there, Angel-Johnson has embarked on a master data management initiative.

IT 144

IT Digital Transformation Internet of Things Strategy 144

Cloudera

NOVEMBER 29, 2021

Data transforms businesses. That’s where the data lifecycle comes into play. Managing data and its flow, from the edge to the cloud, is one of the most important tasks in the process of gaining data intelligence. . This has also improved analytics for ad hoc business report queries.

Data Lake 77

Data Lake Data Warehouse Data Architecture Reporting 77

AWS Big Data

NOVEMBER 15, 2023

Each CDH dataset has three processing layers: source (raw data), prepared (transformed data in Parquet), and semantic (combined datasets). It is possible to define stages (DEV, INT, PROD) in each layer to allow structured release and test without affecting PROD.

Dashboards 107

Dashboards Analytics Metadata Data Warehouse 107

AWS Big Data

NOVEMBER 29, 2023

dbt is an open source, SQL-first templating engine that allows you to write repeatable and extensible data transforms in Python and SQL. dbt is predominantly used by data warehouses (such as Amazon Redshift ) customers who are looking to keep their data transform logic separate from storage and engine.

Data Lake 117

Data Lake Management Metrics Data Warehouse 117

AWS Big Data

NOVEMBER 13, 2023

Queue Usage Concurrency Scaling Mode Concurrency on Main / Memory % Query Monitoring Rules etl For ingestion from multiple data integration auto auto Stop action on: Query runtime (seconds) > 3600 The following table summarizes the new workload management configuration for the consumer cluster.

Data Warehouse 104

Data Warehouse Analytics Data Lake Data Science 104

Cloudera

OCTOBER 11, 2021

Modak Nabu relies on a framework of “Botworks”, a series of micro-jobs to accomplish various data transformation steps from ingestion to profiling, and indexing. Cloudera Data Engineering within CDP provides : Fully managed Spark-on-Kubernetes service that hides the complexity running production DE workloads at scale.

Data Lake 95

Data Lake Cost-Benefit Data-driven Dashboards 95

Cloudera

JUNE 30, 2022

These tools empower analysts and data scientists to easily collaborate on the same data, with their choice of tools and analytic engines. No more lock-in, unnecessary data transformations, or data movement across tools and clouds just to extract insights out of the data.

Data Lake 88

Data Lake Data Warehouse Data Architecture Metadata 88

AWS Big Data

OCTOBER 14, 2024

Enterprise data is brought into data lakes and data warehouses to carry out analytical, reporting, and data science use cases using AWS analytical services like Amazon Athena , Amazon Redshift , Amazon EMR , and so on. A question arises on what level of details we need to include in the table metadata.

Metadata 89

Metadata Data Lake Modeling Data Warehouse 89

Cloudera

JULY 21, 2022

Cloudera Data Warehouse). Efficient batch data processing. Complex data transformations. Built-in workflow: In addition to querying capabilities, Rill includes scheduled exports and alerts to stay on top of regular reporting and provide opportunities to dive deeper. Apache Hive. Joins and subqueries . Apache Druid.

Metrics 87

Metrics Slice and Dice Data Warehouse Dashboards 87

Smarten

MAY 13, 2024

To accomplish this interchange, the method uses data mining and machine learning and it contains components like a data dictionary to define the fields used by the model, and data transformation to map user data and make it easier for the system to mine that data. No programming or scripting required.

Predictive Modeling 64

Predictive Modeling Data mining Predictive Analytics Modeling 64

AWS Big Data

AUGUST 26, 2024

Duplicating data from a production database to a lower or lateral environment and masking personally identifiable information (PII) to comply with regulations enables development, testing, and reporting without impacting critical systems or exposing sensitive customer data. PII detection and scrubbing.

Visualization 107

Visualization Metadata Data Transformation Testing 107