Wayne Yaddow

FEBRUARY 4, 2025

Its EssentialVerifying Data Transformations (Part4) Uncovering the leading problems in data transformation workflowsand practical ways to detect and preventthem In Parts 13 of this series of blogs, categories of data transformations were identified as among the top causes of data quality defects in data pipeline workflows.

Data Transformation 52

Data Transformation Testing Data Quality Strategy 52

Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

Wayne Yaddow

FEBRUARY 21, 2025

Complex Data TransformationsTest Planning Best Practices Ensuring data accuracy with structured testing and best practices Photo by Taylor Vick on Unsplash Introduction Data transformations and conversions are crucial for data pipelines, enabling organizations to process, integrate, and refine raw data into meaningful insights.

Testing 52

Testing Data Transformation Data Quality Data Integration 52

datapine

SEPTEMBER 29, 2022

In this article, we will detail everything which is at stake when we talk about DQM: why it is essential, how to measure data quality, the pillars of good quality management, and some data quality control techniques. But first, let’s define what data quality actually is. 4 – Data Reporting. date, month, and year).

Data Quality 241

Data Quality Metrics Data-driven Management 241

Wayne Yaddow

FEBRUARY 6, 2025

Common challenges and practical mitigation strategies for reliable data transformations. Photo by Mika Baumeister on Unsplash Introduction Data transformations are important processes in data engineering, enabling organizations to structure, enrich, and integrate data for analytics , reporting, and operational decision-making.

Testing 52

Testing Data Transformation Data-driven Manufacturing 52

CIO Business Intelligence

APRIL 8, 2025

Fragmented systems, inconsistent definitions, outdated architecture and manual processes contribute to a silent erosion of trust in data. When financial data is inconsistent, reporting becomes unreliable. A compliance report is rejected because timestamps dont match across systems. Assign domain data stewards.

Data Quality 102

Data Quality Data-driven Key Performance Indicator Metadata 102

Wayne Yaddow

FEBRUARY 26, 2025

AI is transforming how senior data engineers and data scientists validate data transformations and conversions. Artificial intelligence-based verification approaches aid in the detection of anomalies, the enforcement of data integrity, and the optimization of pipelines for improved efficiency.

Data Transformation 52

Data Transformation Testing Data-driven Data Quality 52

AWS Big Data

OCTOBER 30, 2024

After configuring the data source, launch Power BI. Create a blank report or use an existing report to integrate the new visuals. Choose Get Data and select the name of the data source you created. After authorization is complete, you can build your reports in Microsoft Power BI with the subscribed data assets.

Visualization 119

Visualization Data Lake Testing Data Governance 119

CIO Business Intelligence

NOVEMBER 17, 2022

According to the survey, 80% of the top KPIs that CDOs report focusing on are business oriented. The top five KPIs for CDOs include operational efficiency, data privacy and protection, productivity and capacity, innovation and revenue, and customer satisfaction and success.

Data Governance 128

Data Governance Data Transformation Strategy Data Strategy 128

Sisense

OCTOBER 30, 2019

A critical part of effectively exploring your data, transforming it into actionable insights, and enhancing decision-making for your business is being empowered to slice and dice your data, and be less dependent on technical resources for new updates. Analytics reports are a vital part of this process.

Data-driven 70

Data-driven Reporting Management Slice and Dice 70

AWS Big Data

NOVEMBER 22, 2024

Amazon Redshift has launched a session reuse capability for the Data API that can significantly streamline multi-step, stateful workloads such as exchange, transform, and load (ETL) pipelines, reporting processes, and other flows that involve sequential queries.

Data Warehouse 105

Data Warehouse Recreation/Entertainment Cost-Benefit Data-driven 105

Wayne Yaddow

MARCH 5, 2025

In this post, well see the fundamental procedures, tools, and techniques that data engineers, data scientists, and QA/testing teams use to ensure high-quality data as soon as its deployed. First, we look at how unit and integration tests uncover transformation errors at an early stage.

Testing 52

Testing Data Transformation Statistics Metadata 52

Alation

OCTOBER 26, 2021

They need trusted data to drive reliable reporting, decision-making, and risk reduction. A Strong Data Culture Supports Strategic Decision Making. Our successful customers invest in and infuse data and analytics throughout the enterprise. After all, finance is one of the greatest consumers of data within a business.

Finance 52

Finance Data Transformation Enterprise Metrics 52

AWS Big Data

JANUARY 15, 2025

In the following section, two use cases demonstrate how the data mesh is established with Amazon DataZone to better facilitate machine learning for an IoT-based digital twin and BI dashboards and reporting using Tableau. In the past, one-to-one connections were established between Tableau and respective applications.

IoT 111

IoT Machine Learning Metadata Data-driven 111

Cloudera

AUGUST 17, 2021

When we announced the GA of Cloudera Data Engineering back in September of last year, a key vision we had was to simplify the automation of data transformation pipelines at scale. Let’s take a common use-case for Business Intelligence reporting. Figure 2: Example BI reporting data pipeline.

Management 115

Management Cost-Benefit Data Transformation Optimization 115

FineReport

MARCH 7, 2024

This contemplation is paramount in the realm of data analysis reporting, where the practical application of big data takes center stage. Data Analysis Report (by FineReport ) Note: All the data analysis reports in this article are created using the FineReport reporting tool.

Dashboards 52

Dashboards Reporting Advertising Statistics 52

Smart Data Collective

FEBRUARY 20, 2020

Big data is changing the nature of invoicing software in many ways. In 2015, Spend Matters wrote a detailed report on the applications of big data in the e-invoicing industry. Big Data Transforms Invoicing Software Applications. Detailed Reports and Follow-Ups.

Big Data 141

Big Data Software Reporting Interactive 141

erwin

SEPTEMBER 17, 2020

According to erwin’s “2020 State of Data Governance and Automation” report , close to 70 percent of data professional respondents say they spend an average of 10 or more hours per week on data-related activities, and most of that time is spent searching for and preparing data.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

Octopai

MARCH 31, 2021

In other words, kind of like Hansel and Gretel in the forest, your data leaves a trail of breadcrumbs – the metadata – to record where it came from and who it really is. So the first step in any data lineage mapping project is to ensure that all of your data transformation processes do in fact accurately record metadata.

Metadata 133

Metadata Data Transformation Business Intelligence Reporting 133

AWS Big Data

JANUARY 27, 2023

Kinesis Data Firehose uses Lambda to perform data transformation and compression, storing the file in a compressed columnar format (Parquet) in the target S3 bucket. The AWS Glue Data Catalog has the table definitions for the data sources.

Analytics 87

Analytics Reporting Dashboards Visualization 87

CIO Business Intelligence

NOVEMBER 7, 2023

Given the importance of sharing information among diverse disciplines in the era of digital transformation, this concept is arguably as important as ever. The aim is to normalize, aggregate, and eventually make available to analysts across the organization data that originates in various pockets of the enterprise.

Data Warehouse 142

Data Warehouse Digital Transformation Data-driven Reporting 142

AWS Big Data

SEPTEMBER 5, 2024

Federated queries are useful for use cases where organizations want to combine data from their operational systems with data stored in Amazon Redshift. Federated queries allow querying data across Amazon RDS for MySQL and PostgreSQL data sources without the need for extract, transform, and load (ETL) pipelines.

IoT 111

IoT Data Warehouse Cost-Benefit Reporting 111

AWS Big Data

NOVEMBER 9, 2023

In this post, we delve into a case study for a retail use case, exploring how the Data Build Tool (dbt) was used effectively within an AWS environment to build a high-performing, efficient, and modern data platform. It does this by helping teams handle the T in ETL (extract, transform, and load) processes.

Data Warehouse 126

Data Warehouse Testing Data Quality Reporting 126

erwin

APRIL 29, 2020

These tools range from enterprise service bus (ESB) products, data integration tools; extract, transform and load (ETL) tools, procedural code, application program interfaces (API)s, file transfer protocol (FTP) processes, and even business intelligence (BI) reports that further aggregate and transform data.

Key Performance Indicator 111

Key Performance Indicator Metadata Data Governance Data Quality 111

DataKitchen

JULY 27, 2023

Azure Blob Storage serves as the data lake to store raw data. Azure Databricks, a big data analytics platform built on Apache Spark, performs the actual data transformations. As part of these workflows, Azure Functions can be used to perform small pieces of data transformation logic.

Machine Learning 130

Machine Learning Cost-Benefit Data Transformation Testing 130

AWS Big Data

JANUARY 6, 2025

With Amazon AppFlow, you can run data flows at nearly any scale and at the frequency you chooseon a schedule, in response to a business event, or on demand. You can configure data transformation capabilities such as filtering and validation to generate rich, ready-to-use data as part of the flow itself, without additional steps.

Analytics 98

Analytics Data Warehouse Big Data Metrics 98

David Menninger's Analyst Perspectives

NOVEMBER 7, 2024

A side benefit of AI-enabled business applications is the increasing availability of useful, timely and consistent data for forecasting, planning, analysis and reporting. The next important step is creating an enterprise planning and reporting database of record.

Forecasting 130

Forecasting Machine Learning Finance Predictive Analytics 130

AWS Big Data

NOVEMBER 27, 2024

Amazon Redshift is a fast, scalable, secure, and fully managed cloud data warehouse that makes it simple and cost-effective to analyze all your data using standard SQL and your existing ETL, business intelligence (BI), and reporting tools. dbt Cloud is a hosted service that helps data teams productionize dbt deployments.

Data Warehouse 110

Data Warehouse Analytics Testing Sales 110

DataKitchen

OCTOBER 5, 2022

DataOps Observability can help you ensure that your complex data pipelines and processes are accurate and that they deliver as designed. Observability also validates that your data transformations, models, and reports are performing as expected. to monitor your data operations. without replacing staff or systems?to

Testing 173

Testing Risk Data Processing Statistics 173

erwin

MARCH 4, 2021

And when you talk about that question at a high level, he says, you get a very “simple answer,”– which is ‘the only thing we want to have is the right data with the right quality to the right person at the right time at the right cost.’. The Why: Data Governance Drivers. Why should companies care about data governance?

Data Governance 122

Data Governance Digital Transformation Data-driven Cost-Benefit 122

CIO Business Intelligence

JUNE 7, 2022

Data analytics draws from a range of disciplines — including computer programming, mathematics, and statistics — to perform analysis on data in an effort to describe, predict, and improve performance. What are the four types of data analytics? Data analytics and data science are closely related.

Data Analytics 135

Data Analytics Diagnostic Analytics Management Analytics 135

CIO Business Intelligence

AUGUST 3, 2023

At Paytronix, which manages customer loyalty, online ordering, and other systems for its customers, director of data science ­Jesse Marshall wanted to reduce the custom coding of data transformations—the conversion, cleaning, and structuring of data into a form usable for analytics and reports.

Analytics 98

Analytics Data Lake Metadata Cost-Benefit 98

AWS Big Data

DECEMBER 13, 2023

In addition to using native managed AWS services that BMS didn’t need to worry about upgrading, BMS was looking to offer an ETL service to non-technical business users that could visually compose data transformation workflows and seamlessly run them on the AWS Glue Apache Spark-based serverless data integration engine.

Metadata 111

Metadata Data Lake Visualization Data Quality 111

Cloudera

OCTOBER 19, 2021

Airflow has been adopted by many Cloudera Data Platform (CDP) customers in the public cloud as the next generation orchestration service to setup and operationalize complex data pipelines. The same Airflow job can now be used to generate different SQL reports. Looking forward.

Data Transformation 120

Data Transformation Interactive Machine Learning Testing 120

AWS Big Data

AUGUST 1, 2024

However, you might face significant challenges when planning for a large-scale data warehouse migration. Trace the flow of data from its origins in the source systems, through the data warehouse, and ultimately to its consumption by reporting, analytics, and other downstream processes.

Data Warehouse 115

Data Warehouse KPI Optimization Cost-Benefit 115

CIO Business Intelligence

FEBRUARY 20, 2024

Legacy data management is holding back manufacturing transformation Until now, however, this vision has remained out of reach. The data transformation imperative What Denso and other industry leaders realise is that for IT-OT convergence to be realised, and the benefits of AI unlocked, data transformation is vital.

Data Architecture 98

Data Architecture Unstructured Data Manufacturing IT 98

Jet Global

MAY 28, 2019

The success of any business into the next year and beyond will depend entirely on the volume, accuracy, and reportability of the data they collect—and how well the business can analyze, extract insight from, and take action on that data. Enter the Warehouse.

Data Warehouse 85

Data Warehouse Reporting Business Intelligence Sales 85