Wayne Yaddow

FEBRUARY 21, 2025

Complex Data TransformationsTest Planning Best Practices Ensuring data accuracy with structured testing and best practices Photo by Taylor Vick on Unsplash Introduction Data transformations and conversions are crucial for data pipelines, enabling organizations to process, integrate, and refine raw data into meaningful insights.

Testing 52

Testing Data Transformation Data Quality Data Integration 52

Wayne Yaddow

FEBRUARY 4, 2025

Its EssentialVerifying Data Transformations (Part4) Uncovering the leading problems in data transformation workflowsand practical ways to detect and preventthem In Parts 13 of this series of blogs, categories of data transformations were identified as among the top causes of data quality defects in data pipeline workflows.

Data Transformation 52

Data Transformation Testing Data Quality Strategy 52

Wayne Yaddow

FEBRUARY 11, 2025

Managing tests of complex data transformations when automated data testing tools lack important features? Photo by Marvin Meyer on Unsplash Introduction Data transformations are at the core of modern business intelligence, blending and converting disparate datasets into coherent, reliable outputs.

Testing 52

Testing Data Transformation Data Quality Statistics 52

Wayne Yaddow

FEBRUARY 18, 2025

Selecting the strategies and tools for validating data transformations and data conversions in your data pipelines. Introduction Data transformations and data conversions are crucial to ensure that raw data is organized, processed, and ready for useful analysis.

Testing 52

Testing Data Transformation Data-driven Data Quality 52

DataKitchen

NOVEMBER 10, 2023

Extrinsic Control Deficit: Many of these changes stem from tools and processes beyond the immediate control of the data team. Unregulated ETL/ELT Processes: The absence of stringent data quality tests in ETL (Extract, Transform, Load) or ELT (Extract, Load, Transform) processes further exacerbates the problem.

Data Quality 139

Data Quality Testing Data Lake Data Integration 139

Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

Wayne Yaddow

FEBRUARY 26, 2025

AI is transforming how senior data engineers and data scientists validate data transformations and conversions. Artificial intelligence-based verification approaches aid in the detection of anomalies, the enforcement of data integrity, and the optimization of pipelines for improved efficiency.

Data Transformation 52

Data Transformation Testing Data-driven Data Quality 52

Wayne Yaddow

FEBRUARY 6, 2025

Common challenges and practical mitigation strategies for reliable data transformations. Photo by Mika Baumeister on Unsplash Introduction Data transformations are important processes in data engineering, enabling organizations to structure, enrich, and integrate data for analytics , reporting, and operational decision-making.

Testing 52

Testing Data Transformation Data-driven Manufacturing 52

Wayne Yaddow

MARCH 5, 2025

In this post, well see the fundamental procedures, tools, and techniques that data engineers, data scientists, and QA/testing teams use to ensure high-quality data as soon as its deployed. First, we look at how unit and integration tests uncover transformation errors at an early stage.

Testing 52

Testing Data Transformation Statistics Metadata 52

DataKitchen

APRIL 26, 2021

In early April 2021, DataKItchen sat down with Jonathan Hodges, VP Data Management & Analytics, at Workiva ; Chuck Smith, VP of R&D Data Strategy at GlaxoSmithKline (GSK) ; and Chris Bergh, CEO and Head Chef at DataKitchen, to find out about their enterprise DataOps transformation journey, including key successes and lessons learned.

Measurement 246

Measurement Metrics Data-driven Dashboards 246

CIO Business Intelligence

DECEMBER 22, 2022

“This style of organization is useful for any data-oriented work, making it easier to take advantage of the benefits offered by building a global data fabric.” Analytics, Collaboration Software, Data Management, Data Mining, Data Science, IT Strategy, Small and Medium Business.

Analytics 130

Analytics Machine Learning Data mining Data Quality 130

Smart Data Collective

OCTOBER 13, 2021

There are countless examples of big data transforming many different industries. There is no disputing the fact that the collection and analysis of massive amounts of unstructured data has been a huge breakthrough. How does Data Virtualization manage data quality requirements? In forecasting future events.

Visualization 145

Visualization Cost-Benefit Big Data Prescriptive Analytics 145

AWS Big Data

AUGUST 1, 2024

However, you might face significant challenges when planning for a large-scale data warehouse migration. Effective planning, thorough risk assessment, and a well-designed migration strategy are crucial to mitigating these challenges and implementing a successful transition to the new data warehouse environment on Amazon Redshift.

Data Warehouse 114

Data Warehouse KPI Optimization Cost-Benefit 114

Smart Data Collective

APRIL 29, 2020

A survey from Tech Pro Research showed that 70 percent of organisations already have a digital transformation strategy or are developing one. Solutions for the various data management processes need to be carefully considered. The techniques for managing organisational data in a standardised approach that minimises inefficiency.

Management 99

Management Data Warehouse Digital Transformation Dashboards 99

CIO Business Intelligence

DECEMBER 6, 2023

Often, tech vendors act as an extended workforce, providing manpower and technological expertise for their client’s digital transformation journey. When taking this to the next level, vendor partners act as co-innovators, helping businesses craft winning strategies based on innovation.

Digital Transformation 116

Digital Transformation Software Cost-Benefit Manufacturing 116

CIO Business Intelligence

FEBRUARY 16, 2023

But to augment its various businesses with ML and AI, Iyengar’s team first had to break down data silos within the organization and transform the company’s data operations. Digitizing was our first stake at the table in our data journey,” he says. The offensive side?

Unstructured Data 106

Unstructured Data Data Lake Prescriptive Analytics Data Warehouse 106

Ontotext

MARCH 20, 2024

Traditional data integration methods struggle to bridge these gaps, hampered by high costs, data quality concerns, and inconsistencies. Studies reveal that businesses lose significant time and opportunities due to missing integrations and poor data quality and accessibility.

Data-driven 52

Data-driven Strategy Sales Data Integration 52

erwin

SEPTEMBER 17, 2020

Here are six benefits of automating end-to-end data lineage: Reduced Errors and Operational Costs. Data quality is crucial to every organization. Automated data capture can significantly reduce errors when compared to manual entry. However, different types of data need to be treated differently.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

CIO Business Intelligence

NOVEMBER 7, 2023

Given the importance of sharing information among diverse disciplines in the era of digital transformation, this concept is arguably as important as ever. The aim is to normalize, aggregate, and eventually make available to analysts across the organization data that originates in various pockets of the enterprise.

Data Warehouse 142

Data Warehouse Digital Transformation Data-driven Reporting 142

IBM Big Data Hub

FEBRUARY 9, 2023

In our last blog , we delved into the seven most prevalent data challenges that can be addressed with effective data governance. Today we will share our approach to developing a data governance program to drive data transformation and fuel a data-driven culture.

Data Governance 76

Data Governance Business Objectives Data Quality Measurement 76

AWS Big Data

FEBRUARY 8, 2023

It’s common to ingest multiple data sources into Amazon Redshift to perform analytics. Often, each data source will have its own processes of creating and maintaining data, which can lead to data quality challenges within and across sources. Answering questions as simple as “How many unique customers do we have?”

Data Quality 98

Data Quality Testing Data Warehouse Unstructured Data 98

AWS Big Data

APRIL 19, 2023

Background The success of a data-driven organization recognizes data as a key enabler to increase and sustain innovation. The goal of a data product is to solve the long-standing issue of data silos and data quality. Conclusion Data-driven organizations are transitioning to a data product way of thinking.

Technology 105

Technology Data-driven Machine Learning Sales 105

AWS Big Data

JULY 20, 2023

Prior to the creation of the data lake, Orca’s data was distributed among various data silos, each owned by a different team with its own data pipelines and technology stack. Moreover, running advanced analytics and ML on disparate data sources proved challenging.

Data Lake 98

Data Lake Analytics Snapshot Data Quality 98

Ontotext

FEBRUARY 12, 2024

Usually, organizations will combine different domain topologies, depending on the trade-offs, and choose to focus on specific aspects of data mesh. Once accomplished, an effective implementation spurs a mindset in which organizations prioritize and value data for decision-making, formulating strategies, and day-to-day operations.

Data-driven 59

Data-driven Data Lake Data Quality Business Objectives 59

Octopai

JUNE 9, 2024

This challenge is especially critical for executives responsible for data strategy and operations. Here’s how automated data lineage can transform these challenges into opportunities, as illustrated by the journey of a health services company we’ll call “HealthCo.”

IT 52

IT Data-driven Predictive Analytics Data Strategy 52

Alation

NOVEMBER 22, 2022

What Is Data Governance In The Public Sector? Effective data governance for the public sector enables entities to ensure data quality, enhance security, protect privacy, and meet compliance requirements. With so much focus on compliance, democratizing data for self-service analytics can present a challenge.

Data Governance 52

Data Governance Metadata Data-driven Unstructured Data 52

Octopai

JANUARY 19, 2021

Octopai is the first BI Intelligence platform to analyze Azure Data Factory in hybrid BI environments, providing automated data lineage and discovery and will continue to announce the early support of more platforms as part of an overall strategy to have one centralized view of the entire BI landscape. “We

Metadata 52

Metadata ROI Machine Learning Data Quality 52

erwin

JANUARY 11, 2019

Organizations have spent a lot of time and money trying to harmonize data across diverse platforms , including cleansing, uploading metadata, converting code, defining business glossaries, tracking data transformations and so on. So questions linger about whether transformed data can be trusted.

Data Governance 40

Data Governance Risk Metadata Management 40

Peter James Thomas

OCTOBER 30, 2018

Prelude… I recently came across an article in Marketing Week with the clickbait-worthy headline of Why the rise of the chief data officer will be short-lived (their choice of capitalisation). It may well be that one thing that a CDO needs to get going is a data transformation programme. It may be to improve Data Quality.

Marketing 51

Marketing Strategy Data Architecture Data Strategy 51

Jet Global

MAY 28, 2019

A data warehouse is typically used by companies with a high level of data diversity or analytical requirements. As the complexity and volume of data used in the enterprise scales and organizations want to get more out of their analytics efforts, data warehouses are gaining more traction for reporting and analytics over databases.

Data Warehouse 85

Data Warehouse Reporting Business Intelligence Sales 85

IBM Big Data Hub

MARCH 14, 2024

Businesses face significant hurdles when preparing data for artificial intelligence (AI) applications. The existence of data silos and duplication, alongside apprehensions regarding data quality, presents a multifaceted environment for organizations to manage.

Cost-Benefit 75

Cost-Benefit Metadata Optimization Management 75

Alation

AUGUST 11, 2022

But there are only so many data engineers available in the market today; there’s a big skills shortage. So to get away from that lack of data engineers, what data mesh says is, ‘Take those business logic data transformation capabilities and move that to the domains.’ Let’s take data privacy as an example.

Metadata 52

Metadata Data Warehouse Data Quality Data Lake 52

AWS Big Data

MARCH 3, 2023

Additionally, the scale is significant because the multi-tenant data sources provide a continuous stream of testing activity, and our users require quick data refreshes as well as historical context for up to a decade due to compliance and regulatory demands. Finally, data integrity is of paramount importance.

Software 52

Software Data Lake Testing Dashboards 52

DataRobot Blog

MARCH 10, 2022

This is currently a widespread strategy across the industry where we are seeing companies move from reactive to predictive inventory management and capacity planning. The first step in building a model that can predict machine failure and even recommend the next best course of action is to aggregate, clean, and prepare data to train against.

Manufacturing 52

Manufacturing IoT Machine Learning Forecasting 52

Alation

OCTOBER 28, 2021

To make good on this potential, healthcare organizations need to understand their data and how they can use it. These systems should collectively maintain data quality, integrity, and security, so the organization can use data effectively and efficiently. Why Is Data Governance in Healthcare Important?

Data Governance 52

Data Governance Measurement Data Quality Metrics 52

AWS Big Data

APRIL 5, 2023

The company decided to use AWS to unify its business intelligence (BI) and reporting strategy for both internal organization-wide use cases and in-product embedded analytics targeted at its customers. If each tool tells a different story because it has different data, we won’t have alignment within the business on what this data means.”

Dashboards 52

Dashboards Reporting Cost-Benefit Visualization 52

Jet Global

MAY 9, 2024

Data Extraction : The process of gathering data from disparate sources, each of which may have its own schema defining the structure and format of the data and making it available for processing. This can include tasks such as data ingestion, cleansing, filtering, aggregation, or standardization.

Data Lake 52

Data Lake Data Warehouse Business Intelligence Machine Learning 52