Wayne Yaddow

FEBRUARY 4, 2025

Its EssentialVerifying Data Transformations (Part4) Uncovering the leading problems in data transformation workflowsand practical ways to detect and preventthem In Parts 13 of this series of blogs, categories of data transformations were identified as among the top causes of data quality defects in data pipeline workflows.

Data Transformation 52

Data Transformation Testing Data Quality Strategy 52

Wayne Yaddow

FEBRUARY 21, 2025

Complex Data TransformationsTest Planning Best Practices Ensuring data accuracy with structured testing and best practices Photo by Taylor Vick on Unsplash Introduction Data transformations and conversions are crucial for data pipelines, enabling organizations to process, integrate, and refine raw data into meaningful insights.

Testing 52

Testing Data Transformation Data Quality Data Integration 52

Wayne Yaddow

FEBRUARY 11, 2025

Managing tests of complex data transformations when automated data testing tools lack important features? Photo by Marvin Meyer on Unsplash Introduction Data transformations are at the core of modern business intelligence, blending and converting disparate datasets into coherent, reliable outputs.

Testing 52

Testing Data Transformation Data Quality Statistics 52

Alation

DECEMBER 7, 2021

Alation and Bigeye have partnered to bring data observability and data quality monitoring into the data catalog. Read to learn how our newly combined capabilities put more trustworthy, quality data into the hands of those who are best equipped to leverage it. trillion each year due to poor data quality.

Data Quality 52

Data Quality Data-driven Metrics Dashboards 52

DataKitchen

NOVEMBER 10, 2023

Extrinsic Control Deficit: Many of these changes stem from tools and processes beyond the immediate control of the data team. Unregulated ETL/ELT Processes: The absence of stringent data quality tests in ETL (Extract, Transform, Load) or ELT (Extract, Load, Transform) processes further exacerbates the problem.

Data Quality 136

Data Quality Testing Data Lake Data Integration 136

Wayne Yaddow

FEBRUARY 26, 2025

AI is transforming how senior data engineers and data scientists validate data transformations and conversions. Artificial intelligence-based verification approaches aid in the detection of anomalies, the enforcement of data integrity, and the optimization of pipelines for improved efficiency.

Data Transformation 52

Data Transformation Testing Data-driven Data Quality 52

AWS Big Data

JANUARY 15, 2025

Their terminal operations rely heavily on seamless data flows and the management of vast volumes of data. Recently, EUROGATE has developed a digital twin for its container terminal Hamburg (CTH), generating millions of data points every second from Internet of Things (IoT)devices attached to its container handling equipment (CHE).

IoT 107

IoT Machine Learning Metadata Data-driven 107

Wayne Yaddow

MARCH 5, 2025

In this post, well see the fundamental procedures, tools, and techniques that data engineers, data scientists, and QA/testing teams use to ensure high-quality data as soon as its deployed. First, we look at how unit and integration tests uncover transformation errors at an early stage.

Testing 52

Testing Data Transformation Statistics Metadata 52

AWS Big Data

DECEMBER 21, 2023

AWS Glue is a serverless data integration service that makes it straightforward to discover, prepare, and combine data for analytics, machine learning (ML), and application development. AWS Glue provides both visual and code-based interfaces to make data integration effortless.

Analytics 110

Analytics IT Data Lake Visualization 110

erwin

APRIL 29, 2020

Data lineage is the journey data takes from its creation through its transformations over time. It describes a certain dataset’s origin, movement, characteristics and quality. Tracing the source of data is an arduous task. What are the transformation rules?

Key Performance Indicator 111

Key Performance Indicator Metadata Data Governance Data Quality 111

Wayne Yaddow

FEBRUARY 6, 2025

Common challenges and practical mitigation strategies for reliable data transformations. Photo by Mika Baumeister on Unsplash Introduction Data transformations are important processes in data engineering, enabling organizations to structure, enrich, and integrate data for analytics , reporting, and operational decision-making.

Testing 52

Testing Data Transformation Data-driven Manufacturing 52

AWS Big Data

DECEMBER 13, 2023

In addition to using native managed AWS services that BMS didn’t need to worry about upgrading, BMS was looking to offer an ETL service to non-technical business users that could visually compose data transformation workflows and seamlessly run them on the AWS Glue Apache Spark-based serverless data integration engine.

Metadata 107

Metadata Data Lake Visualization Data Quality 107

CIO Business Intelligence

JUNE 23, 2022

Yet as companies fight for skilled analyst roles to utilize data to make better decisions , they often fall short in improving the data supply chain and resulting data quality. Without a solid data supply-chain management practices in place, data quality often suffers. First mile/last mile impacts.

Data Quality 98

Data Quality Key Performance Indicator Metrics KPI 98

AWS Big Data

NOVEMBER 9, 2023

Building a data platform involves various approaches, each with its unique blend of complexities and solutions. In this post, we delve into a case study for a retail use case, exploring how the Data Build Tool (dbt) was used effectively within an AWS environment to build a high-performing, efficient, and modern data platform.

Data Warehouse 124

Data Warehouse Testing Data Quality Reporting 124

CIO Business Intelligence

DECEMBER 22, 2022

DataOps (data operations) is an agile, process-oriented methodology for developing and delivering analytics. It brings together DevOps teams with data engineers and data scientists to provide the tools, processes, and organizational structures to support the data-focused enterprise. What is DataOps?

Analytics 130

Analytics Machine Learning Data mining Software 130

Smart Data Collective

OCTOBER 13, 2021

There are countless examples of big data transforming many different industries. There is no disputing the fact that the collection and analysis of massive amounts of unstructured data has been a huge breakthrough. We would like to talk about data visualization and its role in the big data movement.

Visualization 145

Visualization Cost-Benefit Big Data Prescriptive Analytics 145

CIO Business Intelligence

JANUARY 24, 2024

Unfortunately, the road to data strategy success is fraught with challenges, so CIOs and other technology leaders need to plan and execute carefully. Here are some data strategy mistakes IT leaders would be wise to avoid. Overlooking these data resources is a big mistake. It will not be something they can ignore.

Data Strategy 136

Data Strategy Strategy Unstructured Data Data Governance 136

Sisense

AUGUST 28, 2020

We live in a world of data: There’s more of it than ever before, in a ceaselessly expanding array of forms and locations. Dealing with Data is your window into the ways data teams are tackling the challenges of this new world to help their companies and their customers thrive. What is data integrity?

Data Integration 101

Data Integration Testing Data Quality Data-driven 101

erwin

SEPTEMBER 17, 2020

Replace manual and recurring tasks for fast, reliable data lineage and overall data governance. It’s paramount that organizations understand the benefits of automating end-to-end data lineage. The importance of end-to-end data lineage is widely understood and ignoring it is risky business. Doing Data Lineage Right.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

Smart Data Collective

APRIL 29, 2020

What is data management? Data management can be defined in many ways. The techniques for managing organisational data in a standardised approach that minimises inefficiency. Extraction, Transform, Load (ETL). Data transformation. Data analytics and visualisation. Microsoft Azure.

Management 98

Management Data Warehouse Digital Transformation Dashboards 98

CIO Business Intelligence

FEBRUARY 16, 2023

The company’s orthodontics business, for instance, makes heavy use of image processing to the point that unstructured data is growing at a pace of roughly 20% to 25% per month. For example, imaging data can be used to show patients how an aligner will change their appearance over time. “It The offensive side?

Unstructured Data 105

Unstructured Data Data Lake Prescriptive Analytics Data Warehouse 105

CIO Business Intelligence

DECEMBER 6, 2023

An IDC report estimated the global IT developer shortage will reach four million by 2025, leaving businesses struggling to accelerate digital transformation without the needed workforce. However, this partnership model cannot keep pace with an always-changing technology landscape in which the skill gaps and lack of resources are increasing.

Digital Transformation 116

Digital Transformation Software Cost-Benefit Manufacturing 116

erwin

MARCH 4, 2021

Modern data governance is a strategic, ongoing and collaborative practice that enables organizations to discover and track their data, understand what it means within a business context, and maximize its security, quality and value. The What: Data Governance Defined. Data governance has no standard definition.

Data Governance 122

Data Governance Digital Transformation Data-driven Cost-Benefit 122

Jen Stirrup

SEPTEMBER 30, 2021

They may also learn from evidence, but the data and the modelling fundamentally comes from humans in some way. Data Science – Data science is the field of study that combines domain expertise, programming skills, and knowledge of mathematics and statistics to extract meaningful insights from data. Credit: [link].

Business Intelligence 135

Business Intelligence Data mining Machine Learning Testing 135

IBM Big Data Hub

JANUARY 9, 2024

The emergence of generative AI prompted several prominent companies to restrict its use because of the mishandling of sensitive internal data. Currently, no standardized process exists for overcoming data ingestion’s challenges, but the model’s accuracy depends on it. Increased variance: Variance measures consistency.

Enterprise 71

Enterprise Data Integration Data Quality Contextual Data 71

Octopai

JUNE 9, 2024

.” The Data Strategy HealthCo, like many forward-thinking organizations, recognized early on that data is not just a valuable asset but a strategic imperative. They put data at the forefront of their business, integrating it into decision-making processes, products, and services. The lack of trust in data created inertia.

IT 52

IT Data-driven Predictive Analytics Data Strategy 52

AWS Big Data

AUGUST 1, 2024

However, you might face significant challenges when planning for a large-scale data warehouse migration. Identify all upstream and downstream applications, as well as business processes that rely on the data warehouse. Data transformation experts to convert database stored functions in the producer or consumer.

Data Warehouse 107

Data Warehouse KPI Optimization Cost-Benefit 107

AWS Big Data

FEBRUARY 8, 2023

Amazon Redshift enables you to run complex SQL analytics at scale and performance on terabytes to petabytes of structured and unstructured data, and make the insights widely available through popular business intelligence (BI) and analytics tools. It’s common to ingest multiple data sources into Amazon Redshift to perform analytics.

Data Quality 91

Data Quality Testing Data Warehouse Unstructured Data 91

AWS Big Data

JULY 20, 2023

The Orca Platform is powered by a state-of-the-art anomaly detection system that uses cutting-edge ML algorithms and big data capabilities to detect potential security threats and alert customers in real time, ensuring maximum security for their cloud environment. Why did Orca choose Apache Iceberg?

Data Lake 97

Data Lake Analytics Snapshot Data Quality 97

AWS Big Data

JULY 27, 2023

AWS Glue Studio is a graphical interface that makes it easy to create, run, and monitor extract, transform, and load (ETL) jobs in AWS Glue. DataBrew is a visual data preparation tool that enables you to clean and normalize data without writing any code. This will create a session using a configurable subset of the data.

Visualization 95

Visualization Cost-Benefit Data Quality Publishing 95

IBM Big Data Hub

FEBRUARY 9, 2023

In our last blog , we delved into the seven most prevalent data challenges that can be addressed with effective data governance. Today we will share our approach to developing a data governance program to drive data transformation and fuel a data-driven culture.

Data Governance 76

Data Governance Business Objectives Data Quality Measurement 76

Sisense

OCTOBER 15, 2020

We live in a world of data: There’s more of it than ever before, in a ceaselessly expanding array of forms and locations. Dealing with Data is your window into the ways data teams are tackling the challenges of this new world to help their companies and their customers thrive. Cleaning up dirty data.

Dashboards 98

Dashboards IoT Optimization Internet of Things 98

Ontotext

NOVEMBER 18, 2021

You step onto the market, and if you don’t keep your data, there’s no knowing where you might be swept off to. [1]. Picture this – you start with the perfect use case for your data analytics product. Nowadays, data analytics doesn’t exist on its own. You make a great pitch and you sell well. Sounds unlikely.

Visualization 98

Visualization Reporting Metadata Enterprise 98

DataKitchen

DECEMBER 9, 2022

ChatGPT> DataOps, or data operations, is a set of practices and technologies that organizations use to improve the speed, quality, and reliability of their data analytics processes. One of the key benefits of DataOps observability is the ability to improve collaboration and communication across teams and systems.

Machine Learning 130

Machine Learning Data-driven Optimization Data Analytics 130

Octopai

JUNE 22, 2023

Two prominent approaches have emerged: self-service data management and centralized data management. Each approach has its own merits and trade-offs, and understanding how to leverage data lineage within these frameworks can empower organizations and provide crucial control over their data assets.

Management 52

Management Data-driven Data Quality Data Governance 52

AWS Big Data

APRIL 19, 2023

Background The success of a data-driven organization recognizes data as a key enabler to increase and sustain innovation. The goal of a data product is to solve the long-standing issue of data silos and data quality. A modern data architecture is critical in order to become a data-driven organization.

Technology 100

Technology Data-driven Machine Learning Sales 100

Alation

OCTOBER 18, 2022

Making this data visible in the data catalog will let data teams share their work, support re-use, and empower everyone to better understand and trust data. Data Transformation in the Modern Data Stack. Data engineering plays a critical role in distributing data to a wide audience.

Metadata 52

Metadata Metrics Recreation/Entertainment Data Quality 52