Wayne Yaddow

FEBRUARY 11, 2025

Managing tests of complex data transformations when automated data testing tools lack important features? Photo by Marvin Meyer on Unsplash Introduction Data transformations are at the core of modern business intelligence, blending and converting disparate datasets into coherent, reliable outputs.

Testing 52

Testing Data Transformation Data Quality Statistics 52

Wayne Yaddow

FEBRUARY 18, 2025

Selecting the strategies and tools for validating data transformations and data conversions in your data pipelines. Introduction Data transformations and data conversions are crucial to ensure that raw data is organized, processed, and ready for useful analysis.

Testing 52

Testing Data Transformation Data-driven Data Quality 52

Wayne Yaddow

FEBRUARY 6, 2025

Common challenges and practical mitigation strategies for reliable data transformations. Photo by Mika Baumeister on Unsplash Introduction Data transformations are important processes in data engineering, enabling organizations to structure, enrich, and integrate data for analytics , reporting, and operational decision-making.

Testing 52

Testing Data Transformation Data-driven Manufacturing 52

CIO Business Intelligence

DECEMBER 22, 2022

Research firm Gartner further describes the methodology as one focused on “improving the communication, integration, and automation of data flows between data managers and data consumers across an organization.” Data scientists may also be included as key members of DataOps teams, according to Dunning. “I

Analytics 130

Analytics Machine Learning Data mining Software 130

CIO Business Intelligence

JUNE 23, 2022

Yet as companies fight for skilled analyst roles to utilize data to make better decisions , they often fall short in improving the data supply chain and resulting data quality. Without a solid data supply-chain management practices in place, data quality often suffers.

Data Quality 98

Data Quality Key Performance Indicator Metrics KPI 98

AWS Big Data

DECEMBER 13, 2023

In collaboration with AWS, BMS identified a business need to migrate and modernize their custom extract, transform, and load (ETL) platform to a native AWS solution to reduce complexities, resources, and investment to upgrade when new Spark, Python, or AWS Glue versions are released.

Metadata 105

Metadata Data Lake Visualization Data Quality 105

AWS Big Data

NOVEMBER 9, 2023

It does this by helping teams handle the T in ETL (extract, transform, and load) processes. It allows users to write data transformation code, run it, and test the output, all within the framework it provides. As part of their cloud modernization initiative, they sought to migrate and modernize their legacy data platform.

Data Warehouse 122

Data Warehouse Testing Data Quality Reporting 122

erwin

APRIL 29, 2020

Compliance with these business rules can be tracked through data lineage, incorporating auditability and validation controls across data transformations and pipelines to generate alerts when there are non-compliant data instances. Seeing data pipelines and information flows further supports compliance efforts.

Metadata 111

Metadata Key Performance Indicator Data Governance Data Quality 111

Smart Data Collective

OCTOBER 13, 2021

There are countless examples of big data transforming many different industries. There is no disputing the fact that the collection and analysis of massive amounts of unstructured data has been a huge breakthrough. How does Data Virtualization manage data quality requirements?

Visualization 145

Visualization Cost-Benefit Big Data Prescriptive Analytics 145

CIO Business Intelligence

JANUARY 24, 2024

“Organizations often get services and applications up and running without having put stewardship in place,” says Marc Johnson, CISO and senior advisor at Impact Advisors, a healthcare management consulting firm. Creating data silos Denying business users access to information because of data silos has been a problem for years.

Data Strategy 136

Data Strategy Strategy Unstructured Data Data Governance 136

Sisense

AUGUST 28, 2020

Uncomfortable truth incoming: Most people in your organization don’t think about the quality of their data from intake to production of insights. However, as a data team member, you know how important data integrity (and a whole host of other aspects of data management) is.

Data Integration 101

Data Integration Testing Data Quality Data-driven 101

CIO Business Intelligence

DECEMBER 6, 2023

Despite logistics challenges caused by the global pandemic, the company managed to rapidly scale up its team to over 1,000 people in a period of only 11 months.

Digital Transformation 116

Digital Transformation Software Cost-Benefit Manufacturing 116

Octopai

JUNE 22, 2023

In the era of big data, organizations are grappling with the challenge of effectively managing and leveraging vast amounts of data. Two prominent approaches have emerged: self-service data management and centralized data management.

Management 52

Management Data-driven Data Quality Data Governance 52

erwin

SEPTEMBER 17, 2020

Here are six benefits of automating end-to-end data lineage: Reduced Errors and Operational Costs. Data quality is crucial to every organization. Automated data capture can significantly reduce errors when compared to manual entry. Improved Customer and Employee Satisfaction.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

erwin

MARCH 4, 2021

However, while digital transformation and other data-driven initiatives are desired outcomes, few organizations know what data they have or where it is, and they struggle to integrate known data in various formats and numerous systems – especially if they don’t have a way to automate those processes.

Data Governance 122

Data Governance Digital Transformation Data-driven Cost-Benefit 122

CIO Business Intelligence

FEBRUARY 16, 2023

My vision is that I can give the keys to my businesses to manage their data and run their data on their own, as opposed to the Data & Tech team being at the center and helping them out,” says Iyengar, director of Data & Tech at Straumann Group North America. The offensive side?

Unstructured Data 105

Unstructured Data Data Lake Prescriptive Analytics Data Warehouse 105

AWS Big Data

AUGUST 1, 2024

Large-scale data warehouse migration to the cloud is a complex and challenging endeavor that many organizations undertake to modernize their data infrastructure, enhance data management capabilities, and unlock new business opportunities.

Data Warehouse 105

Data Warehouse KPI Optimization Cost-Benefit 105

AWS Big Data

JULY 20, 2023

Prior to the creation of the data lake, Orca’s data was distributed among various data silos, each owned by a different team with its own data pipelines and technology stack. Moreover, running advanced analytics and ML on disparate data sources proved challenging.

Data Lake 96

Data Lake Analytics Snapshot Data Quality 96

DataKitchen

DECEMBER 9, 2022

DataOps involves close collaboration between data scientists, IT professionals, and business stakeholders, and it often involves the use of automation and other technologies to streamline data-related tasks. One of the key benefits of DataOps is the ability to accelerate the development and deployment of data-driven solutions.

Machine Learning 130

Machine Learning Data-driven Optimization Data Analytics 130

erwin

JANUARY 11, 2019

Organizations have spent a lot of time and money trying to harmonize data across diverse platforms , including cleansing, uploading metadata, converting code, defining business glossaries, tracking data transformations and so on. So questions linger about whether transformed data can be trusted.

Data Governance 40

Data Governance Risk Metadata Management 40

AWS Big Data

DECEMBER 21, 2023

AWS Glue provides both visual and code-based interfaces to make data integration effortless. Using a native AWS Glue connector increases agility, simplifies data movement, and improves data quality. Attach the AWS managed policy GlueServiceRole. Choose Dashboards Management on the navigation menu.

Analytics 110

Analytics IT Data Lake Visualization 110

IBM Big Data Hub

FEBRUARY 9, 2023

In our last blog , we delved into the seven most prevalent data challenges that can be addressed with effective data governance. Today we will share our approach to developing a data governance program to drive data transformation and fuel a data-driven culture.

Data Governance 76

Data Governance Business Objectives Data Quality Measurement 76

Sisense

OCTOBER 15, 2020

Dealing with this challenge requires caching the relevant context on the processing instances (state management) using techniques like sliding time windows. Another goal that teams dealing with streaming data may have is managing and optimizing a file system on object storage. Optimizing object storage. Ori has a B.A.

Dashboards 98

Dashboards IoT Optimization Internet of Things 98

AWS Big Data

FEBRUARY 8, 2023

Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. Amazon Redshift enables you to run complex SQL analytics at scale and performance on terabytes to petabytes of structured and unstructured data, and make the insights widely available through popular business intelligence (BI) and analytics tools.

Data Quality 89

Data Quality Testing Data Warehouse Unstructured Data 89

Data Virtualization

JULY 20, 2023

Reading Time: < 1 minute In this post, I’m going to cover logical data management and its impact on data mesh architectures. But there’s a lot of confusion in the marketplace today between different types of architectures, specifically data mesh and data fabric, so I’ll.

Management 40

Management Data Integration IT Data Quality 40

Octopai

JUNE 9, 2024

These data products were intended to enhance patient outcomes, streamline hospital operations, and provide actionable insights for decision-making. This strategic choice justified further investment into their data team, infrastructure, management, and science. The complexity of their data ecosystem became a major obstacle.

IT 52

IT Data-driven Predictive Analytics Data Strategy 52

AWS Big Data

APRIL 19, 2023

Data mesh is a new approach to data management. Companies across industries are using a data mesh to decentralize data management to improve data agility and get value from data. The goal of a data product is to solve the long-standing issue of data silos and data quality.

Technology 98

Technology Data-driven Machine Learning Sales 98

AWS Big Data

JULY 27, 2023

DataBrew is a visual data preparation tool that enables you to clean and normalize data without writing any code. The over 200 transformations it provides are now available to be used in an AWS Glue Studio visual job. Now that we identified the data quality issues to address, we need to decide how to deal with each case.

Visualization 94

Visualization Cost-Benefit Data Quality Publishing 94

Octopai

JANUARY 19, 2021

With Octopai’s support and analysis of Azure Data Factory, enterprises can now view complete end-to-end data lineage from Azure Data Factory all the way through to reporting for the first time ever. About Octopai: Octopai was founded in 2015 by BI professionals who realized the need for dynamic solutions in a stagnant market.

Metadata 52

Metadata ROI Machine Learning Data Quality 52

IBM Big Data Hub

MARCH 14, 2024

Businesses face significant hurdles when preparing data for artificial intelligence (AI) applications. The existence of data silos and duplication, alongside apprehensions regarding data quality, presents a multifaceted environment for organizations to manage.

Cost-Benefit 75

Cost-Benefit Metadata Optimization Management 75

Alation

MARCH 22, 2022

As the latest iteration in this pursuit of high-quality data sharing, DataOps combines a range of disciplines. It synthesizes all we’ve learned about agile, data quality , and ETL/ELT. In fact, DataOps places an outsized burden on data management teams. How the IDF Supports a Smarter Data Pipeline.

Metadata 52

Metadata Cost-Benefit Data Quality Data Lake 52

Octopai

AUGUST 11, 2024

In essence, data flow lineage is indispensable for ensuring transparency, maintaining data quality, achieving compliance, enabling efficient troubleshooting, conducting impact analysis, and enhancing collaboration within organizations. Resource Intensive : Organizations need skilled personnel to manage and customize these tools.

Metadata 40

Metadata Visualization Data Quality Data Governance 40

Ontotext

MARCH 27, 2024

They also don’t have features for enterprise data management such as schema language, data validation capabilities, interoperable serialization formats, or a proper modeling language. RDF is used extensively for data publishing and data interchange and is based on W3C and other industry standards.

Modeling 53

Modeling Metadata Data Quality Enterprise 53

Alation

AUGUST 11, 2022

But there are only so many data engineers available in the market today; there’s a big skills shortage. So to get away from that lack of data engineers, what data mesh says is, ‘Take those business logic data transformation capabilities and move that to the domains.’ Let’s take data privacy as an example.

Metadata 52

Metadata Data Warehouse Data Quality Data Lake 52

AWS Big Data

MARCH 3, 2023

In this post, we share how the AWS Data Lab helped Tricentis to improve their software as a service (SaaS) Tricentis Analytics platform with insights powered by Amazon Redshift. Although Tricentis has amassed such data over a decade, the data remains untapped for valuable insights.

Software 52

Software Data Lake Testing Cost-Benefit 52

Peter James Thomas

OCTOBER 30, 2018

Where they have, I have normally found the people holding these roles to be better informed about data matters than their peers. The same goes in general for Marketing Managers. It may well be that one thing that a CDO needs to get going is a data transformation programme. It may be to introduce or expand Data Governance.

Marketing 51

Marketing Strategy Data Architecture Data Strategy 51

Alation

OCTOBER 28, 2021

Leaders are asking how they might use data to drive smarter decision making to support this new model and improve medical treatments that lead to better outcomes. This data is also a lucrative target for cyber criminals. To make good on this potential, healthcare organizations need to understand their data and how they can use it.

Data Governance 52

Data Governance Measurement Data Quality Metrics 52