Wayne Yaddow

FEBRUARY 11, 2025

Managing tests of complex data transformations when automated data testing tools lack important features? Photo by Marvin Meyer on Unsplash Introduction Data transformations are at the core of modern business intelligence, blending and converting disparate datasets into coherent, reliable outputs.

Testing 52

Testing Data Transformation Data Quality Statistics 52

Alation

DECEMBER 7, 2021

Alation and Bigeye have partnered to bring data observability and data quality monitoring into the data catalog. Read to learn how our newly combined capabilities put more trustworthy, quality data into the hands of those who are best equipped to leverage it. trillion each year due to poor data quality.

Data Quality 52

Data Quality Data-driven Metrics Dashboards 52

Wayne Yaddow

FEBRUARY 18, 2025

Selecting the strategies and tools for validating data transformations and data conversions in your data pipelines. Introduction Data transformations and data conversions are crucial to ensure that raw data is organized, processed, and ready for useful analysis.

Testing 52

Testing Data Transformation Data-driven Data Quality 52

Wayne Yaddow

FEBRUARY 26, 2025

AI is transforming how senior data engineers and data scientists validate data transformations and conversions. Artificial intelligence-based verification approaches aid in the detection of anomalies, the enforcement of data integrity, and the optimization of pipelines for improved efficiency.

Data Transformation 52

Data Transformation Testing Data-driven Data Quality 52

erwin

APRIL 29, 2020

Business terms and data policies should be implemented through standardized and documented business rules. Compliance with these business rules can be tracked through data lineage, incorporating auditability and validation controls across data transformations and pipelines to generate alerts when there are non-compliant data instances.

Key Performance Indicator 111

Key Performance Indicator Metadata Data Governance Data Quality 111

Smart Data Collective

OCTOBER 13, 2021

There are countless examples of big data transforming many different industries. There is no disputing the fact that the collection and analysis of massive amounts of unstructured data has been a huge breakthrough. Data Virtualization allows accessing them from a single point, replicating them only when strictly necessary.

Visualization 145

Visualization Cost-Benefit Big Data Prescriptive Analytics 145

AWS Big Data

NOVEMBER 9, 2023

In this post, we delve into a case study for a retail use case, exploring how the Data Build Tool (dbt) was used effectively within an AWS environment to build a high-performing, efficient, and modern data platform. It does this by helping teams handle the T in ETL (extract, transform, and load) processes.

Data Warehouse 126

Data Warehouse Testing Data Quality Reporting 126

AWS Big Data

DECEMBER 13, 2023

In addition to using native managed AWS services that BMS didn’t need to worry about upgrading, BMS was looking to offer an ETL service to non-technical business users that could visually compose data transformation workflows and seamlessly run them on the AWS Glue Apache Spark-based serverless data integration engine.

Metadata 110

Metadata Data Lake Visualization Data Quality 110

CIO Business Intelligence

JANUARY 24, 2024

Similar to disaster recovery, business continuity, and information security, data strategy needs to be well thought out and defined to inform the rest, while providing a foundation from which to build a strong business.” Overlooking these data resources is a big mistake.

Data Strategy 136

Data Strategy Strategy Unstructured Data Data Governance 136

CIO Business Intelligence

NOVEMBER 7, 2023

For years, IT and business leaders have been talking about breaking down the data silos that exist within their organizations. Given the importance of sharing information among diverse disciplines in the era of digital transformation, this concept is arguably as important as ever. There’s also the issue of bias.

Data Warehouse 142

Data Warehouse Digital Transformation Data-driven Reporting 142

Sisense

AUGUST 28, 2020

However, as a data team member, you know how important data integrity (and a whole host of other aspects of data management) is. In this article, we’ll dig into the core aspects of data integrity, what processes ensure it, and how to deal with data that doesn’t meet your standards.

Data Integration 101

Data Integration Testing Data Quality Data-driven 101

CIO Business Intelligence

FEBRUARY 16, 2023

But to augment its various businesses with ML and AI, Iyengar’s team first had to break down data silos within the organization and transform the company’s data operations. Digitizing was our first stake at the table in our data journey,” he says. The offensive side?

Unstructured Data 105

Unstructured Data Data Lake Prescriptive Analytics Data Warehouse 105

AWS Big Data

AUGUST 1, 2024

However, you might face significant challenges when planning for a large-scale data warehouse migration. Discovery of workload and integrations Conducting discovery and assessment for migrating a large on-premises data warehouse to Amazon Redshift is a critical step in the migration process.

Data Warehouse 114

Data Warehouse KPI Optimization Cost-Benefit 114

Smart Data Collective

APRIL 29, 2020

The techniques for managing organisational data in a standardised approach that minimises inefficiency. Extraction, Transform, Load (ETL). The extraction of raw data, transforming to a suitable format for business needs, and loading into a data warehouse. Data transformation. Microsoft Azure.

Management 99

Management Data Warehouse Digital Transformation Dashboards 99

erwin

MARCH 4, 2021

But when IT-driven data management and business-oriented data governance work together in terms of both personnel, processes and technology, decisions can be made and their impacts determined based on a full inventory of reliable information. Virginia residents also would be able to opt out of data collection.

Data Governance 122

Data Governance Digital Transformation Data-driven Cost-Benefit 122

erwin

SEPTEMBER 17, 2020

Replace manual and recurring tasks for fast, reliable data lineage and overall data governance. It’s paramount that organizations understand the benefits of automating end-to-end data lineage. Critically, it makes it easier to get a clear view of how information is created and flows into, across and outside an enterprise.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

AWS Big Data

FEBRUARY 8, 2023

It’s common to ingest multiple data sources into Amazon Redshift to perform analytics. Often, each data source will have its own processes of creating and maintaining data, which can lead to data quality challenges within and across sources. Answering questions as simple as “How many unique customers do we have?”

Data Quality 98

Data Quality Testing Data Warehouse Unstructured Data 98

IBM Big Data Hub

JANUARY 9, 2024

The entire generative AI pipeline hinges on the data pipelines that empower it, making it imperative to take the correct precautions. 4 key components to ensure reliable data ingestion Data quality and governance: Data quality means ensuring the security of data sources, maintaining holistic data and providing clear metadata.

Enterprise 71

Enterprise Data Integration Data Quality Contextual Data 71

Jen Stirrup

SEPTEMBER 30, 2021

Before we dive in, let’s define strands of AI, Machine Learning and Data Science: Business intelligence (BI) leverages software and services to transform data into actionable insights that inform an organization’s strategic and tactical business decisions.

Business Intelligence 135

Business Intelligence Data mining Machine Learning Testing 135

AWS Big Data

JULY 20, 2023

Prior to the creation of the data lake, Orca’s data was distributed among various data silos, each owned by a different team with its own data pipelines and technology stack. Moreover, running advanced analytics and ML on disparate data sources proved challenging.

Data Lake 98

Data Lake Analytics Snapshot Data Quality 98

DataKitchen

DECEMBER 9, 2022

By streamlining data-related workflows and enabling real-time collaboration, DataOps can help organizations to quickly turn data into insights, and to put those insights into action. ChatGPT> DataOps observability is a critical aspect of modern data analytics and machine learning.

Machine Learning 130

Machine Learning Data-driven Optimization Data Analytics 130

Ontotext

NOVEMBER 18, 2021

OntoRefine is a data transformation tool that lets you unite plenty of data formats and get them into your triplestore. Now that the data is in the database, we can start benefiting from the RDF technology’s strengths. One of the core upsides of storing your data in that format is inference.

Visualization 98

Visualization Reporting Metadata Enterprise 98

Ontotext

FEBRUARY 12, 2024

This is especially beneficial when teams need to increase data product velocity with trust and data quality, reduce communication costs, and help data solutions align with business objectives. What does data mesh do that other approaches can’t?

Data-driven 59

Data-driven Data Lake Data Quality Business Objectives 59

Alation

OCTOBER 18, 2022

Making this data visible in the data catalog will let data teams share their work, support re-use, and empower everyone to better understand and trust data. Data Transformation in the Modern Data Stack. Data engineering plays a critical role in distributing data to a wide audience.

Metadata 52

Metadata Metrics Recreation/Entertainment Data Quality 52

Octopai

JUNE 22, 2023

On the other hand, centralized data management emphasizes a more structured and governed approach. Data is managed and controlled by a dedicated team of data professionals, ensuring data quality, security, and compliance. This approach offers greater control and reduces the risk of data inconsistencies.

Management 52

Management Data-driven Data Quality Data Governance 52

IBM Big Data Hub

FEBRUARY 9, 2023

In our last blog , we delved into the seven most prevalent data challenges that can be addressed with effective data governance. Today we will share our approach to developing a data governance program to drive data transformation and fuel a data-driven culture.

Data Governance 76

Data Governance Business Objectives Data Quality Measurement 76

Sisense

OCTOBER 15, 2020

Every data professional knows that ensuring data quality is vital to producing usable query results. Streaming data can be extra challenging in this regard, as it tends to be “dirty,” with new fields that are added without warning and frequent mistakes in the data collection process.

Dashboards 98

Dashboards IoT Optimization Internet of Things 98

AWS Big Data

APRIL 19, 2023

Background The success of a data-driven organization recognizes data as a key enabler to increase and sustain innovation. The goal of a data product is to solve the long-standing issue of data silos and data quality. Suppose a consumer is browsing the Customer data product in the data mesh marketplace.

Technology 105

Technology Data-driven Machine Learning Sales 105

AWS Big Data

DECEMBER 21, 2023

AWS Glue provides both visual and code-based interfaces to make data integration effortless. Using a native AWS Glue connector increases agility, simplifies data movement, and improves data quality. For more information, see Setting up networking for development for AWS Glue. Choose Create connection.

Analytics 110

Analytics IT Data Lake Visualization 110

erwin

APRIL 16, 2024

In this blog, we’ll delve into the critical role of governance and data modeling tools in supporting a seamless data mesh implementation and explore how erwin tools can be used in that role. erwin also provides data governance, metadata management and data lineage software called erwin Data Intelligence by Quest.

Metadata 52

Metadata Data Quality Data Governance Modeling 52

Jet Global

MAY 28, 2019

The decision will come down to a database vs a data warehouse—but let’s start by explaining what each is and why they are used. All About That (Data)Base. A database is, by definition, ‘any collection of data organized for storage, accessibility, and retrieval.’ Let’s look at why: Data Quality and Consistency.

Data Warehouse 85

Data Warehouse Reporting Business Intelligence Sales 85

Ontotext

MARCH 20, 2024

In today’s data-driven world, businesses are drowning in a sea of information. Traditional data integration methods struggle to bridge these gaps, hampered by high costs, data quality concerns, and inconsistencies. This allows your teams to make informed decisions based on real data, not just intuition.

Data-driven 52

Data-driven Strategy Sales Data Integration 52

Smarten

AUGUST 4, 2023

As your users become accustomed to augmented analytics, they will want the ability to quickly, and easily, gather data, integrated from disparate data sources. Users can then prepare that data – transforming, shaping, reducing, combining, exploring, cleaning, sampling, and aggregating data, to get the dataset users wish to analyze.

Data Lake 52

Data Lake Machine Learning Data Integration Data Quality 52

Alation

NOVEMBER 22, 2022

What Is Data Governance In The Public Sector? Effective data governance for the public sector enables entities to ensure data quality, enhance security, protect privacy, and meet compliance requirements. With so much focus on compliance, democratizing data for self-service analytics can present a challenge.

Data Governance 52

Data Governance Metadata Data-driven Unstructured Data 52

erwin

JANUARY 11, 2019

Organizations have spent a lot of time and money trying to harmonize data across diverse platforms , including cleansing, uploading metadata, converting code, defining business glossaries, tracking data transformations and so on. So questions linger about whether transformed data can be trusted.

Data Governance 40

Data Governance Risk Metadata Management 40

Octopai

AUGUST 11, 2024

Just as a navigation app provides a detailed map of roads, guiding you from your starting point to your destination while highlighting every turn and intersection, data flow lineage offers a comprehensive view of data movement and transformations throughout its lifecycle. For more information, Contact Us.

Metadata 40

Metadata Visualization Data Quality Data Governance 40

Alation

OCTOBER 28, 2021

Leaders are asking how they might use data to drive smarter decision making to support this new model and improve medical treatments that lead to better outcomes. Healthcare organizations need to manage and protect sensitive information in a consistent, secure, and organized way. Why Is Data Governance in Healthcare Important?

Data Governance 52

Data Governance Measurement Data Quality Metrics 52