Smart Data Collective

APRIL 29, 2020

As the world is gradually becoming more dependent on data, the services, tools and infrastructure are all the more important for businesses in every sector. Data management has become a fundamental business concern, and especially for businesses that are going through a digital transformation. What is data management?

Management 97

Management Data Warehouse Digital Transformation Dashboards 97

Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

AWS Big Data

JANUARY 15, 2025

Their terminal operations rely heavily on seamless data flows and the management of vast volumes of data. With the addition of these technologies alongside existing systems like terminal operating systems (TOS) and SAP, the number of data producers has grown substantially.

IoT 102

IoT Machine Learning Metadata Data-driven 102

Wayne Yaddow

FEBRUARY 11, 2025

Managing tests of complex data transformations when automated data testing tools lack important features? Photo by Marvin Meyer on Unsplash Introduction Data transformations are at the core of modern business intelligence, blending and converting disparate datasets into coherent, reliable outputs.

Testing 52

Testing Data Transformation Data Quality Statistics 52

Wayne Yaddow

FEBRUARY 18, 2025

Selecting the strategies and tools for validating data transformations and data conversions in your data pipelines. Introduction Data transformations and data conversions are crucial to ensure that raw data is organized, processed, and ready for useful analysis.

Testing 52

Testing Data Transformation Data-driven Data Quality 52

DataKitchen

APRIL 26, 2021

In early April 2021, DataKItchen sat down with Jonathan Hodges, VP Data Management & Analytics, at Workiva ; Chuck Smith, VP of R&D Data Strategy at GlaxoSmithKline (GSK) ; and Chris Bergh, CEO and Head Chef at DataKitchen, to find out about their enterprise DataOps transformation journey, including key successes and lessons learned.

Measurement 246

Measurement Metrics Data-driven Dashboards 246

Wayne Yaddow

FEBRUARY 6, 2025

Common challenges and practical mitigation strategies for reliable data transformations. Photo by Mika Baumeister on Unsplash Introduction Data transformations are important processes in data engineering, enabling organizations to structure, enrich, and integrate data for analytics , reporting, and operational decision-making.

Testing 52

Testing Data Transformation Data-driven Manufacturing 52

CIO Business Intelligence

DECEMBER 22, 2022

Research firm Gartner further describes the methodology as one focused on “improving the communication, integration, and automation of data flows between data managers and data consumers across an organization.” Data scientists may also be included as key members of DataOps teams, according to Dunning. “I

Analytics 130

Analytics Machine Learning Data mining Software 130

CIO Business Intelligence

JUNE 23, 2022

Yet as companies fight for skilled analyst roles to utilize data to make better decisions , they often fall short in improving the data supply chain and resulting data quality. Without a solid data supply-chain management practices in place, data quality often suffers.

Data Quality 98

Data Quality Key Performance Indicator Metrics KPI 98

AWS Big Data

DECEMBER 13, 2023

In collaboration with AWS, BMS identified a business need to migrate and modernize their custom extract, transform, and load (ETL) platform to a native AWS solution to reduce complexities, resources, and investment to upgrade when new Spark, Python, or AWS Glue versions are released.

Metadata 103

Metadata Data Lake Visualization Data Quality 103

AWS Big Data

NOVEMBER 9, 2023

It does this by helping teams handle the T in ETL (extract, transform, and load) processes. It allows users to write data transformation code, run it, and test the output, all within the framework it provides. As part of their cloud modernization initiative, they sought to migrate and modernize their legacy data platform.

Data Warehouse 120

Data Warehouse Testing Data Quality Reporting 120

erwin

APRIL 29, 2020

Compliance with these business rules can be tracked through data lineage, incorporating auditability and validation controls across data transformations and pipelines to generate alerts when there are non-compliant data instances. Seeing data pipelines and information flows further supports compliance efforts.

Metadata 111

Metadata Key Performance Indicator Data Governance Data Quality 111

Smart Data Collective

OCTOBER 13, 2021

There are countless examples of big data transforming many different industries. There is no disputing the fact that the collection and analysis of massive amounts of unstructured data has been a huge breakthrough. How does Data Virtualization manage data quality requirements?

Visualization 145

Visualization Cost-Benefit Big Data Prescriptive Analytics 145

CIO Business Intelligence

JANUARY 24, 2024

“Organizations often get services and applications up and running without having put stewardship in place,” says Marc Johnson, CISO and senior advisor at Impact Advisors, a healthcare management consulting firm. Creating data silos Denying business users access to information because of data silos has been a problem for years.

Data Strategy 136

Data Strategy Strategy Unstructured Data Data Governance 136

CIO Business Intelligence

NOVEMBER 7, 2023

Given the importance of sharing information among diverse disciplines in the era of digital transformation, this concept is arguably as important as ever. The aim is to normalize, aggregate, and eventually make available to analysts across the organization data that originates in various pockets of the enterprise.

Data Warehouse 142

Data Warehouse Digital Transformation Data-driven Reporting 142

Sisense

AUGUST 28, 2020

Uncomfortable truth incoming: Most people in your organization don’t think about the quality of their data from intake to production of insights. However, as a data team member, you know how important data integrity (and a whole host of other aspects of data management) is.

Data Integration 101

Data Integration Testing Data Quality Data-driven 101

CIO Business Intelligence

DECEMBER 6, 2023

Despite logistics challenges caused by the global pandemic, the company managed to rapidly scale up its team to over 1,000 people in a period of only 11 months.

Digital Transformation 116

Digital Transformation Software Cost-Benefit Manufacturing 116

Octopai

JUNE 22, 2023

In the era of big data, organizations are grappling with the challenge of effectively managing and leveraging vast amounts of data. Two prominent approaches have emerged: self-service data management and centralized data management.

Management 52

Management Data-driven Data Quality Data Governance 52

erwin

SEPTEMBER 17, 2020

Here are six benefits of automating end-to-end data lineage: Reduced Errors and Operational Costs. Data quality is crucial to every organization. Automated data capture can significantly reduce errors when compared to manual entry. Improved Customer and Employee Satisfaction.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

erwin

MARCH 4, 2021

However, while digital transformation and other data-driven initiatives are desired outcomes, few organizations know what data they have or where it is, and they struggle to integrate known data in various formats and numerous systems – especially if they don’t have a way to automate those processes.

Data Governance 122

Data Governance Digital Transformation Data-driven Cost-Benefit 122

CIO Business Intelligence

FEBRUARY 16, 2023

My vision is that I can give the keys to my businesses to manage their data and run their data on their own, as opposed to the Data & Tech team being at the center and helping them out,” says Iyengar, director of Data & Tech at Straumann Group North America. The offensive side?

Unstructured Data 105

Unstructured Data Data Lake Prescriptive Analytics Data Warehouse 105

AWS Big Data

AUGUST 1, 2024

Large-scale data warehouse migration to the cloud is a complex and challenging endeavor that many organizations undertake to modernize their data infrastructure, enhance data management capabilities, and unlock new business opportunities.

Data Warehouse 101

Data Warehouse KPI Optimization Cost-Benefit 101

AWS Big Data

JULY 20, 2023

Prior to the creation of the data lake, Orca’s data was distributed among various data silos, each owned by a different team with its own data pipelines and technology stack. Moreover, running advanced analytics and ML on disparate data sources proved challenging.

Data Lake 95

Data Lake Analytics Snapshot Data Quality 95

DataKitchen

DECEMBER 9, 2022

DataOps involves close collaboration between data scientists, IT professionals, and business stakeholders, and it often involves the use of automation and other technologies to streamline data-related tasks. One of the key benefits of DataOps is the ability to accelerate the development and deployment of data-driven solutions.

Machine Learning 130

Machine Learning Data-driven Optimization Data Analytics 130

erwin

JANUARY 11, 2019

Organizations have spent a lot of time and money trying to harmonize data across diverse platforms , including cleansing, uploading metadata, converting code, defining business glossaries, tracking data transformations and so on. So questions linger about whether transformed data can be trusted.

Data Governance 40

Data Governance Risk Metadata Management 40

AWS Big Data

DECEMBER 21, 2023

AWS Glue provides both visual and code-based interfaces to make data integration effortless. Using a native AWS Glue connector increases agility, simplifies data movement, and improves data quality. Attach the AWS managed policy GlueServiceRole. Choose Dashboards Management on the navigation menu.

Analytics 108

Analytics IT Data Lake Visualization 108

IBM Big Data Hub

FEBRUARY 9, 2023

In our last blog , we delved into the seven most prevalent data challenges that can be addressed with effective data governance. Today we will share our approach to developing a data governance program to drive data transformation and fuel a data-driven culture.

Data Governance 76

Data Governance Business Objectives Data Quality Measurement 76

Sisense

OCTOBER 15, 2020

Dealing with this challenge requires caching the relevant context on the processing instances (state management) using techniques like sliding time windows. Another goal that teams dealing with streaming data may have is managing and optimizing a file system on object storage. Optimizing object storage. Ori has a B.A.

Dashboards 98

Dashboards IoT Optimization Internet of Things 98

Ontotext

FEBRUARY 12, 2024

It also breaks down the code and data monolith and distributes it across the domain teams, which results in better management and scalability. The data mesh concept will mitigate cognitive overload when building data-driven organizations that require intense technical, domain, and operational knowledge.

Data-driven 59

Data-driven Data Lake Data Quality Business Objectives 59

erwin

APRIL 16, 2024

The data mesh framework In the dynamic landscape of data management, the search for agility, scalability, and efficiency has led organizations to explore new, innovative approaches. One such innovation gaining traction is the data mesh framework. This empowers individual teams to own and manage their data.

Metadata 52

Metadata Data Quality Data Governance Modeling 52

AWS Big Data

FEBRUARY 8, 2023

Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. Amazon Redshift enables you to run complex SQL analytics at scale and performance on terabytes to petabytes of structured and unstructured data, and make the insights widely available through popular business intelligence (BI) and analytics tools.

Data Quality 86

Data Quality Testing Data Warehouse Unstructured Data 86

Data Virtualization

JULY 20, 2023

Reading Time: < 1 minute In this post, I’m going to cover logical data management and its impact on data mesh architectures. But there’s a lot of confusion in the marketplace today between different types of architectures, specifically data mesh and data fabric, so I’ll.

Management 40

Management Data Integration IT Data Quality 40

Octopai

JUNE 9, 2024

These data products were intended to enhance patient outcomes, streamline hospital operations, and provide actionable insights for decision-making. This strategic choice justified further investment into their data team, infrastructure, management, and science. The complexity of their data ecosystem became a major obstacle.

IT 52

IT Data-driven Predictive Analytics Data Strategy 52

AWS Big Data

APRIL 19, 2023

Data mesh is a new approach to data management. Companies across industries are using a data mesh to decentralize data management to improve data agility and get value from data. The goal of a data product is to solve the long-standing issue of data silos and data quality.

Technology 95

Technology Data-driven Machine Learning Sales 95

AWS Big Data

JULY 27, 2023

DataBrew is a visual data preparation tool that enables you to clean and normalize data without writing any code. The over 200 transformations it provides are now available to be used in an AWS Glue Studio visual job. Now that we identified the data quality issues to address, we need to decide how to deal with each case.

Visualization 91

Visualization Cost-Benefit Data Quality Publishing 91

Octopai

JANUARY 19, 2021

With Octopai’s support and analysis of Azure Data Factory, enterprises can now view complete end-to-end data lineage from Azure Data Factory all the way through to reporting for the first time ever. About Octopai: Octopai was founded in 2015 by BI professionals who realized the need for dynamic solutions in a stagnant market.

Metadata 52

Metadata ROI Machine Learning Data Quality 52

Jet Global

MAY 28, 2019

Common databases that most of us use in our every day lives are relational databases, which includes ERP and business process management systems, SQL databases, CRM systems, and even Excel spreadsheets. A data warehouse is typically used by companies with a high level of data diversity or analytical requirements.

Data Warehouse 85

Data Warehouse Reporting Business Intelligence Sales 85