Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

DataKitchen

NOVEMBER 10, 2023

Implement a communication protocol that swiftly informs stakeholders, allowing them to brace for or address the potential impacts of the data change. Building a Culture of Accountability: Encourage a culture where data integrity is everyone’s responsibility.

Data Quality 139

Data Quality Testing Data Lake Data Integration 139

Wayne Yaddow

FEBRUARY 26, 2025

AI is transforming how senior data engineers and data scientists validate data transformations and conversions. Artificial intelligence-based verification approaches aid in the detection of anomalies, the enforcement of data integrity, and the optimization of pipelines for improved efficiency.

Data Transformation 52

Data Transformation Testing Data-driven Data Quality 52

Ontotext

AUGUST 24, 2023

The second approach is to use some Data Integration Platform. As an enterprise-supported tool, it has already established how to make all data transformations. Then the recommended approach is to use one of the many JSON to RDF transformation frameworks to produce RDF data.

Data Integration 52

Data Integration Modeling Business Objectives Optimization 52

DataKitchen

JULY 27, 2023

Let’s go through the ten Azure data pipeline tools Azure Data Factory : This cloud-based data integration service allows you to create data-driven workflows for orchestrating and automating data movement and transformation. Azure Blob Storage serves as the data lake to store raw data.

Machine Learning 130

Machine Learning Cost-Benefit Data Transformation Testing 130

Smart Data Collective

OCTOBER 13, 2021

There are countless examples of big data transforming many different industries. There is no disputing the fact that the collection and analysis of massive amounts of unstructured data has been a huge breakthrough. This is something that you can learn more about in just about any technology blog.

Visualization 145

Visualization Cost-Benefit Big Data Prescriptive Analytics 145

DataKitchen

DECEMBER 9, 2022

ChatGPT> DataOps, or data operations, is a set of practices and technologies that organizations use to improve the speed, quality, and reliability of their data analytics processes. The goal of DataOps is to help organizations make better use of their data to drive business decisions and improve outcomes.

Machine Learning 130

Machine Learning Data-driven Optimization Data Analytics 130

CIO Business Intelligence

NOVEMBER 7, 2023

Given the importance of sharing information among diverse disciplines in the era of digital transformation, this concept is arguably as important as ever. The aim is to normalize, aggregate, and eventually make available to analysts across the organization data that originates in various pockets of the enterprise.

Data Warehouse 142

Data Warehouse Digital Transformation Data-driven Reporting 142

AWS Big Data

JANUARY 30, 2025

AWS Glue A data integration service, AWS Glue consolidates major data integration capabilities into a single service. These include data discovery, modern ETL, cleansing, transforming, and centralized cataloging. Its also serverless, which means theres no infrastructure to manage.

Data Warehouse 54

Data Warehouse Data Architecture Machine Learning Data Transformation 54

IBM Big Data Hub

JANUARY 9, 2024

The emergence of generative AI prompted several prominent companies to restrict its use because of the mishandling of sensitive internal data. According to CNN, some companies imposed internal bans on generative AI tools while they seek to better understand the technology and many have also blocked the use of internal ChatGPT.

Enterprise 71

Enterprise Data Integration Data Quality Contextual Data 71

AWS Big Data

DECEMBER 13, 2023

In addition to using native managed AWS services that BMS didn’t need to worry about upgrading, BMS was looking to offer an ETL service to non-technical business users that could visually compose data transformation workflows and seamlessly run them on the AWS Glue Apache Spark-based serverless data integration engine.

Metadata 111

Metadata Data Lake Visualization Data Quality 111

CIO Business Intelligence

FEBRUARY 16, 2023

The company’s orthodontics business, for instance, makes heavy use of image processing to the point that unstructured data is growing at a pace of roughly 20% to 25% per month. Advances in imaging technology present Straumann Group with the opportunity to provide its customers with new capabilities to offer their clients.

Unstructured Data 105

Unstructured Data Data Lake Prescriptive Analytics Data Warehouse 105

AWS Big Data

AUGUST 13, 2024

In today’s data-driven world, seamless integration and transformation of data across diverse sources into actionable insights is paramount. We will create a glue studio job, add events and venue data from the SFTP server, carry out data transformations and load transformed data to s3.

Data Processing 115

Data Processing Visualization Data Lake Data Processing 115

AWS Big Data

AUGUST 19, 2024

As organizations increasingly rely on data stored across various platforms, such as Snowflake , Amazon Simple Storage Service (Amazon S3), and various software as a service (SaaS) applications, the challenge of bringing these disparate data sources together has never been more pressing.

Analytics 119

Analytics Data-driven Data Integration Data Lake 119

erwin

APRIL 29, 2020

Many large organizations, in their desire to modernize with technology, have acquired several different systems with various data entry points and transformation rules for data as it moves into and across the organization. Seeing data pipelines and information flows further supports compliance efforts.

Key Performance Indicator 111

Key Performance Indicator Metadata Data Governance Data Quality 111

Octopai

NOVEMBER 13, 2022

A given modern data stack will usually include components for data ingestion from your data sources, data transformation, data storage, data analysis and reporting. In the ideal modern data stack, it should be easy to connect these components and no big deal to switch one component out for another.

Enterprise 97

Enterprise Data Warehouse Reporting Metadata 97

AWS Big Data

DECEMBER 21, 2023

Movement of data across data lakes, data warehouses, and purpose-built stores is achieved by extract, transform, and load (ETL) processes using data integration services such as AWS Glue. AWS Glue provides both visual and code-based interfaces to make data integration effortless.

Analytics 111

Analytics IT Data Lake Visualization 111

CIO Business Intelligence

JUNE 7, 2022

It comprises the processes, tools and techniques of data analysis and management, including the collection, organization, and storage of data. The chief aim of data analytics is to apply statistical analysis and technologies on data to find trends and solve problems. What are the four types of data analytics?

Data Analytics 135

Data Analytics Diagnostic Analytics Management Analytics 135

AWS Big Data

SEPTEMBER 13, 2024

The AWS pay-as-you-go model and the constant pace of innovation in data processing technologies enable CFM to maintain agility and facilitate a steady cadence of trials and experimentation. In this post, we share how we built a well-governed and scalable data engineering platform using Amazon EMR for financial features generation.

Interactive 109

Interactive Strategy Cost-Benefit Data Governance 109

CIO Business Intelligence

JANUARY 24, 2024

How enterprises gather, store, cleanse, access, and secure their data can be a major factor in their ability to meet corporate goals. Unfortunately, the road to data strategy success is fraught with challenges, so CIOs and other technology leaders need to plan and execute carefully.

Data Strategy 136

Data Strategy Strategy Unstructured Data Data Governance 136

AWS Big Data

MARCH 15, 2023

The API retrieves data at runtime from an Amazon Aurora PostgreSQL-Compatible Edition database for end-user consumption. To populate the database, the Infomedia team developed a data pipeline using Amazon Simple Storage Service (Amazon S3) for data storage, AWS Glue for data transformations, and Apache Hudi for CDC and record-level updates.

Cost-Benefit 94

Cost-Benefit Data Processing Optimization Data-driven 94

Cloudera

JUNE 1, 2021

These acquisitions usher in a new era of “ self-service ” by automating complex operations so customers can focus on building great data-driven apps instead of managing infrastructure. Datacoral powers fast and easy data transformations for any type of data via a robust multi-tenant SaaS architecture that runs in AWS.

Uncertainty 144

Uncertainty Cost-Benefit Risk Strategy 144

erwin

MARCH 26, 2020

According to Gartner , “data literacy is the ability to read, write and communicate data in context, including an understanding of data sources and constructs, analytical methods and techniques applied — and the ability to describe the use case, application and resulting value.”.

Data-driven 90

Data-driven Unstructured Data Enterprise Machine Learning 90

DataKitchen

OCTOBER 25, 2022

It’s because it’s a hard thing to accomplish when there are so many teams, locales, data sources, pipelines, dependencies, data transformations, models, visualizations, tests, internal customers, and external customers. You can’t quality-control your data integrations or reports with only some details. .

Testing 130

Testing Data-driven Visualization Dashboards 130

Cloudera

NOVEMBER 22, 2021

Open source frameworks such as Apache Impala, Apache Hive and Apache Spark offer a highly scalable programming model that is capable of processing massive volumes of structured and unstructured data by means of parallel execution on a large number of commodity computing nodes. . As a result, alternative data integration technologies (e.g.,

Data Processing 94

Data Processing Data Warehouse Enterprise Visualization 94

AWS Big Data

OCTOBER 11, 2023

Due to this low complexity, the solution uses AWS serverless services to ingest the data, transform it, and make it available for analytics. Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading scalability, data availability, security, and performance.

Visualization 119

Visualization Dashboards Data-driven Gap analysis 119

Ontotext

MARCH 20, 2024

In today’s data-driven world, businesses are drowning in a sea of information. Traditional data integration methods struggle to bridge these gaps, hampered by high costs, data quality concerns, and inconsistencies. Unleashing the Power of Data Connections Zenia Graph isn’t just another data solution company.

Data-driven 52

Data-driven Strategy Sales Data Integration 52

IBM Big Data Hub

JULY 28, 2023

As an independent software vendor (ISV), we at Primeur embed the Open Liberty Java runtime in our flagship data integration platform, DATA ONE. Primeur and DATA ONE As a smart data integration company, we at Primeur believe in simplification. Data Shaper , providing any-to-any data transformations.

Data Integration 55

Data Integration Optimization Software Insurance 55

AWS Big Data

JULY 20, 2023

Orca connects to your environment in minutes with patented SideScanning technology to provide complete coverage across vulnerabilities, malware, misconfigurations, lateral movement risk, weak and leaked passwords, overly permissive identities, and more. In addition, we describe the Orca Platform architecture and the technologies used.

Data Lake 98

Data Lake Analytics Snapshot Data Quality 98

FineReport

JUNE 7, 2024

Criteria for Top Data Visualization Companies Innovation and Technology Cutting-edge technology lies at the core of top data visualization companies. Embrace FanRuan’s transformative technologies to elevate your data visualization experience to unprecedented heights!

Visualization 52

Visualization Predictive Analytics Dashboards Predictive Modeling 52

AWS Big Data

MARCH 3, 2023

Additionally, the scale is significant because the multi-tenant data sources provide a continuous stream of testing activity, and our users require quick data refreshes as well as historical context for up to a decade due to compliance and regulatory demands. Finally, data integrity is of paramount importance.

Software 52

Software Data Lake Testing Cost-Benefit 52

erwin

JANUARY 11, 2019

Organizations have spent a lot of time and money trying to harmonize data across diverse platforms , including cleansing, uploading metadata, converting code, defining business glossaries, tracking data transformations and so on.

Data Governance 40

Data Governance Risk Metadata Management 40

FineReport

MARCH 31, 2024

Furthermore, these tools boast customization options, allowing users to tailor data sources to address areas critical to their business success, thereby generating actionable insights and customizable reports. Best BI Tools for Data Analysts 3.1 Key Features: Extensive library of pre-built connectors for diverse data sources.

Dashboards 52

Dashboards Visualization Data mining Data-driven 52

Alation

JANUARY 17, 2023

The modern data stack started to gain popularity in the early 2010s, as companies began to realize the benefits of big data. Most importantly, cloud computing has gotten much faster and cheaper, emerging as the core technology driving the development of the modern data stack. How Can I Build a Modern Data Stack?

Data Warehouse 52

Data Warehouse Cost-Benefit Data Science Data Transformation 52

Alation

AUGUST 11, 2022

In the technology industry, even the most incremental trends often get framed in next-generation terminology. DataOps sprung up to connect data sources to data consumers. The data warehouse and analytical data stores moved to the cloud and disaggregated into the data mesh. Tools became stacks.

Metadata 52

Metadata Data Warehouse Data Quality Data Lake 52

Data Virtualization

JULY 20, 2023

But there’s a lot of confusion in the marketplace today between different types of architectures, specifically data mesh and data fabric, so I’ll. The post Logical Data Management and Data Mesh appeared first on Data Management Blog - Data Integration and Modern Data Management Articles, Analysis and Information.

Management 40

Management Data Integration IT Data Quality 40

Alation

OCTOBER 28, 2021

Data governance in healthcare refers to how data is collected and used by hospitals, pharmaceutical companies, and other healthcare organizations and service providers. It combines people, process, technology, and data within a system founded on transparency and compliance. Remove Low Quality, Unused, or “Stale” Data.

Data Governance 52

Data Governance Measurement Data Quality Metrics 52