Wayne Yaddow

FEBRUARY 6, 2025

Common challenges and practical mitigation strategies for reliable data transformations. Photo by Mika Baumeister on Unsplash Introduction Data transformations are important processes in data engineering, enabling organizations to structure, enrich, and integrate data for analytics , reporting, and operational decision-making.

Testing 52

Testing Data Transformation Data-driven Manufacturing 52

CIO Business Intelligence

FEBRUARY 27, 2025

Additionally, integrating mainframe data with the cloud enables enterprises to feed information into data lakes and data lake houses, which is ideal for authorized data professionals to easily leverage the best and most modern tools for analytics and forecasting. Four key challenges prevent them from doing so: 1.

Metadata 98

Metadata Data Lake Cost-Benefit Forecasting 98

AWS Big Data

OCTOBER 23, 2024

The Airflow REST API facilitates a wide range of use cases, from centralizing and automating administrative tasks to building event-driven, data-aware data pipelines. In this post, we discuss the enhancement and present several use cases that the enhancement unlocks for your Amazon MWAA environment.

Interactive 115

Interactive Testing Data-driven Data Lake 115

AWS Big Data

NOVEMBER 22, 2024

In this post, we’ll walk through an example ETL process that uses session reuse to efficiently create, populate, and query temporary staging tables across the full data transformation workflow—all within the same persistent Amazon Redshift database session.

Data Warehouse 105

Data Warehouse Recreation/Entertainment Cost-Benefit Data-driven 105

datapine

SEPTEMBER 29, 2022

Once data is deemed high-quality, critical business processes and functions should run more efficiently and accurately, with a higher ROI and lower costs. Data Quality Management Best Practices. This means there are no unintended data errors, and it corresponds to its appropriate designation (e.g., date, month, and year).

Data Quality 241

Data Quality Metrics Data-driven Management 241

datapine

MAY 2, 2023

The rise of SaaS business intelligence tools is answering that need, providing a dynamic vessel for presenting and interacting with essential insights in a way that is digestible and accessible. The future is bright for logistics companies that are willing to take advantage of big data.

Big Data 275

Big Data Internet of Things Cost-Benefit Optimization 275

O'Reilly on Data

OCTOBER 19, 2021

To make the data scientists maximally productive, we want to provide supporting software architecture in terms of APIs and libraries that allow them to focus on data, not on the machines. Data Science Layers. However, none of these layers help with modeling and optimization. Model Development.

IT 364

IT Testing Experimentation Software 364

Cloudera

OCTOBER 19, 2021

Airflow has been adopted by many Cloudera Data Platform (CDP) customers in the public cloud as the next generation orchestration service to setup and operationalize complex data pipelines. This presented challenges for users in building more complex multi-step pipelines that are typical of DE workflows.

Data Transformation 120

Data Transformation Interactive Machine Learning Testing 120

AWS Big Data

MAY 22, 2024

Amazon OpenSearch Ingestion is a fully managed serverless pipeline that allows you to ingest, filter, transform, enrich, and route data to an Amazon OpenSearch Service domain or Amazon OpenSearch Serverless collection. When you create a new pipeline on the OpenSearch Service console, you’re presented with a new catalog page.

Data Architecture 122

Data Architecture Visualization Data Transformation Management 122

AWS Big Data

APRIL 26, 2023

We also split the data transformation into several modules (Data Aggregation, Data Filtering, and Data Preparation) to make the system more transparent and easier to maintain. Although each module is specific to a data source or a particular data transformation, we utilize reusable blocks inside of every job.

Forecasting 145

Forecasting Manufacturing Data Lake Big Data 145

Smart Data Collective

APRIL 29, 2020

The extraction of raw data, transforming to a suitable format for business needs, and loading into a data warehouse. Data transformation. This process helps to transform raw data into clean data that can be analysed and aggregated. Data analytics and visualisation.

Management 98

Management Data Warehouse Digital Transformation Dashboards 98

DataKitchen

OCTOBER 5, 2022

Jason is the CDO for Company X, responsible for six teams of data professionals across several locations. The breadth of his teams’ work and the technologies they use present a significant challenge to his main goal: to deliver new and useful analytics solutions to the business. to monitor your data operations.

Testing 173

Testing Risk Data Processing Statistics 173

CIO Business Intelligence

FEBRUARY 16, 2023

The company’s orthodontics business, for instance, makes heavy use of image processing to the point that unstructured data is growing at a pace of roughly 20% to 25% per month. Advances in imaging technology present Straumann Group with the opportunity to provide its customers with new capabilities to offer their clients.

Unstructured Data 105

Unstructured Data Data Lake Prescriptive Analytics Data Warehouse 105

AWS Big Data

DECEMBER 4, 2024

Division-level metadata management and data governance – A centrally hosted service provides domain teams with the capability to publish their data products along with relevant metadata, like business definitions and lineage. A data portal for consumers to discover data products and access associated metadata.

Metadata 105

Metadata Data Governance Data Quality Data-driven 105

Ontotext

NOVEMBER 29, 2024

If we want to overcome the challenges of such transformations in sustainable ways, we need to look for solutions from multidimensional perspectives. The role of knowledge graphs in AECO transformation At present, knowledge graphs are the best-known technology capable of offering decentralized ways of going beyond existing data silos.

Modeling 59

Modeling Structured Data Technology Data Transformation 59

CIO Business Intelligence

JULY 5, 2022

What is the difference between business analytics and data analytics? Business analytics is a subset of data analytics. Data analytics is used across disciplines to find trends and solve problems using data mining , data cleansing, data transformation, data modeling, and more.

Business Analytics 136

Business Analytics Prescriptive Analytics Data mining Diagnostic Analytics 136

AWS Big Data

JULY 18, 2024

For the updateRequestProcessorChain , OpenSearch provides the ingest pipeline , allowing the enrichment or transformation of data before indexing. Multiple processor stages can be chained to form a pipeline for data transformation. However, it’s strongly recommended to do the data transformation outside OpenSearch.

Dashboards 115

Dashboards Testing Data-driven Visualization 115

AWS Big Data

NOVEMBER 9, 2023

In this post, we delve into a case study for a retail use case, exploring how the Data Build Tool (dbt) was used effectively within an AWS environment to build a high-performing, efficient, and modern data platform. It does this by helping teams handle the T in ETL (extract, transform, and load) processes.

Data Warehouse 126

Data Warehouse Testing Data Quality Reporting 126

Cloudera

JUNE 1, 2021

These acquisitions usher in a new era of “ self-service ” by automating complex operations so customers can focus on building great data-driven apps instead of managing infrastructure. Datacoral powers fast and easy data transformations for any type of data via a robust multi-tenant SaaS architecture that runs in AWS.

Uncertainty 144

Uncertainty Cost-Benefit Risk Strategy 144

Jen Stirrup

SEPTEMBER 30, 2021

Before we dive in, let’s define strands of AI, Machine Learning and Data Science: Business intelligence (BI) leverages software and services to transform data into actionable insights that inform an organization’s strategic and tactical business decisions.

Business Intelligence 135

Business Intelligence Data mining Machine Learning Testing 135

AWS Big Data

NOVEMBER 16, 2023

Data Vault 2.0 allows for the following: Agile data warehouse development Parallel data ingestion A scalable approach to handle multiple data sources even on the same entity A high level of automation Historization Full lineage support However, Data Vault 2.0

Enterprise 111

Enterprise Data Warehouse Data Lake Optimization 111

CIO Business Intelligence

JUNE 7, 2022

Data analytics draws from a range of disciplines — including computer programming, mathematics, and statistics — to perform analysis on data in an effort to describe, predict, and improve performance. What are the four types of data analytics? In business analytics, this is the purview of business intelligence (BI).

Data Analytics 135

Data Analytics Diagnostic Analytics Management Analytics 135

AWS Big Data

OCTOBER 14, 2024

Adding data transformation details to metadata can be challenging because of the dispersed nature of this information across data processing pipelines, making it difficult to extract and incorporate into table-level metadata. Maintaining lists of possible values for the columns requires continuous updates. Choose Stacks.

Metadata 105

Metadata Data Lake Modeling Data Warehouse 105

AWS Big Data

NOVEMBER 15, 2023

The difference lies in when and where data transformation takes place. In ETL, data is transformed before it’s loaded into the data warehouse. In ELT, raw data is loaded into the data warehouse first, then it’s transformed directly within the warehouse.

Analytics 111

Analytics Dashboards Metadata Data Warehouse 111

AWS Big Data

AUGUST 9, 2023

The advent of rapid adoption of serverless data lake architectures—with ever-growing datasets that need to be ingested from a variety of sources, followed by complex data transformation and machine learning (ML) pipelines—can present a challenge.

Data Lake 98

Data Lake Metrics Cost-Benefit Testing 98

DataKitchen

OCTOBER 5, 2021

The data organization wants to run the Value Pipeline as robustly as a six sigma factory, and it must be able to implement and deploy process improvements as rapidly as a Silicon Valley start-up. The data engineer builds data transformations. Their product is the data.

Testing 154

Testing Dashboards Measurement Experimentation 154

AWS Big Data

AUGUST 9, 2024

After the read query validation stage was complete and we were satisfied with the performance, we reconnected our orchestrator so that the data transformation queries could be run in the new cluster. At this point, only one-time queries and those made by Amazon QuickSight reached the new cluster.

Data Lake 111

Data Lake Analytics Data Warehouse Data-driven 111

DataKitchen

OCTOBER 5, 2022

Jason is the CDO for Company X, responsible for six teams of data professionals across several locations. The breadth of his teams’ work and the technologies they use present a significant challenge to his main goal: to deliver new and useful analytics solutions to the business. to monitor your data operations.

Testing 130

Testing Risk Data Processing Statistics 130

AWS Big Data

APRIL 19, 2023

The data science algorithm Valentine is an effective tool for this. Valentine is presented in the paper Valentine: Evaluating Matching Techniques for Dataset Discovery (2021, Koutras et al.). Joel has led data transformation projects on fraud analytics, claims automation, and data governance.

Technology 105

Technology Data-driven Machine Learning Sales 105

CIO Business Intelligence

JANUARY 19, 2024

Whether you’re looking to earn a certification from an accredited university, gain experience as a new grad, hone vendor-specific skills, or demonstrate your knowledge of data analytics, the following certifications (presented in alphabetical order) will work for you. Not finding what you’re looking for?

Data Science 126

Data Science Machine Learning Predictive Modeling Forecasting 126

Alation

FEBRUARY 20, 2020

You can check out a self service data prep flow from catalog to viz in this recorded version here. Rita Sallam Introduces the Data Prep Rodeo. Which brings me to memorable moment #2 on Tuesday when Rita Sallam took the stage for her drill-down presentation on Self-Service Data Prep. To Coming Home, Home on the Range.

Scorecard 52

Scorecard ROI Data-driven Visualization 52

Peter James Thomas

MARCH 9, 2020

Typically I break each of the areas appearing in boxes above into sub-areas, score the organisation against these, roll the results back up and present them back to the client with accompanying commentary; normally also including some sort of benchmark for comparison [2].

Measurement 67

Measurement Data Strategy Strategy Modeling 67

AWS Big Data

APRIL 27, 2023

With these features, you can now build data pipelines completely in standard SQL that are serverless, more simple to build, and able to operate at scale. Typically, data transformation processes are used to perform this operation, and a final consistent view is stored in an S3 bucket or folder.

Data Lake 98

Data Lake Snapshot Optimization Data Transformation 98

AWS Big Data

FEBRUARY 8, 2023

This post presents one possible approach to addressing this challenge in an Amazon Redshift data warehouse. Stored procedures Stored procedures are commonly used to encapsulate logic for data transformation, data validation, and business-specific logic. all URIDs that have are present).

Data Quality 98

Data Quality Testing Data Warehouse Unstructured Data 98

AWS Big Data

OCTOBER 12, 2023

Prerequisites Before you get started, make sure you have the following prerequisites: An AWS account An IAM user with administrator access An S3 bucket Solution Architecture To automate the complete process, we use the following architecture, which integrates Step Functions for orchestration and Amazon EMR Serverless for data transformations.

Big Data 135

Big Data Data-driven Management Visualization 135

AWS Big Data

JULY 6, 2023

It has not been specifically designed for heavy data transformation tasks. The proposed architecture enables extraction and repartitioning of the data in just a few minutes. It’s scalable and cost-effective, and can be adapted to other ETL and data processing use cases.

Machine Learning 145

Machine Learning Visualization IoT Digital Transformation 145

Cloudera

OCTOBER 8, 2024

However, this transition presents significant challenges. Detailed Data and Model Lineage Tracking*: Ensures comprehensive tracking and documentation of data transformations and model lifecycle events, enhancing reproducibility and auditability.

Optimization 75

Optimization Metrics Experimentation Enterprise 75