Wayne Yaddow

FEBRUARY 26, 2025

AI is transforming how senior data engineers and data scientists validate data transformations and conversions. Artificial intelligence-based verification approaches aid in the detection of anomalies, the enforcement of data integrity, and the optimization of pipelines for improved efficiency.

Data Transformation 52

Data Transformation Testing Data-driven Data Quality 52

AWS Big Data

JANUARY 6, 2025

With Amazon AppFlow, you can run data flows at nearly any scale and at the frequency you chooseon a schedule, in response to a business event, or on demand. You can configure data transformation capabilities such as filtering and validation to generate rich, ready-to-use data as part of the flow itself, without additional steps.

Analytics 94

Analytics Data Warehouse Big Data Metrics 94

Alation

OCTOBER 26, 2021

Building a Data Culture Within a Finance Department. Our finance users tell us that their first exposure to the Alation Data Catalog often comes soon after the launch of organization-wide data transformation efforts. After all, finance is one of the greatest consumers of data within a business.

Finance 52

Finance Data Transformation Enterprise Metrics 52

AWS Big Data

NOVEMBER 29, 2023

dbt is an open source, SQL-first templating engine that allows you to write repeatable and extensible data transforms in Python and SQL. dbt is predominantly used by data warehouses (such as Amazon Redshift ) customers who are looking to keep their data transform logic separate from storage and engine.

Data Lake 116

Data Lake Management Metrics Data Warehouse 116

AWS Big Data

NOVEMBER 27, 2024

Together with price-performance, Amazon Redshift offers capabilities such as serverless architecture, machine learning integration within your data warehouse and secure data sharing across the organization. dbt Cloud is a hosted service that helps data teams productionize dbt deployments. or a later version) database.

Data Warehouse 93

Data Warehouse Analytics Testing Modeling 93

AWS Big Data

JANUARY 15, 2025

With a unified catalog, enhanced analytics capabilities, and efficient data transformation processes, were laying the groundwork for future growth. Amazon DataZone empowers EUROGATE by setting the stage for long-term operational excellence and scalability.

IoT 100

IoT Machine Learning Metadata Data-driven 100

DataKitchen

JULY 27, 2023

Here are a few examples that we have seen of how this can be done: Batch ETL with Azure Data Factory and Azure Databricks: In this pattern, Azure Data Factory is used to orchestrate and schedule batch ETL processes. Azure Blob Storage serves as the data lake to store raw data. Azure Machine Learning). So go ahead.

Machine Learning 130

Machine Learning Cost-Benefit Data Transformation Testing 130

DataKitchen

NOVEMBER 10, 2023

Identifying Anomalies: Use advanced algorithms to detect anomalies in data patterns. Establish baseline metrics for normal database operations, enabling the system to flag deviations as potential issues. Monitor for freshness, schema changes, volume, field health/quality, new tables, and usage.

Data Quality 136

Data Quality Testing Data Lake Data Integration 136

O'Reilly on Data

JANUARY 17, 2023

In the past they understood the APIs of TensorFlow and Torch to build models by hand; today they are fluent in the autoML vendor’s APIs to train models, and they understand how to review the metrics. The second is the experienced ML professional who really knows how to build and tune models. It does not exist in the code.

Machine Learning 363

Machine Learning Predictive Modeling Software Modeling 363

CIO Business Intelligence

JUNE 23, 2022

The challenge is to capture source of the data correctly from the outset and ensure data quality does not degrade when moving across the data supply-chain. A key supply chain management metric used to evaluate the performance of physical supply chains is OTIF – On-Time-In-Full. Supply chain complexity.

Data Quality 98

Data Quality Key Performance Indicator Metrics KPI 98

AWS Big Data

OCTOBER 11, 2023

In this post, we discuss ways to modernize your legacy, on-premises, real-time analytics architecture to build serverless data analytics solutions on AWS using Amazon Managed Service for Apache Flink. Near-real-time streaming analytics captures the value of operational data and metrics to provide new insights to create business opportunities.

Management 117

Management Metadata Analytics Dashboards 117

CIO Business Intelligence

APRIL 8, 2025

Build data validation rules directly into ingestion layers so that insufficient data is stopped at the gate and not detected after damage is done. Use lineage tooling to trace data from source to report. Understanding how data transforms and where it breaks is crucial for audibility and root-cause resolution.

Data Quality 102

Data Quality Data-driven Key Performance Indicator Metadata 102

CIO Business Intelligence

JULY 5, 2022

What is the difference between business analytics and data analytics? Business analytics is a subset of data analytics. Data analytics is used across disciplines to find trends and solve problems using data mining , data cleansing, data transformation, data modeling, and more.

Business Analytics 136

Business Analytics Prescriptive Analytics Data mining Diagnostic Analytics 136

AWS Big Data

AUGUST 9, 2023

The advent of rapid adoption of serverless data lake architectures—with ever-growing datasets that need to be ingested from a variety of sources, followed by complex data transformation and machine learning (ML) pipelines—can present a challenge. Notify any failures to a Slack channel.

Data Lake 93

Data Lake Metrics Cost-Benefit Testing 93

AWS Big Data

JUNE 20, 2023

Kinesis Data Firehose is a fully managed service for delivering near-real-time streaming data to various destinations for storage and performing near-real-time analytics. You can perform analytics on VPC flow logs delivered from your VPC using the Kinesis Data Firehose integration with Datadog as a destination.

Dashboards 92

Dashboards Visualization Metrics Data Transformation 92

Alation

OCTOBER 18, 2022

Alation is pleased to be named a dbt Metrics Partner and to announce the start of a partnership with dbt, which will bring dbt data into the Alation data catalog. In the modern data stack, dbt is a key tool to make data ready for analysis. Data Transformation in the Modern Data Stack.

Metadata 52

Metadata Metrics Recreation/Entertainment Data Quality 52

AWS Big Data

DECEMBER 4, 2024

To evaluate the success of the delivery model, ANZ has implemented key metrics, such as cost transparency and domain adoption, to guide the data mesh governance team in refining the delivery approach. For instance, one enhancement involves integrating cross-functional squads to support data literacy.

Metadata 93

Metadata Data Governance Data Quality Data-driven 93

AWS Big Data

NOVEMBER 9, 2023

In this post, we delve into a case study for a retail use case, exploring how the Data Build Tool (dbt) was used effectively within an AWS environment to build a high-performing, efficient, and modern data platform. It does this by helping teams handle the T in ETL (extract, transform, and load) processes.

Data Warehouse 118

Data Warehouse Testing Data Quality Reporting 118

AWS Big Data

AUGUST 1, 2024

However, you might face significant challenges when planning for a large-scale data warehouse migration. The success criteria are the key performance indicators (KPIs) for each component of the data workflow. Data transformation experts to convert database stored functions in the producer or consumer.

Data Warehouse 99

Data Warehouse KPI Optimization Cost-Benefit 99

Sisense

AUGUST 6, 2020

An obvious mechanical answer is: use relevance as a metric. Another important method is to benchmark existing metrics. Know the limitations of your existing dataset and answer these questions: What categories of data are there? What data transformations are needed from your data scientists to prepare the data?

Management 95

Management Machine Learning Key Performance Indicator Cost-Benefit 95

Wayne Yaddow

MARCH 3, 2025

When implementing automated validation, AI-driven regression testing, real-time canary pipelines, synthetic data generation, freshness enforcement, KPI tracking, and CI/CD automation, organizations can shift from reactive data observability to proactive data quality assurance. Summary: Why thisorder?

Data Quality 52

Data Quality Strategy ROI Testing 52

AWS Big Data

NOVEMBER 15, 2023

The difference lies in when and where data transformation takes place. In ETL, data is transformed before it’s loaded into the data warehouse. In ELT, raw data is loaded into the data warehouse first, then it’s transformed directly within the warehouse.

Dashboards 106

Dashboards Analytics Metadata Data Warehouse 106

DataKitchen

OCTOBER 5, 2021

The data organization wants to run the Value Pipeline as robustly as a six sigma factory, and it must be able to implement and deploy process improvements as rapidly as a Silicon Valley start-up. The data engineer builds data transformations. Their product is the data.

Testing 152

Testing Dashboards Measurement Experimentation 152

Cloudera

OCTOBER 8, 2024

Detailed Data and Model Lineage Tracking*: Ensures comprehensive tracking and documentation of data transformations and model lifecycle events, enhancing reproducibility and auditability.

Optimization 75

Optimization Experimentation Metrics Enterprise 75

AWS Big Data

JUNE 6, 2023

Furthermore, it allows for necessary actions to be taken, such as rectifying errors in the data source, refining data transformation processes, and updating data quality rules. The Lambda function is responsible for converting the data quality metrics and dispatching them to the designated email addresses via Amazon SNS.

Data Quality 81

Data Quality Metrics Data-driven Visualization 81

AWS Big Data

AUGUST 9, 2024

After the read query validation stage was complete and we were satisfied with the performance, we reconnected our orchestrator so that the data transformation queries could be run in the new cluster. At this point, only one-time queries and those made by Amazon QuickSight reached the new cluster.

Data Lake 107

Data Lake Analytics Data Warehouse Data-driven 107

AWS Big Data

SEPTEMBER 5, 2024

If storing operational data in a data warehouse is a requirement, synchronization of tables between operational data stores and Amazon Redshift tables is supported. In scenarios where data transformation is required, you can use Redshift stored procedures to modify data in Redshift tables.

IoT 106

IoT Data Warehouse Cost-Benefit Reporting 106

AWS Big Data

MAY 25, 2023

Data transformation plays a pivotal role in providing the necessary data insights for businesses in any organization, small and large. To gain these insights, customers often perform ETL (extract, transform, and load) jobs from their source systems and output an enriched dataset.

Reporting 76

Reporting Metrics Optimization Data Lake 76

AWS Big Data

JULY 20, 2023

Specifically, the system uses Amazon SageMaker Processing jobs to process the data stored in the data lake, employing the AWS SDK for Pandas (previously known as AWS Wrangler) for various data transformation operations, including cleaning, normalization, and feature engineering.

Data Lake 93

Data Lake Analytics Snapshot Data Quality 93

Octopai

NOVEMBER 13, 2022

The modern data stack is a data management system built out of cloud-based data systems. A given modern data stack will usually include components for data ingestion from your data sources, data transformation, data storage, data analysis and reporting.

Enterprise 97

Enterprise Data Warehouse Reporting Metadata 97

DataKitchen

OCTOBER 25, 2022

It’s because it’s a hard thing to accomplish when there are so many teams, locales, data sources, pipelines, dependencies, data transformations, models, visualizations, tests, internal customers, and external customers. It’s not just a fear of change.

Testing 130

Testing Data-driven Visualization Dashboards 130

AWS Big Data

FEBRUARY 6, 2023

Let’s look at some key metrics. After analyzing YARN logs by various metrics, you’re ready to design future EMR architectures. His area of interests are data lakes and cloud modern data architecture delivery. Kalen Zhang was the Global Segment Tech Lead of Partner Data and Analytics at AWS.

Dashboards 83

Dashboards Optimization Data Lake Cost-Benefit 83

AWS Big Data

NOVEMBER 7, 2023

Second, organizations still need transformations like cleansing, deduplication, and combining datasets for analysis and machine learning (ML). For these, AWS Glue provides fast, scalable data transformation. They can now analyze business metrics in near-real time and make data-driven decisions faster than ever before.

Dashboards 80

Dashboards Data-driven Data Integration Data Lake 80

AWS Big Data

APRIL 11, 2023

With the proliferation of IoT devices and the abundance of data generated by them, it has become possible to collect real-time data on inventory levels, customer behavior, and other key metrics. In the inventory management and forecasting solution, AWS Glue is recommended for data transformation.

Forecasting 83

Forecasting Management IoT Data-driven 83

AWS Big Data

SEPTEMBER 19, 2023

To make high-quality decisions, we need to collect user event data from various sources and update risk profiles in real time. We also need to be able to add new fields and metrics to the risk profiles as our team identifies new attacks, without needing engineering intervention or complex deployments.

Analytics 91

Analytics Risk Big Data Machine Learning 91

Alation

DECEMBER 7, 2021

This platform should: Connect to diverse data sources (on-prem, hybrid, legacy, or modern). Extract data quality information. Monitor data anomalies and data drift. Track how data transforms, noting unexpected changes during its lifecycle. Alation’s Data Catalog: Built-in Data Quality Capabilities.

Data Quality 52

Data Quality Data-driven Metrics Dashboards 52

Cloudera

JUNE 17, 2022

The NiFi flow behind the Inbound Connection can not only receive data and forward it to a Kafka topic, but can perform schema validation, format conversions, and data transformation, as well as routing, filtering, and enriching the data.

Cost-Benefit 94

Cost-Benefit IoT Data Warehouse Manufacturing 94