Wayne Yaddow

MARCH 5, 2025

In this post, well see the fundamental procedures, tools, and techniques that data engineers, data scientists, and QA/testing teams use to ensure high-quality data as soon as its deployed. First, we look at how unit and integration tests uncover transformation errors at an early stage. PyTest, JUnit,NUnit).

Testing 52

Testing Data Transformation Statistics Metadata 52

Cloudera

JUNE 24, 2024

In the fast-evolving landscape of data science and machine learning, efficiency is not just desirable—it’s essential. Imagine a world where every data practitioner, from seasoned data scientists to budding developers, has an intelligent assistant at their fingertips.

Machine Learning 72

Machine Learning Data Science Data-driven Testing 72

Jen Stirrup

SEPTEMBER 30, 2021

Although CRISP-DM is not perfect , the CRISP-DM framework offers a pathway for machine learning using AzureML for Microsoft Data Platform professionals. AI vs ML vs Data Science vs Business Intelligence. They may also learn from evidence, but the data and the modelling fundamentally comes from humans in some way.

Business Intelligence 135

Business Intelligence Data mining Machine Learning Testing 135

DataKitchen

JULY 27, 2023

You can use it for big data analytics and machine learning workloads. Azure Databricks Delta Live Table s: These provide a more straightforward way to build and manage Data Pipelines for the latest, high-quality data in Delta Lake. Azure Blob Storage serves as the data lake to store raw data.

Machine Learning 130

Machine Learning Cost-Benefit Data Transformation Testing 130

CIO Business Intelligence

OCTOBER 11, 2022

The goal, she explained, is to knock down data silos between those groups, using multiple data lakes supported by strong security and governance, to drive positive impact across the supply chain, manufacturing, and the clinical trials of new drugs. . Four ways to improve data-driven business transformation .

Machine Learning 145

Machine Learning Data Science Data-driven Testing 145

Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

Wayne Yaddow

FEBRUARY 26, 2025

AI is transforming how senior data engineers and data scientists validate data transformations and conversions. Artificial intelligence-based verification approaches aid in the detection of anomalies, the enforcement of data integrity, and the optimization of pipelines for improved efficiency.

Data Transformation 52

Data Transformation Testing Data-driven Data Quality 52

Wayne Yaddow

MARCH 12, 2025

How GX helps data teams validate, test, and monitor complex data pipelines Introduction Data flows from diverse sources, and transformations are becoming increasingly complex. Great Expectations can enable a wide range of data transformations and conversion operations.

Data Transformation 52

Data Transformation Data Quality Testing Data Warehouse 52

CIO Business Intelligence

JANUARY 19, 2024

The exam tests general knowledge of the platform and applies to multiple roles, including administrator, developer, data analyst, data engineer, data scientist, and system architect. The exam is designed for seasoned and high-achiever data science thought and practice leaders.

Data Science 126

Data Science Machine Learning Predictive Modeling Forecasting 126

CIO Business Intelligence

APRIL 8, 2025

Build data validation rules directly into ingestion layers so that insufficient data is stopped at the gate and not detected after damage is done. Use lineage tooling to trace data from source to report. Understanding how data transforms and where it breaks is crucial for audibility and root-cause resolution.

Data Quality 102

Data Quality Data-driven Key Performance Indicator Metadata 102

CIO Business Intelligence

JUNE 7, 2022

Data analytics draws from a range of disciplines — including computer programming, mathematics, and statistics — to perform analysis on data in an effort to describe, predict, and improve performance. What are the four types of data analytics? Data analytics methods and techniques.

Data Analytics 135

Data Analytics Diagnostic Analytics Management Analytics 135

CIO Business Intelligence

JULY 5, 2022

What is the difference between business analytics and data analytics? Business analytics is a subset of data analytics. Data analytics is used across disciplines to find trends and solve problems using data mining , data cleansing, data transformation, data modeling, and more.

Business Analytics 136

Business Analytics Prescriptive Analytics Data mining Diagnostic Analytics 136

AWS Big Data

DECEMBER 16, 2024

For each service, you need to learn the supported authorization and authentication methods, data access APIs, and framework to onboard and test data sources. To solve for these challenges, we launched Amazon SageMaker Lakehouse unified data connectivity.

Visualization 71

Visualization Data Processing Testing Publishing 71

Cloudera

OCTOBER 7, 2022

We’re excited to announce the general availability of the open source adapters for dbt for all the engines in CDP — Apache Hive , Apache Impala , and Apache Spark, with added support for Apache Livy and Cloudera Data Engineering. This variety can result in a lack of standardization, leading to data duplication and inconsistency.

Data Warehouse 81

Data Warehouse Data Transformation Machine Learning Data Lake 81

Cloudera

OCTOBER 13, 2020

In the beginning, CDP ran only on AWS with a set of services that supported a handful of use cases and workload types: CDP Data Warehouse: a kubernetes-based service that allows business analysts to deploy data warehouses with secure, self-service access to enterprise data. Predict – Data Engineering (Apache Spark).

Data Warehouse 97

Data Warehouse Machine Learning Visualization Data Lake 97

Sisense

AUGUST 6, 2020

AI and machine learning (ML) are not just catchy buzzwords; they’re vital to the future of our planet and your business. Doing it right can mean the difference between thriving in the new world of data and disappearing from it. Be sure test cases represent the diversity of app users. Can a chatbot help improve relations?

Management 95

Management Machine Learning Key Performance Indicator Cost-Benefit 95

AWS Big Data

JANUARY 30, 2025

Our approach The migration initiative consisted of two main parts: building the new architecture and migrating data pipelines from the existing tool to the new architecture. Often, we would work on both in parallel, testing one component of the architecture while developing another at the same time.

Data Warehouse 52

Data Warehouse Data Architecture Machine Learning Data Transformation 52

Cloudera

OCTOBER 19, 2021

Airflow has been adopted by many Cloudera Data Platform (CDP) customers in the public cloud as the next generation orchestration service to setup and operationalize complex data pipelines. With this Technical Preview release, any CDE customer can test drive the new authoring interface by setting up the latest CDE service.

Data Transformation 120

Data Transformation Interactive Machine Learning Testing 120

AWS Big Data

NOVEMBER 15, 2023

Once released, consumers use datasets from different providers for analysis, machine learning (ML) workloads, and visualization. Each CDH dataset has three processing layers: source (raw data), prepared (transformed data in Parquet), and semantic (combined datasets).

Dashboards 106

Dashboards Analytics Metadata Data Warehouse 106

DataRobot Blog

APRIL 4, 2022

Before the data is put into the model comes a process called feature engineering – transforming the original data columns to impose certain business assumptions or simply increase model accuracy. The classical approach is to assume the adstock function (typically linear ) and test out various values of ? Request a demo.

Machine Learning 52

Machine Learning Sales Measurement ROI 52

AWS Big Data

DECEMBER 13, 2023

In addition to using native managed AWS services that BMS didn’t need to worry about upgrading, BMS was looking to offer an ETL service to non-technical business users that could visually compose data transformation workflows and seamlessly run them on the AWS Glue Apache Spark-based serverless data integration engine.

Metadata 101

Metadata Data Lake Visualization Data Quality 101

Cloudera

JUNE 30, 2022

Today’s general availability announcement covers Iceberg running within key data services in the Cloudera Data Platform (CDP) — including Cloudera Data Warehousing ( CDW ), Cloudera Data Engineering ( CDE ), and Cloudera Machine Learning ( CML ). Read why the future of data lakehouses is open.

Data Lake 88

Data Lake Data Warehouse Data Architecture Metadata 88

Cloudera

FEBRUARY 8, 2021

Continuing from my previous blog post about how awesome and easy it is to develop web-based applications backed by Cloudera Operational Database (COD), I started a small project to integrate COD with another CDP cloud experience, Cloudera Machine Learning (CML). . Now, let’s start testing our model! Go to runner.py

Machine Learning 75

Machine Learning Statistics Dashboards Modeling 75

AWS Big Data

APRIL 12, 2023

Also, you can run other types of business applications, such as web applications and machine learning (ML) TensorFlow workloads, on the same EKS cluster. We also share a Spark benchmark solution that suits all Amazon EMR deployment options, so you can replicate the process in your environment for your own performance test cases.

Testing 85

Testing Big Data Metadata Optimization 85

AWS Big Data

AUGUST 9, 2023

The advent of rapid adoption of serverless data lake architectures—with ever-growing datasets that need to be ingested from a variety of sources, followed by complex data transformation and machine learning (ML) pipelines—can present a challenge. Disable the rules after testing to avoid repeated messages.

Data Lake 93

Data Lake Metrics Testing Cost-Benefit 93

Cloudera

OCTOBER 11, 2021

Modak Nabu relies on a framework of “Botworks”, a series of micro-jobs to accomplish various data transformation steps from ingestion to profiling, and indexing. Cloudera Data Engineering within CDP provides : Fully managed Spark-on-Kubernetes service that hides the complexity running production DE workloads at scale.

Data Lake 95

Data Lake Cost-Benefit Data-driven Dashboards 95

Cloudera

AUGUST 8, 2022

The general availability covers Iceberg running within some of the key data services in CDP, including Cloudera Data Warehouse ( CDW ), Cloudera Data Engineering ( CDE ), and Cloudera Machine Learning ( CML ). Cloudera Data Engineering (Spark 3) with Airflow enabled. Cloudera Machine Learning .

Snapshot 94

Snapshot Data Warehouse Machine Learning Cost-Benefit 94

Cloudera

OCTOBER 14, 2020

Apache Spark unifies batch processing, real-time processing, stream analytics, machine learning, and interactive query in one-platform. The Test and Development queue have fixed resource limits. Background. Why choose K8s for Apache Spark. All other queues are only limited by the size of the cluster. Acknowledgments.

Machine Learning 118

Machine Learning Management Big Data Optimization 118

AWS Big Data

JULY 6, 2023

It has not been specifically designed for heavy data transformation tasks. Step Functions helps developers use AWS services to build distributed applications, automate processes, orchestrate microservices, and create data and machine learning (ML) pipelines. Note that Lambda is a general purpose serverless engine.

Machine Learning 136

Machine Learning Visualization IoT Digital Transformation 136

AWS Big Data

OCTOBER 14, 2024

Amazon Redshift is used to analyze structured and semi-structured data across data warehouses, operational databases, and data lakes. Amazon EMR provides a big data environment for data processing, interactive analysis, and machine learning using open source frameworks such as Apache Spark, Apache Hive, and Presto.

Metadata 88

Metadata Data Lake Modeling Data Warehouse 88

Cloudera

NOVEMBER 29, 2021

Data transforms businesses. That’s where the data lifecycle comes into play. Managing data and its flow, from the edge to the cloud, is one of the most important tasks in the process of gaining data intelligence. .

Data Lake 77

Data Lake Data Warehouse Data Architecture Reporting 77

AWS Big Data

AUGUST 1, 2024

Migrating to Amazon Redshift offers organizations the potential for improved price-performance, enhanced data processing, faster query response times, and better integration with technologies such as machine learning (ML) and artificial intelligence (AI). Platform architects define a well-architected platform.

Data Warehouse 99

Data Warehouse KPI Optimization Cost-Benefit 99

AWS Big Data

JULY 26, 2023

To grow the power of data at scale for the long term, it’s highly recommended to design an end-to-end development lifecycle for your data integration pipelines. The following are common asks from our customers: Is it possible to develop and test AWS Glue data integration jobs on my local laptop?

Data Integration 79

Data Integration Snapshot Testing Visualization 79

CIO Business Intelligence

JUNE 13, 2022

According to Evanta’s 2022 CIO Leadership Perspectives study, CIOs’ second top priority within the IT function is around data and analytics, with CIOs seeing advancing organizational use of data as key to reaching enterprise objectives. Others also list data initiatives as a top issue for CIOs.

IT 144

IT Digital Transformation Internet of Things Strategy 144

Smarten

MAY 13, 2024

To accomplish this interchange, the method uses data mining and machine learning and it contains components like a data dictionary to define the fields used by the model, and data transformation to map user data and make it easier for the system to mine that data. Validation summary of models.

Predictive Modeling 64

Predictive Modeling Data mining Predictive Analytics Modeling 64

AWS Big Data

OCTOBER 18, 2023

They use various AWS analytics services, such as Amazon EMR, to enable their analysts and data scientists to apply advanced analytics techniques to interactively develop and test new surveillance patterns and improve investor protection. Melody Yang is a Senior Big Data Solutions Architect for Amazon EMR at AWS.

Big Data 91

Big Data Data Processing Interactive Testing 91

Domino Data Lab

AUGUST 20, 2019

Many thanks to AWP Pearson for the permission to excerpt “Manual Feature Engineering: Manipulating Data for Fun and Profit” from the book, Machine Learning with Python for Everyone by Mark E. Missing values can be filled in based on expert knowledge, heuristics, or by some machine learning techniques.

Testing 68

Testing Modeling Interactive Measurement 68