Wayne Yaddow

MARCH 12, 2025

Great Expectations can be integrated directly into existing data pipelines to define, test, and document expectations about the appearance of transformed or converted data. Data quality rules are codified into structured Expectation Suites by Great Expectations instead of relying on ad-hoc scripts or manual checks.

Data Transformation 52

Data Transformation Data Quality Testing Data Warehouse 52

Wayne Yaddow

FEBRUARY 18, 2025

Selecting the strategies and tools for validating data transformations and data conversions in your data pipelines. Introduction Data transformations and data conversions are crucial to ensure that raw data is organized, processed, and ready for useful analysis.

Testing 52

Testing Data Transformation Data-driven Data Quality 52

Wayne Yaddow

FEBRUARY 6, 2025

Common challenges and practical mitigation strategies for reliable data transformations. Photo by Mika Baumeister on Unsplash Introduction Data transformations are important processes in data engineering, enabling organizations to structure, enrich, and integrate data for analytics , reporting, and operational decision-making.

Testing 52

Testing Data Transformation Data-driven Manufacturing 52

erwin

SEPTEMBER 17, 2020

For example, automatically importing mappings from developers’ Excel sheets, flat files, Access and ETL tools into a comprehensive mappings inventory, complete with auto generated and meaningful documentation of the mappings, is a powerful way to support overall data governance. Data quality is crucial to every organization.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

AWS Big Data

JULY 25, 2024

With CloudSearch, you can search large collections of data such as webpages, document files, forum posts, or product information. You send your documents to OpenSearch Serverless, which indexes them for search using the OpenSearch REST API. With OpenSearch Serverless , you get improved, out-of-the-box, hands-free operation.

Cost-Benefit 126

Cost-Benefit Machine Learning Dashboards Management 126

AWS Big Data

OCTOBER 14, 2024

Adding data transformation details to metadata can be challenging because of the dispersed nature of this information across data processing pipelines, making it difficult to extract and incorporate into table-level metadata. The AWS Glue crawler will then populate the additional metadata in AWS Glue Data Catalog.

Metadata 105

Metadata Data Lake Modeling Data Warehouse 105

datapine

SEPTEMBER 29, 2022

Data processes that depended upon the previously defective data will likely need to be re-initiated, especially if their functioning was at risk or compromised by the defected data. These processes could include reports, campaigns, or financial documentation. Accuracy should be measured through source documentation (i.e.,

Data Quality 241

Data Quality Metrics Data-driven Management 241

Cloudera

JUNE 1, 2021

These acquisitions usher in a new era of “ self-service ” by automating complex operations so customers can focus on building great data-driven apps instead of managing infrastructure. Datacoral powers fast and easy data transformations for any type of data via a robust multi-tenant SaaS architecture that runs in AWS.

Uncertainty 144

Uncertainty Cost-Benefit Risk Strategy 144

Alation

OCTOBER 26, 2021

Building a Data Culture Within a Finance Department. Our finance users tell us that their first exposure to the Alation Data Catalog often comes soon after the launch of organization-wide data transformation efforts. After all, finance is one of the greatest consumers of data within a business.

Finance 52

Finance Data Transformation Enterprise Metrics 52

AWS Big Data

OCTOBER 30, 2024

Get started with our technical documentation. Joel Farvault is Principal Specialist SA Analytics for AWS with 25 years’ experience working on enterprise architecture, data governance and analytics, mainly in the financial services industry.

Visualization 119

Visualization Data Lake Testing Data Governance 119

CIO Business Intelligence

APRIL 8, 2025

Build data validation rules directly into ingestion layers so that insufficient data is stopped at the gate and not detected after damage is done. Use lineage tooling to trace data from source to report. Understanding how data transforms and where it breaks is crucial for audibility and root-cause resolution.

Data Quality 102

Data Quality Data-driven Key Performance Indicator Metadata 102

AWS Big Data

NOVEMBER 29, 2023

dbt is an open source, SQL-first templating engine that allows you to write repeatable and extensible data transforms in Python and SQL. dbt is predominantly used by data warehouses (such as Amazon Redshift ) customers who are looking to keep their data transform logic separate from storage and engine.

Data Lake 122

Data Lake Management Metrics Data Warehouse 122

AWS Big Data

NOVEMBER 9, 2023

In this post, we delve into a case study for a retail use case, exploring how the Data Build Tool (dbt) was used effectively within an AWS environment to build a high-performing, efficient, and modern data platform. It does this by helping teams handle the T in ETL (extract, transform, and load) processes.

Data Warehouse 126

Data Warehouse Testing Data Quality Reporting 126

erwin

APRIL 29, 2020

Business terms and data policies should be implemented through standardized and documented business rules. Compliance with these business rules can be tracked through data lineage, incorporating auditability and validation controls across data transformations and pipelines to generate alerts when there are non-compliant data instances.

Key Performance Indicator 111

Key Performance Indicator Metadata Data Governance Data Quality 111

Cloudera

OCTOBER 7, 2022

We’re excited to announce the general availability of the open source adapters for dbt for all the engines in CDP — Apache Hive , Apache Impala , and Apache Spark, with added support for Apache Livy and Cloudera Data Engineering. The Open Data Lakehouse . Cloudera builds dbt adaptors for all engines in the open data lakehouse.

Data Warehouse 81

Data Warehouse Data Transformation Machine Learning Data Lake 81

O'Reilly on Data

JANUARY 17, 2023

Instead of invoking the open-source scikit-learn or Keras calls to build models, your team now goes from Pandas data transforms straight to … the API calls for AWS AutoPilot or GCP Vertex AI. It does not exist in the code. AutoML drives this point home. And it’s available to everyone. What if we go the other way?

Machine Learning 363

Machine Learning Predictive Modeling Software Modeling 363

AWS Big Data

JULY 18, 2024

OpenSearch is an open source, distributed search engine suitable for a wide array of use-cases such as ecommerce search, enterprise search (content management search, document search, knowledge management search, and so on), site search, application search, and semantic search. OpenSearch also includes capabilities to ingest and analyze data.

Dashboards 115

Dashboards Testing Data-driven Visualization 115

Smart Data Collective

APRIL 29, 2020

The techniques for managing organisational data in a standardised approach that minimises inefficiency. Extraction, Transform, Load (ETL). The extraction of raw data, transforming to a suitable format for business needs, and loading into a data warehouse. Data transformation.

Management 98

Management Data Warehouse Digital Transformation Dashboards 98

CIO Business Intelligence

NOVEMBER 7, 2023

Given the importance of sharing information among diverse disciplines in the era of digital transformation, this concept is arguably as important as ever. The aim is to normalize, aggregate, and eventually make available to analysts across the organization data that originates in various pockets of the enterprise.

Data Warehouse 142

Data Warehouse Digital Transformation Data-driven Reporting 142

AWS Big Data

JANUARY 6, 2025

With Amazon AppFlow, you can run data flows at nearly any scale and at the frequency you chooseon a schedule, in response to a business event, or on demand. You can configure data transformation capabilities such as filtering and validation to generate rich, ready-to-use data as part of the flow itself, without additional steps.

Analytics 98

Analytics Data Warehouse Big Data Metrics 98

AWS Big Data

SEPTEMBER 13, 2024

In recent years, driven by the commoditization of data storage and processing solutions, the industry has seen a growing number of systematic investment management firms switch to alternative data sources to drive their investment decisions. The bulk of our data scientists are heavy users of Jupyter Notebook. or later.

Interactive 109

Interactive Strategy Cost-Benefit Data Governance 109

AWS Big Data

APRIL 30, 2024

Amazon Q Developer can now generate complex data integration jobs with multiple sources, destinations, and data transformations. Generated jobs can use a variety of data transformations, including filter, project, union, join, and custom user-supplied SQL.

Data Integration 128

Data Integration Data Lake Data Warehouse Software 128

AWS Big Data

JUNE 19, 2024

SUPER data type columns in Amazon Redshift contain semi-structured data like JSON documents. Previously, data masking in Amazon Redshift only worked with regular table columns, but now you can apply masking policies specifically to elements within SUPER columns. All columns should masked for them.

Data Warehouse 115

Data Warehouse Testing Sales Structured Data 115

Cloudera

JUNE 24, 2024

This integration empowers developers and data scientists alike with advanced capabilities for code completion, generation, and troubleshooting. Whether you’re tackling data transformation challenges or refining intricate machine learning models, our Copilot is designed to be your reliable partner in innovation.

Machine Learning 72

Machine Learning Data Science Data-driven Testing 72

Sisense

AUGUST 6, 2020

Take Grammarly as an example: This popular program checks the grammar, tone, and style of documents. Getting this AI properly trained required a huge learning dataset with countless documents that were tagged according to specific criteria. Accurately prepared data is the base of AI. What will it take to build your MVP?

Management 95

Management Machine Learning Key Performance Indicator Cost-Benefit 95

IBM Big Data Hub

NOVEMBER 20, 2023

Today’s healthcare providers use a wide variety of applications and data across a broad ecosystem of partners to manage their daily workflows. Integrating these applications and data is critical to their success, allowing them to deliver patient care efficiently and effectively.

Cost-Benefit 87

Cost-Benefit Data-driven Data Transformation Management 87

IBM Big Data Hub

JANUARY 9, 2024

ELT tools such as IBM® DataStage® facilitate fast and secure transformations through parallel processing engines. In 2023, the average enterprise receives hundreds of disparate data streams, making efficient and accurate data transformations crucial for traditional and new AI model development.

Enterprise 71

Enterprise Data Integration Data Quality Contextual Data 71

AWS Big Data

AUGUST 1, 2024

However, you might face significant challenges when planning for a large-scale data warehouse migration. As part of the success criteria for operational service levels, you need to document the expected service levels for the new Amazon Redshift data warehouse environment. Platform architects define a well-architected platform.

Data Warehouse 115

Data Warehouse KPI Optimization Cost-Benefit 115

erwin

MARCH 26, 2020

Increased data variety, balancing structured, semi-structured and unstructured data, as well as data originating from a widening array of external sources. Reducing the IT bottleneck that creates barriers to data accessibility. Hybrid on-premises/cloud environments that complicate data integration and preparation.

Data-driven 90

Data-driven Unstructured Data Enterprise Machine Learning 90

Cloudera

AUGUST 3, 2022

By leveraging Hive to apply Ranger FGAC, Spark obtains secure access to the data in a protected staging area. Since Spark has direct access to the staged data, any Spark APIs can be used, from complex data transformations to data science and machine learning. . so stay tuned! .

Snapshot 90

Snapshot Cost-Benefit Machine Learning Data Science 90

Alation

OCTOBER 27, 2022

Few actors in the modern data stack have inspired the enthusiasm and fervent support as dbt. This data transformation tool enables data analysts and engineers to transform, test and document data in the cloud data warehouse. But what does this mean from a practitioner perspective?

Metrics 52

Metrics Dashboards Sales Reporting 52

AWS Big Data

AUGUST 26, 2024

Solution overview The following diagram illustrates the solution architecture: The solution uses AWS Glue as an ETL engine to extract data from the source Amazon RDS database. Built-in data transformations then scrub columns containing PII using pre-defined masking functions. PII detection and scrubbing.

Visualization 116

Visualization Metadata Data Transformation Testing 116

AWS Big Data

OCTOBER 5, 2023

With Amazon AppFlow, you can run data flows at nearly any scale at the frequency you choose—on a schedule, in response to a business event, or on demand. You can configure data transformation capabilities such as filtering and validation to generate rich, ready-to-use data as part of the flow itself, without additional steps.

Data Warehouse 118

Data Warehouse Machine Learning Data Integration Data-driven 118

IBM Big Data Hub

OCTOBER 16, 2023

It includes processes that trace and document the origin of data, models and associated metadata and pipelines for audits. A data store lets a business connect existing data with new data and discover new insights with real-time analytics and business intelligence. Track models and drive transparent processes.

Risk 70

Risk Modeling Management Metadata 70

AWS Big Data

JULY 6, 2023

It has not been specifically designed for heavy data transformation tasks. Additionally, check out the official documentation of AWS Glue , Lambda , and Step Functions. You also use AWS Glue to consolidate the files produced by the parallel tasks. Note that Lambda is a general purpose serverless engine.

Machine Learning 145

Machine Learning Visualization IoT Digital Transformation 145

Cloudera

OCTOBER 8, 2024

Detailed Data and Model Lineage Tracking*: Ensures comprehensive tracking and documentation of data transformations and model lifecycle events, enhancing reproducibility and auditability.

Optimization 75

Optimization Metrics Experimentation Enterprise 75

AWS Big Data

AUGUST 13, 2024

We will create a glue studio job, add events and venue data from the SFTP server, carry out data transformations and load transformed data to s3. For further details on the SFTP connector, see the SFTP Connector for Glue documentation. Select Visual ETL in the central pane.

Data Processing 115

Data Processing Visualization Data Lake Data Processing 115