article thumbnail

Announcing Open Source DataOps Data Quality TestGen 3.0

DataKitchen

Now With Actionable, Automatic, Data Quality Dashboards Imagine a tool that can point at any dataset, learn from your data, screen for typical data quality issues, and then automatically generate and perform powerful tests, analyzing and scoring your data to pinpoint issues before they snowball. New Quality Dashboard & Score Explorer.

article thumbnail

Monitoring Apache Iceberg metadata layer using AWS Lambda, AWS Glue, and AWS CloudWatch

AWS Big Data

In this blog post, we’ll discuss how the metadata layer of Apache Iceberg can be used to make data lakes more efficient. You will learn about an open-source solution that can collect important metrics from the Iceberg metadata layer. This ensures that each change is tracked and reversible, enhancing data governance and auditability.

Metadata 126
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

How EUROGATE established a data mesh architecture using Amazon DataZone

AWS Big Data

In the following section, two use cases demonstrate how the data mesh is established with Amazon DataZone to better facilitate machine learning for an IoT-based digital twin and BI dashboards and reporting using Tableau. From here, the metadata is published to Amazon DataZone by using AWS Glue Data Catalog.

IoT 111
article thumbnail

Deploy Amazon QuickSight dashboards to monitor AWS Glue ETL job metrics and set alarms

AWS Big Data

You have metrics available per job run within the AWS Glue console, but they don’t cover all available AWS Glue job metrics, and the visuals aren’t as interactive compared to the QuickSight dashboard. In the following sections, we show you how to set up the solution, explore the dashboards, and configure alarms.

Metrics 115
article thumbnail

What Is Active Metadata Management and How Does It Work?

Octopai

First, what active metadata management isn’t : “Okay, you metadata! Now, what active metadata management is (well, kind of): “Okay, you metadata! Metadata are the details on those tools: what they are, what to use them for, what to use them with. . That takes active metadata management. Quit lounging around!

article thumbnail

Enhance data governance with enforced metadata rules in Amazon DataZone

AWS Big Data

We’re excited to announce a new feature in Amazon DataZone that offers enhanced metadata governance for your subscription approval process. With this update, domain owners can define and enforce metadata requirements for data consumers when they request access to data assets. Key benefits The feature benefits multiple stakeholders.

article thumbnail

Write queries faster with Amazon Q generative SQL for Amazon Redshift

AWS Big Data

Amazon Q generative SQL for Amazon Redshift uses generative AI to analyze user intent, query patterns, and schema metadata to identify common SQL query patterns directly within Amazon Redshift, accelerating the query authoring process for users and reducing the time required to derive actionable data insights. Choose Query data.

Metadata 105