AWS Big Data

JANUARY 27, 2023

Verify all table metadata is stored in the AWS Glue Data Catalog. Consume data with Athena or Amazon EMR Trino for business analysis. Update and delete source records in Amazon RDS for MySQL and validate the reflection of the data lake tables. the Flink table API/SQL can integrate with the AWS Glue Data Catalog.

Data Lake 92

Data Lake Metadata Business Analysis Data-driven 92

AWS Big Data

JANUARY 17, 2024

With Amazon EMR 6.15, we launched AWS Lake Formation based fine-grained access controls (FGAC) on Open Table Formats (OTFs), including Apache Hudi, Apache Iceberg, and Delta lake. Many large enterprise companies seek to use their transactional data lake to gain insights and improve decision-making.

Data Lake 116

Data Lake Snapshot Big Data Data-driven 116

AWS Big Data

MAY 15, 2024

By extracting detailed information from CloudTrail and querying it using Athena, this solution streamlines the process of data collection, analysis, and reporting of EIP usage within an AWS account. Download the CloudFormation template from the repository. It then determines the frequency of EIP attachments to resources.

Snapshot 115

Snapshot Optimization Data Lake Reporting 115

AWS Big Data

JUNE 21, 2023

When setting out to build a data warehouse, it’s a common pattern to have a data lake as the source of the data warehouse. The data lake in this context serves a number of important functions: It acts as a central source for multiple applications, not just exclusively for data warehousing purposes.

Data Warehouse 105

Data Warehouse Data Lake OLAP Cost-Benefit 105

AWS Big Data

JULY 28, 2023

Amazon Redshift now makes it easier for you to run queries in AWS data lakes by automatically mounting the AWS Glue Data Catalog. You no longer have to create an external schema in Amazon Redshift to use the data lake tables cataloged in the Data Catalog.

Data Lake 96

Data Lake Data Governance Data Warehouse Data-driven 96

AWS Big Data

JULY 25, 2023

SQL Notebook You can download the SQL notebook with most used system views queries. Choose your level of metrics to monitor: Workgroup Namespace Snapshot storage If we select Workgroup , we can choose from the workgroup-level metrics shown in the following screenshot. How to monitor queries based on status?

Metrics 98

Metrics Data Warehouse Dashboards Snapshot 98

AWS Big Data

AUGUST 1, 2024

We can determine the following are needed: An open data format ingestion architecture processing the source dataset and refining the data in the S3 data lake. This requires a dedicated team of 3–7 members building a serverless data lake for all data sources. You can import this in Query Editor V2.0.

Data Warehouse 115

Data Warehouse KPI Optimization Cost-Benefit 115

AWS Big Data

JUNE 12, 2023

Prerequisites Create and download a valid key to SSH into an Amazon Elastic Compute Cloud (Amazon EC2) instance from your local machine. If the check boxes on the Lake Formation Data Catalog settings page are unselected (see the following screenshot), that means that the Data Catalog permissions are being managed by LakeFormation.

Management 98

Management Metadata Internet of Things Testing 98

AWS Big Data

DECEMBER 9, 2024

Today, many customers build data quality validation pipelines using its Data Quality Definition Language (DQDL) because with static rules, dynamic rules , and anomaly detection capability , its fairly straightforward. One of its key features is the ability to manage data using branches. Download wap.ipynb.

Data Quality 126

Data Quality Publishing Snapshot Data Lake 126

AWS Big Data

MAY 9, 2025

As data use cases become more complex, data engineering teams require sophisticated tooling to handle versioning, increasing data volumes, and schema changes across multiple data sources and applications. For example, they might need to compare historical snapshots with current data to analyze trends over time.

Snapshot 91

Snapshot Analytics Data-driven Data Processing 91

AWS Big Data

DECEMBER 19, 2024

Data lakes were originally designed to store large volumes of raw, unstructured, or semi-structured data at a low cost, primarily serving big data and analytics use cases. Enabling automatic compaction on Iceberg tables reduces metadata overhead on your Iceberg tables and improves query performance.

Data Lake 98

Data Lake IoT Metadata Testing 98