AWS Big Data

SEPTEMBER 12, 2024

The AWS Glue Data Catalog now enhances managed table optimization of Apache Iceberg tables by automatically removing data files that are no longer needed. Along with the Glue Data Catalog’s automated compaction feature, these storage optimizations can help you reduce metadata overhead, control storage costs, and improve query performance.

Optimization 129

Optimization Snapshot Metadata Metrics 129

AWS Big Data

APRIL 9, 2025

Whether youre a data analyst seeking a specific metric or a data steward validating metadata compliance, this update delivers a more precise, governed, and intuitive search experience. This supports data hygiene and infrastructure cost optimization.

Metadata 111

Metadata Metrics Data-driven Cost-Benefit 111

AWS Big Data

NOVEMBER 19, 2024

Customers maintain multiple MWAA environments to separate development stages, optimize resources, manage versions, enhance security, ensure redundancy, customize settings, improve scalability, and facilitate experimentation. Refer to Amazon Managed Workflows for Apache Airflow Pricing for rates and more details.

Metadata 111

Metadata Cost-Benefit Metrics Optimization 111

AWS Big Data

NOVEMBER 6, 2024

This workload imbalance presents a challenge for customers seeking to optimize their resource utilization and stream processing efficiency. reduces the Amazon DynamoDB cost associated with KCL by optimizing read operations on the DynamoDB table storing metadata. x benefits, refer to Use features of the AWS SDK for Java 2.x.

Cost-Benefit 116

Cost-Benefit Metadata Optimization Publishing 116

Cloudera

NOVEMBER 13, 2020

Impala Optimizations for Small Queries. We’ll discuss the various phases Impala takes a query through and how small query optimizations are incorporated into the design of each phase. For a more in-depth description of these phases please refer to Impala: A Modern, Open-Source SQL Engine for Hadoop. Query Planner Design.

Optimization 143

Optimization Metadata Statistics Cost-Benefit 143

AWS Big Data

OCTOBER 3, 2023

Iceberg tables store metadata in manifest files. As the number of data files increase, the amount of metadata stored in these manifest files also increases, leading to longer query planning time. The query runtime also increases because it’s proportional to the number of data or metadata file read operations.

Optimization 126

Optimization Snapshot Data Lake Metadata 126

AWS Big Data

NOVEMBER 20, 2024

We’re excited to announce a new feature in Amazon DataZone that offers enhanced metadata governance for your subscription approval process. With this update, domain owners can define and enforce metadata requirements for data consumers when they request access to data assets. Key benefits The feature benefits multiple stakeholders.

Metadata 96

Metadata Data Governance Metrics Marketing 96

AWS Big Data

NOVEMBER 17, 2023

Starting today, the Athena SQL engine uses a cost-based optimizer (CBO), a new feature that uses table and column statistics stored in the AWS Glue Data Catalog as part of the table’s metadata. Let’s discuss some of the cost-based optimization techniques that contributed to improved query performance.

Optimization 130

Optimization Statistics Metadata Data Lake 130

AWS Big Data

APRIL 24, 2025

Bob now knows that he can quickly build Amazon QuickSight dashboards with queries that are optimized using Redshifts cost-based optimizer. Ava defines the user attributes as static IAM tags that could also include attributes stored in the identity provider (IdP) or as session tags dynamically to represent the user metadata.

Sales 89

Sales Data Lake Management Data-driven 89

AWS Big Data

NOVEMBER 7, 2024

BladeBridge provides a configurable framework to seamlessly convert legacy metadata and code into more modern services such as Amazon Redshift. For more details, refer to the BladeBridge Analyzer Demo. Refer to this BladeBridge documentation to get more details on SQL and expression conversion.

Data Warehouse 105

Data Warehouse Reporting Big Data Data Lake 105

AWS Big Data

NOVEMBER 19, 2024

Unfiltered Table Metadata This tab displays the response of the AWS Glue API GetUnfilteredTableMetadata policies for the selected table. Get table data and metadata for this user to see how Lake Formation permissions are enforced and so the two users can see different data (on the Authorized Data tab).

Data Processing 98

Data Processing Metadata Publishing Testing 98

AWS Big Data

FEBRUARY 6, 2023

For instructions, refer to How to Set Up a MongoDB Cluster. Choose the table to view the schema and other metadata. Conclusion In this post, we showed how to set up an AWS Glue crawler to crawl over a MongoDB Atlas collection, gathering metadata and creating table records in the AWS Glue Data Catalog.

Metadata 98

Metadata Data Lake Machine Learning Big Data 98

AWS Big Data

MARCH 22, 2024

When you use Trino on Amazon EMR or Athena, you get the latest open source community innovations along with proprietary, AWS developed optimizations. and Athena engine version 2, AWS has been developing query plan and engine behavior optimizations that improve query performance on Trino. Starting from Amazon EMR 6.8.0

Metadata 119

Metadata Statistics Broadcasting Optimization 119

erwin

JUNE 26, 2020

Organizations with particularly deep data stores might need a data catalog with advanced capabilities, such as automated metadata harvesting to speed up the data preparation process. The most optimal and streamlined way to achieve this is by using a data catalog, which can provide a first stop for users ahead of working in BI platforms.

Metadata 132

Metadata Cost-Benefit Measurement Data-driven 132

AWS Big Data

OCTOBER 21, 2024

Let’s briefly describe the capabilities of the AWS services we referred above: AWS Glue is a fully managed, serverless, and scalable extract, transform, and load (ETL) service that simplifies the process of discovering, preparing, and loading data for analytics. As stated earlier, the first step involves data ingestion.

Sales 115

Sales Data-driven Data Processing Key Performance Indicator 115

O'Reilly on Data

DECEMBER 19, 2018

We won’t be writing code to optimize scheduling in a manufacturing plant; we’ll be training ML algorithms to find optimum performance based on historical data. With machine learning, the challenge isn’t writing the code; the algorithms are implemented in a number of well-known and highly optimized libraries.

Machine Learning 279

Machine Learning Software Metadata Testing 279

AWS Big Data

SEPTEMBER 11, 2024

Each file arrives as a pair with a tail metadata file in CSV format containing the size and name of the file. This metadata file is later used to read source file names during processing into the staging layer. These files follow the same naming pattern, with a daily system-generated timestamp appended to each file name.

Data Architecture 111

Data Architecture Optimization Data Warehouse Metadata 111

AWS Big Data

OCTOBER 1, 2024

Over the last year, Amazon Redshift added several performance optimizations for data lake queries across multiple areas of query engine such as rewrite, planning, scan execution and consuming AWS Glue Data Catalog column statistics. Some of the queries in our benchmark experienced up to 12x speed up.

Data Lake 116

Data Lake Statistics Broadcasting Optimization 116

AWS Big Data

NOVEMBER 3, 2023

Although we don’t cover optimizing your jobs for costs in this post, you can refer to Monitor and optimize cost on AWS Glue for Apache Spark to learn how to fine-tune your AWS Glue jobs for performance, efficiency ,and cost-optimization. Let’s dive in! If the tables don’t exist, Athena creates them.

Metrics 115

Metrics Dashboards Metadata Visualization 115

AWS Big Data

JANUARY 24, 2024

Each storage format implements this functionality in slightly different ways; for a comparison, refer to Choosing an open table format for your transactional data lake on AWS. For more information, refer to Amazon S3: Allows read and write access to objects in an S3 Bucket.

Snapshot 126

Snapshot Data Lake Metadata Optimization 126

AWS Big Data

JANUARY 18, 2024

We group the new capabilities into four categories: Discover and secure Connect with data sharing Scale and optimize Audit and monitor Let’s dive deeper and discuss the new capabilities introduced in 2023. To learn more about DataZone, refer to the User Guide. This enhancement simplifies many use cases to avoid metadata duplication.

Data Lake 119

Data Lake Metadata Data Governance Statistics 119

IBM Big Data Hub

NOVEMBER 28, 2022

Metadata management performs a critical role within the modern data management stack. However, as data volumes continue to grow, manual approaches to metadata management are sub-optimal and can result in missed opportunities. This puts into perspective the role of active metadata management. Improve data discovery.

Metadata 56

Metadata Data Quality Data-driven Data Governance 56

datapine

SEPTEMBER 29, 2022

Data quality refers to the assessment of the information you have, relative to its purpose and its ability to serve that purpose. While the digital age has been successful in prompting innovation far and wide, it has also facilitated what is referred to as the “data crisis” – low-quality data. 2 – Data profiling.

Data Quality 241

Data Quality Metrics Data-driven Management 241

AWS Big Data

FEBRUARY 13, 2025

To learn more about Express brokers, refer to Introducing Express brokers for Amazon MSK to deliver high throughput and faster scaling for your Kafka clusters. However, you can use Amazon MSK Replicator to copy all data and metadata from your existing MSK cluster to a new cluster comprising of Express brokers.

Metrics 67

Metrics Metadata Strategy Management 67

AWS Big Data

DECEMBER 4, 2024

Use case Consider a large company that relies heavily on data-driven insights to optimize its customer support processes. The data is also registered in the Glue Data Catalog , a metadata repository. The database will be used to store the metadata related to the data integrations performed by zero-ETL.

Data Integration 115

Data Integration Data Lake Statistics Data-driven 115

Ontotext

FEBRUARY 16, 2023

Ontotext’s approach is to optimize models and algorithms through human contribution and benchmarking in order to create better and more accurate AI. To be able to annotate the specified content consistently and unambiguously, these experts usually follow a set of specific conventions, which are referred to as “annotation guidelines”.

Metadata 59

Metadata Measurement Metrics Modeling 59

CIO Business Intelligence

OCTOBER 13, 2023

They understand data modeling, including conceptualization and database optimization, and demonstrate a commitment to continuing education. According to Dataversity , good data architects have a solid understanding of the cloud, databases, and the applications and programs used by those databases.

Data Architecture 132

Data Architecture Data Warehouse Statistics Visualization 132

CIO Business Intelligence

MARCH 24, 2023

While data management has become a common term for the discipline, it is sometimes referred to as data resource management or enterprise information management (EIM). Programs must support proactive and reactive change management activities for reference data values and the structure/use of master data and metadata.

Data Governance 142

Data Governance Management Metadata Data Quality 142

AWS Big Data

JUNE 22, 2023

If you simply run queries without considering the optimal data layout on Amazon S3, it results in a high volume of data scanned, long-running queries, and increased cost. Partitioning is a common technique to lay out your data optimally for distributed analytics engines. We also can see the partition metadata on the AWS Glue console.

Metadata 105

Metadata Management Recreation/Entertainment Optimization 105

AWS Big Data

AUGUST 31, 2023

The Iceberg specification allows seamless table evolution such as schema and partition evolution, and its design is optimized for usage on Amazon S3. Iceberg stores the metadata pointer for all the metadata files. For more details on Iceberg format versions, refer to Format Versioning.

Data Lake 119

Data Lake Data Warehouse Metadata Data Architecture 119

Cloudera

JANUARY 15, 2021

Pushing down column predicate filters to Kudu allows for optimized execution by skipping reading column values for filtered out rows and reducing network IO between a client, like the distributed query engine Apache Impala, and Kudu. See the references section below for details on the table schema, loading process, and queries that were run.

Optimization 72

Optimization Broadcasting Testing Metadata 72

AWS Big Data

JUNE 9, 2023

Despite these capabilities, data lakes are not databases, and object storage does not provide support for ACID processing semantics, which you may require to effectively optimize and manage your data at scale across hundreds or thousands of users using a multitude of different technologies.

Data Lake 130

Data Lake Metadata Statistics Optimization 130

AWS Big Data

NOVEMBER 15, 2023

BMW Group uses 4,500 AWS Cloud accounts across the entire organization but is faced with the challenge of reducing unnecessary costs, optimizing spend, and having a central place to monitor costs. For more information on this foundation, refer to A Detailed Overview of the Cost Intelligence Dashboard.

Analytics 111

Analytics Dashboards Metadata Data Warehouse 111

AWS Big Data

MARCH 2, 2023

Apache Iceberg is an open table format for very large analytic datasets, which captures metadata information on the state of datasets as they evolve and change over time. Apache Iceberg addresses customer needs by capturing rich metadata information about the dataset at the time the individual data files are created.

Data Lake 137

Data Lake Data Processing Metadata Snapshot 137

AWS Big Data

OCTOBER 16, 2023

The individual pieces of data within these streams are often referred to as records. Store large records in Amazon S3 with a reference in Kinesis Data Streams A useful approach for storing large records involves utilizing an alternative storage solution while employing a reference within Kinesis Data Streams.

Cost-Benefit 126

Cost-Benefit Testing Optimization Strategy 126

AWS Big Data

JUNE 7, 2023

Through their unique position in ports, at sea, and on roads, they optimize global cargo flows and create sustainable customer value. To share the datasets, they needed a way to share access to the data and access to catalog metadata in the form of tables and views. An AWS Glue job (metadata exporter) runs daily on the source account.

Metadata 59

Metadata Data Lake Machine Learning Big Data 59

Ontotext

JULY 29, 2021

KGs bring the Semantic Web paradigm to the enterprises, by introducing semantic metadata to drive data management and content management to new levels of efficiency and breaking silos to let them synergize with various forms of knowledge management. Take this restaurant, for example. used across different systems in the enterprise.

Enterprise 124

Enterprise Metadata Knowledge Discovery Management 124