AWS Big Data

JUNE 10, 2024

The open table format accelerates companies’ adoption of a modern data strategy because it allows them to use various tools on top of a single copy of the data. A solution based on Apache Iceberg encompasses complete data management, featuring simple built-in table optimization capabilities within an existing storage solution.

Data Lake 125

Data Lake Metadata Snapshot Analytics 125

AWS Big Data

JUNE 9, 2023

A modern data architecture enables companies to ingest virtually any type of data through automated pipelines into a data lake, which provides highly durable and cost-effective object storage at petabyte or exabyte scale. and Delta Lake 2.3.0. Apache Iceberg 1.2.0,

Data Lake 129

Data Lake Metadata Statistics Optimization 129

AWS Big Data

OCTOBER 19, 2023

Data lakes and data warehouses are two of the most important data storage and management technologies in a modern data architecture. Data lakes store all of an organization’s data, regardless of its format or structure.

Data Lake 129

Data Lake Data Warehouse Visualization Snapshot 129

AWS Big Data

DECEMBER 4, 2024

With this new functionality, customers can create up-to-date replicas of their data from applications such as Salesforce, ServiceNow, and Zendesk in an Amazon SageMaker Lakehouse and Amazon Redshift. SageMaker Lakehouse gives you the flexibility to access and query your data in-place with all Apache Iceberg compatible tools and engines.

Data Integration 105

Data Integration Data Lake Statistics Data-driven 105

AWS Big Data

MARCH 10, 2023

Since the deluge of big data over a decade ago, many organizations have learned to build applications to process and analyze petabytes of data. Data lakes have served as a central repository to store structured and unstructured data at any scale and in various formats.

Data Lake 120

Data Lake Sales Data Warehouse Snapshot 120

AWS Big Data

MAY 24, 2023

When you build your transactional data lake using Apache Iceberg to solve your functional use cases, you need to focus on operational use cases for your S3 data lake to optimize the production environment. availability. The examples are run on a Jupyter Notebook environment attached to the EMR cluster.

Data Lake 96

Data Lake Snapshot Metadata Optimization 96

AWS Big Data

JULY 29, 2024

In the era of big data, data lakes have emerged as a cornerstone for storing vast amounts of raw data in its native format. They support structured, semi-structured, and unstructured data, offering a flexible and scalable environment for data ingestion from multiple sources.

Metadata 124

Metadata Snapshot Data Lake Metrics 124

AWS Big Data

DECEMBER 12, 2024

In the context of comprehensive data governance, Amazon DataZone offers organization-wide data lineage visualization using Amazon Web Services (AWS) services, while dbt provides project-level lineage through model analysis and supports cross-project integration between data lakes and warehouses.

Snapshot 87

Snapshot Recreation/Entertainment Experimentation Data Lake 87

AWS Big Data

APRIL 27, 2023

Amazon Athena supports the MERGE command on Apache Iceberg tables, which allows you to perform inserts, updates, and deletes in your data lake at scale using familiar SQL statements that are compliant with ACID (Atomic, Consistent, Isolated, Durable).

Data Lake 89

Data Lake Snapshot Optimization Data Transformation 89

AWS Big Data

JULY 20, 2023

With data becoming the driving force behind many industries today, having a modern data architecture is pivotal for organizations to be successful. In this post, we describe Orca’s journey building a transactional data lake using Amazon Simple Storage Service (Amazon S3), Apache Iceberg, and AWS Analytics.

Data Lake 97

Data Lake Analytics Snapshot Data Quality 97

AWS Big Data

SEPTEMBER 13, 2023

A modern data architecture is an evolutionary architecture pattern designed to integrate a data lake, data warehouse, and purpose-built stores with a unified governance model. The company wanted the ability to continue processing operational data in the secondary Region in the rare event of primary Region failure.

Data Lake 109

Data Lake Data Processing Metadata Snapshot 109

AWS Big Data

MARCH 4, 2024

As enterprises collect increasing amounts of data from various sources, the structure and organization of that data often need to change over time to meet evolving analytical needs. Schema evolution enables adding, deleting, renaming, or modifying columns without needing to rewrite existing data.

Snapshot 129

Snapshot Data Lake Metadata Recreation/Entertainment 129

AWS Big Data

SEPTEMBER 12, 2024

The AWS Glue Data Catalog now enhances managed table optimization of Apache Iceberg tables by automatically removing data files that are no longer needed. Iceberg creates a new version called a snapshot for every change to the data in the table. As more table changes are made, more data files are created.

Optimization 122

Optimization Snapshot Metadata Metrics 122

O'Reilly on Data

OCTOBER 19, 2021

Since software engineers manage to build ordinary software without experiencing as much pain as their counterparts in the ML department, it begs the question: should we just start treating ML projects as software engineering projects as usual, maybe educating ML practitioners about the existing best practices? Orchestration. Versioning.

IT 364

IT Testing Experimentation Software 364

AWS Big Data

JANUARY 24, 2024

AWS-powered data lakes, supported by the unmatched availability of Amazon Simple Storage Service (Amazon S3), can handle the scale, agility, and flexibility required to combine different data and analytics approaches. For more information, refer to Amazon S3: Allows read and write access to objects in an S3 Bucket.

Snapshot 121

Snapshot Data Lake Metadata Optimization 121

AWS Big Data

DECEMBER 9, 2024

In todays data-driven world, tracking and analyzing changes over time has become essential. As organizations process vast amounts of data, maintaining an accurate historical record is crucial. History management in data systems is fundamental for compliance, business intelligence, data quality, and time-based analysis.

Snapshot 64

Snapshot Data Warehouse Data Lake Data Quality 64

AWS Big Data

OCTOBER 3, 2023

In our previous post Improve operational efficiencies of Apache Iceberg tables built on Amazon S3 data lakes , we discussed how you can implement solutions to improve operational efficiencies of your Amazon Simple Storage Service (Amazon S3) data lake that is using the Apache Iceberg open table format and running on the Amazon EMR big data platform.

Optimization 125

Optimization Snapshot Data Lake Metadata 125

AWS Big Data

MARCH 27, 2023

Amazon Redshift Serverless makes it simple to run and scale analytics without having to manage your data warehouse infrastructure. You can define your own key and value for your resource tag, so that you can easily manage and filter your resources. Tags allows you to assign metadata to your AWS resources. Create cost reports.

Data Warehouse 96

Data Warehouse Management Snapshot Data Lake 96

AWS Big Data

MARCH 28, 2023

As organizations across the globe are modernizing their data platforms with data lakes on Amazon Simple Storage Service (Amazon S3), handling SCDs in data lakes can be challenging.

Data Lake 88

Data Lake Testing Snapshot Sales 88

AWS Big Data

JULY 3, 2023

Index rebalancing arbitrage takes advantage of short-term price discrepancies resulting from ETF managers’ efforts to minimize index tracking error. Amazon Simple Storage Service (Amazon S3) is a popular cloud-based object storage service that can be used as the foundation for building a data lake.

Snapshot 101

Snapshot Data Lake Testing Strategy 101

AWS Big Data

NOVEMBER 22, 2023

Apache Hudi is an open table format that brings database and data warehouse capabilities to data lakes. Apache Hudi helps data engineers manage complex challenges, such as managing continuously evolving datasets with transactions while maintaining query performance.

Data Lake 112

Data Lake Snapshot Metadata Optimization 112

AWS Big Data

JULY 8, 2024

Amazon Managed Service for Apache Flink offers a fully managed, serverless experience in running Apache Flink applications and now supports Apache Flink 1.19.1 , the latest stable version of Apache Flink at the time of writing. Managed Service for Apache Flink currently uses the Python 3.11 support Python 3.11 Python 3.11

Management 111

Management Consulting Dashboards Snapshot 111

AWS Big Data

MARCH 20, 2023

In the first post of this series , we described how AWS Glue for Apache Spark works with Apache Hudi, Linux Foundation Delta Lake, and Apache Iceberg datasets tables using the native support of those data lake formats. Even without prior experience using Hudi, Delta Lake or Iceberg, you can easily achieve typical use cases.

Visualization 94

Visualization Data Lake Snapshot Big Data 94

AWS Big Data

JANUARY 17, 2024

With Amazon EMR 6.15, we launched AWS Lake Formation based fine-grained access controls (FGAC) on Open Table Formats (OTFs), including Apache Hudi, Apache Iceberg, and Delta lake. Many large enterprise companies seek to use their transactional data lake to gain insights and improve decision-making.

Data Lake 113

Data Lake Snapshot Big Data Data-driven 113

AWS Big Data

MAY 15, 2024

It is a serverless service, eliminating the need for infrastructure management and costing you only for the queries you run. By extracting detailed information from CloudTrail and querying it using Athena, this solution streamlines the process of data collection, analysis, and reporting of EIP usage within an AWS account.

Snapshot 110

Snapshot Optimization Data Lake Reporting 110

AWS Big Data

FEBRUARY 12, 2024

About Redshift and some relevant features for the use case Amazon Redshift is a fully managed, petabyte-scale, massively parallel data warehouse that offers simple operations and high performance. It makes it fast, simple, and cost-effective to analyze all your data using standard SQL and your existing business intelligence (BI) tools.

Analytics 118

Analytics Data Warehouse Snapshot Cost-Benefit 118

AWS Big Data

AUGUST 9, 2024

This trend is no exception for Dafiti , an ecommerce company that recognizes the importance of using data to drive strategic decision-making processes. Amazon Redshift is widely used for Dafiti’s data analytics, supporting approximately 100,000 daily queries from over 400 users across three countries. TB of data.

Data Lake 110

Data Lake Analytics Data Warehouse Data-driven 110

AWS Big Data

JUNE 27, 2024

Amazon Redshift is a fully managed, petabyte-scale data warehouse service in the cloud. You can start with just a few hundred gigabytes of data and scale to a petabyte or more. This enables you to use your data to acquire new insights for your business and customers. For additional details, refer to Automated snapshots.

Snapshot 87

Snapshot Data Warehouse Data Processing Strategy 87

AWS Big Data

NOVEMBER 16, 2023

Amazon Redshift is a popular cloud data warehouse, offering a fully managed cloud-based service that seamlessly integrates with an organization’s Amazon Simple Storage Service (Amazon S3) data lake, real-time streams, machine learning (ML) workflows, transactional workflows, and much more—all while providing up to 7.9x

Enterprise 104

Enterprise Data Warehouse Snapshot Cost-Benefit 104

AWS Big Data

MARCH 25, 2024

In this post, we discuss why data streaming is a crucial component of generative AI applications due to its real-time nature. In-context learning LLMs are trained with point-in-time data and have no inherent ability to access fresh data at inference time. For more information, refer to Dynamic Tables.

Data Lake 112

Data Lake Unstructured Data Management Snapshot 112

AWS Big Data

JANUARY 24, 2023

This solution only replicates metadata in the Data Catalog, not the actual underlying data. To have a redundant data lake using Lake Formation and AWS Glue in an additional Region, we recommend replicating the Amazon S3-based storage using S3 replication , S3 sync, aws-s3-copy-sync-using-batch or S3 Batch replication process.

Data Architecture 80

Data Architecture Metadata Data Lake Snapshot 80

AWS Big Data

NOVEMBER 10, 2023

The data sourcing problem To ensure the reliability of PySpark data pipelines, it’s essential to have consistent record-level data from both dimensional and fact tables stored in the Enterprise Data Warehouse (EDW). These tables are then joined with tables from the Enterprise Data Lake (EDL) at runtime.

Data Processing 109

Data Processing Data Lake Data Warehouse Optimization 109

AWS Big Data

AUGUST 16, 2023

Iceberg has become very popular for its support for ACID transactions in data lakes and features like schema and partition evolution, time travel, and rollback. AWS Glue crawlers will extract schema information and update the location of Iceberg metadata and schema updates in the Data Catalog. Choose Next.

Data Lake 93

Data Lake Metadata Snapshot Management 93

Cloudera

APRIL 3, 2023

Every table change creates an Iceberg snapshot, this helps to resolve concurrency issues and allows readers to scan a stable table state every time. During queries the query engines scan both the data files and delete files belonging to the same snapshot and merge them together (i.e. ID, TBL_ICEBERG_PART_2.NAME,

Data Warehouse 82

Data Warehouse Snapshot Metadata Cost-Benefit 82

AWS Big Data

MARCH 23, 2023

Lake Formation provides an authorization and governance layer on data stored in Amazon S3. Large organizations often have lines of businesses (LoBs) that operate with autonomy in managing their business data. It makes sharing data across LoBs non-trivial. The Iceberg table keeps track of the snapshots.

Interactive 83

Interactive Snapshot Data Lake Software 83

AWS Big Data

JUNE 21, 2023

Amazon DynamoDB is a fully managed NoSQL service that delivers single-digit millisecond performance at any scale. The goal of a data warehouse is to enable businesses to analyze their data fast; this is important because it means they are able to gain valuable insights in a timely manner.

Data Warehouse 98

Data Warehouse Data Lake OLAP Cost-Benefit 98

AWS Big Data

JANUARY 8, 2024

For high availability and durability, Kinesis Data Streams achieves high durability by synchronously replicating the streamed data across three Availability Zones in an AWS Region and gives you the option to retain data for up to 365 days. The raw data can be streamed to Amazon S3 for archiving.

Analytics 130

Analytics IoT Data-driven Snapshot 130