AWS Big Data

OCTOBER 19, 2023

Data lakes and data warehouses are two of the most important data storage and management technologies in a modern data architecture. Data lakes store all of an organization’s data, regardless of its format or structure.

Data Lake 114

Data Lake Data Warehouse Visualization Snapshot 114

AWS Big Data

MAY 24, 2023

When you build your transactional data lake using Apache Iceberg to solve your functional use cases, you need to focus on operational use cases for your S3 data lake to optimize the production environment. You can use either the AWS Glue Data Catalog (recommended) or a Hive catalog for Iceberg tables.

Data Lake 84

Data Lake Snapshot Metadata Optimization 84

AWS Big Data

JULY 20, 2023

With data becoming the driving force behind many industries today, having a modern data architecture is pivotal for organizations to be successful. In this post, we describe Orca’s journey building a transactional data lake using Amazon Simple Storage Service (Amazon S3), Apache Iceberg, and AWS Analytics.

Data Lake 87

Data Lake Analytics Snapshot Data Quality 87

AWS Big Data

JULY 3, 2023

Terminology Let’s first discuss some of the terminology used in this post: Research data lake on Amazon S3 – A data lake is a large, centralized repository that allows you to manage all your structured and unstructured data at any scale. This is where the tagging feature in Apache Iceberg comes in handy.

Snapshot 87

Snapshot Data Lake Testing Strategy 87

AWS Big Data

FEBRUARY 12, 2024

It makes it fast, simple, and cost-effective to analyze all your data using standard SQL and your existing business intelligence (BI) tools. Moreover, no separate effort is required to process historical data versus live streaming data. Apart from incremental analytics, Redshift simplifies a lot of operational aspects.

Analytics 107

Analytics Data Warehouse Snapshot Cost-Benefit 107

AWS Big Data

JANUARY 24, 2023

The utility for cloning and experimentation is available in the open-sourced GitHub repository. This solution only replicates metadata in the Data Catalog, not the actual underlying data. This ensures that the data lake will still be functional in another Region if Lake Formation has an availability issue.

Data Architecture 66

Data Architecture Metadata Data Lake Snapshot 66

O'Reilly on Data

OCTOBER 19, 2021

It has far-reaching implications as to how such applications should be developed and by whom: ML applications are directly exposed to the constantly changing real world through data, whereas traditional software operates in a simplified, static, abstract world which is directly constructed by the developer. This approach is not novel.

IT 350

IT Testing Experimentation Software 350