AWS Big Data

APRIL 27, 2023

Amazon Athena supports the MERGE command on Apache Iceberg tables, which allows you to perform inserts, updates, and deletes in your data lake at scale using familiar SQL statements that are compliant with ACID (Atomic, Consistent, Isolated, Durable).

Data Lake 96

Data Lake Snapshot Optimization Data Transformation 96

AWS Big Data

MARCH 10, 2023

Since the deluge of big data over a decade ago, many organizations have learned to build applications to process and analyze petabytes of data. Data lakes have served as a central repository to store structured and unstructured data at any scale and in various formats.

Data Lake 122

Data Lake Sales Data Warehouse Snapshot 122

AWS Big Data

JULY 20, 2023

With data becoming the driving force behind many industries today, having a modern data architecture is pivotal for organizations to be successful. In this post, we describe Orca’s journey building a transactional data lake using Amazon Simple Storage Service (Amazon S3), Apache Iceberg, and AWS Analytics.

Data Lake 98

Data Lake Analytics Snapshot Data Quality 98

Cloudera

OCTOBER 10, 2024

It combines the flexibility and scalability of data lake storage with the data analytics, data governance, and data management functionality of the data warehouse. Let’s take a look at some of the features in Cloudera Lakehouse Optimizer, the benefits they provide, and the road ahead for this service.

Optimization 90

Optimization Snapshot Data Lake Cost-Benefit 90

AWS Big Data

SEPTEMBER 13, 2023

A modern data architecture is an evolutionary architecture pattern designed to integrate a data lake, data warehouse, and purpose-built stores with a unified governance model. Moreover, the framework should consume compute resources as optimally as possible per the size of the operational tables.

Data Lake 111

Data Lake Data Processing Metadata Snapshot 111

AWS Big Data

NOVEMBER 21, 2024

The AWS Glue Data Catalog supports automatic table optimization of Apache Iceberg tables, including compaction , snapshots, and orphan data management. These storage optimizations can help you reduce metadata overhead, control storage costs, and improve query performance. Choose iceberg_optimizer_vpc_db.

Optimization 111

Optimization Snapshot Metadata Software 111

AWS Big Data

JANUARY 24, 2024

AWS-powered data lakes, supported by the unmatched availability of Amazon Simple Storage Service (Amazon S3), can handle the scale, agility, and flexibility required to combine different data and analytics approaches. Compacting files speeds up the read operation when queried.

Snapshot 126

Snapshot Data Lake Metadata Optimization 126

AWS Big Data

DECEMBER 4, 2024

Open table formats are emerging in the rapidly evolving domain of big data management, fundamentally altering the landscape of data storage and analysis. By providing a standardized framework for data representation, open table formats break down data silos, enhance data quality, and accelerate analytics at scale.

Snapshot 84

Snapshot Metadata Data Lake Optimization 84

O'Reilly on Data

OCTOBER 19, 2021

ML use cases rarely dictate the master data management solution, so the ML stack needs to integrate with existing data warehouses. To manage the dynamism, we can resort to taking snapshots that represent immutable points in time: of models, of data, of code, and of internal state. Enter the software development layers.

IT 350

IT Testing Experimentation Software 350

AWS Big Data

NOVEMBER 22, 2023

Apache Hudi is an open table format that brings database and data warehouse capabilities to data lakes. Apache Hudi helps data engineers manage complex challenges, such as managing continuously evolving datasets with transactions while maintaining query performance. For CoW tables, queries see the latest data committed.

Data Lake 115

Data Lake Snapshot Metadata Optimization 115

AWS Big Data

JULY 3, 2023

Backtesting is a process used in quantitative finance to evaluate trading strategies using historical data. This helps traders determine the potential profitability of a strategy and identify any risks associated with it, enabling them to optimize it for better performance.

Snapshot 105

Snapshot Data Lake Testing Strategy 105

Cloudera

FEBRUARY 14, 2024

Introduction Apache Iceberg has recently grown in popularity because it adds data warehouse-like capabilities to your data lake making it easier to analyze all your data — structured and unstructured. Problem with too many snapshots Everytime a write operation occurs on an Iceberg table, a new snapshot is created.

Optimization 59

Optimization Strategy Snapshot Metadata 59

AWS Big Data

JANUARY 17, 2024

With Amazon EMR 6.15, we launched AWS Lake Formation based fine-grained access controls (FGAC) on Open Table Formats (OTFs), including Apache Hudi, Apache Iceberg, and Delta lake. Many large enterprise companies seek to use their transactional data lake to gain insights and improve decision-making.

Data Lake 115

Data Lake Snapshot Big Data Data-driven 115

AWS Big Data

MAY 15, 2024

You can use this solution regularly as part of your cost-optimization efforts to safely remove unused EIPs to reduce your costs. By extracting detailed information from CloudTrail and querying it using Athena, this solution streamlines the process of data collection, analysis, and reporting of EIP usage within an AWS account.

Snapshot 115

Snapshot Optimization Data Lake Reporting 115

AWS Big Data

DECEMBER 12, 2024

Amazon Athena offers serverless, flexible SQL analytics for one-time queries, enabling direct querying of Amazon Simple Storage Service (Amazon S3) data for rapid, cost-effective instant analysis. In this post, we use dbt for data modeling on both Amazon Athena and Amazon Redshift.

Snapshot 90

Snapshot Recreation/Entertainment Experimentation Data Lake 90

AWS Big Data

AUGUST 9, 2024

These processes retrieve data from around 90 different data sources, resulting in updating roughly 2,000 tables in the data warehouse and 3,000 external tables in Parquet format, accessed through Amazon Redshift Spectrum and a data lake on Amazon Simple Storage Service (Amazon S3). TB of data.

Data Lake 111

Data Lake Analytics Data Warehouse Data-driven 111

AWS Big Data

NOVEMBER 10, 2023

Your applications can seamlessly read from and write to your Amazon Redshift data warehouse while maintaining optimal performance and transactional consistency. Additionally, you’ll benefit from performance improvements through pushdown optimizations, further enhancing the efficiency of your operations. options(**read_config).option("query",

Data Processing 111

Data Processing Data Lake Data Warehouse Optimization 111

AWS Big Data

NOVEMBER 16, 2023

Amazon Redshift is a popular cloud data warehouse, offering a fully managed cloud-based service that seamlessly integrates with an organization’s Amazon Simple Storage Service (Amazon S3) data lake, real-time streams, machine learning (ML) workflows, transactional workflows, and much more—all while providing up to 7.9x

Enterprise 110

Enterprise Data Warehouse Snapshot Cost-Benefit 110

Cloudera

FEBRUARY 8, 2024

Queries containing joins, filters, projections, group-by, or aggregations without group-by can be transparently rewritten by the Hive optimizer to use one or more eligible materialized views. Subsequently, these snapshot IDs are used to determine the delta changes that should be applied to the materialized view rows.

Snapshot 96

Snapshot Metadata Cost-Benefit Data Warehouse 96

AWS Big Data

DECEMBER 4, 2024

With this new functionality, customers can create up-to-date replicas of their data from applications such as Salesforce, ServiceNow, and Zendesk in an Amazon SageMaker Lakehouse and Amazon Redshift. SageMaker Lakehouse gives you the flexibility to access and query your data in-place with all Apache Iceberg compatible tools and engines.

Data Integration 113

Data Integration Data Lake Statistics Data-driven 113

AWS Big Data

FEBRUARY 1, 2023

With data volumes exhibiting a double-digit percentage growth rate year on year and the COVID pandemic disrupting global logistics in 2021, it became more critical to scale and generate near-real-time data. This introduces the need for both polling and pushing the data to access and analyze in near-real time.

Optimization 52

Optimization Forecasting Data Lake Metadata 52

AWS Big Data

JANUARY 24, 2023

This solution only replicates metadata in the Data Catalog, not the actual underlying data. To have a redundant data lake using Lake Formation and AWS Glue in an additional Region, we recommend replicating the Amazon S3-based storage using S3 replication , S3 sync, aws-s3-copy-sync-using-batch or S3 Batch replication process.

Data Architecture 90

Data Architecture Metadata Data Lake Snapshot 90

Cloudera

APRIL 3, 2023

Every table change creates an Iceberg snapshot, this helps to resolve concurrency issues and allows readers to scan a stable table state every time. During queries the query engines scan both the data files and delete files belonging to the same snapshot and merge them together (i.e. ID, TBL_ICEBERG_PART_2.NAME,

Data Warehouse 87

Data Warehouse Snapshot Metadata Cost-Benefit 87

AWS Big Data

JUNE 21, 2023

Built on highly curated structured data, it provides the flexibility and speed to run aggregations across an entire dataset to derive insights. To house our data, we need to define a data model. An optimal design choice is to use a dimensional model. This is achieved by partitioning the data.

Data Warehouse 104

Data Warehouse Data Lake OLAP Cost-Benefit 104

AWS Big Data

MARCH 25, 2024

Furthermore, data events are filtered, enriched, and transformed to a consumable format using a stream processor. The result is made available to the application by querying the latest snapshot. For more details, refer to Create a low-latency source-to-data lake pipeline using Amazon MSK Connect, Apache Flink, and Apache Hudi.

Data Lake 119

Data Lake Unstructured Data Management Snapshot 119

AWS Big Data

AUGUST 1, 2024

This includes the ETL processes that capture source data, the functional refinement and creation of data products, the aggregation for business metrics, and the consumption from analytics, business intelligence (BI), and ML. This will enable right-sizing the Redshift data warehouse to meet workload demands cost-effectively.

Data Warehouse 113

Data Warehouse KPI Optimization Cost-Benefit 113

AWS Big Data

JANUARY 8, 2024

Stream Processing – An application created with Amazon Managed Service for Apache Flink can read the records from the data stream to detect and clean any errors in the time series data and enrich the data with specific metadata to optimize operational analytics.

Analytics 134

Analytics IoT Data-driven Snapshot 134

Cloudera

MARCH 5, 2024

With built-in features like time travel, schema evolution, and streamlined data discovery, Iceberg empowers data teams to enhance data lake management while upholding data integrity. Zero Downtime Upgrades Beyond improvements to Iceberg and Ozone, the platform now boasts Zero Downtime Upgrades (ZDU).

Snapshot 92

Snapshot Data Lake Enterprise Data Governance 92

AWS Big Data

DECEMBER 9, 2024

Inventory management benefits from historical data for analyzing sales patterns and optimizing stock levels. In fraud detection, historical data helps identify anomalous patterns in transactions or user behaviors. Anytime when you need SCD Type-2 snapshot of your Iceberg table, you can create the corresponding representation.

Snapshot 64

Snapshot Data Warehouse Data Lake Data Quality 64

AWS Big Data

JULY 8, 2024

In Apache Flink 1.19, you can now specify different checkpointing intervals based on whether a source operator is processing backlog data. This flexibility optimizes job performance by reducing checkpoint frequency during backlog phases, enhancing overall throughput. This feature only involves source connectors.

Management 115

Management Consulting Snapshot Dashboards 115

AWS Big Data

DECEMBER 19, 2024

Data lakes were originally designed to store large volumes of raw, unstructured, or semi-structured data at a low cost, primarily serving big data and analytics use cases. With this new feature, as you enable the Data Catalog optimizer.

Data Lake 52

Data Lake IoT Metadata Testing 52

AWS Big Data

DECEMBER 20, 2024

After the tables are created, the second task transfers HDFS data to a landing bucket in Amazon S3 using AWS DataSync to sync customer data. This process brings data from all the different layers of the data lake. Its highly recommended to regularly expire snapshots that are no longer needed.

Metadata 52

Metadata Data-driven Snapshot Data Lake 52

Cloudera

NOVEMBER 15, 2024

A Better Way Forward: Cloudera’s Open Data Lakehouse Cloudera offers a solution to these challenges with its open data lakehouse, which combines the flexibility and scalability of data lake storage with data warehouse functionality to unify and simplify the management of cyber log data.

Analytics 90

Analytics Metadata Snapshot Data-driven 90

Cloudera

SEPTEMBER 12, 2022

We have delivered the performance and reliability of the data warehouse with the flexibility and scale of a data lake with our data service engines and the Hive metastore. Applying the Iceberg table format to all the organization’s data in the data lake makes it more performant and usable at scale.

Unstructured Data 80

Unstructured Data Data Architecture Data Lake Snapshot 80

Cloudera

MARCH 23, 2022

Being multi-function also means integrated end-to-end data pipelines that break siloes, piecing together analytics as a coherent life-cycle where business value can be extracted at each and every stage. Users should be able to choose their tool of choice and take advantage of its workload specific optimizations. 4: Enterprise grade.

Metadata 97

Metadata Data Architecture Machine Learning Cost-Benefit 97

Corinium

APRIL 25, 2019

Most enterprises in the 21st century regard data as an incredibly valuable asset – Insurance is no exception - to know your customers better, know your market better, operate more efficiently and other business benefits. In data-driven organizations, data is flowing.

Insurance 150

Insurance Risk IoT Cost-Benefit 150

AWS Big Data

MARCH 3, 2023

Building data lakes from continuously changing transactional data of databases and keeping data lakes up to date is a complex task and can be an operational challenge. You can then apply transformations and store data in Delta format for managing inserts, updates, and deletes.

Data Lake 98

Data Lake Dashboards Metrics Metadata 98