article thumbnail

Achieve data resilience using Amazon OpenSearch Service disaster recovery with snapshot and restore

AWS Big Data

This post focuses on introducing an active-passive approach using a snapshot and restore strategy. Snapshot and restore in OpenSearch Service The snapshot and restore strategy in OpenSearch Service involves creating point-in-time backups, known as snapshots , of your OpenSearch domain.

Snapshot 107
article thumbnail

Build a high-performance quant research platform with Apache Iceberg

AWS Big Data

Iceberg provides time travel and snapshotting capabilities out of the box to manage lookahead bias that could be embedded in the data (such as delayed data delivery). Icebergs time travel capability is driven by a concept called snapshots , which are recorded in metadata files. groupBy("exchange_code", "instrument").count().orderBy("count",

Metadata 111
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Chart Snapshot: Barcode Plot

The Data Visualisation Catalogue

— VizWiz ‘Avengers’ characters’ appearances over time How the ‘Avengers’ Line-up Has Changed Over the Years — Wall Street Journal Multiple Income Households Flowingdata / Nathan Yau The Corruption Perceptions Index 2023 Week 35 | Power BI: Create a Faceted Instance Chart — Workout Wednesday / Meagan Longoria The post Chart (..)

article thumbnail

Amazon OpenSearch Service H1 2023 in review

AWS Big Data

Since its release in January 2021, the OpenSearch project has released 14 versions through June 2023. In this post, we provide a review of all the exciting features releases in OpenSearch Service in the first half of 2023. In July 2023, we previewed support for a third collection type: vector search.

article thumbnail

Increase flexibility and enable a cyber-resilient IT infrastructure

CIO Business Intelligence

For example, 59% of organizations were hit by ransomware in 2023 and 70% of them suffered data encryption. 1 Sophos State of Ransomware 2024 2 Forrester Opportunity Snapshot: Organizations Are Missing Critical Ransomware Recovery Capabilities, July 2024 About the author: Belu de Arbelaiz is the Sr.

IT 99
article thumbnail

Run Apache XTable in AWS Lambda for background conversion of open table formats

AWS Big Data

Originally open sourced in November 2023 under the name OneTable, with contributions from amongst others OneHouse , it was licensed under Apache 2.0. Querying all snapshots, we can see that we created three snapshots with overwrites after the initial one.

Metadata 103
article thumbnail

Use Amazon Athena with Spark SQL for your open-source transactional table formats

AWS Big Data

These formats enable ACID (atomicity, consistency, isolation, durability) transactions, upserts, and deletes, and advanced features such as time travel and snapshots that were previously only available in data warehouses. It will never remove files that are still required by a non-expired snapshot.

Snapshot 123