AWS Big Data

APRIL 27, 2023

Amazon Athena supports the MERGE command on Apache Iceberg tables, which allows you to perform inserts, updates, and deletes in your data lake at scale using familiar SQL statements that are compliant with ACID (Atomic, Consistent, Isolated, Durable). Create a table to point to the CDC data. Upload 20220922-184314489.csv

Data Lake 97

Data Lake Snapshot Optimization Data Transformation 97

AWS Big Data

OCTOBER 23, 2024

result = client.trigger_run( dag_id="any_bash_command", conf={ "command": "echo 'Hello from external trigger!'" Let’s expand the use case to run your data pipeline and perform extract, transform, and load (ETL) jobs when a new file lands in an Amazon Simple Storage Service (Amazon S3) bucket in your data lake.

Interactive 115

Interactive Testing Data-driven Data Lake 115

AWS Big Data

NOVEMBER 29, 2023

dbt is an open source, SQL-first templating engine that allows you to write repeatable and extensible data transforms in Python and SQL. dbt is predominantly used by data warehouses (such as Amazon Redshift ) customers who are looking to keep their data transform logic separate from storage and engine.

Data Lake 122

Data Lake Management Metrics Data Warehouse 122

AWS Big Data

AUGUST 9, 2024

These processes retrieve data from around 90 different data sources, resulting in updating roughly 2,000 tables in the data warehouse and 3,000 external tables in Parquet format, accessed through Amazon Redshift Spectrum and a data lake on Amazon Simple Storage Service (Amazon S3). We started with 115 dc2.large

Data Lake 111

Data Lake Analytics Data Warehouse Data-driven 111

Alation

JULY 18, 2022

Are your data users overwhelmed by silos and frustrated by untrusted data? Tell them to grab a catalog … and go jump in a lake. That was the message — delivered a little more elegantly than that — at Databricks’ Data+AI Summit 2022. Automatically tracking data lineage across queries executed in any language.

ROI 52

ROI Metadata Data Lake Digital Transformation 52

AWS Big Data

DECEMBER 21, 2023

To bring their customers the best deals and user experience, smava follows the modern data architecture principles with a data lake as a scalable, durable data store and purpose-built data stores for analytical processing and data consumption.

Data Lake 115

Data Lake Data Warehouse Data-driven B2B 115

Cloudera

JUNE 30, 2022

These tools empower analysts and data scientists to easily collaborate on the same data, with their choice of tools and analytic engines. No more lock-in, unnecessary data transformations, or data movement across tools and clouds just to extract insights out of the data.

Data Lake 88

Data Lake Data Warehouse Data Architecture Metadata 88