AWS Big Data

NOVEMBER 9, 2023

In this post, we delve into a case study for a retail use case, exploring how the Data Build Tool (dbt) was used effectively within an AWS environment to build a high-performing, efficient, and modern data platform. It does this by helping teams handle the T in ETL (extract, transform, and load) processes.

Data Warehouse 126

Data Warehouse Testing Data Quality Reporting 126

AWS Big Data

AUGUST 1, 2024

However, you might face significant challenges when planning for a large-scale data warehouse migration. The following diagram illustrates a scalable migration pattern for extract, transform, and load (ETL) scenario. The success criteria are the key performance indicators (KPIs) for each component of the data workflow.

Data Warehouse 114

Data Warehouse KPI Optimization Cost-Benefit 114

Cloudera

JUNE 30, 2022

These tools empower analysts and data scientists to easily collaborate on the same data, with their choice of tools and analytic engines. No more lock-in, unnecessary data transformations, or data movement across tools and clouds just to extract insights out of the data.

Data Lake 88

Data Lake Data Warehouse Data Architecture Metadata 88

CIO Business Intelligence

NOVEMBER 7, 2023

Given the importance of sharing information among diverse disciplines in the era of digital transformation, this concept is arguably as important as ever. The aim is to normalize, aggregate, and eventually make available to analysts across the organization data that originates in various pockets of the enterprise.

Data Warehouse 142

Data Warehouse Digital Transformation Data-driven Reporting 142

AWS Big Data

NOVEMBER 13, 2023

Consequently, there was a fivefold rise in data integrations and a fivefold increase in ad hoc queries submitted to the Redshift cluster. Also, over time the number of BI dashboards (both scheduled and live) increased, which contributed to more queries being submitted to the Redshift cluster.

Data Warehouse 110

Data Warehouse Analytics Data Lake Data Science 110

Cloudera

NOVEMBER 29, 2021

Data transforms businesses. That’s where the data lifecycle comes into play. Managing data and its flow, from the edge to the cloud, is one of the most important tasks in the process of gaining data intelligence. . The company needed a modern data architecture to manage the growing traffic effectively. .

Data Lake 77

Data Lake Data Warehouse Data Architecture Reporting 77

AWS Big Data

SEPTEMBER 5, 2024

If storing operational data in a data warehouse is a requirement, synchronization of tables between operational data stores and Amazon Redshift tables is supported. In scenarios where data transformation is required, you can use Redshift stored procedures to modify data in Redshift tables.

IoT 110

IoT Data Warehouse Cost-Benefit Reporting 110

AWS Big Data

MARCH 3, 2023

Data ingestion – Steps 1 and 2 use AWS DMS, which connects to the source database and moves full and incremental data (CDC) to Amazon S3 in Parquet format. Data transformation – Steps 3 and 4 represent an EMR Serverless Spark application (Amazon EMR 6.9 Monjumi Sarma is a Data Lab Solutions Architect at AWS.

Data Lake 98

Data Lake Dashboards Metrics Metadata 98

TDAN

DECEMBER 3, 2019

Data is the key to unlocking insight— the secret sauce that will help you get predictive, the fuel for business intelligence. The transformative potential in AI? It relies on data. The thing that powers your CRM, your monthly report, your Tableau dashboard. The good news is that data has never […].

Business Intelligence 72

Business Intelligence Dashboards Reporting Data Architecture 72

FineReport

MARCH 31, 2024

Furthermore, these tools boast customization options, allowing users to tailor data sources to address areas critical to their business success, thereby generating actionable insights and customizable reports. Best BI Tools for Data Analysts 3.1 Why is Choosing the Best BI Tools Important?

Dashboards 52

Dashboards Visualization Data mining Data-driven 52

AWS Big Data

APRIL 5, 2023

Showpad built new customer-facing embedded dashboards within Showpad eOSTM and migrated its legacy dashboards to Amazon QuickSight , a unified BI service providing modern interactive dashboards, natural language querying, paginated reports, machine learning (ML) insights, and embedded analytics at scale.

Dashboards 52

Dashboards Reporting Cost-Benefit Visualization 52

AWS Big Data

FEBRUARY 6, 2023

The results of the log analyzer reveal Hadoop workload insights with various views and metrics of the Hadoop applications shown in Amazon QuickSight dashboards, which leads to the design of a future EMR cluster. The QuickSight timeline dashboard shows the peak time job runs because of the daily batch job.

Cost-Benefit 52

Cost-Benefit Data Lake Dashboards Big Data 52

Birst BI

FEBRUARY 25, 2020

Customers such as Crossmark , DJO Global and others use Birst with Snowflake to deliver the ultimate modern data architecture. Data never leaves Snowflake with Birst’s ability to support the reporting and self-service needs of both centralized IT and decentralized LOB teams.

Data Warehouse 52

Data Warehouse Cost-Benefit Data Architecture Enterprise 52

AWS Big Data

NOVEMBER 16, 2023

Data Vault 2.0 allows for the following: Agile data warehouse development Parallel data ingestion A scalable approach to handle multiple data sources even on the same entity A high level of automation Historization Full lineage support However, Data Vault 2.0

Enterprise 110

Enterprise Data Warehouse Data Lake Optimization 110

CIO Business Intelligence

AUGUST 7, 2024

The company started its New Analytics Era initiative by migrating its data from outdated SQL servers to a modern AWS data lake. It then built a cutting-edge cloud-based analytics platform, designed with an innovative data architecture. It also crafted multiple machine learning and AI models to tackle business challenges.

IT 119

IT Insurance Cost-Benefit Testing 119

Jet Global

MAY 1, 2023

Their dashboards were visually stunning. In turn, end users were thrilled with the bells and whistles of charts, graphs, and dashboards. As rich, data-driven user experiences are increasingly intertwined with our daily lives, end users are demanding new standards for how they interact with their business data.

Analytics 52

Analytics Cost-Benefit Visualization Dashboards 52

Jet Global

OCTOBER 1, 2024

Trino allows users to run ad hoc queries across massive datasets, making real-time decision-making a reality without needing extensive data transformations. This is particularly valuable for teams that require instant answers from their data. Data Lake Analytics: Trino doesn’t just stop at databases.

Dashboards 52

Dashboards Data Lake Reporting Cost-Benefit 52

AWS Big Data

APRIL 29, 2025

While enabling organization-wide efficiency, the team also applied these principles to the data architecture, making sure that CLEA itself operates frugally. After evaluating various tools, we built a serverless data transformation pipeline using Amazon Athena and dbt.

Data Transformation 54

Data Transformation Cost-Benefit Testing Data Lake 54

AWS Big Data

MAY 12, 2025

Streaming pipelines used Spark Streaming to ingest real-time data from Kafka, writing raw datasets to an Amazon Simple Storage Service (Amazon S3) data lake while simultaneously loading them into BigQuery and Google Cloud Storage to build logical data layers. The following figure is an example of a lineage diagram from DataHub.

Metrics 80

Metrics Cost-Benefit Metadata Data Lake 80

AWS Big Data

APRIL 28, 2025

Technical recap The AWS Glue Data Catalog served as the primary source of truth for schema and table updates, with Amazon EventBridge capturing Data Catalog events to trigger synchronization workflows. AWS Lambda parsed event metadata and managed schema synchronization, while Apache Kafka buffered events for real-time processing.

Data Lake 68

Data Lake Metadata Cost-Benefit Snapshot 68