AWS Big Data

NOVEMBER 9, 2023

In this post, we delve into a case study for a retail use case, exploring how the Data Build Tool (dbt) was used effectively within an AWS environment to build a high-performing, efficient, and modern data platform. It does this by helping teams handle the T in ETL (extract, transform, and load) processes.

Data Warehouse 126

Data Warehouse Testing Data Quality Reporting 126

AWS Big Data

DECEMBER 21, 2023

For the downstream consumption by all departments across the organization, smava’s Data Platform team prepares curated data products following the extract, load, and transform (ELT) pattern. The data products from the Business Vault and Data Mart stages are now available for consumers.

Data Lake 115

Data Lake Data Warehouse Data-driven B2B 115

AWS Big Data

MARCH 13, 2024

You can also use the data transformation feature of Data Firehose to invoke a Lambda function to perform data transformation in batches. Query the data using Athena Athena is a serverless, interactive analytics service built to analyze unstructured, semi-structured, and structured data where it is hosted.

Analytics 122

Analytics IoT Metadata Internet of Things 122

AWS Big Data

NOVEMBER 13, 2023

Amazon Redshift is a fully managed data warehousing service that offers both provisioned and serverless options, making it more efficient to run and scale analytics without having to manage your data warehouse.

Data Warehouse 111

Data Warehouse Analytics Data Lake Data Science 111

FineReport

MARCH 7, 2024

In the realm of big data utilization , we often romanticize its profound impact, envisioning scenarios like precision-targeted advertising, streamlined social security management, and the intelligent evolution of the pharmaceutical sector. Why Big Data Analysis Report? Try FineReport Now 1.

Dashboards 52

Dashboards Reporting Advertising Statistics 52

IBM Big Data Hub

JANUARY 18, 2023

In another decade, the internet and mobile started the generate data of unforeseen volume, variety and velocity. It required a different data platform solution. Hence, Data Lake emerged, which handles unstructured and structured data with huge volume. Data lakehouse was created to solve these problems.

Data Lake 52

Data Lake Data Warehouse Data-driven Metadata 52

AWS Big Data

JULY 31, 2023

You can use AWS Glue Studio to create jobs that extract structured or semi-structured data from a data source, perform a transformation of that data, and save the result set in a data target. This concludes creating data sources on the AWS Glue job canvas. Under Transforms , choose SQL Query.

Sales 52

Sales Data Warehouse Visualization Testing 52

Jet Global

MAY 9, 2024

Data Extraction : The process of gathering data from disparate sources, each of which may have its own schema defining the structure and format of the data and making it available for processing. This can include tasks such as data ingestion, cleansing, filtering, aggregation, or standardization.

Data Lake 52

Data Lake Data Warehouse Business Intelligence Machine Learning 52

AWS Big Data

DECEMBER 17, 2024

As business and data volume grew over time, DeNA started to face the following challenges: Performance Data quality tests took days to weeks to complete because engineers hadnt designed the batch jobs to handle big data. The implementation required loading data into memory for processing.

Data Quality 90

Data Quality Testing Metrics Optimization 90

AWS Big Data

FEBRUARY 25, 2025

Based on the configuration file, the input data is fetched and technical validations are applied. If data mapping has been enabled within the data processing job, then the structured data is prepared based on the given schema.

Data Processing 56

Data Processing Machine Learning Data-driven Cost-Benefit 56

AWS Big Data

NOVEMBER 14, 2024

We use the built-in features of Data Firehose, including AWS Lambda for necessary data transformation and Amazon Simple Notification Service (Amazon SNS) for near real-time alerts. Each AWS account has one Data Catalog per AWS Region. Each Data Catalog is a highly scalable collection of tables organized into databases.

Data Lake 105

Data Lake Metadata Testing Data-driven 105