KDnuggets

OCTOBER 30, 2023

A comparative overview of data warehouses, data lakes, and data marts to help you make informed decisions on data storage solutions for your data architecture.

Data Lake 140

Data Lake Data Warehouse Data Architecture 140

AWS Big Data

OCTOBER 3, 2023

A data lake is a centralized repository that you can use to store all your structured and unstructured data at any scale. You can store your data as-is, without having to first structure the data and then run different types of analytics for better business insights.

Data Lake 122

Data Lake Metadata Snapshot Recreation/Entertainment 122

AWS Big Data

MARCH 18, 2024

Many organizations operate data lakes spanning multiple cloud data stores. In these cases, you may want an integrated query layer to seamlessly run analytical queries across these diverse cloud stores and streamline your data analytics processes. Upload the data file to the S3 bucket created by the CloudFormation stack.

Data Lake 131

Data Lake Analytics Cost-Benefit Management 131

AWS Big Data

OCTOBER 19, 2023

Data lakes and data warehouses are two of the most important data storage and management technologies in a modern data architecture. Data lakes store all of an organization’s data, regardless of its format or structure.

Data Lake 138

Data Lake Data Warehouse Visualization Snapshot 138

AWS Big Data

MARCH 2, 2023

Apache Iceberg is an open table format for very large analytic datasets, which captures metadata information on the state of datasets as they evolve and change over time. Iceberg has become very popular for its support for ACID transactions in data lakes and features like schema and partition evolution, time travel, and rollback.

Data Lake 137

Data Lake Data Processing Metadata Snapshot 137

DataKitchen

DECEMBER 9, 2024

They opted for Snowflake, a cloud-native data platform ideal for SQL-based analysis. The team landed the data in a Data Lake implemented with cloud storage buckets and then loaded into Snowflake, enabling fast access and smooth integrations with analytical tools.

Data Quality 161

Data Quality Data Lake Testing Statistics 161

AWS Big Data

OCTOBER 1, 2024

Amazon Redshift enables you to efficiently query and retrieve structured and semi-structured data from open format files in Amazon S3 data lake without having to load the data into Amazon Redshift tables. Amazon Redshift extends SQL capabilities to your data lake, enabling you to run analytical queries.

Data Lake 116

Data Lake Statistics Broadcasting Optimization 116

AWS Big Data

SEPTEMBER 26, 2024

Organizations are collecting and storing vast amounts of structured and unstructured data like reports, whitepapers, and research documents. By consolidating this information, analysts can discover and integrate data from across the organization, creating valuable data products based on a unified dataset.

Data Lake 115

Data Lake Cost-Benefit Unstructured Data Modeling 115

AWS Big Data

AUGUST 15, 2024

Unlocking the true value of data often gets impeded by siloed information. Traditional data management—wherein each business unit ingests raw data in separate data lakes or warehouses—hinders visibility and cross-functional analysis.

Data Lake 122

Data Lake Data Warehouse Data Governance Publishing 122

AWS Big Data

NOVEMBER 26, 2024

Initially, data warehouses were the go-to solution for structured data and analytical workloads but were limited by proprietary storage formats and their inability to handle unstructured data. Eventually, transactional data lakes emerged to add transactional consistency and performance of a data warehouse to the data lake.

Metadata 105

Metadata Data Lake Snapshot Data Warehouse 105

AWS Big Data

JUNE 9, 2023

A modern data architecture enables companies to ingest virtually any type of data through automated pipelines into a data lake, which provides highly durable and cost-effective object storage at petabyte or exabyte scale.

Data Lake 130

Data Lake Metadata Statistics Optimization 130

AWS Big Data

DECEMBER 17, 2024

Today, Amazon Redshift is used by customers across all industries for a variety of use cases, including data warehouse migration and modernization, near real-time analytics, self-service analytics, data lake analytics, machine learning (ML), and data monetization.

Data Lake 98

Data Lake Data Warehouse Data-driven Optimization 98

AWS Big Data

JULY 18, 2024

Over the years, organizations have invested in creating purpose-built, cloud-based data lakes that are siloed from one another. A major challenge is enabling cross-organization discovery and access to data across these multiple data lakes, each built on different technology stacks.

Data Lake 130

Data Lake Publishing Metadata Data-driven 130

AWS Big Data

AUGUST 31, 2023

Amazon Redshift enables you to directly access data stored in Amazon Simple Storage Service (Amazon S3) using SQL queries and join data across your data warehouse and data lake. With Amazon Redshift, you can query the data in your S3 data lake using a central AWS Glue metastore from your Redshift data warehouse.

Data Lake 119

Data Lake Data Warehouse Metadata Data Architecture 119

AWS Big Data

JUNE 10, 2024

For detailed information on managing your Apache Hive metastore using Lake Formation permissions, refer to Query your Apache Hive metastore with AWS Lake Formation permissions. In this post, we present a methodology for deploying a data mesh consisting of multiple Hive data warehouses across EMR clusters.

Data Lake 117

Data Lake Metadata Data Warehouse Data Processing 117

AWS Big Data

SEPTEMBER 10, 2024

We often see requests from customers who have started their data journey by building data lakes on Microsoft Azure, to extend access to the data to AWS services. In such scenarios, data engineers face challenges in connecting and extracting data from storage containers on Microsoft Azure.

Data Lake 111

Data Lake Metadata Management Software 111

AWS Big Data

OCTOBER 9, 2023

Data lakes have been gaining popularity for storing vast amounts of data from diverse sources in a scalable and cost-effective way. As the number of data consumers grows, data lake administrators often need to implement fine-grained access controls for different user profiles.

Data Lake 131

Data Lake Testing Big Data Management 131

AWS Big Data

DECEMBER 4, 2024

While traditional extract, transform, and load (ETL) processes have long been a staple of data integration due to its flexibility, for common use cases such as replication and ingestion, they often prove time-consuming, complex, and less adaptable to the fast-changing demands of modern data architectures.

Data Integration 115

Data Integration Data Lake Statistics Data-driven 115

Dataiku

SEPTEMBER 20, 2024

In this series of blog posts, we have been exploring the problem of augmenting a table using information contained in a data lake (a large collection of datasets).

Data Lake 98

Data Lake 98

AWS Big Data

JULY 31, 2024

In the current industry landscape, data lakes have become a cornerstone of modern data architecture, serving as repositories for vast amounts of structured and unstructured data. Maintaining data consistency and integrity across distributed data lakes is crucial for decision-making and analytics.

Data Lake 116

Data Lake Marketing Data Processing Management 116

CIO Business Intelligence

DECEMBER 20, 2024

Chief among these is United ChatGPT for secure employee experimental use and an external-facing LLM that better informs customers about flight delays, known as Every Flight Has a Story, that has already boosted customer satisfaction by 6%, Birnbaum notes. Historically United storytellers had to manually edit templates, which took time.

IT 131

IT Unstructured Data Experimentation Data Lake 131

AWS Big Data

DECEMBER 15, 2023

With the exponential growth of data, companies are handling huge volumes and a wide variety of data including personally identifiable information (PII). PII is a legal term pertaining to information that can identify, contact, or locate a single person. For our solution, we use Amazon Redshift to store the data.

Data Lake 119

Data Lake Data Warehouse Big Data Structured Data 119

CIO Business Intelligence

FEBRUARY 27, 2025

According to a study from Rocket Software and Foundry , 76% of IT decision-makers say challenges around accessing mainframe data and contextual metadata are a barrier to mainframe data usage, while 64% view integrating mainframe data with cloud data sources as the primary challenge.

Metadata 98

Metadata Data Lake Cost-Benefit Forecasting 98

AWS Big Data

JANUARY 15, 2025

For container terminal operators, data-driven decision-making and efficient data sharing are vital to optimizing operations and boosting supply chain efficiency. In addition to real-time analytics and visualization, the data needs to be shared for long-term data analytics and machine learning applications.

IoT 111

IoT Machine Learning Metadata Data-driven 111

David Menninger's Analyst Perspectives

JANUARY 21, 2025

The company added support for vector search in 2023, which can help organizations improve trust in the output of GenAI by providing access to factually accurate and up-to-date information.

Data Lake 130

Data Lake IoT Cost-Benefit Enterprise 130

AWS Big Data

APRIL 24, 2025

Prerequisites To follow the steps in this post, you must complete the following prerequisites: AWS account with access to the following AWS services: Amazon S3 AWS Lake Formation and AWS Glue Data Catalog Amazon Redshift Amazon Athena Amazon EMR AWS Identity and Access Management (IAM) Set up an admin user for Ava. Choose Grant.

Sales 93

Sales Data Lake Management Data-driven 93

DataKitchen

AUGUST 19, 2021

Figure 3 shows an example processing architecture with data flowing in from internal and external sources. Each data source is updated on its own schedule, for example, daily, weekly or monthly. The data scientists and analysts have what they need to build analytics for the user. The new Recipes run, and BOOM! Conclusion.

Data Warehouse 246

Data Warehouse Data Lake Manufacturing Testing 246

David Menninger's Analyst Perspectives

SEPTEMBER 28, 2021

Its data intelligence cloud platform can automatically classify data from various sources such as online transaction processing databases, master repositories and Excel files without moving the data, so the information assets stay protected.

Data Governance 349

Data Governance Metadata Software Management 349

CIO Business Intelligence

FEBRUARY 20, 2025

This allows for the extraction and integration of data into AI models without overhauling entire platforms, Erolin says. CIOs should also use data lakes to aggregate information from multiple sources, he adds. AI models can then access the data they need without direct reliance on outdated apps.

Insurance 108

Insurance Cost-Benefit Unstructured Data Data Lake 108

AWS Big Data

JANUARY 26, 2023

AWS Glue provides an extensible architecture that enables users with different data processing use cases. A common use case is building data lakes on Amazon Simple Storage Service (Amazon S3) using AWS Glue extract, transform, and load (ETL) jobs. Refer to AWS Glue job parameters for more details.

Data Lake 136

Data Lake Big Data Software Interactive 136

AWS Big Data

OCTOBER 14, 2024

Enterprise data is brought into data lakes and data warehouses to carry out analytical, reporting, and data science use cases using AWS analytical services like Amazon Athena , Amazon Redshift , Amazon EMR , and so on. Can it also help write SQL queries? The answer is yes.

Metadata 105

Metadata Data Lake Modeling Data Warehouse 105

AWS Big Data

DECEMBER 12, 2024

In the context of comprehensive data governance, Amazon DataZone offers organization-wide data lineage visualization using Amazon Web Services (AWS) services, while dbt provides project-level lineage through model analysis and supports cross-project integration between data lakes and warehouses. hasLabel('lineage_node').outE('lineage_edge').inV().hasLabel('lineage_node').path().by(elementMap())

Snapshot 98

Snapshot Recreation/Entertainment Experimentation Data Lake 98

AWS Big Data

JULY 29, 2024

In the era of big data, data lakes have emerged as a cornerstone for storing vast amounts of raw data in its native format. They support structured, semi-structured, and unstructured data, offering a flexible and scalable environment for data ingestion from multiple sources.

Metadata 126

Metadata Snapshot Data Lake Metrics 126

AWS Big Data

JUNE 15, 2023

In today’s world, customers manage vast amounts of data in their Amazon Simple Storage Service (Amazon S3) data lakes, which requires convoluted data pipelines to continuously understand the changes in the data layout and make them available to consuming systems.

Data Lake 105

Data Lake Metadata Cost-Benefit Management 105

AWS Big Data

DECEMBER 4, 2024

Open table formats are emerging in the rapidly evolving domain of big data management, fundamentally altering the landscape of data storage and analysis. By providing a standardized framework for data representation, open table formats break down data silos, enhance data quality, and accelerate analytics at scale.

Snapshot 84

Snapshot Metadata Data Lake Optimization 84

AWS Big Data

OCTOBER 21, 2024

However, enterprises often encounter challenges with data silos, insufficient access controls, poor governance, and quality issues. Embracing data as a product is the key to address these challenges and foster a data-driven culture. They would like to get a cross-organization visibility of supply chain and inventory data.

Sales 116

Sales Data-driven Data Processing Key Performance Indicator 116

CIO Business Intelligence

SEPTEMBER 20, 2023

The data can also help us enrich our commodity products. How are you populating your data lake? We’ve decided to take a practical approach, led by Kyle Benning, who runs our data function. Then our analytics team, an IT group, makes sure we build the data lake in the right sequence.

Data Lake 135

Data Lake Data-driven Dashboards Risk 135