AWS Big Data

MARCH 13, 2025

Governance features including fine-grained access control are built into SageMaker Unified Studio using Amazon SageMaker Catalog to help you meet enterprise security requirements across your entire data estate.

Analytics 96

Analytics Data Lake Data Warehouse Data-driven 96

AWS Big Data

APRIL 3, 2024

Businesses are constantly evolving, and data leaders are challenged every day to meet new requirements. For many enterprises and large organizations, it is not feasible to have one processing engine or tool to deal with the various business requirements. This post is co-written with Andries Engelbrecht and Scott Teal from Snowflake.

Data Lake 112

Data Lake Snapshot Metadata Data Architecture 112

AWS Big Data

JUNE 10, 2024

Enterprises and organizations across the globe want to harness the power of data to make better decisions by putting data at the center of every decision-making process. This post is co-written with Amit Gilad, Alex Dickman and Itay Takersman from Cloudinary.

Data Lake 125

Data Lake Metadata Snapshot Analytics 125

AWS Big Data

SEPTEMBER 10, 2024

We often see requests from customers who have started their data journey by building data lakes on Microsoft Azure, to extend access to the data to AWS services. In such scenarios, data engineers face challenges in connecting and extracting data from storage containers on Microsoft Azure.

Data Lake 110

Data Lake Metadata Management Software 110

AWS Big Data

DECEMBER 4, 2024

Amazon SageMaker Lakehouse , now generally available, unifies all your data across Amazon Simple Storage Service (Amazon S3) data lakes and Amazon Redshift data warehouses, helping you build powerful analytics and AI/ML applications on a single copy of data. The tools to transform your business are here.

Data Analytics 92

Data Analytics Analytics Data Lake Data Quality 92

CIO Business Intelligence

APRIL 25, 2024

For instance, a Data Cloud-triggered flow could update an account manager in Slack when shipments in an external data lake are marked as delayed. Sharing Customer 360 insights back without data replication. CRM Systems, Data Management, Salesforce.com

Data Integration 119

Data Integration Data Lake Data Warehouse Metadata 119

IBM Big Data Hub

JUNE 26, 2020

Organizations need to collect, organize, and analyze their data across multi-cloud, hybrid cloud, and data lakes. In turn, enterprises are increasingly looking for machine-learning-powered integration tools to synchronize data for analytics, improve employee productivity, and prepare data for analytics.

Data Integration 107

Data Integration Data Lake Machine Learning Enterprise 107

DataKitchen

JANUARY 25, 2022

There’s no shortage of consultants who will promise to manage the end-to-end lifecycle of data from integration to transformation to visualization. . The challenge is that data engineering and analytics are incredibly complex. The data requirements of a thriving business are never complete.

Consulting 244

Consulting Testing Data Lake Data Quality 244

AWS Big Data

JUNE 10, 2024

Use cases for Hive metastore federation for Amazon EMR Hive metastore federation for Amazon EMR is applicable to the following use cases: Governance of Amazon EMR-based data lakes – Producers generate data within their AWS accounts using an Amazon EMR-based data lake supported by EMRFS on Amazon Simple Storage Service (Amazon S3)and HBase.

Data Lake 103

Data Lake Metadata Data Warehouse Data Processing 103

CIO Business Intelligence

OCTOBER 9, 2024

The rise of generative AI (GenAI) felt like a watershed moment for enterprises looking to drive exponential growth with its transformative potential. As the technology subsists on data, customer trust and their confidential information are at stake—and enterprises cannot afford to overlook its pitfalls.

Strategy 143

Strategy Modeling Management Data Lake 143

Cloudera

JUNE 2, 2022

Since 2015, the Cloudera DataFlow team has been helping the largest enterprise organizations in the world adopt Apache NiFi as their enterprise standard data movement tool. That’s why we love that Cloudera uses NiFi and the way it integrates between all systems. What is the modern data stack?

Enterprise 104

Enterprise Data Lake Data Collection Data-driven 104

CIO Business Intelligence

JULY 13, 2022

Since 2015, the Cloudera DataFlow team has been helping the largest enterprise organizations in the world adopt Apache NiFi as their enterprise standard data movement tool. That’s why we love that Cloudera uses NiFi and the way it integrates between all systems. What is the modern data stack?

Enterprise 88

Enterprise Data Lake Data Collection Data-driven 88

Data Virtualization

MARCH 16, 2023

Reading Time: 6 minutes Data lake, by combining the flexibility of object storage with the scalability and agility of cloud platforms, are becoming an increasingly popular choice as an enterprise data repository. Whether you are on Amazon Web Services (AWS) and leverage AWS S3.

Data Lake 52

Data Lake Strategy Data Integration Enterprise 52

Data Virtualization

MARCH 16, 2023

Reading Time: 6 minutes Data lake, by combining the flexibility of object storage with the scalability and agility of cloud platforms, are becoming an increasingly popular choice as an enterprise data repository. Whether you are on Amazon Web Services (AWS) and leverage AWS S3.

Data Lake 52

Data Lake Strategy Data Integration Enterprise 52

AWS Big Data

NOVEMBER 11, 2024

Samir works directly with enterprise customers to design and build customized solutions catered to their data analytics and cybersecurity needs. Sesha Sanjana Mylavarapu is an Associate Data Lake Consultant at AWS Professional Services.

Snapshot 104

Snapshot Strategy Dashboards Data Lake 104

David Menninger's Analyst Perspectives

OCTOBER 24, 2024

Data fabric refers to technology products that can be used to integrate, manage and govern data across distributed environments, supporting the cultural and organizational data ownership and access goals of data mesh.

Management 130

Management Data-driven Data Governance Data Lake 130

AWS Big Data

JANUARY 9, 2025

These features allow efficient data corrections, gap-filling in time series, and historical data updates without disrupting ongoing analyses or compromising data integrity. Unlike direct Amazon S3 access, Iceberg supports these operations on petabyte-scale data lakes without requiring complex custom code.

Metadata 110

Metadata Snapshot Cost-Benefit Optimization 110

AWS Big Data

JUNE 25, 2024

This post is co-authored by Vijay Gopalakrishnan, Director of Product, Salesforce Data Cloud. In today’s data-driven business landscape, organizations collect a wealth of data across various touch points and unify it in a central data warehouse or a data lake to deliver business insights.

Data Lake 117

Data Lake Cost-Benefit Data-driven Data Warehouse 117

AWS Big Data

AUGUST 22, 2024

The infrastructure provides an analytics experience to hundreds of in-house analysts, data scientists, and student-facing frontend specialists. The data engineering team is on a mission to modernize its data integration platform to be agile, adaptive, and straightforward to use.

Data Warehouse 115

Data Warehouse Data Lake Data Integration Management 115

Smart Data Collective

OCTOBER 17, 2022

A point of data entry in a given pipeline. Examples of an origin include storage systems like data lakes, data warehouses and data sources that include IoT devices, transaction processing applications, APIs or social media. The final point to which the data has to be eventually transferred is a destination.

Data Warehouse 145

Data Warehouse Data Lake Visualization Big Data 145

AWS Big Data

OCTOBER 20, 2023

Today, we are pleased to announce new AWS Glue connectors for Azure Blob Storage and Azure Data Lake Storage that allow you to move data bi-directionally between Azure Blob Storage, Azure Data Lake Storage, and Amazon Simple Storage Service (Amazon S3). option("header","true").load("wasbs://yourblob@youraccountname.blob.core.windows.net/loadingtest-input/100mb")

Data Lake 104

Data Lake Big Data Consulting Data Warehouse 104

AWS Big Data

JANUARY 12, 2024

Ingestion: Data lake batch, micro-batch, and streaming Many organizations land their source data into their data lake in various ways, including batch, micro-batch, and streaming jobs. Amazon AppFlow can be used to transfer data from different SaaS applications to a data lake.

Data Lake 114

Data Lake Cost-Benefit Visualization Structured Data 114

CIO Business Intelligence

APRIL 25, 2022

The data lakehouse is a relatively new data architecture concept, first championed by Cloudera, which offers both storage and analytics capabilities as part of the same solution, in contrast to the concepts for data lake and data warehouse which, respectively, store data in native format, and structured data, often in SQL format.

Recreation/Entertainment 119

Recreation/Entertainment Data Lake Data Warehouse Unstructured Data 119

erwin

OCTOBER 8, 2020

By using automated and repeatable capabilities, you can quickly and safely migrate data to the cloud and govern it along the way. But transforming and migrating enterprise data to the cloud is only half the story – once there, it needs to be governed for completeness and compliance. GDPR, CCPA, HIPAA, SOX, PIC DSS).

Data Governance 145

Data Governance Metadata Testing Data Lake 145

AWS Big Data

FEBRUARY 29, 2024

Data is your generative AI differentiator, and a successful generative AI implementation depends on a robust data strategy incorporating a comprehensive data governance approach. Data governance is a critical building block across all these approaches, and we see two emerging areas of focus.

Data Governance 116

Data Governance Unstructured Data Metadata Data Lake 116

CIO Business Intelligence

AUGUST 9, 2024

The original proof of concept was to have one data repository ingesting data from 11 sources, including flat files and data stored via APIs on premises and in the cloud, Pruitt says. There are a lot of variables that determine what should go into the data lake and what will probably stay on premise,” Pruitt says.

Data Transformation 126

Data Transformation Machine Learning Data Lake Dashboards 126

AWS Big Data

DECEMBER 13, 2023

In addition to using native managed AWS services that BMS didn’t need to worry about upgrading, BMS was looking to offer an ETL service to non-technical business users that could visually compose data transformation workflows and seamlessly run them on the AWS Glue Apache Spark-based serverless data integration engine.

Metadata 107

Metadata Data Lake Visualization Data Quality 107

AWS Big Data

DECEMBER 15, 2023

Even after identification, it’s cumbersome to implement redaction, masking, or encryption of sensitive data at scale. In this post, we provide an automated solution to detect PII data in Amazon Redshift using AWS Glue. For our solution, we use Amazon Redshift to store the data.

Data Lake 118

Data Lake Data Warehouse Big Data Structured Data 118

AWS Big Data

FEBRUARY 26, 2025

Analytics remained one of the key focus areas this year, with significant updates and innovations aimed at helping businesses harness their data more efficiently and accelerate insights. From enhancing data lakes to empowering AI-driven analytics, AWS unveiled new tools and services that are set to shape the future of data and analytics.

Analytics 59

Analytics Data Lake Metadata Data Warehouse 59

AWS Big Data

NOVEMBER 10, 2023

The data sourcing problem To ensure the reliability of PySpark data pipelines, it’s essential to have consistent record-level data from both dimensional and fact tables stored in the Enterprise Data Warehouse (EDW). These tables are then joined with tables from the Enterprise Data Lake (EDL) at runtime.

Data Processing 109

Data Processing Data Lake Data Warehouse Optimization 109

AWS Big Data

NOVEMBER 29, 2023

In 2013, Amazon Web Services revolutionized the data warehousing industry by launching Amazon Redshift , the first fully-managed, petabyte-scale, enterprise-grade cloud data warehouse. Amazon Redshift made it simple and cost-effective to efficiently analyze large volumes of data using existing business intelligence tools.

Data Warehouse 120

Data Warehouse Analytics Data Lake Machine Learning 120

CIO Business Intelligence

APRIL 1, 2024

GenAI requires high-quality data. Ensure that data is cleansed, consistent, and centrally stored, ideally in a data lake. Data preparation, including anonymizing, labeling, and normalizing data across sources, is key. 2024 2 The State of Generative AI in the Enterprise, Deloitte, Jan.

Uncertainty 142

Uncertainty Data Lake Consulting Risk 142

CIO Business Intelligence

APRIL 29, 2022

This would be straightforward task were it not for the fact that, during the digital-era, there has been an explosion of data – collected and stored everywhere – much of it poorly governed, ill-understood, and irrelevant. Further, data management activities don’t end once the AI model has been developed. Addressing the Challenge.

Data Governance 145

Data Governance IT Risk Data Lake 145

AWS Big Data

MAY 3, 2023

In today’s data-driven business environment, organizations face the challenge of efficiently preparing and transforming large amounts of data for analytics and data science purposes. Businesses need to build data warehouses and data lakes based on operational data.

Data Lake 99

Data Lake Data Warehouse Data-driven Optimization 99

AWS Big Data

NOVEMBER 22, 2024

Keerthi Chadalavada is a Senior Software Development Engineer at AWS Glue, focusing on combining generative AI and data integration technologies to design and build comprehensive solutions for customers’ data and analytics needs. In his spare time, he enjoys cycling with his new road bike.

Cost-Benefit 101

Cost-Benefit Data-driven Software Testing 101

AWS Big Data

JANUARY 6, 2025

Amazon Redshift is a fast, scalable, and fully managed cloud data warehouse that allows you to process and run your complex SQL analytics workloads on structured and semi-structured data. It also helps you securely access your data in operational databases, data lakes, or third-party datasets with minimal movement or copying of data.

Analytics 97

Analytics Data Warehouse Metrics Big Data 97

AWS Big Data

MARCH 20, 2023

In the first post of this series , we described how AWS Glue for Apache Spark works with Apache Hudi, Linux Foundation Delta Lake, and Apache Iceberg datasets tables using the native support of those data lake formats. Even without prior experience using Hudi, Delta Lake or Iceberg, you can easily achieve typical use cases.

Visualization 93

Visualization Data Lake Snapshot Big Data 93