AWS Big Data

DECEMBER 4, 2024

They’re taking data they’ve historically used for analytics or business reporting and putting it to work in machine learning (ML) models and AI-powered applications. This innovation drives an important change: you’ll no longer have to copy or move data between data lake and data warehouses.

Data Analytics 101

Data Analytics Analytics Data Lake Data Quality 101

AWS Big Data

OCTOBER 29, 2024

The CDH is used to create, discover, and consume data products through a central metadata catalog, while enforcing permission policies and tightly integrating data engineering, analytics, and machine learning services to streamline the user journey from data to insight.

Data Lake 104

Data Lake Sales Metadata Machine Learning 104

AWS Big Data

DECEMBER 17, 2024

Today, Amazon Redshift is used by customers across all industries for a variety of use cases, including data warehouse migration and modernization, near real-time analytics, self-service analytics, data lake analytics, machine learning (ML), and data monetization.

Data Lake 95

Data Lake Data Warehouse Data-driven Optimization 95

AWS Big Data

MARCH 13, 2025

At AWS re:Invent 2024, we announced the next generation of Amazon SageMaker , the center for all your data, analytics, and AI. Governance features including fine-grained access control are built into SageMaker Unified Studio using Amazon SageMaker Catalog to help you meet enterprise security requirements across your entire data estate.

Analytics 87

Analytics Data Lake Data Warehouse Data-driven 87

CIO Business Intelligence

DECEMBER 20, 2024

Data architecture definition Data architecture describes the structure of an organizations logical and physical data assets, and data management resources, according to The Open Group Architecture Framework (TOGAF). An organizations data architecture is the purview of data architects. Curate the data.

Data Architecture 122

Data Architecture Management Consulting Internet of Things 122

AWS Big Data

NOVEMBER 26, 2024

Initially, data warehouses were the go-to solution for structured data and analytical workloads but were limited by proprietary storage formats and their inability to handle unstructured data. Eventually, transactional data lakes emerged to add transactional consistency and performance of a data warehouse to the data lake.

Metadata 105

Metadata Data Lake Snapshot Data Warehouse 105

O'Reilly on Data

OCTOBER 19, 2021

Much has been written about struggles of deploying machine learning projects to production. As with many burgeoning fields and disciplines, we don’t yet have a shared canonical infrastructure stack or best practices for developing and deploying data-intensive applications. However, the concept is quite abstract.

IT 364

IT Testing Experimentation Software 364

Cloudera

FEBRUARY 16, 2022

In this example, the Machine Learning (ML) model struggles to differentiate between a chihuahua and a muffin. Will the model correctly determine it is a muffin or get confused and think it is a chihuahua? The extent to which we can predict how the model will classify an image given a change input (e.g.

Machine Learning 85

Machine Learning Modeling Metadata Recreation/Entertainment 85

AWS Big Data

MARCH 10, 2023

Since the deluge of big data over a decade ago, many organizations have learned to build applications to process and analyze petabytes of data. Data lakes have served as a central repository to store structured and unstructured data at any scale and in various formats.

Data Lake 122

Data Lake Sales Data Warehouse Snapshot 122

AWS Big Data

JUNE 9, 2023

A modern data architecture enables companies to ingest virtually any type of data through automated pipelines into a data lake, which provides highly durable and cost-effective object storage at petabyte or exabyte scale.

Data Lake 130

Data Lake Metadata Statistics Optimization 130

Jet Global

SEPTEMBER 4, 2020

Consultants and developers familiar with the AX data model could query the database using any number of different tools, including a myriad of different report writers. Data entities are more secure and arguably easier to master than the relational database model, but one downside is there are lots of them! Data Lakes.

Data Lake 88

Data Lake OLAP Data Warehouse Unstructured Data 88

Cloudera

DECEMBER 8, 2020

Federated Learning is a paradigm in which machine learning models are trained on decentralized data. Instead of collecting data on a single server or data lake, it remains in place — on smartphones, industrial sensing equipment, and other edge devices — and models are trained on-device.

Machine Learning 85

Machine Learning Data Lake Reporting Data Collection 85

Jet Global

NOVEMBER 5, 2020

That stands for “bring your own database,” and it refers to a model in which core ERP data are replicated to a separate standalone database used exclusively for reporting. OLAP reporting based on a data warehouse model is a well-proven solution for companies with robust reporting requirements. Option 3: Azure Data Lakes.

Data Lake 98

Data Lake OLAP Data Warehouse Unstructured Data 98

AWS Big Data

JUNE 10, 2024

Use cases for Hive metastore federation for Amazon EMR Hive metastore federation for Amazon EMR is applicable to the following use cases: Governance of Amazon EMR-based data lakes – Producers generate data within their AWS accounts using an Amazon EMR-based data lake supported by EMRFS on Amazon Simple Storage Service (Amazon S3)and HBase.

Data Lake 112

Data Lake Metadata Data Warehouse Data Processing 112

Sisense

DECEMBER 23, 2019

Artificial Intelligence and machine learning are the future of every industry, especially data and analytics. Let’s talk about AI and machine learning (ML). AI and ML are the only ways to derive value from massive data lakes, cloud-native data warehouses, and other huge stores of information.

Data Lake 86

Data Lake Machine Learning Data Warehouse Digital Transformation 86

AWS Big Data

DECEMBER 9, 2024

Given the importance of data in the world today, organizations face the dual challenges of managing large-scale, continuously incoming data while vetting its quality and reliability. One of its key features is the ability to manage data using branches. The data is visualized using matplotlib for interactive data analysis.

Data Quality 123

Data Quality Publishing Snapshot Data Lake 123

AWS Big Data

JULY 20, 2023

With data becoming the driving force behind many industries today, having a modern data architecture is pivotal for organizations to be successful. In this post, we describe Orca’s journey building a transactional data lake using Amazon Simple Storage Service (Amazon S3), Apache Iceberg, and AWS Analytics.

Data Lake 98

Data Lake Analytics Snapshot Data Quality 98

Jen Stirrup

MAY 5, 2020

Azure allows you to protect your enterprise data assets, using Azure Active Directory and setting up your virtual network. Other technologies, such as Azure Data Factory, can help process large amounts of data around in the cloud. The data is also distributed. So, Azure Databricks connects to many different data sources.

Machine Learning 89

Machine Learning Data Science Data Lake Big Data 89

CIO Business Intelligence

APRIL 26, 2024

billion acquisition of data and analytics company Neustar in 2021, TransUnion has expanded into other services such as marketing, fraud detection and prevention, and robust analytical services. The multilayered data platform will enable TransUnion’s customers to perform deep analytics and build complex AI models.

Modeling 114

Modeling IT Machine Learning Data Governance 114

AWS Big Data

OCTOBER 14, 2024

Enterprise data is brought into data lakes and data warehouses to carry out analytical, reporting, and data science use cases using AWS analytical services like Amazon Athena , Amazon Redshift , Amazon EMR , and so on. foundation model (FM) in Amazon Bedrock as the LLM. Can it also help write SQL queries?

Metadata 103

Metadata Data Lake Modeling Data Warehouse 103

AWS Big Data

SEPTEMBER 13, 2023

A modern data architecture is an evolutionary architecture pattern designed to integrate a data lake, data warehouse, and purpose-built stores with a unified governance model. Of those tables, some are larger (such as in terms of record volume) than others, and some are updated more frequently than others.

Data Lake 111

Data Lake Data Processing Metadata Snapshot 111

DataKitchen

NOVEMBER 29, 2021

In 2022, data organizations will institute robust automated processes around their AI systems to make them more accountable to stakeholders. Model developers will test for AI bias as part of their pre-deployment testing. Continuous testing, monitoring and observability will prevent biased models from deploying or continuing to operate.

Testing 245

Testing Data Lake Data Architecture Manufacturing 245

CIO Business Intelligence

FEBRUARY 20, 2025

When building a machine-learning-powered tool to predict the maintenance needs of its customers, Ensono found that its customers used multiple old apps to collect incident tickets, but those apps stored incident data in very different formats, with inconsistent types of data collected, he says.

Insurance 108

Insurance Cost-Benefit Unstructured Data Data Lake 108

DataKitchen

AUGUST 3, 2021

First-generation – expensive, proprietary enterprise data warehouse and business intelligence platforms maintained by a specialized team drowning in technical debt. Second-generation – gigantic, complex data lake maintained by a specialized team drowning in technical debt.

Data Architecture 246

Data Architecture Data Lake Cost-Benefit Data Warehouse 246

Jet Global

JULY 2, 2019

To address the flood of data and the needs of enterprise businesses to store, sort, and analyze that data, a new storage solution has evolved: the data lake. What’s in a Data Lake? Data warehouses do a great job of standardizing data from disparate sources for analysis. Taking a Dip.

Data Lake 73

Data Lake Data Warehouse Big Data Machine Learning 73

David Menninger's Analyst Perspectives

JANUARY 21, 2025

I assert that through 2026, almost all enterprises developing applications based on GenAI will explore vector search and retrieval-augmented generation (RAG) to complement foundation models with proprietary data and content.

Data Lake 130

Data Lake IoT Cost-Benefit Enterprise 130

AWS Big Data

JUNE 23, 2023

Events and many other security data types are stored in Imperva’s Threat Research Multi-Region data lake. Imperva harnesses data to improve their business outcomes. As part of their solution, they are using Amazon QuickSight to unlock insights from their data.

Data Lake 92

Data Lake Dashboards Cost-Benefit Data Warehouse 92

Corinium

JUNE 6, 2019

Some of the work is very foundational, such as building an enterprise data lake and migrating it to the cloud, which enables other more direct value-added activities such as self-service. It is also important to have a strong test and learn culture to encourage rapid experimentation.

Insurance 250

Insurance Analytics Forecasting Deep Learning 250

CIO Business Intelligence

AUGUST 9, 2024

At Atlanta’s Hartsfield-Jackson International Airport, an IT pilot has led to a wholesale data journey destined to transform operations at the world’s busiest airport, fueled by machine learning and generative AI. Data integrity presented a major challenge for the team, as there were many instances of duplicate data.

Data Transformation 126

Data Transformation Machine Learning Data Lake Dashboards 126

AWS Big Data

JANUARY 9, 2025

These features allow efficient data corrections, gap-filling in time series, and historical data updates without disrupting ongoing analyses or compromising data integrity. Unlike direct Amazon S3 access, Iceberg supports these operations on petabyte-scale data lakes without requiring complex custom code.

Metadata 111

Metadata Snapshot Cost-Benefit Optimization 111

AWS Big Data

JULY 29, 2024

In the era of big data, data lakes have emerged as a cornerstone for storing vast amounts of raw data in its native format. They support structured, semi-structured, and unstructured data, offering a flexible and scalable environment for data ingestion from multiple sources.

Metadata 126

Metadata Snapshot Data Lake Metrics 126

AWS Big Data

APRIL 24, 2023

Building a data lake on Amazon Simple Storage Service (Amazon S3) provides numerous benefits for an organization. However, many use cases, like performing change data capture (CDC) from an upstream relational database to an Amazon S3-based data lake, require handling data at a record level.

Data Lake 98

Data Lake Data Governance Machine Learning Cost-Benefit 98

CIO Business Intelligence

MARCH 29, 2024

That’s why Rocket Mortgage has been a vigorous implementor of machine learning and AI technologies — and why CIO Brian Woodring emphasizes a “human in the loop” AI strategy that will not be pinned down to any one generative AI model. The rest are on premises.

Data Lake 134

Data Lake Machine Learning Data Warehouse Unstructured Data 134

DataKitchen

SEPTEMBER 18, 2024

These leaders are expected to influence organizational behavior without direct authority, leading to what DataKitchen CEO Christopher Bergh described as “data nags”—individuals who know what’s wrong but struggle to get others to act. Who should make the change (data engineers, system owners, or data quality professionals).

Scorecard 177

Scorecard Data Quality Measurement Testing 177

CIO Business Intelligence

APRIL 12, 2023

Real-time AI involves processing data for making decisions within a given time frame. Real-time AI brings together streaming data and machine learning algorithms to make fast and automated decisions; examples include recommendations, fraud detection, security monitoring, and chatbots. It isn’t easy.

Machine Learning 145

Machine Learning Cost-Benefit Data-driven Strategy 145

AWS Big Data

JUNE 20, 2023

It manages large collections of files as tables, and it supports modern analytical data lake operations such as record-level insert, update, delete, and time travel queries. Data labeling is required for various use cases, including forecasting, computer vision, natural language processing, and speech recognition.

Data Lake 99

Data Lake Data Science Recreation/Entertainment Experimentation 99

CIO Business Intelligence

JUNE 29, 2022

Taking the broadest possible interpretation of data analytics , Azure offers more than a dozen services — and that’s before you include Power BI, with its AI-powered analysis and new datamart option , or governance-oriented approaches such as Microsoft Purview. Azure Data Factory. Azure Data Lake Analytics.

Data Lake 116

Data Lake Analytics Data Warehouse Machine Learning 116