AWS Big Data

MARCH 10, 2023

Since the deluge of big data over a decade ago, many organizations have learned to build applications to process and analyze petabytes of data. Data lakes have served as a central repository to store structured and unstructured data at any scale and in various formats.

Data Lake 116

Data Lake Sales Data Warehouse Snapshot 116

AWS Big Data

NOVEMBER 26, 2024

This post was co-written with Dipankar Mazumdar, Staff Data Engineering Advocate with AWS Partner OneHouse. Data architecture has evolved significantly to handle growing data volumes and diverse workloads. The mechanism periodically scans a data catalog like the AWS Glue Data Catalog for tables to convert with XTable.

Metadata 95

Metadata Data Lake Snapshot Data Warehouse 95

AWS Big Data

JUNE 9, 2023

A modern data architecture enables companies to ingest virtually any type of data through automated pipelines into a data lake, which provides highly durable and cost-effective object storage at petabyte or exabyte scale.

Data Lake 125

Data Lake Metadata Statistics Optimization 125

CIO Business Intelligence

APRIL 8, 2025

These strategies, such as investing in AI-powered cleansing tools and adopting federated governance models, not only address the current data quality challenges but also pave the way for improved decision-making, operational efficiency and customer satisfaction. When financial data is inconsistent, reporting becomes unreliable.

Data Quality 102

Data Quality Data-driven Key Performance Indicator Metadata 102

Cloudera

MARCH 28, 2024

At a time when AI is exploding in popularity and finding its way into nearly every facet of business operations, data has arguably never been more valuable. More recently, that value has been made clear by the emergence of AI-powered technologies like generative AI (GenAI) and the use of Large Language Models (LLMs).

Data Architecture 94

Data Architecture Strategy Data Lake Data-driven 94

AWS Big Data

SEPTEMBER 13, 2023

The Analytics specialty practice of AWS Professional Services (AWS ProServe) helps customers across the globe with modern data architecture implementations on the AWS Cloud. Of those tables, some are larger (such as in terms of record volume) than others, and some are updated more frequently than others.

Data Lake 104

Data Lake Data Processing Metadata Snapshot 104

AWS Big Data

JUNE 10, 2024

Use cases for Hive metastore federation for Amazon EMR Hive metastore federation for Amazon EMR is applicable to the following use cases: Governance of Amazon EMR-based data lakes – Producers generate data within their AWS accounts using an Amazon EMR-based data lake supported by EMRFS on Amazon Simple Storage Service (Amazon S3)and HBase.

Data Lake 96

Data Lake Metadata Data Warehouse Data Processing 96

AWS Big Data

JULY 20, 2023

With data becoming the driving force behind many industries today, having a modern data architecture is pivotal for organizations to be successful. In this post, we describe Orca’s journey building a transactional data lake using Amazon Simple Storage Service (Amazon S3), Apache Iceberg, and AWS Analytics.

Data Lake 94

Data Lake Analytics Snapshot Data Quality 94

AWS Big Data

JANUARY 15, 2025

Need for a data mesh architecture Because entities in the EUROGATE group generate vast amounts of data from various sourcesacross departments, locations, and technologiesthe traditional centralized data architecture struggles to keep up with the demands for real-time insights, agility, and scalability.

IoT 101

IoT Machine Learning Metadata Data-driven 101

AWS Big Data

APRIL 8, 2025

In modern data architectures, Apache Iceberg has emerged as a popular table format for data lakes, offering key features including ACID transactions and concurrent write support. This post demonstrates how to implement reliable concurrent write handling mechanisms in Iceberg tables.

Snapshot 117

Snapshot Management Metadata Big Data 117

AWS Big Data

FEBRUARY 26, 2025

This evaluation, we feel, critically examines vendors capabilities to address key service needs, including data engineering, operational data integration, modern data architecture delivery, and enabling less-technical data integration across various deployment models.

Data Integration 81

Data Integration Data Lake Data Warehouse Unstructured Data 81

CIO Business Intelligence

NOVEMBER 25, 2022

As part of that transformation, Agusti has plans to integrate a data lake into the company’s data architecture and expects two AI proofs of concept (POCs) to be ready to move into production within the quarter. Today, we backflush our data lake through our data warehouse. We’re still in that journey.”

Data Lake 128

Data Lake Data Warehouse Unstructured Data Data Architecture 128

AWS Big Data

JULY 21, 2023

This leads to having data across many instances of data warehouses and data lakes using a modern data architecture in separate AWS accounts. We recently announced the integration of Amazon Redshift data sharing with AWS Lake Formation. Take note of this role’s ARN to use later in the steps.

Data Lake 93

Data Lake Data Warehouse Marketing Management 93

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI.

Metadata 84

Metadata Data Lake Dashboards Interactive 84

CIO Business Intelligence

OCTOBER 13, 2023

Data architecture is a complex and varied field and different organizations and industries have unique needs when it comes to their data architects. Solutions data architect: These individuals design and implement data solutions for specific business needs, including data warehouses, data marts, and data lakes.

Data Architecture 132

Data Architecture Data Warehouse Statistics Visualization 132

CIO Business Intelligence

MARCH 9, 2022

However, they do contain effective data management, organization, and integrity capabilities. As a result, users can easily find what they need, and organizations avoid the operational and cost burdens of storing unneeded or duplicate data copies. Warehouse, data lake convergence. Meet the data lakehouse.

Data Lake 119

Data Lake Unstructured Data Data Warehouse Big Data 119

DataKitchen

JANUARY 3, 2022

Business analytic teams have ongoing deliverables – a dashboard, a PowerPoint, or a model that they refresh and renew. There’s a recent trend toward people creating data lake or data warehouse patterns and calling it data enablement or a data hub. IT-created infrastructure such as a data lake/warehouse).

Business Analytics 245

Business Analytics Analytics Testing Dashboards 245

AWS Big Data

APRIL 24, 2023

Building a data lake on Amazon Simple Storage Service (Amazon S3) provides numerous benefits for an organization. However, many use cases, like performing change data capture (CDC) from an upstream relational database to an Amazon S3-based data lake, require handling data at a record level.

Data Lake 88

Data Lake Data Governance Machine Learning Cost-Benefit 88

IBM Big Data Hub

JANUARY 5, 2023

Several factors determine the quality of your enterprise data like accuracy, completeness, consistency, to name a few. But there’s another factor of data quality that doesn’t get the recognition it deserves: your data architecture. How the right data architecture improves data quality.

Data Architecture 52

Data Architecture Data Quality Strategy Data Lake 52

IBM Big Data Hub

AUGUST 4, 2023

Today, the way businesses use data is much more fluid; data literate employees use data across hundreds of apps, analyze data for better decision-making, and access data from numerous locations. Then, it applies these insights to automate and orchestrate the data lifecycle.

Data Architecture 59

Data Architecture Data Lake Machine Learning Data Governance 59

CIO Business Intelligence

APRIL 12, 2023

After walking his executive team through the data hops, flows, integrations, and processing across different ingestion software, databases, and analytical platforms, they were shocked by the complexity of their current data architecture and technology stack. How is data, process, and model drift managed for reliability?

Machine Learning 145

Machine Learning Cost-Benefit Data-driven Strategy 145

AWS Big Data

FEBRUARY 26, 2025

Analytics remained one of the key focus areas this year, with significant updates and innovations aimed at helping businesses harness their data more efficiently and accelerate insights. From enhancing data lakes to empowering AI-driven analytics, AWS unveiled new tools and services that are set to shape the future of data and analytics.

Analytics 59

Analytics Data Lake Metadata Data Warehouse 59

AWS Big Data

JANUARY 31, 2023

We have collected some of the key talks and solutions on data governance, data mesh, and modern data architecture published and presented in AWS re:Invent 2022, and a few data lake solutions built by customers and AWS Partners for easy reference. Starting with Amazon EMR release 6.7.0,

Data Lake 90

Data Lake Data Governance Data Architecture Machine Learning 90

CIO Business Intelligence

APRIL 25, 2022

The other 10% represents the effort of initial deployment, data-loading, configuration and the setup of administrative tasks and analysis that is specific to the customer, the Henschen said. They require specific data inputs, models, algorithms and they deliver very specific recommendations.

Recreation/Entertainment 119

Recreation/Entertainment Data Lake Data Warehouse Unstructured Data 119

CDW Research Hub

JULY 18, 2022

One modern data platform solution that provides simplicity and flexibility to grow is Snowflake’s data cloud and platform. Sirius has created a lightweight development tool to rapidly build and deploy best-practice data models. Security Data Lake. Learn more about our Security Data Lake Solution.

Optimization 87

Optimization Data Lake Data Warehouse Manufacturing 87

CIO Business Intelligence

JULY 26, 2023

The CIO delights in detailing the work of Re/Max’s technology team, which is building the pipelines and cloud-native applications to deliver agents in the field the most refined and insightful data from more than 500 MLS listing serivces in the US and Canada as quickly as possible.

Data Lake 98

Data Lake Digital Transformation Machine Learning Data Architecture 98

Cloudera

JUNE 30, 2022

With Cloudera’s vision of hybrid data , enterprises adopting an open data lakehouse can easily get application interoperability and portability to and from on premises environments and any public cloud without worrying about data scaling. Why integrate Apache Iceberg with Cloudera Data Platform?

Data Lake 88

Data Lake Data Warehouse Data Architecture Metadata 88

AWS Big Data

FEBRUARY 16, 2024

Many customers are extending their data warehouse capabilities to their data lake with Amazon Redshift. They are looking to further enhance their security posture where they can enforce access policies on their data lakes based on Amazon Simple Storage Service (Amazon S3). Choose Create endpoint.

Data Lake 112

Data Lake Data Warehouse Testing Business Objectives 112

AWS Big Data

OCTOBER 23, 2024

Amazon Redshift enables data warehousing by seamlessly integrating with other data stores and services in the modern data organization through features such as Zero-ETL , data sharing , streaming ingestion , data lake integration , and Redshift ML.

Data Warehouse 97

Data Warehouse Cost-Benefit Data Lake Data-driven 97

BI-Survey

AUGUST 21, 2020

Leaders rely less on data mart deployment than on lean, flexible architectures and usable data based on cloud services, a complementary data lake, data governance, data hubs and data catalogs. Leaders obviously strive towards a leaner architecture and a flexible infrastructure.

Data Warehouse 98

Data Warehouse Data Lake Data Governance Data Architecture 98

AWS Big Data

NOVEMBER 29, 2023

Zero-ETL integration also enables you to load and analyze data from multiple operational database clusters in a new or existing Amazon Redshift instance to derive holistic insights across many applications. Use one click to access your data lake tables using auto-mounted AWS Glue data catalogs on Amazon Redshift for a simplified experience.

Data Warehouse 113

Data Warehouse Analytics Data Lake Machine Learning 113

CIO Business Intelligence

AUGUST 3, 2023

But reaching all these goals, as well as using enterprise data for generative AI to streamline the business and develop new services, requires a proper foundation. That hard, ongoing work includes integrating siloed data, modeling, and understanding it, as well as maintaining and securing it over time.

Analytics 98

Analytics Data Lake Metadata Cost-Benefit 98

AWS Big Data

OCTOBER 10, 2023

Data governance is the process of ensuring the integrity, availability, usability, and security of an organization’s data. Due to the volume, velocity, and variety of data being ingested in data lakes, it can get challenging to develop and maintain policies and procedures to ensure data governance at scale for your data lake.

Data Quality 104

Data Quality Data Governance Data Lake Testing 104

AWS Big Data

FEBRUARY 27, 2024

The following are the key components of the Bluestone Data Platform: Data mesh architecture – Bluestone adopted a data mesh architecture, a paradigm that distributes data ownership across different business units. This enables data-driven decision-making across the organization.

Data-driven 106

Data-driven Data Lake Data Quality Data Governance 106

CIO Business Intelligence

OCTOBER 18, 2022

She decided to bring Resultant in to assist, starting with the firm’s strategic data assessment (SDA) framework, which evaluates a client’s data challenges in terms of people and processes, data models and structures, data architecture and platforms, visual analytics and reporting, and advanced analytics.

Data Transformation 141

Data Transformation Consulting Data Lake Reporting 141

AWS Big Data

OCTOBER 9, 2024

Today, customers are embarking on data modernization programs by migrating on-premises data warehouses and data lakes to the AWS Cloud to take advantage of the scale and advanced analytical capabilities of the cloud. Compare ongoing data that is replicated from the source on-premises database to the target S3 data lake.

Data Quality 96

Data Quality Data Lake Data Warehouse Metrics 96

Cloudera

AUGUST 20, 2024

Trusted data is what makes the outputs of AI not just accurate, but impactful in decision making. Ensuring data is trustworthy comes with its own complications. Cloudera’s State of Enterprise AI and Modern Data Architecture survey identified several challenges when it comes to data.

Data Architecture 82

Data Architecture Data Lake Data Governance Data Warehouse 82