AWS Big Data

JANUARY 15, 2025

Their terminal operations rely heavily on seamless data flows and the management of vast volumes of data. Recently, EUROGATE has developed a digital twin for its container terminal Hamburg (CTH), generating millions of data points every second from Internet of Things (IoT)devices attached to its container handling equipment (CHE).

IoT 109

IoT Machine Learning Metadata Data-driven 109

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI.

Metadata 84

Metadata Data Lake Dashboards Interactive 84

Data Virtualization

DECEMBER 8, 2022

Reading Time: 3 minutes While cleaning up our archive recently, I found an old article published in 1976 about data dictionary/directory systems (DD/DS). Nowadays, we no longer use the term DD/DS, but “data catalog” or simply “metadata system”. It was written by L.

Metadata 69

Metadata IT Publishing Data Integration 69

AWS Big Data

SEPTEMBER 11, 2024

This post describes how HPE Aruba automated their Supply Chain management pipeline, and re-architected and deployed their data solution by adopting a modern data architecture on AWS. Each file arrives as a pair with a tail metadata file in CSV format containing the size and name of the file.

Data Architecture 109

Data Architecture Optimization Data Warehouse Metadata 109

Cloudera

JUNE 29, 2023

To improve the way they model and manage risk, institutions must modernize their data management and data governance practices. Implementing a modern data architecture makes it possible for financial institutions to break down legacy data silos, simplifying data management, governance, and integration — and driving down costs.

Data Architecture 85

Data Architecture Risk Management Risk Management 85

CIO Business Intelligence

FEBRUARY 20, 2024

Manufacturers have long held a data-driven vision for the future of their industry. It’s one where near real-time data flows seamlessly between IT and operational technology (OT) systems. Legacy data management is holding back manufacturing transformation Until now, however, this vision has remained out of reach.

Data Architecture 98

Data Architecture Unstructured Data Manufacturing IT 98

Cloudera

AUGUST 30, 2022

But while state and local governments seek to improve policies, decision making, and the services constituents rely upon, data silos create accessibility and sharing challenges that hinder public sector agencies from transforming their data into a strategic asset and leveraging it for the common good. . Modern data architectures.

Data Architecture 79

Data Architecture Data Lake Data Warehouse Metadata 79

Cloudera

AUGUST 6, 2024

And although there is clear momentum behind the data lakehouse as the ideal architecture for multi-function analytics, the demand for open table formats including Apache Iceberg is a clear signal that data leaders value interoperability and engine freedom. It no longer matters where the data is. Open data is the future.

Metadata 82

Metadata Cost-Benefit Management Enterprise 82

Cloudera

JUNE 7, 2022

Only a fraction of data created is actually stored and managed, with analysts estimating it to be between 4 – 6 ZB in 2020. Clearly, hybrid data presents a massive opportunity and a tough challenge. Capitalizing on the potential requires the ability to harness the value of all of that data, no matter where it is.

IT 112

IT Data Architecture Unstructured Data Big Data 112

AWS Big Data

JANUARY 24, 2023

AWS Lake Formation helps with enterprise data governance and is important for a data mesh architecture. It works with the AWS Glue Data Catalog to enforce data access and governance. This solution only replicates metadata in the Data Catalog, not the actual underlying data.

Data Architecture 93

Data Architecture Metadata Data Lake Snapshot 93

Cloudera

DECEMBER 6, 2024

The AI Forecast: Data and AI in the Cloud Era , sponsored by Cloudera, aims to take an objective look at the impact of AI on business, industry, and the world at large. AI is only as successful as the data behind it. It could be metadata that you weren’t capturing before. But what does that future look like?

Manufacturing 82

Manufacturing Forecasting Metadata Data Processing 82

erwin

JANUARY 9, 2020

For decades, data modeling has been the optimal way to design and deploy new relational databases with high-quality data sources and support application development. Today’s data modeling is not your father’s data modeling software. So here’s why data modeling is so critical to data governance.

Data Governance 144

Data Governance Modeling Metadata Unstructured Data 144

CIO Business Intelligence

JUNE 23, 2022

Only a fraction of data created is actually stored and managed, with analysts estimating it to be between 4 – 6 ZB in 2020. Clearly, hybrid data presents a massive opportunity and a tough challenge. Capitalizing on the potential requires the ability to harness the value of all of that data, no matter where it is.

IT 97

IT Data Architecture Unstructured Data Big Data 97

Cloudera

SEPTEMBER 29, 2022

Each of these trends claim to be complete models for their data architectures to solve the “everything everywhere all at once” problem. Data teams are confused as to whether they should get on the bandwagon of just one of these trends or pick a combination. First, we describe how data mesh and data fabric could be related.

Data Architecture 74

Data Architecture Data Warehouse Metadata Sales 74

TDAN

MAY 31, 2022

The main goal of creating an enterprise data fabric is not new. It is the ability to deliver the right data at the right time, in the right shape, and to the right data consumer, irrespective of how and where it is stored. Data fabric is the common “net” that stitches integrated data from multiple data […].

Metadata 98

Metadata Enterprise IT Data Architecture 98

AWS Big Data

APRIL 3, 2024

They understand that a one-size-fits-all approach no longer works, and recognize the value in adopting scalable, flexible tools and open data formats to support interoperability in a modern data architecture to accelerate the delivery of new solutions.

Data Lake 125

Data Lake Snapshot Metadata Data Architecture 125

CIO Business Intelligence

OCTOBER 13, 2023

Data architecture is a complex and varied field and different organizations and industries have unique needs when it comes to their data architects. Solutions data architect: These individuals design and implement data solutions for specific business needs, including data warehouses, data marts, and data lakes.

Data Architecture 132

Data Architecture Data Warehouse Statistics Visualization 132

AWS Big Data

DECEMBER 4, 2024

They’re taking data they’ve historically used for analytics or business reporting and putting it to work in machine learning (ML) models and AI-powered applications. You’ll get a single unified view of all your data for your data and AI workers, regardless of where the data sits, breaking down your data siloes.

Data Analytics 102

Data Analytics Analytics Data Lake Data Quality 102

AWS Big Data

DECEMBER 4, 2024

While traditional extract, transform, and load (ETL) processes have long been a staple of data integration due to its flexibility, for common use cases such as replication and ingestion, they often prove time-consuming, complex, and less adaptable to the fast-changing demands of modern data architectures.

Data Integration 112

Data Integration Data Lake Statistics Data-driven 112

IBM Big Data Hub

JANUARY 5, 2023

Several factors determine the quality of your enterprise data like accuracy, completeness, consistency, to name a few. But there’s another factor of data quality that doesn’t get the recognition it deserves: your data architecture. How the right data architecture improves data quality.

Data Architecture 52

Data Architecture Data Quality Strategy Data Lake 52

erwin

DECEMBER 12, 2019

Metadata is an important part of data governance, and as a result, most nascent data governance programs are rife with project plans for assessing and documenting metadata. But in many scenarios, it seems that the underlying driver of metadata collection projects is that it’s just something you do for data governance.

Metadata 52

Metadata Data Governance Digital Transformation Data Quality 52

AWS Big Data

OCTOBER 25, 2023

Why it’s challenging to process and manage unstructured data Unstructured data makes up a large proportion of the data in the enterprise that can’t be stored in a traditional relational database management systems (RDBMS). Understanding the data, categorizing it, storing it, and extracting insights from it can be challenging.

Unstructured Data 133

Unstructured Data Metadata Management Analytics 133

Cloudera

FEBRUARY 22, 2022

Over the past decade, the successful deployment of large scale data platforms at our customers has acted as a big data flywheel driving demand to bring in even more data, apply more sophisticated analytics, and on-board many new data practitioners from business analysts to data scientists. Key Design Goals .

Snapshot 110

Snapshot Metadata Cost-Benefit Data Architecture 110

Octopai

APRIL 19, 2021

Aptly named, metadata management is the process in which BI and Analytics teams manage metadata, which is the data that describes other data. In other words, data is the context and metadata is the content. Without metadata, BI teams are unable to understand the data’s full story.

Metadata 59

Metadata Management Business Intelligence Data Governance 59

CIO Business Intelligence

MARCH 24, 2023

Data governance definition Data governance is a system for defining who within an organization has authority and control over data assets and how those data assets may be used. It encompasses the people, processes, and technologies required to manage and protect data assets.

Data Governance 142

Data Governance Management Metadata Data Quality 142

AWS Big Data

JUNE 10, 2024

A modern data strategy redefines and enables sharing data across the enterprise and allows for both reading and writing of a singular instance of the data using an open table format. Why Cloudinary chose Apache Iceberg Apache Iceberg is a high-performance table format for huge analytic workloads.

Data Lake 125

Data Lake Metadata Snapshot Analytics 125

erwin

JULY 12, 2019

In light of recent, high-profile data breaches, it’s past-time we re-examined strategic data governance and its role in managing regulatory requirements. for alleged violations of the European Union’s General Data Protection Regulation (GDPR). Complexity. Five Steps to GDPR/CCPA Compliance. Govern PII “in motion”.

Data Governance 110

Data Governance Management Metadata Risk Management 110

Alation

AUGUST 18, 2022

I’m convinced that a fundamental shift in approach to lineage is needed to drive both the value of data (and the analytics culture) to a new level of effectiveness. What Is Data Lineage Creation & Maintenance? But what data things are interconnected? Simply put, I find it fascinating! Why Focus on Lineage? Look familiar?

Metadata 69

Metadata Visualization Statistics Data Architecture 69

erwin

DECEMBER 6, 2019

In the data-driven era, CIO’s need a solid understanding of data governance 2.0 … Data governance (DG) is no longer about just compliance or relegated to the confines of IT. Today, data governance needs to be a ubiquitous part of your organization’s culture. It also requires funding.

Data Governance 145

Data Governance Metadata Enterprise Data-driven 145

AWS Big Data

FEBRUARY 26, 2025

Amazon SageMaker Lakehouse provides an open data architecture that reduces data silos and unifies data across Amazon Simple Storage Service (Amazon S3) data lakes, Redshift data warehouses, and third-party and federated data sources. With AWS Glue 5.0, AWS Glue 5.0 AWS Glue 5.0 Apache Iceberg 1.6.1,

Analytics 60

Analytics Data Lake Metadata Data Warehouse 60

AWS Big Data

JUNE 7, 2023

They chose AWS Glue as their preferred data integration tool due to its serverless nature, low maintenance, ability to control compute resources in advance, and scale when needed. To share the datasets, they needed a way to share access to the data and access to catalog metadata in the form of tables and views.

Metadata 57

Metadata Data Lake Machine Learning Big Data 57

DataKitchen

OCTOBER 11, 2021

First, you must understand the existing challenges of the data team, including the data architecture and end-to-end toolchain. The final step is designing a data solution and its implementation. The biggest challenge is broken data pipelines due to highly manual processes. List of Challenges. Definition of Done.

Testing 154

Testing Metadata Dashboards Statistics 154

AWS Big Data

OCTOBER 3, 2023

Over the years, data lakes on Amazon Simple Storage Service (Amazon S3) have become the default repository for enterprise data and are a common choice for a large set of users who query data for a variety of analytics and machine leaning use cases. Analytics use cases on data lakes are always evolving.

Data Lake 120

Data Lake Metadata Snapshot Recreation/Entertainment 120

IBM Big Data Hub

AUGUST 4, 2023

Data democratization, much like the term digital transformation five years ago, has become a popular buzzword throughout organizations, from IT departments to the C-suite. It’s often described as a way to simply increase data access, but the transition is about far more than that.

Data Architecture 59

Data Architecture Data Lake Machine Learning Data Governance 59

erwin

SEPTEMBER 17, 2020

Replace manual and recurring tasks for fast, reliable data lineage and overall data governance. It’s paramount that organizations understand the benefits of automating end-to-end data lineage. The importance of end-to-end data lineage is widely understood and ignoring it is risky business. Doing Data Lineage Right.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

erwin

MAY 13, 2020

The role of data modeling (DM) has expanded to support enterprise data management, including data governance and intelligence efforts. Metadata management is the key to managing and governing your data and drawing intelligence from it. Types of Data Models: Conceptual, Logical and Physical.

Data Governance 93

Data Governance Enterprise Modeling Management 93

Cloudera

OCTOBER 7, 2021

Within the context of a data mesh architecture, I will present industry settings / use cases where the particular architecture is relevant and highlight the business value that it delivers against business and technology areas. Introduction to the Data Mesh Architecture and its Required Capabilities.

Metadata 124

Metadata Cost-Benefit Enterprise Interactive 124