AWS Big Data

FEBRUARY 29, 2024

Data is your generative AI differentiator, and a successful generative AI implementation depends on a robust data strategy incorporating a comprehensive data governance approach. Data governance is a critical building block across all these approaches, and we see two emerging areas of focus.

Data Governance 124

Data Governance Unstructured Data Metadata Data Lake 124

AWS Big Data

AUGUST 15, 2024

Unlocking the true value of data often gets impeded by siloed information. Traditional data management—wherein each business unit ingests raw data in separate data lakes or warehouses—hinders visibility and cross-functional analysis. Amazon DataZone natively supports data sharing for Amazon Redshift data assets.

Data Lake 122

Data Lake Data Warehouse Data Governance Publishing 122

AWS Big Data

JUNE 10, 2024

In this post, we delve into the key aspects of using Amazon EMR for modern data management, covering topics such as data governance, data mesh deployment, and streamlined data discovery. Organizations have multiple Hive data warehouses across EMR clusters, where the metadata gets generated.

Data Lake 117

Data Lake Metadata Data Warehouse Data Processing 117

CIO Business Intelligence

APRIL 29, 2022

This would be straightforward task were it not for the fact that, during the digital-era, there has been an explosion of data – collected and stored everywhere – much of it poorly governed, ill-understood, and irrelevant. Many organisations focus too heavily on fine tuning their computational models in their pursuit of ‘quick-wins.’

Data Governance 145

Data Governance IT Data Lake Risk 145

AWS Big Data

MARCH 10, 2023

Since the deluge of big data over a decade ago, many organizations have learned to build applications to process and analyze petabytes of data. Data lakes have served as a central repository to store structured and unstructured data at any scale and in various formats.

Data Lake 122

Data Lake Sales Data Warehouse Snapshot 122

erwin

OCTOBER 31, 2019

The Regulatory Rationale for Integrating Data Management & Data Governance. Now, as Cybersecurity Awareness Month comes to a close – and ghosts and goblins roam the streets – we thought it a good time to resurrect some guidance on how data governance can make data security less scary.

Data Governance 105

Data Governance Metadata Risk Data Lake 105

AWS Big Data

JUNE 9, 2023

A modern data architecture enables companies to ingest virtually any type of data through automated pipelines into a data lake, which provides highly durable and cost-effective object storage at petabyte or exabyte scale.

Data Lake 130

Data Lake Metadata Statistics Optimization 130

AWS Big Data

DECEMBER 19, 2024

Initially, the data inventories of different services were siloed within isolated environments, making data discovery and sharing across services manual and time-consuming for all teams involved. Implementing robust data governance is challenging. The following figure illustrates the data mesh architecture.

Data Governance 52

Data Governance Publishing Data-driven Metadata 52

DataKitchen

DECEMBER 7, 2021

In the book, author Zhamak Dehghani reveals that, despite the time, money, and effort poured into them, data warehouses and data lakes fail when applied at the scale and speed of today’s organizations. A distributed data mesh is a better choice. Disrupting Data Governance: A Call to Action, by Laura B.

Data-driven 246

Data-driven Data Governance Big Data Data Science 246

AWS Big Data

DECEMBER 12, 2024

In the context of comprehensive data governance, Amazon DataZone offers organization-wide data lineage visualization using Amazon Web Services (AWS) services, while dbt provides project-level lineage through model analysis and supports cross-project integration between data lakes and warehouses.

Snapshot 98

Snapshot Recreation/Entertainment Experimentation Data Lake 98

AWS Big Data

OCTOBER 10, 2023

Data governance is the process of ensuring the integrity, availability, usability, and security of an organization’s data. Due to the volume, velocity, and variety of data being ingested in data lakes, it can get challenging to develop and maintain policies and procedures to ensure data governance at scale for your data lake.

Data Quality 111

Data Quality Data Governance Data Lake Testing 111

erwin

DECEMBER 13, 2018

This past year witnessed a data governance awakening – or as the Wall Street Journal called it, a “global data governance reckoning.” There was tremendous data drama and resulting trauma – from Facebook to Equifax and from Yahoo to Marriott. So what’s on the horizon for data governance in the year ahead?

Data Governance 91

Data Governance IoT Internet of Things Data-driven 91

AWS Big Data

DECEMBER 4, 2024

Under the federated mesh architecture, each divisional mesh functions as a node within the broader enterprise data mesh, maintaining a degree of autonomy in managing its data products. This model balances node or domain-level autonomy with enterprise-level oversight, creating a scalable and consistent framework across ANZ.

Metadata 105

Metadata Data Governance Data Quality Data-driven 105

CIO Business Intelligence

APRIL 26, 2024

That data is in the process of being unified on a multilayered platform that offers a variety of data services, including data ingestion, data management, data governance, and data security. We’re modernizing existing products to get to this entire data analytics value chain.”

Modeling 114

Modeling IT Machine Learning Data Governance 114

Jet Global

JULY 2, 2019

To address the flood of data and the needs of enterprise businesses to store, sort, and analyze that data, a new storage solution has evolved: the data lake. What’s in a Data Lake? Data warehouses do a great job of standardizing data from disparate sources for analysis. Taking a Dip.

Data Lake 73

Data Lake Data Warehouse Big Data Machine Learning 73

AWS Big Data

APRIL 24, 2023

Building a data lake on Amazon Simple Storage Service (Amazon S3) provides numerous benefits for an organization. However, many use cases, like performing change data capture (CDC) from an upstream relational database to an Amazon S3-based data lake, require handling data at a record level.

Data Lake 98

Data Lake Data Governance Machine Learning Cost-Benefit 98

AWS Big Data

JULY 21, 2023

Data-driven organizations treat data as an asset and use it across different lines of business (LOBs) to drive timely insights and better business decisions. This leads to having data across many instances of data warehouses and data lakes using a modern data architecture in separate AWS accounts.

Data Lake 98

Data Lake Data Warehouse Marketing Management 98

Cloudera

AUGUST 4, 2021

And if data security tops IT concerns, data governance should be their second priority. Not only is it critical to protect data, but data governance is also the foundation for data-driven businesses and maximizing value from data analytics. But it’s still not easy. But it’s still not easy.

Data Governance 86

Data Governance Metadata Data-driven Enterprise 86

AWS Big Data

JULY 29, 2024

In the era of big data, data lakes have emerged as a cornerstone for storing vast amounts of raw data in its native format. They support structured, semi-structured, and unstructured data, offering a flexible and scalable environment for data ingestion from multiple sources.

Metadata 126

Metadata Snapshot Data Lake Metrics 126

AWS Big Data

OCTOBER 21, 2024

However, enterprises often encounter challenges with data silos, insufficient access controls, poor governance, and quality issues. Embracing data as a product is the key to address these challenges and foster a data-driven culture. This data platform is managed by Amazon Data Zone.

Sales 116

Sales Data-driven Data Processing Key Performance Indicator 116

CIO Business Intelligence

OCTOBER 2, 2024

According to Kari Briski, VP of AI models, software, and services at Nvidia, successfully implementing gen AI hinges on effective data management and evaluating how different models work together to serve a specific use case. But some IT leaders are getting it right because they focus on three key aspects.

Management 141

Management Data Governance Cost-Benefit Structured Data 141

erwin

AUGUST 31, 2023

Q: Is data modeling cool again? In today’s fast-paced digital landscape, data reigns supreme. The data-driven enterprise relies on accurate, accessible, and actionable information to make strategic decisions and drive innovation. A: It always was and is getting cooler!!

Data-driven 98

Data-driven Modeling Enterprise Structured Data 98

David Menninger's Analyst Perspectives

OCTOBER 26, 2021

Databricks is a data engineering and analytics cloud platform built on top of Apache Spark that processes and transforms huge volumes of data and offers data exploration capabilities through machine learning models. The platform supports streaming data, SQL queries, graph processing and machine learning.

Big Data 327

Big Data Data Processing Machine Learning Modeling 327

erwin

DECEMBER 17, 2020

What is Microsoft’s Common Data Model (CDM), and why is it so powerful? The same is true for data, with a number of vendors creating data models by vertical industry (financial services, healthcare, etc.) and making them commercially available to improve how organizations understand and work with their data assets.

Modeling 90

Modeling Metadata Data-driven Data Lake 90

CIO Business Intelligence

AUGUST 9, 2024

The original proof of concept was to have one data repository ingesting data from 11 sources, including flat files and data stored via APIs on premises and in the cloud, Pruitt says. There are a lot of variables that determine what should go into the data lake and what will probably stay on premise,” Pruitt says.

Data Transformation 126

Data Transformation Machine Learning Data Lake Dashboards 126

Cloudera

OCTOBER 23, 2024

One of the most important innovations in data management is open table formats, specifically Apache Iceberg , which fundamentally transforms the way data teams manage operational metadata in the data lake.

Metadata 84

Metadata Data Lake Dashboards Interactive 84

AWS Big Data

JANUARY 31, 2023

Data governance is the collection of policies, processes, and systems that organizations use to ensure the quality and appropriate handling of their data throughout its lifecycle for the purpose of generating business value.

Data Lake 98

Data Lake Data Governance Data Architecture Machine Learning 98

Alation

FEBRUARY 20, 2020

For many enterprises, a hybrid cloud data lake is no longer a trend, but becoming reality. With a cloud deployment, enterprises can leverage a “pay as you go” model; reducing the burden of incurring capital costs. With an on-premise deployment, enterprises have full control over data security, data access, and data governance.

Data Lake 52

Data Lake ROI Metadata Cost-Benefit 52

AWS Big Data

FEBRUARY 27, 2024

The following are the key components of the Bluestone Data Platform: Data mesh architecture – Bluestone adopted a data mesh architecture, a paradigm that distributes data ownership across different business units. This enables data-driven decision-making across the organization.

Data-driven 111

Data-driven Data Lake Data Quality Data Governance 111

AWS Big Data

JULY 19, 2023

You can structure your data, measure business processes, and get valuable insights quickly can be done by using a dimensional model. Amazon Redshift provides built-in features to accelerate the process of modeling, orchestrating, and reporting from a dimensional model. Declare the grain of your data.

Modeling 97

Modeling Sales Data Warehouse Snapshot 97

Alation

JANUARY 27, 2022

People might not understand the data, the data they chose might not be ideal for their application, or there might be better, more current, or more accurate data available. An effective data governance program ensures data consistency and trustworthiness. It can also help prevent data misuse.

Data Governance 52

Data Governance Analytics Cost-Benefit Data-driven 52

AWS Big Data

FEBRUARY 26, 2025

Analytics remained one of the key focus areas this year, with significant updates and innovations aimed at helping businesses harness their data more efficiently and accelerate insights. From enhancing data lakes to empowering AI-driven analytics, AWS unveiled new tools and services that are set to shape the future of data and analytics.

Analytics 62

Analytics Data Lake Metadata Data Warehouse 62

Cloudera

FEBRUARY 16, 2022

In this example, the Machine Learning (ML) model struggles to differentiate between a chihuahua and a muffin. Will the model correctly determine it is a muffin or get confused and think it is a chihuahua? The extent to which we can predict how the model will classify an image given a change input (e.g. Model Visibility.

Machine Learning 85

Machine Learning Modeling Metadata Recreation/Entertainment 85

Alation

FEBRUARY 20, 2020

The rise of data lakes, IOT analytics, and big data pipelines has introduced a new world of fast, big data. How Data Catalogs Can Help. Data catalogs evolved as a key component of the data governance revolution by creating a bridge between the new world and old world of data governance.

Data Governance 52

Data Governance Dashboards Data Lake Big Data 52

erwin

JULY 17, 2019

Part Two of the Digital Transformation Journey … In our last blog on driving digital transformation , we explored how enterprise architecture (EA) and business process (BP) modeling are pivotal factors in a viable digital transformation strategy. Digital Transformation Strategy: Smarter Data. The solution is data intelligence.

Digital Transformation 98

Digital Transformation Strategy Metadata Data-driven 98

CIO Business Intelligence

OCTOBER 13, 2023

Solutions data architect: These individuals design and implement data solutions for specific business needs, including data warehouses, data marts, and data lakes. Application data architect: The application data architect designs and implements data models for specific software applications.

Data Architecture 132

Data Architecture Data Warehouse Statistics Visualization 132

BI-Survey

AUGUST 21, 2020

Leaders rely less on data mart deployment than on lean, flexible architectures and usable data based on cloud services, a complementary data lake, data governance, data hubs and data catalogs. Data virtualization and data hubs make it possible to access data sources faster and more flexibly.

Data Warehouse 98

Data Warehouse Data Lake Data Governance Data Architecture 98