AWS Big Data

OCTOBER 30, 2024

Amazon DataZone now launched authentication supports through the Amazon Athena JDBC driver, allowing data users to seamlessly query their subscribed data lake assets via popular business intelligence (BI) and analytics tools like Tableau, Power BI, Excel, SQL Workbench, DBeaver, and more.

Visualization 115

Visualization Data Lake Testing Data Governance 115

AWS Big Data

OCTOBER 10, 2024

Amazon Redshift has established itself as a highly scalable, fully managed cloud data warehouse trusted by tens of thousands of customers for its superior price-performance and advanced data analytics capabilities.

Data Lake 101

Data Lake Data Warehouse Recreation/Entertainment Data-driven 101

CIO Business Intelligence

DECEMBER 20, 2024

Data architecture definition Data architecture describes the structure of an organizations logical and physical data assets, and data management resources, according to The Open Group Architecture Framework (TOGAF). An organizations data architecture is the purview of data architects. Cloud storage.

Data Architecture 122

Data Architecture Management Consulting Internet of Things 122

AWS Big Data

AUGUST 15, 2024

Unlocking the true value of data often gets impeded by siloed information. Traditional data management—wherein each business unit ingests raw data in separate data lakes or warehouses—hinders visibility and cross-functional analysis. Amazon DataZone natively supports data sharing for Amazon Redshift data assets.

Data Lake 108

Data Lake Data Warehouse Data Governance Publishing 108

AWS Big Data

OCTOBER 29, 2024

With over 10 PB of data across 1,500 data assets, 1,000 data use cases, and more than 9000 users, the BMW CDH has become a resounding success since BMW decided to build it in a strategic collaboration with Amazon Web Services (AWS) in 2020. This led to inefficiencies in data governance and access control.

Data Lake 88

Data Lake Sales Metadata Machine Learning 88

AWS Big Data

DECEMBER 19, 2024

However, many companies today still struggle to effectively harness and use their data due to challenges such as data silos, lack of discoverability, poor data quality, and a lack of data literacy and analytical capabilities to quickly access and use data across the organization.

Data Governance 52

Data Governance Publishing Data-driven Metadata 52

AWS Big Data

JANUARY 15, 2025

Their terminal operations rely heavily on seamless data flows and the management of vast volumes of data. Recently, EUROGATE has developed a digital twin for its container terminal Hamburg (CTH), generating millions of data points every second from Internet of Things (IoT)devices attached to its container handling equipment (CHE).

IoT 100

IoT Machine Learning Metadata Data-driven 100

David Menninger's Analyst Perspectives

SEPTEMBER 22, 2022

But collecting data is only half of the equation. As the data grows, it becomes challenging to find the right data at the right time. Many organizations can’t take full advantage of their data lakes because they don’t know what data actually exists.

Data Lake 278

Data Lake Business Intelligence Analytics IT 278

AWS Big Data

JUNE 9, 2023

A modern data architecture enables companies to ingest virtually any type of data through automated pipelines into a data lake, which provides highly durable and cost-effective object storage at petabyte or exabyte scale.

Data Lake 125

Data Lake Metadata Statistics Optimization 125

AWS Big Data

JULY 18, 2024

Over the years, organizations have invested in creating purpose-built, cloud-based data lakes that are siloed from one another. A major challenge is enabling cross-organization discovery and access to data across these multiple data lakes, each built on different technology stacks.

Data Lake 122

Data Lake Publishing Metadata Data-driven 122

erwin

OCTOBER 31, 2019

Do you know where your data is? What data you have? Add to the mix the potential for a data breach followed by non-compliance, reputational damage and financial penalties and a real horror story could unfold. s Information Commissioner’s Office had levied against both Facebook and Equifax for their data breaches.

Data Governance 105

Data Governance Metadata Risk Data Lake 105

erwin

APRIL 2, 2021

How can companies protect their enterprise data assets, while also ensuring their availability to stewards and consumers while minimizing costs and meeting data privacy requirements? Data Security Starts with Data Governance. Lack of a solid data governance foundation increases the risk of data-security incidents.

Data Governance 106

Data Governance Cost-Benefit Metadata Risk 106

AWS Big Data

MARCH 10, 2023

Since the deluge of big data over a decade ago, many organizations have learned to build applications to process and analyze petabytes of data. Data lakes have served as a central repository to store structured and unstructured data at any scale and in various formats.

Data Lake 116

Data Lake Sales Data Warehouse Snapshot 116

AWS Big Data

FEBRUARY 29, 2024

Data is your generative AI differentiator, and a successful generative AI implementation depends on a robust data strategy incorporating a comprehensive data governance approach. Data governance is a critical building block across all these approaches, and we see two emerging areas of focus.

Data Governance 109

Data Governance Unstructured Data Metadata Data Lake 109

DataKitchen

DECEMBER 7, 2021

Fail Fast, Learn Faster: Lessons in Data-Driven Leadership in an Age of Disruption, Big Data, and AI, by Randy Bean. This book is for any data leader looking to get the most out their data and their data teams. A distributed data mesh is a better choice. How did we get here? Author Laura B.

Data-driven 246

Data-driven Data Governance Big Data Data Science 246

AWS Big Data

OCTOBER 30, 2024

Amazon DataZone is a data management service that makes it faster and easier for customers to catalog, discover, share, and govern data stored across AWS, on premises, and from third-party sources. When you’re connected, you can query, visualize, and share data—governed by Amazon DataZone—within Tableau.

Analytics 103

Analytics Visualization Data Governance Data-driven 103

AWS Big Data

JUNE 10, 2024

In this post, we delve into the key aspects of using Amazon EMR for modern data management, covering topics such as data governance, data mesh deployment, and streamlined data discovery. Organizations have multiple Hive data warehouses across EMR clusters, where the metadata gets generated.

Data Lake 95

Data Lake Metadata Data Warehouse Data Processing 95

CIO Business Intelligence

APRIL 26, 2024

billion acquisition of data and analytics company Neustar in 2021, TransUnion has expanded into other services such as marketing, fraud detection and prevention, and robust analytical services. At the core of its strategy is the mountain of data that TransUnion has acquired — along with more than 25 companies — over decades.

Modeling 114

Modeling IT Machine Learning Data Governance 114

Alation

JUNE 17, 2021

Leading companies like Cisco, Nielsen, and Finnair turn to Alation + Snowflake for data governance and analytics. By joining forces, we can build more potent, tailored solutions that leverage data governance as a competitive asset. Lastly, active data governance simplifies stewardship tasks of all kinds.

Data Governance 98

Data Governance Data Lake Insurance Enterprise 98

AWS Big Data

OCTOBER 10, 2023

Data governance is the process of ensuring the integrity, availability, usability, and security of an organization’s data. Due to the volume, velocity, and variety of data being ingested in data lakes, it can get challenging to develop and maintain policies and procedures to ensure data governance at scale for your data lake.

Data Quality 103

Data Quality Data Governance Data Lake Testing 103

erwin

DECEMBER 13, 2018

This past year witnessed a data governance awakening – or as the Wall Street Journal called it, a “global data governance reckoning.” There was tremendous data drama and resulting trauma – from Facebook to Equifax and from Yahoo to Marriott. So what’s on the horizon for data governance in the year ahead?

Data Governance 91

Data Governance IoT Internet of Things Data-driven 91

David Menninger's Analyst Perspectives

FEBRUARY 21, 2020

Ventana Research recently announced its 2020 research agenda for data, continuing the guidance we’ve offered for nearly two decades to help organizations derive optimal value and improve business outcomes. Data volumes continue to grow while data latency requirements continue to shrink.

Data Governance 263

Data Governance Optimization Data Lake IT 263

AWS Big Data

SEPTEMBER 26, 2024

In this blog post, there are three personas: Data Lake Administrator (with admin level access) User Silver from the Data Engineering group User Lead Auditor from the Auditor group. You will see how different personas in an organization can access the data without the need to modify their existing enterprise entitlements.

Data Governance 104

Data Governance Enterprise Management Data Lake 104

Data Virtualization

JUNE 16, 2022

Reading Time: 6 minutes Data Governance as a concept and practice has been around for as long as data management has been around. It, however is gaining prominence and interest in recent years due to the increasing volume of data that needs to be.

Data Governance 98

Data Governance Data Integration Management Data Lake 98

Jet Global

JULY 2, 2019

To address the flood of data and the needs of enterprise businesses to store, sort, and analyze that data, a new storage solution has evolved: the data lake. What’s in a Data Lake? Remember that data stream/river analogy earlier? Remember that data stream/river analogy earlier? Taking a Dip.

Data Lake 73

Data Lake Data Warehouse Big Data Machine Learning 73

CIO Business Intelligence

MAY 10, 2023

It was also a lot of churning for the different groups to come up with those data on the weekly, monthly and quarterly basis.” So from the business side, there was a lot of inefficiency getting data to the point where it was presentable to different audiences, which presented in its own right a big business problem for Betadam.

Data-driven 120

Data-driven IT Digital Transformation Data Governance 120

Cloudera

AUGUST 4, 2021

With hackers now working overtime to expose business data or implant ransomware processes, data security is largely IT managers’ top priority. And if data security tops IT concerns, data governance should be their second priority. Effective data governance must extend beyond the IT organization.

Data Governance 86

Data Governance Metadata Data-driven Enterprise 86

CIO Business Intelligence

DECEMBER 21, 2022

Every day, organizations of every description are deluged with data from a variety of sources, and attempting to make sense of it all can be overwhelming. By 2025, it’s estimated we’ll have 463 million terabytes of data created every day,” says Lisa Thee, data for good sector lead at Launch Consulting Group in Seattle.

IT 131

IT Business Intelligence Sales Key Performance Indicator 131

AWS Big Data

JULY 21, 2023

Data-driven organizations treat data as an asset and use it across different lines of business (LOBs) to drive timely insights and better business decisions. This leads to having data across many instances of data warehouses and data lakes using a modern data architecture in separate AWS accounts.

Data Lake 92

Data Lake Data Warehouse Marketing Management 92

AWS Big Data

APRIL 24, 2023

Building a data lake on Amazon Simple Storage Service (Amazon S3) provides numerous benefits for an organization. However, many use cases, like performing change data capture (CDC) from an upstream relational database to an Amazon S3-based data lake, require handling data at a record level.

Data Lake 87

Data Lake Data Governance Machine Learning Cost-Benefit 87

CIO Business Intelligence

SEPTEMBER 20, 2023

The data can also help us enrich our commodity products. How are you populating your data lake? We’ve decided to take a practical approach, led by Kyle Benning, who runs our data function. Then our analytics team, an IT group, makes sure we build the data lake in the right sequence.

Data Lake 135

Data Lake Data-driven Dashboards Risk 135

AWS Big Data

DECEMBER 12, 2024

The combination of these three services provides a powerful, comprehensive solution for end-to-end data lineage analysis. In this post, we use dbt for data modeling on both Amazon Athena and Amazon Redshift. The lineage data generated by dbt on Amazon Redshift includes partial lineage diagrams, as illustrated in the following image.

Snapshot 78

Snapshot Recreation/Entertainment Experimentation Data Lake 78

AWS Big Data

DECEMBER 4, 2024

In today’s rapidly evolving financial landscape, data is the bedrock of innovation, enhancing customer and employee experiences and securing a competitive edge. Like many large financial institutions, ANZ Institutional Division operated with siloed data practices and centralized data management teams.

Metadata 93

Metadata Data Governance Data Quality Data-driven 93

IBM Big Data Hub

JULY 5, 2023

Data Lakes have been around for well over a decade now, supporting the analytic operations of some of the largest world corporations. Such data volumes are not easy to move, migrate or modernize. The challenges of a monolithic data lake architecture Data lakes are, at a high level, single repositories of data at scale.

Data Lake 60

Data Lake Metadata Cost-Benefit Data Warehouse 60

CIO Business Intelligence

OCTOBER 2, 2024

And the other is retrieval augmented generation (RAG) models, where pieces of data from a larger source are vectorized to allow users to “talk” to the data. Data management, when done poorly, results in both diminished returns and extra costs. Having automated and scalable data checks is key.”

Management 141

Management Data Governance Cost-Benefit Structured Data 141

AWS Big Data

JULY 29, 2024

In the era of big data, data lakes have emerged as a cornerstone for storing vast amounts of raw data in its native format. They support structured, semi-structured, and unstructured data, offering a flexible and scalable environment for data ingestion from multiple sources.

Metadata 118

Metadata Snapshot Data Lake Metrics 118

Cloudera

OCTOBER 23, 2024

One of the most important innovations in data management is open table formats, specifically Apache Iceberg , which fundamentally transforms the way data teams manage operational metadata in the data lake. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 84

Metadata Data Lake Dashboards Interactive 84