AWS Big Data

APRIL 28, 2023

The first post of this series describes the overall architecture and how Novo Nordisk built a decentralized data mesh architecture, including Amazon Athena as the data query engine. The third post will show how end-users can consume data from their tool of choice, without compromising data governance.

Data Governance 72

Data Governance Management Data-driven Analytics 72

AWS Big Data

AUGUST 16, 2023

Iceberg has become very popular for its support for ACID transactions in data lakes and features like schema and partition evolution, time travel, and rollback. Solution overview For our example use case, a customer uses Amazon EMR for data processing and Iceberg format for the transactional data. Choose Create.

Data Lake 98

Data Lake Metadata Snapshot Management 98

AWS Big Data

AUGUST 13, 2024

With AWS Glue, you can discover and connect to hundreds of diverse data sources and manage your data in a centralized data catalog. It enables you to visually create, run, and monitor extract, transform, and load (ETL) pipelines to load data into your data lakes. Choose Store a new secret.

Data Processing 113

Data Processing Visualization Data Lake Data Processing 113

Cloudera

NOVEMBER 18, 2020

The technological linchpin of its digital transformation has been its Enterprise Data Architecture & Governance platform. It hosts over 150 big data analytics sandboxes across the region with over 200 users utilizing the sandbox for data discovery.

Internet Publishing and Broadcasting 123

Internet Publishing and Broadcasting Data-driven Broadcasting Digital Transformation 123

CIO Business Intelligence

MARCH 10, 2022

Next up: AI and data lake decisions. To that end, UAB’s next step is to tackle big decisions around expanding its AI and data analytics platforms, says Carver, who is not handling the long-term planning alone. Carver says UAB is “leaning toward” using ServiceNow but is also considering Salesforce for its digital marketplace.

IT 98

IT Data Lake Digital Transformation Data Governance 98

AWS Big Data

AUGUST 19, 2024

This involves creating VPC endpoints in both the AWS and Snowflake VPCs, making sure data transfer remains within the AWS network. Use Amazon Route 53 to create a private hosted zone that resolves the Snowflake endpoint within your VPC. This unlocks scalable analytics while maintaining data governance, compliance, and access control.

Analytics 119

Analytics Data-driven Data Integration Data Lake 119

Cloudera

SEPTEMBER 6, 2022

Previously, there were three types of data structures in telco: . Entity data sets — i.e. marketing data lakes . There are three major architectures under the modern data architecture umbrella. . Application-based datasets — i.e. billing or contact center support systems .

Data Architecture 84

Data Architecture Cost-Benefit Digital Transformation Business Driver 84

AWS Big Data

MARCH 26, 2024

In this post, we discuss how you can use purpose-built AWS services to create an end-to-end data strategy for C360 to unify and govern customer data that address these challenges. The AWS modern data architecture shows a way to build a purpose-built, secure, and scalable data platform in the cloud.

Data Strategy 94

Data Strategy Strategy Data Warehouse Prescriptive Analytics 94

Cloudera

MAY 12, 2021

2020 saw us hosting our first ever fully digital Data Impact Awards ceremony, and it certainly was one of the highlights of our year. We saw a record number of entries and incredible examples of how customers were using Cloudera’s platform and services to unlock the power of data.

Digital Transformation 70

Digital Transformation Machine Learning Optimization Data Lake 70

AWS Big Data

JANUARY 22, 2024

Discussions with users showed they were happier to have faster access to data in a simpler way, a more structured data organization, and a clear mapping of who the producer is. A lot of progress has been made to advance their data-driven culture (data literacy, data sharing, and collaboration across business units).

Data-driven 116

Data-driven Advertising Metadata Data Architecture 116

CIO Business Intelligence

AUGUST 14, 2024

Start where your data is Using your own enterprise data is the major differentiator from open access gen AI chat tools, so it makes sense to start with the provider already hosting your enterprise data. It’s the contextual information supporting the use of these tools,” Curran says.

Enterprise 136

Enterprise Cost-Benefit Experimentation Modeling 136

Cloudera

NOVEMBER 12, 2020

For example, the bank from our example might have separate destination data lakes for their perpetual and periodic workloads to support addressing these VIP workloads separately. Alternatively, you can also spin up a different compute cluster and access the data by using CDP’s Shared Data Experience.

Data Warehouse 104

Data Warehouse Reporting Risk Cost-Benefit 104

Alation

NOVEMBER 28, 2022

It is also hard to know whether one can trust the data within a spreadsheet. And they rarely, if ever, host the most current data available. Sathish Raju, cofounder & CTO, Kloudio and senior director of engineering, Alation: This presents challenges for both business users and data teams.

Metadata 52

Metadata Enterprise Cost-Benefit Finance 52

Domino Data Lab

APRIL 3, 2019

Paco Nathan ‘s latest column dives into data governance. This month’s article features updates from one of the early data conferences of the year, Strata Data Conference – which was held just last week in San Francisco. In particular, here’s my Strata SF talk “Overview of Data Governance” presented in article form.

Machine Learning 84

Machine Learning Data Governance Metadata Data Science 84

AWS Big Data

OCTOBER 30, 2024

Amazon DataZone now launched authentication supports through the Amazon Athena JDBC driver, allowing data users to seamlessly query their subscribed data lake assets via popular business intelligence (BI) and analytics tools like Tableau, Power BI, Excel, SQL Workbench, DBeaver, and more.

Visualization 119

Visualization Data Lake Testing Data Governance 119

AWS Big Data

OCTOBER 21, 2024

However, enterprises often encounter challenges with data silos, insufficient access controls, poor governance, and quality issues. Embracing data as a product is the key to address these challenges and foster a data-driven culture. To incorporate this third-party data, AWS Data Exchange is the logical choice.

Sales 115

Sales Data-driven Data Processing Key Performance Indicator 115

AWS Big Data

DECEMBER 19, 2024

Each service is hosted in a dedicated AWS account and is built and maintained by a product owner and a development team, as illustrated in the following figure. Implementing robust data governance is challenging. In a data mesh architecture, this complexity is amplified by the organizations decentralized nature.

Data Governance 52

Data Governance Publishing Data-driven Metadata 52

Alation

FEBRUARY 17, 2023

This past week, I had the pleasure of hosting Data Governance for Dummies author Jonathan Reichental for a fireside chat , along with Denise Swanson , Data Governance lead at Alation. Can you have proper data management without establishing a formal data governance program?

Data Governance 52

Data Governance Data Quality Metadata Cost-Benefit 52

AWS Big Data

DECEMBER 4, 2024

Under the federated mesh architecture, each divisional mesh functions as a node within the broader enterprise data mesh, maintaining a degree of autonomy in managing its data products. These nodes can implement analytical platforms like data lake houses, data warehouses, or data marts, all united by producing data products.

Metadata 105

Metadata Data Governance Data Quality Data-driven 105

AWS Big Data

FEBRUARY 27, 2024

Each data producer within the organization has its own data lake in Apache Hudi format, ensuring data sovereignty and autonomy. This enables data-driven decision-making across the organization. Lake Formation – Lake Formation emerged as a cornerstone in Bluestone’s data governance strategy.

Data-driven 111

Data-driven Data Lake Data Quality Data Governance 111

AWS Big Data

DECEMBER 21, 2023

To bring their customers the best deals and user experience, smava follows the modern data architecture principles with a data lake as a scalable, durable data store and purpose-built data stores for analytical processing and data consumption.

Data Lake 115

Data Lake Data Warehouse Data-driven B2B 115

AWS Big Data

JULY 14, 2023

In this post, we discuss how the Amazon Finance Automation team used AWS Lake Formation and the AWS Glue Data Catalog to build a data mesh architecture that simplified data governance at scale and provided seamless data access for analytics, AI, and machine learning (ML) use cases.

Finance 98

Finance Metadata Big Data Recreation/Entertainment 98

Cloudera

OCTOBER 7, 2021

Those decentralization efforts appeared under different monikers through time, e.g., data marts versus data warehousing implementations (a popular architectural debate in the era of structured data) then enterprise-wide data lakes versus smaller, typically BU-Specific, “data ponds”.

Metadata 124

Metadata Cost-Benefit Enterprise Interactive 124

IBM Big Data Hub

JUNE 15, 2023

It is comprised of commodity cloud object storage, open data and open table formats, and high-performance open-source query engines. To help organizations scale AI workloads, we recently announced IBM watsonx.data , a data store built on an open data lakehouse architecture and part of the watsonx AI and data platform.

Data Warehouse 86

Data Warehouse Data Lake Optimization Data-driven 86

Andrew White

JANUARY 11, 2021

On January 4th I had the pleasure of hosting a webinar. It was titled, The Gartner 2021 Leadership Vision for Data & Analytics Leaders. This was for the Chief Data Officer, or head of data and analytics. Does Data warehouse as a software tool will play role in future of Data & Analytics strategy?

Data Analytics 52

Data Analytics Analytics Data-driven Finance 52

CIO Business Intelligence

JANUARY 20, 2023

Of course, cost is a big consideration, says Orlandini, as well as deciding where to host the data, and having it available in a fiscally responsible way. An organization might also question if the data should be maintained on-premises due to security concerns in the public cloud. They have data swamps,” he says.

Data Lake 135

Data Lake Data-driven Finance Data Architecture 135

AWS Big Data

MARCH 13, 2025

Open source Pinot requires in-house expertise that can challenge well-established technical teams to provision hardware, configure environments, tune performance, maintain security, adhere to data governance requirements, manage software updates, and constantly monitor for system issues.

Management 75

Management Analytics OLAP Online Analytical Processing 75

CIO Business Intelligence

FEBRUARY 18, 2025

If we revisit our durable goods industry example and consider prioritizing data quality through aggregation in a multi-tier architecture and cloud data platform first, we can achieve the prerequisite needed to build data quality and data trust first. Agentic AI is here to stay and will gain tremendous momentum in 2024.

Machine Learning 90

Machine Learning Data Quality Enterprise Sales 90

Jet Global

FEBRUARY 23, 2024

An on-premise solution provides a high level of control and customization as it is hosted and managed within the organization’s physical infrastructure, but it can be expensive to set up and maintain. Next, identify the data sources that will be involved in the mapping.

Data Warehouse 52

Data Warehouse Reporting Data Transformation Visualization 52