Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI.

Metadata 84

Metadata Data Lake Dashboards Interactive 84

AWS Big Data

OCTOBER 30, 2024

Amazon DataZone now launched authentication supports through the Amazon Athena JDBC driver, allowing data users to seamlessly query their subscribed data lake assets via popular business intelligence (BI) and analytics tools like Tableau, Power BI, Excel, SQL Workbench, DBeaver, and more.

Visualization 119

Visualization Data Lake Testing Data Governance 119

AWS Big Data

NOVEMBER 7, 2024

Amazon Q generative SQL for Amazon Redshift uses generative AI to analyze user intent, query patterns, and schema metadata to identify common SQL query patterns directly within Amazon Redshift, accelerating the query authoring process for users and reducing the time required to derive actionable data insights. Choose Query data.

Metadata 105

Metadata Sales Data Warehouse Optimization 105

AWS Big Data

APRIL 24, 2025

Bob now knows that he can quickly build Amazon QuickSight dashboards with queries that are optimized using Redshifts cost-based optimizer. Ava defines the user attributes as static IAM tags that could also include attributes stored in the identity provider (IdP) or as session tags dynamically to represent the user metadata.

Sales 93

Sales Data Lake Management Data-driven 93

AWS Big Data

FEBRUARY 26, 2025

Analytics remained one of the key focus areas this year, with significant updates and innovations aimed at helping businesses harness their data more efficiently and accelerate insights. From enhancing data lakes to empowering AI-driven analytics, AWS unveiled new tools and services that are set to shape the future of data and analytics.

Analytics 62

Analytics Data Lake Metadata Data Warehouse 62

Jet Global

JULY 2, 2019

To address the flood of data and the needs of enterprise businesses to store, sort, and analyze that data, a new storage solution has evolved: the data lake. What’s in a Data Lake? Data warehouses do a great job of standardizing data from disparate sources for analysis. Taking a Dip.

Data Lake 73

Data Lake Data Warehouse Big Data Machine Learning 73

DataKitchen

AUGUST 9, 2021

In essence, a domain is an integrated data set and a set of views, reports, dashboards, and artifacts created from the data. The domain also includes code that acts upon the data, including tools, pipelines, and other artifacts that drive analytics execution.

Testing 246

Testing Data Lake Metadata Publishing 246

AWS Big Data

MARCH 7, 2024

At the same time, they need to optimize operational costs to unlock the value of this data for timely insights and do so with a consistent performance. With this massive data growth, data proliferation across your data stores, data warehouse, and data lakes can become equally challenging.

Data Lake 131

Data Lake Analytics Dashboards Metrics 131

DataKitchen

OCTOBER 11, 2021

She applies some calculations and forwards the file to a data engineer who loads the data into a database and runs a Talend job that performs ETL to dimensionalize the data and produce a Data Mart. The data engineer then emails the BI Team, who refreshes a Tableau dashboard. Monitoring Job Metadata.

Testing 154

Testing Metadata Dashboards Statistics 154

CIO Business Intelligence

APRIL 8, 2025

We also examine how centralized, hybrid and decentralized data architectures support scalable, trustworthy ecosystems. As data-centric AI, automated metadata management and privacy-aware data sharing mature, the opportunity to embed data quality into the enterprises core has never been more significant.

Data Quality 102

Data Quality Data-driven Key Performance Indicator Metadata 102

AWS Big Data

JUNE 5, 2024

The integration is new way for customers to query operational logs in Amazon S3 and Amazon S3-based data lakes without needing to switch between tools to analyze operational data. Amazon S3 is an object storage service offering industry-leading scalability, data availability, security, and performance.

Data Lake 122

Data Lake Dashboards Cost-Benefit Visualization 122

AWS Big Data

FEBRUARY 22, 2023

In this post, we show how Ruparupa implemented an incrementally updated data lake to get insights into their business using Amazon Simple Storage Service (Amazon S3), AWS Glue , Apache Hudi , and Amazon QuickSight. An AWS Glue ETL job, using the Apache Hudi connector, updates the S3 data lake hourly with incremental data.

Data Lake 52

Data Lake Dashboards Cost-Benefit Data Warehouse 52

AWS Big Data

MAY 7, 2024

It also makes it easier for engineers, data scientists, product managers, analysts, and business users to access data throughout an organization to discover, use, and collaborate to derive data-driven insights. Note that a managed data asset is an asset for which Amazon DataZone can manage permissions.

Metadata 109

Metadata Data Lake Data Processing Data-driven 109

AWS Big Data

NOVEMBER 29, 2023

Zero-ETL integration also enables you to load and analyze data from multiple operational database clusters in a new or existing Amazon Redshift instance to derive holistic insights across many applications. Use one click to access your data lake tables using auto-mounted AWS Glue data catalogs on Amazon Redshift for a simplified experience.

Data Warehouse 130

Data Warehouse Analytics Data Lake Machine Learning 130

AWS Big Data

OCTOBER 4, 2023

An Amazon DataZone domain contains an associated business data catalog for search and discovery, a set of metadata definitions to decorate the data assets that are used for discovery purposes, and data projects with integrated analytics and ML tools for users and groups to consume and publish data assets.

Metadata 119

Metadata Data Lake Publishing Data Governance 119

AWS Big Data

MARCH 29, 2024

Grafana provides powerful customizable dashboards to view pipeline health. QuickSight makes it straightforward for business users to visualize data in interactive dashboards and reports. An AWS Glue crawler scans data on the S3 bucket and populates table metadata on the AWS Glue Data Catalog.

Metrics 128

Metrics Visualization Dashboards Publishing 128

AWS Big Data

AUGUST 5, 2024

These business units have varying landscapes, where a data lake is managed by Amazon Simple Storage Service (Amazon S3) and analytics workloads are run on Amazon Redshift , a fast, scalable, and fully managed cloud data warehouse that allows you to process and run your complex SQL analytics workloads on structured and semi-structured data.

Data Lake 105

Data Lake Finance Sales Management 105

AWS Big Data

MARCH 7, 2023

A data hub contains data at multiple levels of granularity and is often not integrated. It differs from a data lake by offering data that is pre-validated and standardized, allowing for simpler consumption by users. Data hubs and data lakes can coexist in an organization, complementing each other.

Analytics 97

Analytics Data Warehouse Data Lake Metadata 97

AWS Big Data

DECEMBER 5, 2023

HR&A has used Amazon Redshift Serverless and CARTO to process survey findings more efficiently and create custom interactive dashboards to facilitate understanding of the results. A combination of Amazon Redshift Spectrum and COPY commands are used to ingest the survey data stored as CSV files.

Measurement 105

Measurement Dashboards Data Warehouse Analytics 105

CIO Business Intelligence

DECEMBER 21, 2022

“The number-one issue for our BI team is convincing people that business intelligence will help to make true data-driven decisions,” says Diana Stout, senior business analyst at Schellman, a global cybersecurity assessor based in Tampa, Fl. But what they really need to do is fundamentally rethink how data is managed and accessed,” he says.

IT 131

IT Business Intelligence Sales Key Performance Indicator 131

AWS Big Data

MAY 1, 2024

To achieve data-driven management, we built OneData, a data utilization platform used in the four global AWS Regions, which started operation in April 2022. The platform consists of approximately 370 dashboards, 360 tables registered in the data catalog, and 40 linked systems. Promote and expand the use of databases.

Dashboards 124

Dashboards Publishing Data-driven Cost-Benefit 124

Cloudera

JANUARY 21, 2021

With CDW, as an integrated service of CDP, your line of business gets immediate resources needed for faster application launches and expedited data access, all while protecting the company’s multi-year investment in centralized data management, security, and governance. Proprietary file formats mean no one else is invited in!

Data Warehouse 94

Data Warehouse Data Lake IT Analytics 94

Cloudera

JUNE 30, 2022

With Cloudera’s vision of hybrid data , enterprises adopting an open data lakehouse can easily get application interoperability and portability to and from on premises environments and any public cloud without worrying about data scaling. Why integrate Apache Iceberg with Cloudera Data Platform?

Data Lake 88

Data Lake Data Warehouse Data Architecture Metadata 88

AWS Big Data

APRIL 6, 2023

In this post, Morningstar’s Data Lake Team Leads discuss how they utilized tag-based access control in their data lake with AWS Lake Formation and enabled similar controls in Amazon Redshift. This way, our existing data lake consumers could easily transition to Amazon Redshift.

Data Warehouse 98

Data Warehouse Data Lake Management Data-driven 98

Cloudera

OCTOBER 11, 2021

The platform converges data cataloging, data ingestion, data profiling, data tagging, data discovery, and data exploration into a unified platform, driven by metadata. Modak Nabu automates repetitive tasks in the data preparation process and thus accelerates the data preparation by 4x.

Data Lake 95

Data Lake Cost-Benefit Data-driven Dashboards 95

AWS Big Data

MARCH 3, 2023

Building data lakes from continuously changing transactional data of databases and keeping data lakes up to date is a complex task and can be an operational challenge. You can then apply transformations and store data in Delta format for managing inserts, updates, and deletes.

Data Lake 98

Data Lake Dashboards Metrics Metadata 98

Cloudera

JUNE 16, 2022

CSP was recently recognized as a leader in the 2022 GigaOm Radar for Streaming Data Platforms report. The DevOps/app dev team wants to know how data flows between such entities and understand the key performance metrics (KPMs) of these entities. Without context, streaming data is useless.”

Data Lake 88

Data Lake Manufacturing Metadata Dashboards 88

AWS Big Data

DECEMBER 18, 2023

Amazon Redshift powers data-driven decisions for tens of thousands of customers every day with a fully managed, AI-powered cloud data warehouse, delivering the best price-performance for your analytics workloads.

Data Warehouse 111

Data Warehouse Machine Learning Data-driven Data Lake 111

AWS Big Data

MARCH 26, 2024

Profile aggregation – When you’ve uniquely identified a customer, you can build applications in Managed Service for Apache Flink to consolidate all their metadata, from name to interaction history. Then, you transform this data into a concise format. The following diagram shows a sample C360 dashboard built on Amazon QuickSight.

Data Strategy 103

Data Strategy Strategy Data Warehouse Prescriptive Analytics 103

Andrew White

DECEMBER 9, 2019

The client had recently engaged with a well-known consulting company that had recommended a large data catalog effort to collect all enterprise metadata to help identify all data and business issues. Modern data (and analytics) governance does not necessarily need: Wall-to-wall discovery of your data and metadata.

Analytics 66

Analytics Data Lake Data Governance Data Warehouse 66

AWS Big Data

NOVEMBER 29, 2023

You also need services to store data for analysis and machine learning (ML) like Amazon Simple Storage Service (Amazon S3). Customers have created hundreds of thousands of data lakes on Amazon S3. Amazon DataZone uses ML to automatically add metadata to your data catalog, making all of your data more discoverable.

Data Warehouse 116

Data Warehouse Data Lake Dashboards Data Integration 116

AWS Big Data

NOVEMBER 16, 2023

Amazon Redshift is a popular cloud data warehouse, offering a fully managed cloud-based service that seamlessly integrates with an organization’s Amazon Simple Storage Service (Amazon S3) data lake, real-time streams, machine learning (ML) workflows, transactional workflows, and much more—all while providing up to 7.9x

Enterprise 111

Enterprise Data Warehouse Data Lake Optimization 111

AWS Big Data

MARCH 27, 2023

Developers, data scientists, and analysts can work across databases, data warehouses, and data lakes to build reporting and dashboarding applications, perform real-time analytics, share and collaborate on data, and even build and train machine learning (ML) models with Redshift Serverless.

Data Warehouse 98

Data Warehouse Management Snapshot Data Lake 98

AWS Big Data

JULY 14, 2023

The FinAuto team built AWS Cloud Development Kit (AWS CDK), AWS CloudFormation , and API tools to maintain a metadata store that ingests from domain owner catalogs into the global catalog. This global catalog captures new or updated partitions from the data producer AWS Glue Data Catalogs.

Finance 98

Finance Metadata Big Data Recreation/Entertainment 98

AWS Big Data

JANUARY 8, 2024

In this post, we will review the common architectural patterns of two use cases: Time Series Data Analysis and Event Driven Microservices. All these architecture patterns are integrated with Amazon Kinesis Data Streams. The raw data can be streamed to Amazon S3 for archiving.

Analytics 134

Analytics IoT Data-driven Snapshot 134

AWS Big Data

AUGUST 6, 2024

This has led to the emergence of real-time OLAP solutions, which are particularly relevant in the following use cases: User-facing analytics – Incorporating analytics into products or applications that consumers use to gain insights, sometimes referred to as data products. Anomaly detection – Identifying outliers or unusual behavior patterns.

OLAP 111

OLAP Analytics Visualization Dashboards 111

AWS Big Data

MARCH 25, 2024

Streaming jobs constantly ingest new data to synchronize across systems and can perform enrichment, transformations, joins, and aggregations across windows of time more efficiently. OpenSearch Service offers visualization capabilities powered by OpenSearch Dashboards and Kibana (1.5

Data Lake 122

Data Lake Unstructured Data Management Snapshot 122