AWS Big Data

JANUARY 18, 2024

AWS Lake Formation and the AWS Glue Data Catalog form an integral part of a data governance solution for data lakes built on Amazon Simple Storage Service (Amazon S3) with multiple AWS analytics services integrating with them. In 2022 , we talked about the enhancements we had done to these services.

Data Lake 119

Data Lake Metadata Data Governance Statistics 119

AWS Big Data

JULY 21, 2023

Data-driven organizations treat data as an asset and use it across different lines of business (LOBs) to drive timely insights and better business decisions. This leads to having data across many instances of data warehouses and data lakes using a modern data architecture in separate AWS accounts.

Data Lake 98

Data Lake Data Warehouse Marketing Management 98

Cloudera

DECEMBER 16, 2022

We are pleased to announce that Cloudera has been named a Leader in the 2022 Gartner ® Magic Quadrant for Cloud Database Management Systems. Cloudera has long had the capabilities of a data lakehouse, if not the label. 4-Ready for modern data fabric architectures. 4-Ready for modern data fabric architectures.

Management 92

Management Metadata Machine Learning Data Lake 92

AWS Big Data

JULY 3, 2023

Terminology Let’s first discuss some of the terminology used in this post: Research data lake on Amazon S3 – A data lake is a large, centralized repository that allows you to manage all your structured and unstructured data at any scale. This is where the tagging feature in Apache Iceberg comes in handy.

Snapshot 105

Snapshot Data Lake Testing Strategy 105

CIO Business Intelligence

NOVEMBER 4, 2022

MetaBio, which received a 2022 CIO 100 Award , provides a single source for datasets in a unified format, enabling researchers to quickly extract information about various therapeutic functions without having to worry about how to prepare or find the data. Much of Regeneron’s data, of course, is confidential.

Data Lake 124

Data Lake IT Experimentation Data-driven 124

AWS Big Data

APRIL 19, 2023

We split the solution into two primary components: generating Spark job metadata and running the SQL on Amazon EMR. The first component (metadata setup) consumes existing Hive job configurations and generates metadata such as number of parameters, number of actions (steps), and file formats. sql_path SQL file name.

Metadata 98

Metadata Data Lake Testing Consulting 98

CIO Business Intelligence

JUNE 23, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. On data warehouses and data lakes.

Data Lake 98

Data Lake Data Warehouse Machine Learning Data-driven 98

CIO Business Intelligence

OCTOBER 13, 2023

Data architect Armando Vázquez identifies eight common types of data architects: Enterprise data architect: These data architects oversee an organization’s overall data architecture, defining data architecture strategy and designing and implementing architectures. Are data architects in demand?

Data Architecture 132

Data Architecture Data Warehouse Statistics Visualization 132

Cloudera

JUNE 18, 2022

Cloudera customers run some of the biggest data lakes on earth. These lakes power mission critical large scale data analytics, business intelligence (BI), and machine learning use cases, including enterprise data warehouses. On data warehouses and data lakes.

Data Lake 94

Data Lake Data Warehouse Machine Learning Data-driven 94

IBM Big Data Hub

OCTOBER 4, 2023

Why does AI need an open data lakehouse architecture? from 2022 to 2026. Another IDC study showed that while 2/3 of respondents reported using AI-driven data analytics, most reported that less than half of the data under management is available for this type of analytics. All of this supports the use of AI.

Data Lake 77

Data Lake Metadata Data Warehouse Cost-Benefit 77

CIO Business Intelligence

SEPTEMBER 12, 2023

The hype around generative AI since ChatGPT’s launch in November 2022 has driven some software vendors to rush to incorporate the technology into their applications. To that end, Salesforce is leveraging Data Cloud as a central data hub for enterprise implementations of Einstein Copilot.

IT 116

IT Metadata Data Lake Cost-Benefit 116

Cloudera

JUNE 30, 2022

With Cloudera’s vision of hybrid data , enterprises adopting an open data lakehouse can easily get application interoperability and portability to and from on premises environments and any public cloud without worrying about data scaling. Why integrate Apache Iceberg with Cloudera Data Platform?

Data Lake 88

Data Lake Data Warehouse Data Architecture Metadata 88

Alation

JULY 18, 2022

Are your data users overwhelmed by silos and frustrated by untrusted data? Tell them to grab a catalog … and go jump in a lake. That was the message — delivered a little more elegantly than that — at Databricks’ Data+AI Summit 2022. Learn more about our partner network.

ROI 52

ROI Metadata Data Lake Digital Transformation 52

Cloudera

NOVEMBER 17, 2022

The Solution: CDP Private Cloud brings a next-generation hybrid architecture with cloud-native benefits to HBL’s data platform. HBL started their data journey in 2019 when data lake initiative was started to consolidate complex data sources and enable the bank to use single version of truth for decision making.

Management 85

Management Data Lake Consulting Unstructured Data 85

AWS Big Data

NOVEMBER 29, 2023

We have seen a strong customer demand to expand its scope to cloud-based data lakes because data lakes are increasingly the enterprise solution for large-scale data initiatives due to their power and capabilities. Let’s say that this company is located in Europe and the data product must comply with the GDPR.

Data Lake 122

Data Lake Management Metrics Data Warehouse 122

AWS Big Data

DECEMBER 21, 2023

To bring their customers the best deals and user experience, smava follows the modern data architecture principles with a data lake as a scalable, durable data store and purpose-built data stores for analytical processing and data consumption.

Data Lake 115

Data Lake Data Warehouse Data-driven B2B 115

IBM Big Data Hub

APRIL 27, 2022

Gartner defines a data fabric as “a design concept that serves as an integrated layer of data and connecting processes. The data fabric architectural approach can simplify data access in an organization and facilitate self-service data consumption at scale. 2 “Exposing The Data Mesh Blind Side ” Forrester.

Management 79

Management Metadata Data Architecture Data Lake 79

Cloudera

SEPTEMBER 9, 2022

This is the promise of the modern data lakehouse architecture. analyst Sumit Pal, in “Exploring Lakehouse Architecture and Use Cases,” published January 11, 2022: “Data lakehouses integrate and unify the capabilities of data warehouses and data lakes, aiming to support AI, BI, ML, and data engineering on a single platform.”

Metadata 102

Metadata Machine Learning Unstructured Data Data Lake 102

AWS Big Data

MAY 1, 2024

To achieve data-driven management, we built OneData, a data utilization platform used in the four global AWS Regions, which started operation in April 2022. The platform consists of approximately 370 dashboards, 360 tables registered in the data catalog, and 40 linked systems.

Dashboards 124

Dashboards Publishing Data-driven Cost-Benefit 124

Alation

JANUARY 13, 2022

Thoughtworks says data mesh is key to moving beyond a monolithic data lake. Spoiler alert: data fabric and data mesh are independent design concepts that are, in fact, quite complementary. Thoughtworks says data mesh is key to moving beyond a monolithic data lake 2. Gartner on Data Fabric.

Data Lake 59

Data Lake Metadata Data-driven Data Governance 59

Cloudera

JUNE 16, 2022

CSP was recently recognized as a leader in the 2022 GigaOm Radar for Streaming Data Platforms report. The DevOps/app dev team wants to know how data flows between such entities and understand the key performance metrics (KPMs) of these entities. Without context, streaming data is useless.”

Data Lake 88

Data Lake Manufacturing Metadata Dashboards 88

AWS Big Data

MARCH 26, 2024

This view is used to identify patterns and trends in customer behavior, which can inform data-driven decisions to improve business outcomes. In 2022, AWS commissioned a study conducted by the American Productivity and Quality Center (APQC) to quantify the Business Value of Customer 360.

Data Strategy 103

Data Strategy Strategy Data Warehouse Prescriptive Analytics 103

Cloudera

APRIL 3, 2023

The new feature this blog post is aiming to discuss about Iceberg V2 format (version 2), as the Iceberg table specification explains, the V1 format aimed to support large analytic data tables, while V2 aimed to add row level deletes and updates. CDW also simplifies administration by making multi-tenancy secure and manageable.

Data Warehouse 82

Data Warehouse Snapshot Metadata Cost-Benefit 82

Cloudera

SEPTEMBER 15, 2022

July brings summer vacations, holiday gatherings, and for the first time in two years, the return of the Massachusetts Institute of Technology (MIT) Chief Data Officer symposium as an in-person event. A key area of focus for the symposium this year was the design and deployment of modern data platforms.

Data Lake 83

Data Lake Data Architecture Data-driven Data Warehouse 83

CIO Business Intelligence

SEPTEMBER 6, 2022

Very has come full circle as a business built on catalog data, but it took some introspection in order to figure out the best way to get there. Understanding what data you’ve got locked in all these different stores is a big part of the jigsaw puzzle.”.

IT 98

IT Forecasting Data Lake Data Warehouse 98

AWS Big Data

MARCH 3, 2023

Building data lakes from continuously changing transactional data of databases and keeping data lakes up to date is a complex task and can be an operational challenge. You can then apply transformations and store data in Delta format for managing inserts, updates, and deletes.

Data Lake 98

Data Lake Dashboards Metrics Metadata 98

Cloudera

AUGUST 4, 2021

A top-notch system will include an easy-to-navigate data catalog that provides a single-pane view to administer and discover all data assets. The data is profiled and enhanced with rich metadata—including operational, social, and business context—creating trusted and reusable data assets and making them discoverable.

Data Governance 86

Data Governance Metadata Data-driven Enterprise 86

CIO Business Intelligence

JANUARY 30, 2024

As the internal technology provider for parent company Allianz SE with 15,000 employees, the entity employs more than 100 ESG experts who spend several weeks each year heads down collecting and reporting ESG data manually. Data quality is key, but if we’re doing it manually there’s the potential for mistakes.

Reporting 115

Reporting Data Quality Strategy Data-driven 115

IBM Big Data Hub

OCTOBER 12, 2023

Among the tasks necessary for internal and external compliance is the ability to report on the metadata of an AI model. Metadata includes details specific to an AI model such as: The AI model’s creation (when it was created, who created it, etc.) Learn more about IBM watsonx 1.

Cost-Benefit 76

Cost-Benefit Metadata Data Governance Modeling 76

Alation

AUGUST 11, 2022

Today, the brightest minds in our industry are targeting the massive proliferation of data volumes and the accompanying but hard-to-find value locked within all that data. I recently had the opportunity to connect with Mohan at Snowflake Summit 2022 in Las Vegas. Data fabric is a technology architecture.

Metadata 52

Metadata Data Warehouse Data Quality Data Lake 52

Alation

OCTOBER 19, 2022

Today, they have issued The Data Management Survey 23 , a report based on a survey of more than 1,200 data management end-users of 23 products (or groups of products). The survey was conducted from January to April 2022 and examined user feedback on product experience across 18 criteria.

Management 52

Management KPI Data Governance Reporting 52

Alation

JULY 7, 2022

In 2013 I joined American Family Insurance as a metadata analyst. I had always been fascinated by how people find, organize, and access information, so a metadata management role after school was a natural choice. The use cases for metadata are boundless, offering opportunities for innovation in every sector.

Metadata 52

Metadata Data-driven Insurance Statistics 52

AWS Big Data

APRIL 25, 2024

In the era of data, organizations are increasingly using data lakes to store and analyze vast amounts of structured and unstructured data. Data lakes provide a centralized repository for data from various sources, enabling organizations to unlock valuable insights and drive data-driven decision-making.

Optimization 72

Optimization Data Lake Cost-Benefit Reporting 72

Alation

NOVEMBER 28, 2022

You founded Kloudio to address the spreadsheet problem, and Alation acquired Kloudio in February of 2022. But refreshing this analysis with the latest data was impossible… unless you were proficient in SQL or Python. Each analyst has their own version of analysis in their spreadsheets due to differences in formulas and data used.

Metadata 52

Metadata Enterprise Cost-Benefit Finance 52

Jet Global

SEPTEMBER 19, 2023

In fact, a recent Gartner report on cloud expenditure found that cross-industry cloud spend has risen from 8% as a percentage of total IT spend in 2018 to 16% in 2022. What are the best practices for analyzing cloud ERP data? Data Management How do we create a data warehouse or data lake in the cloud using our cloud ERP?

Cost-Benefit 52

Cost-Benefit Data Warehouse Reporting Enterprise 52

CIO Business Intelligence

JANUARY 16, 2025

The mega-vendor era By 2020, the basis of competition for what are now referred to as mega-vendors was interoperability, automation and intra-ecosystem participation and unlocking access to data to drive business capabilities, value and manage risk.

Enterprise 105

Enterprise Digital Transformation Scorecard Interactive 105

AWS Big Data

NOVEMBER 14, 2024

Amazon Data Firehose – Data Firehose is an extract, transform, and load (ETL) service that reliably captures, transforms, and delivers streaming data to data lakes, data stores, and analytics services. AWS Glue – The AWS Glue Data Catalog is your persistent technical metadata store in the AWS Cloud.

Data Lake 105

Data Lake Metadata Testing Data-driven 105