DataKitchen

MAY 2, 2025

Is Your Team in Denial of Data Quality? Here’s How to Tell In many organizations, data quality problems fester in the shadowsignored, rationalized, or swept aside with confident-sounding statements that mask a deeper dysfunction. That doesn’t mean the data inside was correct. A pipeline ran “all green”?

Data Quality 126

Data Quality Snapshot Testing Data-driven 126

AWS Big Data

DECEMBER 4, 2024

This week on the keynote stages at AWS re:Invent 2024, you heard from Matt Garman, CEO, AWS, and Swami Sivasubramanian, VP of AI and Data, AWS, speak about the next generation of Amazon SageMaker , the center for all of your data, analytics, and AI. The relationship between analytics and AI is rapidly evolving.

Data Analytics 104

Data Analytics Analytics Data Lake Data Quality 104

Rocket-Powered Data Science

FEBRUARY 15, 2023

Third, any commitment to a disruptive technology (including data-intensive and AI implementations) must start with a business strategy. I suggest that the simplest business strategy starts with answering three basic questions: What? That is: (1) What is it you want to do and where does it fit within the context of your organization?

Strategy 290

Strategy Experimentation Uncertainty Machine Learning 290

AWS Big Data

OCTOBER 14, 2024

Enterprise data is brought into data lakes and data warehouses to carry out analytical, reporting, and data science use cases using AWS analytical services like Amazon Athena , Amazon Redshift , Amazon EMR , and so on. Can it also help write SQL queries? The answer is yes. The solution architecture and workflow.

Metadata 105

Metadata Data Lake Modeling Data Warehouse 105

TDAN

MARCH 16, 2021

No matter where data comes from, becoming data-driven depends on every member of your organization being able to find, access, and use the data they need.

Business Objectives 87

Business Objectives Data-driven IT Metadata 87

O'Reilly on Data

MARCH 23, 2023

What is it, how does it work, what can it do, and what are the risks of using it? Many of these are unsurprising: you can ask it to write a letter, you can ask it to make up a story, you can ask it to write descriptive entries for products in a catalog. What Software Are We Talking About? It’s much more.

IT 346

IT Modeling Testing Risk 346

DataKitchen

AUGUST 3, 2021

The data mesh design pattern breaks giant, monolithic enterprise data architectures into subsystems or domains, each managed by a dedicated team. DataOps helps the data mesh deliver greater business agility by enabling decentralized domains to work in concert. . But first, let’s define the data mesh design pattern.

Data Architecture 246

Data Architecture Data Lake Cost-Benefit Data Warehouse 246

DataKitchen

APRIL 13, 2021

Read the complete blog below for a more detailed description of the vendors and their capabilities. This is not surprising given that DataOps enables enterprise data teams to generate significant business value from their data. Testing and Data Observability. Download the 2021 DataOps Vendor Landscape here.

Testing 304

Testing Machine Learning Consulting Data Science 304

DataKitchen

APRIL 28, 2021

Industry analysts who follow the data and analytics industry tell DataKitchen that they are receiving inquiries about “data fabrics” from enterprise clients on a near-daily basis. Gartner included data fabrics in their top ten trends for data and analytics in 2019. What is a Data Fabric?

Statistics 246

Statistics Optimization Data Analytics Technology 246

O'Reilly on Data

NOVEMBER 28, 2023

We wanted to find out what people are actually doing, so in September we surveyed O’Reilly’s users. Our survey focused on how companies use generative AI, what bottlenecks they see in adoption, and what skills gaps need to be addressed. AI users say that AI programming (66%) and data analysis (59%) are the most needed skills.

Enterprise 339

Enterprise Modeling Testing Reporting 339

erwin

OCTOBER 8, 2020

So if you’re going to move from your data from on-premise legacy data stores and warehouse systems to the cloud, you should do it right the first time. And as you make this transition, you need to understand what data you have, know where it is located, and govern it along the way. Then you must bulk load the legacy data.

Data Governance 145

Data Governance Metadata Testing Data Lake 145

erwin

DECEMBER 9, 2024

When it comes to using AI and machine learning across your organization, there are many good reasons to provide your data and analytics community with an intelligent data foundation. For instance, Large Language Models (LLMs) are known to ultimately perform better when data is structured.

Metadata 52

Metadata Data Quality Sales Modeling 52

erwin

JUNE 26, 2020

If you’re serious about a data-driven strategy , you’re going to need a data catalog. Organizations need a data catalog because it enables them to create a seamless way for employees to access and consume data and business assets in an organized manner. Three Types of Metadata in a Data Catalog.

Metadata 132

Metadata Cost-Benefit Measurement Data-driven 132

erwin

AUGUST 28, 2020

Organizations with a solid understanding of data governance (DG) are better equipped to keep pace with the speed of modern business. In this post, the erwin Experts address: What Is Data Governance? Why Is Data Governance Important? What Is Good Data Governance? What Are the Key Benefits of Data Governance?

Data Governance 145

Data Governance IT Cost-Benefit Metadata 145

Cloudera

MARCH 2, 2023

Recently, we announced enhanced multi-function analytics support in Cloudera Data Platform (CDP) with Apache Iceberg. Iceberg is a high-performance open table format for huge analytic data sets. This enables you to maximize utilization of streaming data at scale. The Catalog Type should be set to Hive.

Snapshot 114

Snapshot Data Processing Metadata Data Processing 114

Cloudera

JUNE 7, 2024

A constant flow of breaking news from the data lakehouse space is making notable tech headlines this week. On Tuesday, Databricks announced that it will acquire Tabular, a data management company founded by the creators of Apache Iceberg, Ryan Blue, Daniel Weeks, and Jason Reidfor. How Open is Your Open Source?

Management 109

Management Reporting Marketing Data Analytics 109

DataKitchen

NOVEMBER 4, 2021

Data organizations often have a mix of centralized and decentralized activity. DataOps concerns itself with the complex flow of data across teams, data centers and organizational boundaries. It expands beyond tools and data architecture and views the data organization from the perspective of its processes and workflows.

Data Processing 246

Data Processing Data Lake Cost-Benefit Testing 246

AWS Big Data

MARCH 2, 2023

Iceberg has become very popular for its support for ACID transactions in data lakes and features like schema and partition evolution, time travel, and rollback. Iceberg has become very popular for its support for ACID transactions in data lakes and features like schema and partition evolution, time travel, and rollback. AWS Glue 3.0

Data Lake 137

Data Lake Data Processing Metadata Snapshot 137

AWS Big Data

DECEMBER 4, 2024

Open table formats are emerging in the rapidly evolving domain of big data management, fundamentally altering the landscape of data storage and analysis. By providing a standardized framework for data representation, open table formats break down data silos, enhance data quality, and accelerate analytics at scale.

Snapshot 84

Snapshot Metadata Data Lake Optimization 84

AWS Big Data

OCTOBER 4, 2023

Amazon DataZone enables customers to discover, access, share, and govern data at scale across organizational boundaries, reducing the undifferentiated heavy lifting of making data and analytics tools accessible to everyone in the organization. This is challenging because access to data is managed differently by each of the tools.

Metadata 119

Metadata Data Lake Publishing Data Governance 119

Cloudera

AUGUST 16, 2021

We just announced Cloudera DataFlow for the Public Cloud (CDF-PC), the first cloud-native runtime for Apache NiFi data flows. In this blog post we’re revisiting the challenges that come with running Apache NiFi at scale before we take a closer look at the architecture and core features of CDF-PC.

Dashboards 122

Dashboards Metrics KPI Data-driven 122

erwin

FEBRUARY 6, 2020

As organizations deal with managing ever more data, the need to automate data management becomes clear. Last week erwin issued its 2020 State of Data Governance and Automation (DGA) Report. One piece of the research that stuck with me is that 70% of respondents spend 10 or more hours per week on data-related activities.

Management 144

Management Data Governance Cost-Benefit Metadata 144

erwin

MARCH 17, 2020

Teams need to urgently respond to everything from massive changes in workforce access and management to what-if planning for a variety of grim scenarios, in addition to building and documenting new applications and providing fast, accurate access to data for smart decision-making. Data Modeling. Data Governance.

Data Governance 138

Data Governance Enterprise Modeling Metadata 138

IBM Big Data Hub

MARCH 25, 2022

To simplify data access and empower users to leverage trusted information, organizations need a better approach that provides better insights and business outcomes faster, without sacrificing data access controls. There are many different approaches, but you’ll want an architecture that can be used regardless of your data estate.

Metadata 104

Metadata Data Quality Data Governance Data Integration 104

AWS Big Data

AUGUST 27, 2024

Director of Product, Salesforce Data Cloud. In today’s ever-evolving business landscape, organizations must harness and act on data to fuel analytics, generate insights, and make informed decisions to deliver exceptional customer experiences. What is Salesforce Data Cloud? What is Amazon Redshift?

Data Lake 111

Data Lake Analytics Data-driven Management 111

Cloudera

NOVEMBER 13, 2020

This is part of our series of blog posts on recent enhancements to Impala. Apache Impala is synonymous with high-performance processing of extremely large datasets, but what if our data isn’t huge? What if our queries are very selective? It turns out that Apache Impala scales down with data just as well as it scales up.

Optimization 143

Optimization Metadata Statistics Cost-Benefit 143

Cloudera

JUNE 7, 2021

Since then, we have been working hard to expose the full power of Apache Flink SQL and the existing Data Warehousing tools in CDP to combine it into a state-of-the-art real-time analytics platform. Flink SQL DDL and Catalog support. Catalog Integrations. This step significantly speeds up query development and data exploration.

Analytics 106

Analytics Management IT 106

erwin

FEBRUARY 13, 2020

When an organization’s data governance and metadata management programs work in harmony, then everything is easier. Data governance is a complex but critical practice. There’s always more data to handle, much of it unstructured; more data sources, like IoT, more points of integration, and more regulatory compliance requirements.

Metadata 135

Metadata Data Governance Management Cost-Benefit 135

Alation

JULY 15, 2021

Many data catalog initiatives fail. How can prospective buyers ensure they partner with the right catalog to drive success? According to the latest report from Eckerson Group, Deep Dive on Data Catalogs , shoppers must match the goals of their organizations to the capabilities of their chosen catalog.

Reporting 98

Reporting Data Governance Recreation/Entertainment Metadata 98

Cloudera

SEPTEMBER 29, 2020

Performance is one of the key, if not the most important deciding criterion, in choosing a Cloud Data Warehouse service. In today’s fast changing world, enterprises have to make data driven decisions quickly and for that they rely heavily on their data warehouse service. . Cloudera Data Warehouse vs HDInsight.

Data Warehouse 120

Data Warehouse Metadata Data-driven Machine Learning 120

Alation

APRIL 19, 2021

The cloud supports this new workforce, connecting remote workers to vital data, no matter their location. And what are the benefits? Data Cloud Migration Challenges and Solutions. Cloud migration is the process of moving enterprise data and infrastructure from on premise to off premise. But why migrate at all?

Cost-Benefit 111

Cost-Benefit Strategy Enterprise Data Strategy 111

AWS Big Data

SEPTEMBER 7, 2023

Customers of all sizes and industries use Amazon Simple Storage Service (Amazon S3) to store data globally for a variety of use cases. Customers want to know how their data is being accessed, when it is being accessed, and who is accessing it. With exponential growth in data volume, centralized monitoring becomes challenging.

Metadata 116

Metadata Dashboards Metrics Visualization 116

AWS Big Data

JULY 29, 2024

In the era of big data, data lakes have emerged as a cornerstone for storing vast amounts of raw data in its native format. They support structured, semi-structured, and unstructured data, offering a flexible and scalable environment for data ingestion from multiple sources.

Metadata 126

Metadata Snapshot Data Lake Metrics 126

AWS Big Data

JULY 18, 2024

Over the years, organizations have invested in creating purpose-built, cloud-based data lakes that are siloed from one another. A major challenge is enabling cross-organization discovery and access to data across these multiple data lakes, each built on different technology stacks.

Data Lake 130

Data Lake Publishing Metadata Data-driven 130

Cloudera

DECEMBER 11, 2020

In a previous blog post on CDW performance, we compared Azure HDInsight to CDW. In this blog post, we compare Cloudera Data Warehouse (CDW) on Cloudera Data Platform (CDP) using Apache Hive-LLAP to EMR 6.0 (also powered by Apache Hive-LLAP) on Amazon using the TPC-DS 2.9 More on this later in the blog.

Data Warehouse 110

Data Warehouse Metadata Machine Learning Measurement 110

AWS Big Data

AUGUST 8, 2024

Thousands of organizations build data integration pipelines to extract and transform data. They establish data quality rules to ensure the extracted data is of high quality for accurate business decisions. These rules commonly assess the data based on fixed criteria reflecting the current business state.

Data Quality 116

Data Quality Statistics Visualization Metrics 116

Cloudera

MARCH 11, 2021

Data flows are an integral part of every modern enterprise. At Cloudera, we’re helping our customers implement data flows on-premises and in the public cloud using Apache NiFi , a core component of Cloudera DataFlow. In this blog post, I want to share the top three requirements for data flows in 2021 that we hear from our customers.

Data Integration 119

Data Integration Data Warehouse Testing Enterprise 119