KDnuggets

JUNE 26, 2025

Whats the overall data quality score? Most data scientists spend 15-30 minutes manually exploring each new dataset—loading it into pandas, running.info() ,describe() , and.isnull().sum() sum() , then creating visualizations to understand missing data patterns. Perfect for on-demand data quality checks.

Data Quality

DataKitchen

FEBRUARY 20, 2025

Announcing DataOps Data Quality TestGen 3.0: Open-Source, Generative Data Quality Software. It assesses your data, deploys production testing, monitors progress, and helps you build a constituency within your company for lasting change. New Quality Dashboard & Score Explorer.

Data Quality

DataKitchen

JUNE 18, 2025

In this exciting webinar , Christopher Bergh discussed various types of data quality dashboards, emphasizing that effective dashboards make data health visible and drive targeted improvements by relying on concrete, actionable tests. Each type serves a unique role in driving changes in data quality.

Data Quality

AWS Big Data

DECEMBER 17, 2024

This blog was co-authored by DeNA Co., Among these, the healthcare & medical business handles particularly sensitive data. Prevent the inclusion of invalid values in categorical data and process data without any data loss. The challenge Data quality tests require performing 1,300 tests on 10 TB of data monthly.

Data Quality

DataKitchen

DECEMBER 9, 2024

A Drug Launch Case Study in the Amazing Efficiency of a Data Team Using DataOps How a Small Team Powered the Multi-Billion Dollar Acquisition of a Pharma Startup When launching a groundbreaking pharmaceutical product, the stakes and the rewards couldnt be higher. data engineers delivered over 100 lines of code and 1.5

Data Quality

DataKitchen

JULY 8, 2025

Scaling Data Reliability: The Definitive Guide to Test Coverage for Data Engineers The parallels between software development and data analytics have never been more apparent. Let us show you how to implement full-coverage automatic data checks on every table, column, tool, and step in your delivery process.

Testing

DataKitchen

JULY 8, 2025

The Hidden Crisis in Data Usability … and How DataOps Data Quality TestGen Can Help Fix It In many data organizations, there’s a silent crisis: data usability is broken. Whether you work in data quality or engineering, you’ve probably said one of these things: “That’s the way the data came to us.”

Scorecard

DataKitchen

JULY 31, 2025

These aren’t just any data points—they are the backbone of your operations, the foundation of your decision-making, and often the difference between business success and failure. Identifying CDEs is a vital step in data governance because it changes how organizations handle data quality.

Data Governance

DataKitchen

MAY 2, 2025

Is Your Team in Denial of Data Quality? Here’s How to Tell In many organizations, data quality problems fester in the shadowsignored, rationalized, or swept aside with confident-sounding statements that mask a deeper dysfunction. That’s not testing; that’s wishful thinking. Is your team ready?

Data Quality

KDnuggets

JUNE 12, 2025

When you understand distributions, you can spot data quality issues instantly. When you know hypothesis testing, you know whether your A/B test results actually mean something. Hypothesis testing gives you the framework to make valid and provable claims. Learn t-tests, chi-square tests, and confidence intervals.

Data Science

DataKitchen

MARCH 20, 2025

We’ve identified two distinct types of data teams: process-centric and data-centric. Understanding this framework offers valuable insights into team efficiency, operational excellence, and data quality. Process-centric data teams focus their energies predominantly on orchestrating and automating workflows.

Data Quality

DataKitchen

MAY 17, 2025

At the moment, I was flying home from a data quality conference. He slacked back data. An embedded test had failed. And I was tempted, so tempted, as the clock kept ticking, to disable the test and let it go. Then it dawned on me that this test wasnt even ours. These tests werent easy to define or implement.

Data Quality

AWS Big Data

NOVEMBER 13, 2024

Identify pioneer cases – The team identified the pioneer use cases that onboard the data solution first, to validate its business value. The first use case helps predict test results during the car assembly process. Reuse of consumer-based data saves cost in extract, transform, and load (ETL) implementation and system maintenance.

Metadata

KDnuggets

JULY 4, 2025

Step 5: Run Your API To launch the server, use uvicorn like this: uvicorn app.main:app --reload Visit: [link] You’ll see an interactive Swagger UI where you can test the API. That keeps the API responsive and avoids delays.

Machine Learning

DataKitchen

FEBRUARY 17, 2025

No Python, No SQL Templates, No YAML: Why Your Open Source Data Quality Tool Should Generate 80% Of Your Data Quality Tests Automatically As a data engineer, ensuring data quality is both essential and overwhelming. Everyone wants to write more tests, yet they somehow never get it done.

Data Quality

KDnuggets

JULY 4, 2025

random_state=42) Preprocessing the data and making it suitable for the PCA algorithm is as important as applying the algorithm itself. Meanwhile, the transform method is utilized on the test data, which applies the same transformation "learned" from the training data to the test set.

Machine Learning

KDnuggets

AUGUST 8, 2025

The Code node examines data types, calculates distributions, identifies correlations, and detects patterns that inform AI recommendations. Next Steps: Scaling AI-Assisted Data Science // 1.

Data Science

DataKitchen

APRIL 9, 2025

Why Data Quality Isnt Worth The Effort : Data Quality Coffee With Uncle Chip Data quality has become one of the most discussed challenges in modern data teams, yet it remains one of the most thankless and frustrating responsibilities. How Does Open Source DataOps Data Quality TestGen Help?

Data Quality

KDnuggets

JULY 16, 2025

Key activities during this phase include: Exploratory Data Analysis (EDA) : Use visualizations and summary statistics to understand distributions, relationships, and anomalies. Data audit : Identify variable types (e.g., numeric, categorical, text), check for missing or inconsistent values, and assess overall data quality.

Modeling

KDnuggets

AUGUST 4, 2025

How it helps : When youre tweaking hyperparameters and testing different algorithms, keeping track of what worked becomes impossible without proper tooling. Great Expectations: Data Quality Assurance What it solves : Great Expectations is for data validation and quality assurance for ML pipelines How it helps : Bad data breaks models.

Machine Learning

DataKitchen

DECEMBER 6, 2024

Like an Olympic athlete training for the gold, your data needs a continuous, iterative process to maintain peak performance. We covered how Data Quality Testing, Observability, and Scorecards turn data quality into a dynamic process, helping you build accuracy, consistency, and trust at each layerBronze, Silver, and Gold.

Data Quality

DataKitchen

DECEMBER 17, 2024

The Gartner presentation, How Can You Leverage Technologies to Solve Data Quality Challenges? by Melody Chien, underscores the critical role of data quality in modern business operations. Poor data quality, on average, costs organizations $12.9 million annually , or 7% of their total revenue.

Data Quality

Cloudera

DECEMBER 9, 2024

Many enterprises encounter bottlenecks related to data quality, model deployment, and infrastructure requirements that hinder scaling efforts. The post Scaling AI Solutions with Cloudera: A Deep Dive into AI Inference and Solution Patterns appeared first on Cloudera Blog.

Data Processing

erwin

DECEMBER 9, 2024

A data catalog providing automated data profiling does just this and, when tied in with data lineage, your organization can easily see metadatas pathway back to all sources feeding your AI model. Monitoring sales revenue data will provide you with regional patterns and anomalies globally.

Metadata

AWS Big Data

DECEMBER 19, 2024

As data lakes increasingly handle sensitive business data and transactional workloads, maintaining strong data quality, governance, and compliance becomes vital to maintaining trust and regulatory alignment. This comparison will help guide you in making informed decisions on enhancing your data lake environments.

Data Lake

AWS Big Data

JULY 15, 2025

Self-managing Iceberg tables with large volumes of data poses several challenges, including managing concurrent transactions, processing real-time data streams, handling small file proliferation, maintaining data quality and governance, and ensuring compliance. AWS Lake Formation and AWS Glue Data Catalog.

Optimization

Teradata

MAY 30, 2025

Fortunately, Teradata offers integrations to many modular tools that facilitate routine processes allowing data engineers to focus on high-value tasks such as governance, data quality, and efficiency. schema.yml`: YAML file defining metadata, tests, and descriptions for the models in this directory.

Data Integration

Jen Stirrup

JULY 13, 2025

Attempting to build advanced analytics or AI capabilities on shaky data foundations is akin to constructing a skyscraper on quicksand—it may look impressive at first, but it simply won’t stand the test of time. The Solution: Data Strategy as well as AI It is tempting to think that the answer to succesful AI is to implement more AI!

Digital Transformation

DataKitchen

APRIL 18, 2025

Welcome to the Data Quality Coffee Series with Uncle Chip Pull up a chair, pour yourself a fresh cup, and get ready to talk shopbecause its time for Data Quality Coffee with Uncle Chip. This video series is where decades of data experience meet real-world challenges, a dash of humor, and zero fluff.

Data Quality

Smart Data Collective

JUNE 22, 2025

Conclusion: Key Takeaways for Data Teams Embracing AI Web Scrapers You can’t overstate the damage poor data quality causes. AI’s Role in Cleaning and Structuring Data There are many ways AI helps clean up large datasets, especially in eliminating duplicates, correcting formats, and filling in gaps. businesses over $3.1

Big Data

KDnuggets

JULY 2, 2025

Oh, yeah, SQL is also heavily tested, especially for analyst and mid-level data science roles. Behavioral questions test how you think and communicate. Tie your answers to data and metrics (e.g., Underestimating SQL Mistake: Not practicing SQL enough, because “it’s easy compared to Python or machine learning”.

Machine Learning

CIO Business Intelligence

NOVEMBER 25, 2024

“Today’s CIOs inherit highly customized ERPs and struggle to lead change management efforts, especially with systems that [are the] backbone of all the enterprise’s operations,” wrote Isaac Sacolick, founder and president of StarCIO, a digital transformation consultancy, in a recent blog post.

Digital Transformation

DataKitchen

APRIL 18, 2025

Data Quality When You Dont Understand the Data : Data Quality Coffee With Uncle Chip #3 Lets be honestdata quality feels impossible when you dont understand the data. Thats the crux of the issue: you cant test for what you dont understand. Thats where DataOps Data Quality TestGen comes in.

Data Quality

AWS Big Data

MARCH 21, 2025

Traditionally, answering this question would involve multiple data exports, complex extract, transform, and load (ETL) processes, and careful data synchronization across systems. Amazon SageMaker Catalog serves as a central repository hub to store both technical and business catalog information of the data product.

Data Warehouse

Wayne Yaddow

MARCH 28, 2025

Unleashing GenAIEnsuring Data Quality at Scale (Part1) Transitioning from isolated repository systems to consolidated AI LLM pipelines Photo by Joshua Sortino on Unsplash Introduction This blog is based on insights from articles in Database Trends and Applications, Feb/Mar 2025 ( DBTA Journal ).

Data Quality

DataKitchen

MAY 14, 2024

Data Observability and Data Quality Testing Certification Series We are excited to invite you to a free four-part webinar series that will elevate your understanding and skills in Data Observation and Data Quality Testing. Don’t miss this opportunity to transform your data practices.

Data Quality

datapine

SEPTEMBER 29, 2022

1) What Is Data Quality Management? 4) Data Quality Best Practices. 5) How Do You Measure Data Quality? 6) Data Quality Metrics Examples. 7) Data Quality Control: Use Case. 8) The Consequences Of Bad Data Quality. 9) 3 Sources Of Low-Quality Data.

Data Quality