O'Reilly on Data

MARCH 31, 2020

If you’re already a software product manager (PM), you have a head start on becoming a PM for artificial intelligence (AI) or machine learning (ML). But there’s a host of new challenges when it comes to managing AI projects: more unknowns, non-deterministic outcomes, new infrastructures, new processes and new tools.

Management 363

Management Machine Learning Experimentation Metrics 363

O'Reilly on Data

MARCH 25, 2025

Weve seen this across dozens of companies, and the teams that break out of this trap all adopt some version of Evaluation-Driven Development (EDD), where testing, monitoring, and evaluation drive every decision from the start. What breaks your app in production isnt always what you tested for in dev! The way out?

Testing 174

Testing Data-driven Software Measurement 174

O'Reilly on Data

OCTOBER 12, 2021

We know how to test whether or not code is correct (at least up to a certain limit). Given enough unit tests and acceptance tests, we can imagine a system for automatically generating code that is correct. But we don’t have methods to test for code that’s “good.” There are lots of ways to sort.

Testing 363

Testing Measurement Consulting Modeling 363

datapine

OCTOBER 17, 2019

Management reporting is a source of business intelligence that helps business leaders make more accurate, data-driven decisions. In this blog post, we’re going to give a bit of background and context about management reports, and then we’re going to outline 10 essential best practices you can use to make sure your reports are effective.

Reporting 263

Reporting Management Dashboards KPI 263

datapine

OCTOBER 11, 2023

This is no different in the logistics industry, where warehouse managers track a range of KPIs that help them efficiently manage inventory, transportation, employee safety, and order fulfillment, among others. These powerful measurements will allow you to track all activities in real-time to ensure everything runs smoothly and safely.

Metrics 217

Metrics KPI Management Cost-Benefit 217

datapine

FEBRUARY 9, 2024

Table of Contents 1) What Is KPI Management? 4) How to Select Your KPIs 5) Avoid These KPI Mistakes 6) How To Choose A KPI Management Solution 7) KPI Management Examples Fact: 100% of statistics strategically placed at the top of blog posts are a direct result of people studying the dynamics of Key Performance Indicators, or KPIs.

KPI 221

KPI Management Key Performance Indicator Measurement 221

O'Reilly on Data

JULY 15, 2019

In recent posts, we described requisite foundational technologies needed to sustain machine learning practices within organizations, and specialized tools for model development, model governance, and model operations/testing/monitoring. Note that the emphasis of SR 11-7 is on risk management.). Model risk management.

Machine Learning 51

Machine Learning Management Enterprise Risk Management 51

CIO Business Intelligence

JANUARY 14, 2025

How does our AI strategy support our business objectives, and how do we measure its value? Meanwhile, he says establishing how the organization will measure the value of its AI strategy ensures that it is poised to deliver impactful outcomes because, to create such measures, teams must name desired outcomes and the value they hope to get.

Strategy 141

Strategy ROI Experimentation Risk 141

O'Reilly on Data

JUNE 15, 2023

If we want prosocial outcomes, we need to design and report on the metrics that explicitly aim for those outcomes and measure the extent to which they have been achieved. They were not imposed from without, but were adopted because they allowed merchants to track and manage their own trading ventures.

Metrics 359

Metrics Reporting Measurement Finance 359

CIO Business Intelligence

JANUARY 14, 2025

In todays digital economy, business objectives like becoming a leading global wealth management firm or being a premier destination for top talent demand more than just technical excellence. Most importantly, architects make difficult problems manageable. The stakes have never been higher.

Enterprise 119

Enterprise Technology Metrics Measurement 119

DataKitchen

APRIL 13, 2021

Testing and Data Observability. Sandbox Creation and Management. It orchestrates complex pipelines, toolchains, and tests across teams, locations, and data centers. Apache Oozie — An open-source workflow scheduler system to manage Apache Hadoop jobs. Testing and Data Observability. Meta-Orchestration.

Testing 300

Testing Machine Learning Consulting Data Science 300

CIO Business Intelligence

JANUARY 6, 2025

Balancing the rollout with proper training, adoption, and careful measurement of costs and benefits is essential, particularly while securing company assets in tandem, says Ted Kenney, CIO of tech company Access. Our success will be measured by user adoption, a reduction in manual tasks, and an increase in sales and customer satisfaction.

IT 140

IT Cost-Benefit Measurement Experimentation 140

DataKitchen

SEPTEMBER 20, 2021

Figure 2: The DataKitchen Platform helps you reduce time spent managing errors and executing manual processes from about half to 15%. The other 78% of their time is devoted to managing errors, manually executing production pipelines and other supporting activities. Start with just a few critical tests and build gradually.

Testing 246

Testing Metrics Measurement Dashboards 246

datapine

SEPTEMBER 29, 2022

1) What Is Data Quality Management? 5) How Do You Measure Data Quality? However, with all good things comes many challenges and businesses often struggle with managing their information in the correct way. Enters data quality management. What Is Data Quality Management (DQM)? Table of Contents.

Data Quality 241

Data Quality Metrics Data-driven Management 241

CIO Business Intelligence

MARCH 25, 2025

Even modest investments in database tooling and paying down some data management debt can relieve database administrators of the tedium of manual updates or reactive monitoring, says Graham McMillan, CTO of Redgate. Another concern is if regulations force holistic model retraining, forcing CIOs to switch to alternatives to remain compliant.

Risk 140

Risk Cost-Benefit Data-driven Digital Transformation 140

CIO Business Intelligence

NOVEMBER 20, 2023

Measuring developer productivity has long been a Holy Grail of business. In addition, system, team, and individual productivity all need to be measured. The inner loop comprises activities directly related to creating the software product: coding, building, and unit testing. And like the Holy Grail, it has been elusive.

Measurement 126

Measurement Metrics Testing Software 126

O'Reilly on Data

JUNE 19, 2019

One sure sign that companies are getting serious about machine learning is the growing popularity of tools designed specifically for managing the ML model development lifecycle, such as MLflow and Comet.ml. hyperparameter tuning, NAS ) while emphasizing the ease with which one can manage, track, and reproduce such experiments.

Modeling 249

Modeling Machine Learning Testing Metrics 249

CIO Business Intelligence

APRIL 5, 2023

IT is no longer perceived as a cost factor or a pure support function at many organizations, according to management consultancy 4C Group’s Markus Matschi. IT must be able to show the departments and the management team their possibilities and the added value, says Heiko Weigelt, CIO of Funke Media Group.

Measurement 133

Measurement IT Metrics Consulting 133

O'Reilly on Data

OCTOBER 19, 2021

Since software engineers manage to build ordinary software without experiencing as much pain as their counterparts in the ML department, it begs the question: should we just start treating ML projects as software engineering projects as usual, maybe educating ML practitioners about the existing best practices? Why did something break?

IT 364

IT Testing Experimentation Software 364

CIO Business Intelligence

OCTOBER 23, 2024

CIOs should create proofs of concept that test how costs will scale, not just how the technology works.” Governance and human challenges further complicate AI rollouts Another formidable challenge is the governance and data management complexity brought on by the decentralization of AI capabilities.

Enterprise 141

Enterprise Cost-Benefit Broadcasting Risk 141

David Menninger's Analyst Perspectives

NOVEMBER 19, 2024

Maintaining quality and trust is a perennial data management challenge, the importance of which has come into sharper focus in recent years thanks to the rise of artificial intelligence (AI). The ability to monitor and measure improvements in data quality relies on instrumentation.

Data Quality 130

Data Quality Dashboards Data-driven Software 130

O'Reilly on Data

JUNE 18, 2024

The data to answer hyperlocal questions about topics like fertilization and pest management exists but it’s spread across many databases with many owners: governments, NGOs, and corporations, in addition to local knowledge about what works. Many farmers measure their yield in bags of rice, but what is “a bag of rice”?

Testing 318

Testing Software Modeling Measurement 318

CIO Business Intelligence

DECEMBER 20, 2023

This has spurred interest around understanding and measuring developer productivity, says Keith Mann, senior director, analyst, at Gartner. Therefore, engineering leadership should measure software developer productivity, says Mann, but also understand how to do so effectively and be wary of pitfalls.

Measurement 124

Measurement Software IT Metrics 124

CIO Business Intelligence

SEPTEMBER 17, 2024

CISOs can only know the performance and maturity of their security program by actively measuring it themselves; after all, to measure is to know. However, CISOs aren’t typically measuring their security program proactively or methodically to understand their current security program. people, processes, and technology).

Measurement 111

Measurement Metrics Risk Dashboards 111

CIO Business Intelligence

FEBRUARY 7, 2024

Key AI companies have told the UK government to speed up its safety testing for their systems, raising questions about future government initiatives that too may hinge on technology providers opening up generative AI models to tests before new releases hit the public.

Testing 124

Testing Reporting Risk Publishing 124

CIO Business Intelligence

MARCH 5, 2025

The next thing is to make sure they have an objective way of testing the outcome and measuring success. Large software vendors are used to solving the integration problems that enterprises deal with on a daily basis, says Lee McClendon, chief digital and technology officer at software testing company Tricentis.

Testing 99

Testing Measurement Technology Experimentation 99

DataKitchen

APRIL 26, 2021

In early April 2021, DataKItchen sat down with Jonathan Hodges, VP Data Management & Analytics, at Workiva ; Chuck Smith, VP of R&D Data Strategy at GlaxoSmithKline (GSK) ; and Chris Bergh, CEO and Head Chef at DataKitchen, to find out about their enterprise DataOps transformation journey, including key successes and lessons learned.

Measurement 246

Measurement Metrics Data-driven Dashboards 246

DataKitchen

NOVEMBER 29, 2021

Model developers will test for AI bias as part of their pre-deployment testing. Quality test suites will enforce “equity,” like any other performance metric. Continuous testing, monitoring and observability will prevent biased models from deploying or continuing to operate. Companies Commit to Remote. Data Gets Meshier.

Testing 245

Testing Data Lake Data Architecture Manufacturing 245

CIO Business Intelligence

DECEMBER 18, 2024

Over the past year, the focus on risk management has evolved significantly, says Meerah Rajavel, CIO of Palo Alto Networks. Resilience frameworks have measurable ROI, but they require a holistic, platform-based approach to curtail threats and guide the safe use of AI, he adds.

Risk 105

Risk Risk Management Cost-Benefit Measurement 105

CIO Business Intelligence

APRIL 8, 2025

As data-centric AI, automated metadata management and privacy-aware data sharing mature, the opportunity to embed data quality into the enterprises core has never been more significant. Instead, organizations resort to manual workarounds often managed by overburdened analysts or domain experts. Accountability and embedded SLAs.

Data Quality 102

Data Quality Data-driven Key Performance Indicator Metadata 102

CIO Business Intelligence

JUNE 13, 2024

They don’t train to fight in zero gravity, though: They are mostly computer experts charged with things like preventing cyberattacks, maintaining computer networks, and managing satellite communications.) (Guardians are enlisted members of the US Space Force, a service created under the DAF umbrella in 2019.

Testing 119

Testing Experimentation Data Processing Modeling 119

DataKitchen

JANUARY 20, 2021

As DataOps activity takes root within an enterprise, managers face the question of whether to build centralized or decentralized DataOps capabilities. Centralizing analytics helps the organization standardize enterprise-wide measurements and metrics. Develop/execute regression testing . Agile ticketing/Kanban tools.

Metrics 243

Metrics Experimentation Measurement Testing 243

datapine

APRIL 15, 2020

When it comes to implementing and managing a successful BI strategy we have always proclaimed: start small, use the right BI tools , and involve your team. Your Chance: Want to test an agile business intelligence solution? Without further ado, let’s begin. Try our business intelligence software for 14 days, completely free!

Business Intelligence 276

Business Intelligence Analytics Testing Dashboards 276

DataKitchen

MAY 10, 2021

Some will argue that observability is nothing more than testing and monitoring applications using tests, metrics, logs, and other artifacts. Below we will explain how to virtually eliminate data errors using DataOps automation and the simple building blocks of data and analytics testing and monitoring. . Tie tests to alerts.

Testing 214

Testing Manufacturing Data Quality Statistics 214

CIO Business Intelligence

MAY 23, 2024

More recently, products have become increasingly digital, with software that manages patient flows, tools for surgery planning, and sterile management processes that optimize inventory and ensure that surgical instruments are delivered at the right time to the right place.

Digital Transformation 111

Digital Transformation Measurement Cost-Benefit Data Warehouse 111

DataKitchen

SEPTEMBER 18, 2024

The DataOps methodology offers a solution by providing a structured, iterative approach to managing data quality at scale. Agile and Iterative Approach to Data Quality Traditional approaches to data quality often resemble waterfall project management: detailed plans, lengthy analysis phases, and slow execution.

Scorecard 177

Scorecard Data Quality Measurement Testing 177

DataKitchen

NOVEMBER 20, 2021

A DataOps Engineer can make test data available on demand. DataOps Engineers have tools that we apply to all of the pipeline orchestrations that we manage. We have automated testing and a system for exception reporting, where tests identify issues that need to be addressed. We don’t want to embarrass anyone.

Testing 246

Testing Machine Learning Data Warehouse Analytics 246