Remove Measurement Remove Metrics Remove Uncertainty
article thumbnail

Escaping POC Purgatory: Evaluation-Driven Development for AI Systems

O'Reilly on Data

Business value : Once we have a rubric for evaluating our systems, how do we tie our macro-level business value metrics to our micro-level LLM evaluations? How will you measure success? So now we have a user persona, several scenarios, and a way to measure success. How do we do so? We asked them: Who are you building it for?

Testing 174
article thumbnail

The Lean Analytics Cycle: Metrics > Hypothesis > Experiment > Act

Occam's Razor

To win in business you need to follow this process: Metrics > Hypothesis > Experiment > Act. We are far too enamored with data collection and reporting the standard metrics we love because others love them because someone else said they were nice so many years ago. That metric is tied to a KPI.

Metrics 157
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

You Can’t Regulate What You Don’t Understand

O'Reilly on Data

If we want prosocial outcomes, we need to design and report on the metrics that explicitly aim for those outcomes and measure the extent to which they have been achieved. But alignment will be impossible without robust institutions for disclosure and auditing. That is a crucial first step, and we should take it immediately.

Metrics 360
article thumbnail

What you need to know about product management for AI

O'Reilly on Data

Machine learning adds uncertainty. Underneath this uncertainty lies further uncertainty in the development process itself. There are strategies for dealing with all of this uncertainty–starting with the proverb from the early days of Agile: “ do the simplest thing that could possibly work.”

article thumbnail

AI Product Management After Deployment

O'Reilly on Data

Ideally, AI PMs would steer development teams to incorporate I/O validation into the initial build of the production system, along with the instrumentation needed to monitor model accuracy and other technical performance metrics. But in practice, it is common for model I/O validation steps to be added later, when scaling an AI product.

article thumbnail

Why HR professionals struggle with big data

CIO Business Intelligence

This is due, on the one hand, to the uncertainty associated with handling confidential, sensitive data and, on the other hand, to a number of structural problems. Solid reporting provides transparent, consistent and combined HR metrics essential for strategic planning, risk management and the management of HR measures.

article thumbnail

Uncertainties: Statistical, Representational, Interventional

The Unofficial Google Data Science Blog

by AMIR NAJMI & MUKUND SUNDARARAJAN Data science is about decision making under uncertainty. Some of that uncertainty is the result of statistical inference, i.e., using a finite sample of observations for estimation. But there are other kinds of uncertainty, at least as important, that are not statistical in nature.