Remove Consulting Remove Measurement Remove Testing
article thumbnail

Escaping POC Purgatory: Evaluation-Driven Development for AI Systems

O'Reilly on Data

Weve seen this across dozens of companies, and the teams that break out of this trap all adopt some version of Evaluation-Driven Development (EDD), where testing, monitoring, and evaluation drive every decision from the start. What breaks your app in production isnt always what you tested for in dev! How will you measure success?

Testing 174
article thumbnail

The Quality of Auto-Generated Code

O'Reilly on Data

We know how to test whether or not code is correct (at least up to a certain limit). Given enough unit tests and acceptance tests, we can imagine a system for automatically generating code that is correct. But we don’t have methods to test for code that’s “good.” There are lots of ways to sort.

Testing 360
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

The DataOps Vendor Landscape, 2021

DataKitchen

Testing and Data Observability. It orchestrates complex pipelines, toolchains, and tests across teams, locations, and data centers. Prefect Technologies — Open-source data engineering platform that builds, tests, and runs data workflows. Testing and Data Observability. Production Monitoring and Development Testing.

Testing 304
article thumbnail

Web Analytics Consulting: A Simple Framework For Smarter Decisions

Occam's Razor

and in this post I want to share a really, really simple framework for structuring web analytics consulting contracts. The Web Analytics Consultant Quandary. BB sent this query: If I take on a consulting project then what could be expectations out of me? The Optimal Web Analytics Consulting Framework: DC – DR – DA.

article thumbnail

6 steps to measure the business value of IT

CIO Business Intelligence

IT is no longer perceived as a cost factor or a pure support function at many organizations, according to management consultancy 4C Group’s Markus Matschi. In order to make progress along this path, the consultants developed a process model with several development and evaluation phases, using current scientific findings and speaking to CIOs.

article thumbnail

Measuring Incrementality: Controlled Experiments to the Rescue!

Occam's Razor

This: You understand all the environmental variables currently in play, you carefully choose more than one group of "like type" subjects, you expose them to a different mix of media, measure differences in outcomes, prove / disprove your hypothesis (DO FACEBOOK NOW!!!), The nice thing is that you can also test that!

article thumbnail

Our Favorite Questions

O'Reilly on Data

We do so out of genuine curiosity as well as professional necessity: Q is an ML/AI consultant, Chris is a product manager in the AI space, and Shane is an attorney. The opinions presented here are personal, do not reflect the view of our employers, and are not professional product, consulting, or legal advice. A lot of questions.