Occam's Razor

MAY 22, 2006

This post is a primer on the delightful world of testing and experimentation (A/B, Multivariate, and a new term from me: Experience Testing). Experimentation and testing help us figure out we are wrong, quickly and repeatedly and if you think about it that is a great thing for our customers, and for our employers.

Experimentation 103

Experimentation Testing Optimization Measurement 103

O'Reilly on Data

MARCH 31, 2020

If you’re already a software product manager (PM), you have a head start on becoming a PM for artificial intelligence (AI) or machine learning (ML). You already know the game and how it is played: you’re the coordinator who ties everything together, from the developers and designers to the executives. Why AI software development is different.

Management 363

Management Machine Learning Experimentation Metrics 363

Occam's Razor

SEPTEMBER 19, 2011

This: You understand all the environmental variables currently in play, you carefully choose more than one group of "like type" subjects, you expose them to a different mix of media, measure differences in outcomes, prove / disprove your hypothesis (DO FACEBOOK NOW!!!) Maybe it is Search, Email and Facebook. Controlled experiments!

Measurement 120

Measurement Advertising Testing Marketing 120

Rocket-Powered Data Science

FEBRUARY 15, 2023

encouraging and rewarding) a culture of experimentation across the organization. Source: [link] Every business wants to get on board with ChatGPT, to implement it, operationalize it, and capitalize on it. It is important to realize that the usual “hype cycle” rules prevail in such cases as this.

Strategy 290

Strategy Experimentation Uncertainty Machine Learning 290

CIO Business Intelligence

JANUARY 21, 2025

Deloittes State of Generative AI in the Enterprise reports nearly 70% have moved 30% or fewer of their gen AI experiments into production, and 41% of organizations have struggled to define and measure the impacts of their gen AI efforts. Why should CIOs bet on unifying their data and AI practices?

Cost-Benefit 142

Cost-Benefit Data-driven Strategy Marketing 142

CIO Business Intelligence

AUGUST 20, 2024

While the focus at these three levels differ, CIOs should provide a consistent definition of high performance and how it’s measured. These teams focused on delivering reliable technology capabilities, improving end-user experiences, and establishing data and analytics capabilities.

IT 141

IT Digital Transformation Experimentation Risk 141

DataKitchen

JANUARY 20, 2021

Centralizing analytics helps the organization standardize enterprise-wide measurements and metrics. Central DataOps process measurement function with reports. A COE typically has a full-time staff that focuses on delivering value for customers in an experimentation-driven, iterative, result-oriented, customer-focused way.

Metrics 243

Metrics Experimentation Measurement Testing 243

CIO Business Intelligence

MARCH 5, 2025

Two years of experimentation may have given rise to several valuable use cases for gen AI , but during the same period, IT leaders have also learned that the new, fast-evolving technology isnt something to jump into blindly. The next thing is to make sure they have an objective way of testing the outcome and measuring success.

Testing 99

Testing Measurement Technology Experimentation 99

O'Reilly on Data

SEPTEMBER 15, 2020

Technical sophistication: Sophistication measures a team’s ability to use advanced tools and techniques (e.g., Technical competence: Competence measures a team’s ability to successfully deliver on initiatives and projects. Goals should be defined specifically and at a granular level for each stakeholder and relevant use case.

Advertising 362

Advertising Cost-Benefit ROI Machine Learning 362

CIO Business Intelligence

OCTOBER 1, 2024

Measure everything Looking for ROI too soon is often a product of poor planning, says Rowan Curran, an AI and data science analyst at Forrester. Measure everything Looking for ROI too soon is often a product of poor planning, says Rowan Curran, an AI and data science analyst at Forrester. But an AI reset is underway.

ROI 131

ROI IT Measurement Experimentation 131

CIO Business Intelligence

JANUARY 10, 2025

High expectations, but ROI challenges persist Despite significant investments, only 31% of organizations expect to measure generative AIs return on investment in the next six months. The dynamic nature of AI demands new ways to measure value beyond the limits of a conventional business case, Chase said.

ROI 81

ROI Enterprise Broadcasting Metrics 81

CIO Business Intelligence

JUNE 11, 2024

A 1958 Harvard Business Review article coined the term information technology, focusing their definition on rapidly processing large amounts of information, using statistical and mathematical methods in decision-making, and simulating higher order thinking through applications.

IT 136

IT Digital Transformation Broadcasting Business Objectives 136

CIO Business Intelligence

DECEMBER 19, 2024

One of the fastest-growing industries in the world, climate tech and its companion area of nature tech require a wide range of skills to help solve significant environmental problems. In especially high demand are IT pros with software development, data science and machine learning skills.

IT 52

IT IoT Machine Learning Optimization 52

Occam's Razor

NOVEMBER 12, 2006

Mostly because short term goals drive a lot of what we do and if you are selling something on your website then it only seems to make logical sense that we measure conversion rate and get it up as high as we can as fast as we can. So measure Bounce Rate of your website. Even though we should not obsess about conversion rate we do.

Measurement 99

Measurement Analytics Marketing Optimization 99

CIO Business Intelligence

MARCH 4, 2025

This article goes behind the scenes on whats fueling Blocks investment in developer experience, key initiatives including the role of an engineering intelligence platform , and how the company measures and drives success. We want engineering velocity to remain our competitive advantage, says Azra Coburn, Blocks Head of Developer Experience.

Experimentation 69

Experimentation Metrics Data-driven Measurement 69

Smart Data Collective

APRIL 13, 2023

Resilient IT teams are better capable of incorporating changes, they’re more flexible in the face of adversity, and they can achieve more and less time. Building a resilient IT team during an economic downturn seems like an impossible challenge, especially if you’re facing budget limitations and staffing shortages. Hire wisely.

IT 124

IT KPI Experimentation Strategy 124

Corinium

JUNE 6, 2019

A properly set framework will ensure quality, timeliness, scalability, consistency, and industrialization in measuring and driving the return on investment. It is also important to have a strong test and learn culture to encourage rapid experimentation. What is the most common mistake people make around data? It is fast and slow.

Insurance 250

Insurance Analytics Forecasting Deep Learning 250

CIO Business Intelligence

MAY 2, 2024

When a necessary change hits your IT team, getting everyone to buy in and adapt is key — and challenging. Some team members will, by nature, fight any change. Even those once adaptable may have become change resistant due to years of near-constant turbulence. This is true of mergers, acquisitions, divestitures — any type of change.” Why fix it?’”

IT 138

IT Experimentation Software Management 138

O'Reilly on Data

OCTOBER 13, 2020

In an incident management blog post , Atlassian defines SLOs as: “the individual promises you’re making to that customer… SLOs are what set customer expectations and tell IT and DevOps teams what goals they need to hit and measure themselves against. Proper AI product monitoring is essential to this outcome. I/O validation.

Management 308

Management Machine Learning Metrics Modeling 308

The Unofficial Google Data Science Blog

APRIL 23, 2024

the weight given to Likes in our video recommendation algorithm) while $Y$ is a vector of outcome measures such as different metrics of user experience (e.g., the weight given to Likes in our video recommendation algorithm) while $Y$ is a vector of outcome measures such as different metrics of user experience (e.g.,

Experimentation 52

Experimentation Optimization Uncertainty Metrics 52

AWS Big Data

JANUARY 15, 2025

EUROGATE is a leading independent container terminal operator in Europe, known for its reliable and professional container handling services. Every day, EUROGATE handles thousands of freight containers moving in and out of ports as part of global supply chains. From here, the metadata is published to Amazon DataZone by using AWS Glue Data Catalog.

IoT 107

IoT Machine Learning Metadata Data-driven 107

CIO Business Intelligence

APRIL 26, 2023

Experimentation broadens expertise, particularly in a rapidly evolving field like technology where being able to learn many new skills is key to both career and enterprise success, he says. Is your organization giving its teams the training they need to keep pace with the latest industry developments?

IT 105

IT Experimentation Forecasting Machine Learning 105

Occam's Razor

APRIL 8, 2013

First, you figure out what you want to improve; then you create an experiment; then you run the experiment; then you measure the results and decide what to do. We are far too enamored with data collection and reporting the standard metrics we love because others love them because someone else said they were nice so many years ago.

Metrics 157

Metrics KPI Analytics Key Performance Indicator 157

DataKitchen

APRIL 13, 2021

To date, we count over 100 companies in the DataOps ecosystem. However, the rush to rebrand existing products with a DataOps message has created some marketplace confusion. Because it is such a new category, both overly narrow and overly broad definitions of DataOps abound. Meta-Orchestration . DevOps Infrastructure Tools.

Testing 300

Testing Machine Learning Consulting Data Quality 300

Occam's Razor

JULY 18, 2011

You just have to have the right mental model (see Seth Godin above) and you have to… wait for it… wait for it… measure everything you do! For everything you do it is important to measure your effectiveness of all three phases of your effort: Acquisition. You’re trying to measure how well you are doing to: Send emails.

Metrics 138

Metrics Marketing Measurement Cost-Benefit 138

datapine

FEBRUARY 4, 2020

Management thinker Peter Drucker once stated, “if you can’t measure it, you can’t improve it” – and he couldn’t be more right. Let’s face it: every serious business that wants to generate leads and revenue needs to have a marketing strategy that will help them in their quest for profit. How do you know that? How To Write A Marketing Report?

Reporting 280

Reporting Marketing Advertising Metrics 280

Occam's Razor

AUGUST 22, 2011

Key To Your Digital Success: Web Analytics Measurement Model. " Measuring Incrementality: Controlled Experiments to the Rescue! Barriers To An Effective Web Measurement Strategy [+ Solutions!]. Measuring Online Engagement: What Role Does Web Analytics Play? "Engagement" How Do I Measure Success?

KPI 125

KPI Metrics Measurement ROI 125

CIO Business Intelligence

DECEMBER 11, 2024

Research from IDC predicts that we will move from the experimentation phase, the GenAI scramble that we saw in 2023 and 2024, and mature into the adoption phase in 2025/26 before moving into AI-fuelled businesses in 2027 and beyond. So what are the leaders doing differently? All of these relate to the lack of experience with AI.

ROI 59

ROI Experimentation Insurance Strategy 59

CIO Business Intelligence

SEPTEMBER 12, 2024

Unmonitored AI tools can lead to decisions or actions that undermine regulatory and corporate compliance measures, particularly in sectors where data handling and processing are tightly regulated, such as finance and healthcare. A routine audit uncovers severe compliance issues with how the tool accesses and stores data.

Experimentation 111

Experimentation Risk Cost-Benefit Strategy 111

CIO Business Intelligence

MAY 1, 2024

Pilots can offer value beyond just experimentation, of course. McKinsey reports that industrial design teams using LLM-powered summaries of user research and AI-generated images for ideation and experimentation sometimes see a reduction upward of 70% in product development cycle times.

Cost-Benefit 143

Cost-Benefit Metrics Insurance Measurement 143

CIO Business Intelligence

JUNE 13, 2024

Proof that even the most rigid of organizations are willing to explore generative AI arrived this week when the US Department of the Air Force (DAF) launched an experimental initiative aimed at Guardians, Airmen, civilian employees, and contractors. For now, AFRL is experimenting with self-hosted open-source LLMs in a controlled environment.

Testing 119

Testing Experimentation Data Processing Modeling 119

CIO Business Intelligence

FEBRUARY 1, 2024

Prioritising and measuring is key Generative AI represents a welcome shot in the arm for a sector in desperate need of efficiency and productivity gains. In the short term, healthcare CIOs need to focus on prioritising their use cases and ensuring they have a robust measuring framework in place to assess the results of trial deployment.

Experimentation 115

Experimentation Forecasting Measurement Cost-Benefit 115

The Unofficial Google Data Science Blog

JULY 22, 2020

Instead, we focus on the case where an experimenter has decided to run a full traffic ramp-up experiment and wants to use the data from all of the epochs in the analysis. When there are changing assignment weights and time-based confounders, this complication must be considered either in the analysis or the experimental design.

Experimentation 96

Experimentation Statistics Testing Knowledge Discovery 96

CIO Business Intelligence

NOVEMBER 13, 2023

So if you are seeking to lead transformational change at your organization, it’s worth knowing the 10 most common reasons why digital transformation fails and what you as an IT leader can learn from those failures. Without a clear understanding of what their digital transformation should achieve, it’s easy for companies to get lost in the weeds.

Digital Transformation 143

Digital Transformation Cost-Benefit Experimentation Measurement 143

Occam's Razor

MARCH 23, 2010

Too many new things are happening too fast and those of us charged with measuring it have to change the wheels while the bicycle is moving at 30 miles per hour (and this bicycle will become a car before we know it – all while it keeps moving, ever faster). Part of it fueled by Vendors. What a competitive bunch! This is sad.

Analytics 119

Analytics Consulting Metrics Experimentation 119

Domino Data Lab

SEPTEMBER 10, 2020

This means many projects get stuck in endless research and experimentation. Identify metrics that measure this variability. Data scientists can spend weeks just trying to find, capture and transform data into decent features for models, not to mention many cycles of training, tuning, and tweaking models so they’re performant.

Modeling 101

Modeling Data Science IT Metrics 101

Data Science 101

JANUARY 15, 2020

Some pitfalls of this type of experimentation include: Suppose an experiment is performed to observe the relationship between the snack habit of a person while watching TV. Reliability: It means measurements should have repeatable results. For eg: you measure the blood pressure of a person. Let us understand this in brief.

Experimentation 96

Experimentation Statistics Measurement Testing 96