Forecasting uncertainty at Airbnb
O'Reilly on Data
MARCH 28, 2019
Theresa Johnson outlines the AI powering Airbnb’s metrics forecasting platform. Continue reading Forecasting uncertainty at Airbnb.
The Lean Analytics Cycle: Metrics > Hypothesis > Experiment > Act
Occam's Razor
APRIL 8, 2013
To win in business you need to follow this process: Metrics > Hypothesis > Experiment > Act. We are far too enamored with data collection and reporting the standard metrics we love because others love them because someone else said they were nice so many years ago. That metric is tied to a KPI.
Metrics 
157
157 Join 42,000+
Insiders
Sign Up for our Newsletter
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.
Trending Sources
Escaping POC Purgatory: Evaluation-Driven Development for AI Systems
O'Reilly on Data
MARCH 25, 2025
Business value : Once we have a rubric for evaluating our systems, how do we tie our macro-level business value metrics to our micro-level LLM evaluations? Business value : Align outputs with business metrics and optimize workflows to achieve measurable ROI. LLM-powered software amplifies this uncertainty further.
Testing 168
168 
Highlights from the Strata Data Conference in San Francisco 2019
O'Reilly on Data
MARCH 27, 2019
Forecasting uncertainty at Airbnb. Theresa Johnson outlines the AI powering Airbnb’s metrics forecasting platform. Watch " Forecasting uncertainty at Airbnb.". Watch " Winners of the Strata Data Awards 2019.". It’s in the game: A rare look into how EA brought data science into the creative process of game design.
Uncertainty 273
273 
What you need to know about product management for AI
O'Reilly on Data
MARCH 31, 2020
Machine learning adds uncertainty. Underneath this uncertainty lies further uncertainty in the development process itself. There are strategies for dealing with all of this uncertainty–starting with the proverb from the early days of Agile: “ do the simplest thing that could possibly work.”
Management 363
363 
You Can’t Regulate What You Don’t Understand
O'Reilly on Data
JUNE 15, 2023
If we want prosocial outcomes, we need to design and report on the metrics that explicitly aim for those outcomes and measure the extent to which they have been achieved. There is no simple way to solve the alignment problem. But alignment will be impossible without robust institutions for disclosure and auditing.
Metrics 359
359 
AI Product Management After Deployment
O'Reilly on Data
OCTOBER 13, 2020
Ideally, AI PMs would steer development teams to incorporate I/O validation into the initial build of the production system, along with the instrumentation needed to monitor model accuracy and other technical performance metrics. But in practice, it is common for model I/O validation steps to be added later, when scaling an AI product.
Management 308
308 
iostudio delivers key metrics to public sector recruiters with Amazon QuickSight
AWS Big Data
JUNE 27, 2023
Our previous solution offered visualization of key metrics, but point-in-time snapshots produced only in PDF format. In this post, we discuss how we built a solution using QuickSight that delivers real-time visibility of key metrics to public sector recruiters.
Metrics 98
98 
3 ways to avoid the generative AI ROI doom loop
CIO Business Intelligence
NOVEMBER 12, 2024
Make ‘soft metrics’ matter Imagine an experienced manager with an “open door policy.” Meaningful improvement is likely to include some quantifiable metrics like time savings or employee satisfaction. He specializes in removing fear, uncertainty, and doubt from strategic decision-making through empirical data and market sensing.
ROI 72
72 
How CIOs can be pillars of stability in an uncertain world
CIO Business Intelligence
JANUARY 6, 2023
The economists lament, “ A thick fog of uncertainty still surrounds us.” Uncertainty is our jam.” minutes of downtime per year), and expanding digital capabilities in a world characterized by massive economic, political, social, and technological uncertainty. After all, uncertainty is the one certainty.
Uncertainty 124
124 
Uncertainties: Statistical, Representational, Interventional
The Unofficial Google Data Science Blog
DECEMBER 14, 2021
by AMIR NAJMI & MUKUND SUNDARARAJAN Data science is about decision making under uncertainty. Some of that uncertainty is the result of statistical inference, i.e., using a finite sample of observations for estimation. But there are other kinds of uncertainty, at least as important, that are not statistical in nature.
Uncertainty 56
56 
What to Expect From Uncertainty Quantification Methods: An Untold Story of ML Evaluation
Dataiku
JULY 6, 2022
Which metrics should be reported? In this article, we relate our experience in building an evaluation framework for Uncertainty Quantification (UQ) in ML. How can we ensure enough diversity in the selected datasets? Which baselines should be included for fair comparison?
Uncertainty 64
64 
Infographic: Optimal Real Estate, Property Management, and Investment Metrics
Jet Global
MAY 15, 2020
Tracking real estate industry metrics is crucial to your organization’s success. With current market uncertainty, it’s important to understand and track the benchmarks that drive your firm—industry trends, regulatory changes, new accounting standards, cost and profitability, as well as guidance on cash flows. Ready to learn more?
Metrics 59
59 
Leveraging Data Science To Grow And Manage Your Team
Smart Data Collective
AUGUST 18, 2021
Although widely used, keyword scanning software alone simply doesn’t generate sufficient success metrics when sifting through candidate resumes. Even though you may hit it off with a candidate in an interview and be impressed with their resume, that doesn’t necessarily mean they’re the right fit for your business. Retaining staff.
Data Science 139
139 
Salesforce to hire 3,300 staffers as it eyes generative AI opportunity
CIO Business Intelligence
SEPTEMBER 15, 2023
Salesforce sees boomerang hires as a new success metric, the top executives said, adding that there still might be strategic layoffs in the future. The top executives said that some of the new positions are more likely to be filled by what the company terms as “boomerang” hires.
IT 132
132 
5 signs your agile practices will lead to digital disaster
CIO Business Intelligence
SEPTEMBER 20, 2023
They are afraid of failure and the uncertainty of knowledge work, and so that’s stressful. Agile is an amazing risk management tool for managing uncertainty, but that’s not always obvious.” The key is recognizing that planning must be an agile discipline, not a standalone activity performed independently of agile teams.
Bridging the Gap: How ‘Data in Place’ and ‘Data in Use’ Define Complete Data Observability
DataKitchen
SEPTEMBER 21, 2023
The uncertainty of not knowing where data issues will crop up next and the tiresome game of ‘who’s to blame’ when pinpointing the failure. Moreover, advanced metrics like Percentage Regional Sales Growth can provide nuanced insights into business performance. One of the primary sources of tension?
Testing 169
169 
Why HR professionals struggle with big data
CIO Business Intelligence
FEBRUARY 20, 2025
This is due, on the one hand, to the uncertainty associated with handling confidential, sensitive data and, on the other hand, to a number of structural problems. Solid reporting provides transparent, consistent and combined HR metrics essential for strategic planning, risk management and the management of HR measures.
Big Data 84
84 
Three Emerging Analytics Products Derived from Value-driven Data Innovation and Insights Discovery in the Enterprise
Rocket-Powered Data Science
JULY 19, 2023
For example, an exploration of historical data may reveal that an increase in customer satisfaction (or dissatisfaction) with one particular product is correlated with some other satisfaction (or dissatisfaction) metric downstream at a later date.
Data-driven 188
188 
Humans-in-the-loop forecasting: integrating data science and business planning
The Unofficial Google Data Science Blog
DECEMBER 4, 2019
This classification is based on the purpose, horizon, update frequency and uncertainty of the forecast. A single model may also not shed light on the uncertainty range we actually face. These characteristics of the problem drive the forecasting approaches.
Forecasting 115
115 
In AI we trust? Why we Need to Talk About Ethics and Governance (part 2 of 2)
Cloudera
DECEMBER 3, 2021
Systems should be designed with bias, causality and uncertainty in mind. Uncertainty is a measure of our confidence in the predictions made by a system. We need to understand and provide the greatest human oversight on systems with the greatest levels of uncertainty. System Design. Human Judgement & Oversight. Model Drift.
Uncertainty 102
102 
Why CIOs should invest in digital through economic headwinds
CIO Business Intelligence
NOVEMBER 15, 2022
Resilient cybersecurity Despite the clamour for new digital investments, Gartner’s analysts did recognise that this would represent a new cybersecurity risk, with some attributing the increased spending in security over the next year down to ongoing uncertainty regarding Russia’s invasion of Ukraine.
Top Five Financial KPIs for Utilities Industry
Jet Global
SEPTEMBER 28, 2020
COVID-19 introduces another level of uncertainty. Therefore, decision-makers need to track this metric constantly (ideally in real time) rather than getting weekly, daily, or even hourly updates. Lenders use this metric to inform financing decisions, which many utilities will need in the coming years to cover new construction costs.
Customer service, especially field service, helps companies outperform competitors and drive growth
CIO Business Intelligence
FEBRUARY 7, 2023
In economic uncertainty, it’s natural for executives to explore where to reduce spending, trim the fat , so to speak, and cut enterprising investments as a matter of caution. But this thinking is also counter-productive for all the reasons that make uncertainty so predictable. There’s much to be done.
Uncertainty 116
116 
Save Time and Reduce Errors in Your PeopleSoft Reporting
Jet Global
DECEMBER 3, 2019
PeopleSoft data automatically upload into dozens of pre-built templates designed to track the most important metrics and processes related to enterprise performance. The reports themselves even have drill-down capabilities that show users the data behind each metric. Taking Full Advantage of Better Data.
Reporting 98
98 
IT leader’s survival guide: 11 ways to thrive in the years ahead
CIO Business Intelligence
JUNE 8, 2022
Digital disruption, global pandemic, geopolitical crises, economic uncertainty — volatility has thrown into question time-honored beliefs about how best to lead IT. Thriving amid uncertainty means staying flexible, he argues. . Demand transparency on everything — especially on metrics — and be transparent back.
IT 133
133 
CIOs must beware committing ‘AI washing’ themselves
CIO Business Intelligence
JULY 23, 2024
Gen AI can still hallucinate, even if tuned, creating a level of uncertainty when more traditional tools would be more consistent. It’s a scary level of uncertainty and risk, and that makes it difficult to use as a rip and replace for existing technologies.”
Uncertainty 131
131 
What to Look for in a Modern DevOps Solution
CIO Business Intelligence
JUNE 2, 2022
Businesses today have faced greater levels of uncertainty than ever before. Time saved is an important metric for any DevOps administrator. By Milan Shetti, CEO Rocket Software. Ease of implementation and use.
Uncertainty 130
130 
Promising Benefits of Predictive Analytics in Asset Management
Smart Data Collective
FEBRUARY 14, 2023
Their research showed that economic policy uncertainty indicators and CBOE volatility index (VIX) indicators are better for predicting market volatility than financial indices. Therefore, these metrics are likely to be used by most predictive analytics tools used to ascertain risk. Are there any risks associated with asset allocation?
Top Five Financial KPIs for the Construction Industry
Jet Global
MARCH 27, 2020
Now that every industry is facing a prolonged period of uncertainty, having access to accurate, up-to-date information is more critical than ever. With so much uncertainty on the horizon, everyone in the construction industry needs to be at the top of their game , especially in terms of finance and accounting. Bonding Capacity.
Uncertainty 64
64 
Integrate sparse and dense vectors to enhance knowledge retrieval in RAG using Amazon OpenSearch Service
AWS Big Data
SEPTEMBER 5, 2024
Although the absolute metrics of the sparse vector model can’t surpass those of the best dense vector models, it possesses unique and advantageous characteristics. Performance evaluation of retrieval In RAG knowledge retrieval, we usually focus on the relevance of top results, so our evaluation uses recall@4 as the metric indicator.
Metrics 105
105 
How AI Can Improve Your Annotation Quality?
Smart Data Collective
JULY 1, 2023
Consistency and agreement Establish an agreement metric (e.g., Address their questions and clarify any uncertainties promptly. Establish metrics for review Create specific metrics for measuring annotation quality and use them to evaluate both individual annotators and the overall annotation process.
Rebooting expectations to connect and lead in more meaningful ways
CIO Business Intelligence
SEPTEMBER 22, 2022
And they introduced this idea, the VUCA world: Volatile, Uncertainty, Complex, and Ambiguous. So while vision will be disproportionately rewarded, uncertainty yields to understanding, complexity yields to clarity, and ambiguity yields to agility. On metrics: It’s been said a lot that data is the new oil.
Uncertainty 111
111 
Businesses Discover the Importance of Merging Analytics and Content Marketing
Smart Data Collective
AUGUST 19, 2021
Several metrics exist to help you figure this out. Analytics opens up a whole new world for you and takes the uncertainty off identifying your target audience. Analytics can also help you collate key metrics such as click-through rates, returning customers on the site, time spent on pages, etc. What are their patterns?
Marketing 134
134 
Estimating the prevalence of rare events — theory and practice
The Unofficial Google Data Science Blog
AUGUST 27, 2019
Of course, any mistakes by the reviewers would propagate to the accuracy of the metrics, and the metrics calculation should take into account human errors. If we could separate bad videos from good videos perfectly, we could simply calculate the metrics directly without sampling. The missing verdicts create two problems.
Metrics 98
98 
Self-Service Reporting for a Remote Staff: How and Why
Jet Global
JUNE 23, 2020
As companies grapple with economic shutdowns, plummeting revenues, market instability, and economic uncertainty now and into the foreseeable future, they need financial insights more than ever before. Every accounting department handles financial reporting differently depending on its metrics, strategy, or audience.
Reporting 98
98 
Automation for all—managing and scaling networks has never been easier
CIO Business Intelligence
APRIL 27, 2023
At this time of dynamic business and market changes, uncertainty, and quickly evolving consumption models for IT infrastructure, every IT executive understands the benefits and necessity of network agility. Agile networks can respond quickly to changes in the market, customer demands, employee requirements, and technology advances.
Management 98
98 
Operational Finance in the Age of Covid-19: Time to Change the Basics?
Jet Global
OCTOBER 8, 2020
And it’s possible to become lost in the minutiae of the many different metrics available to measure an organisation’s AR capabilities. Analysing the necessary data is a massive undertaking, and one that can draw finance professionals away from other tasks.
Finance 98
98 
Lack of Time for Analysis, Adoption of New Technology and Data Limitations are Top Challenges Facing Finance Teams, According to insightsoftware Survey
Jet Global
MAY 11, 2021
In terms of the key metrics finance teams are monitoring to signal a broader financial recovery, respondents named sales and revenue (35%), operating cash flow (29%) and net profit margin (27%) as the top three. COVID-19 Response & Economic Recovery Indicators.
Finance 98
98 
8 ways to retain top developer talent
CIO Business Intelligence
MARCH 10, 2023
Because of this, the importance of code quality shows why project velocity is not a metric to be seen in isolation as it often is. Mitigate DX-killing red tape The business longs for metrics and insights into what’s happening in the dark interior of software creation. But too much intrusion into developer workflow is a real DX killer.
Software 122
122 
Towards optimal experimentation in online systems
The Unofficial Google Data Science Blog
APRIL 23, 2024
the weight given to Likes in our video recommendation algorithm) while $Y$ is a vector of outcome measures such as different metrics of user experience (e.g., Crucially, it takes into account the uncertainty inherent in our experiments. Here, $X$ is a vector of tuning parameters that control the system's operating characteristics (e.g.
How to Ensure Single Source of Truth Reporting for PeopleSoft
Jet Global
NOVEMBER 15, 2019
Members of the finance or IT teams have to go hunting through multiple data sources, identifying and integrating the metrics they need to build reports. Users can drill down into the facts and figures that make up key metrics to understand the forces having the biggest impact on enterprise performance. Download Now. How accessible?
Reporting 75
75 
PODCAST: COVID19 | Redefining Digital Enterprises – Episode 10: COVID-19 | Evolving Trends of Digital Transformation
bridgei2i
JULY 8, 2020
This is probably the first time ever that we are witnessing a demand, a supply, and also a resource uncertainty. Effectiveness, which is, how can any form of digital information help us drive business metrics. Vignesh: Ganesh, you really highlighted some very important themes. These are strange times. And all at the same time.
How Data Can Help You Manage Extreme Change
Sisense
MARCH 26, 2020
Now is the time to apply the full force of business intelligence used by analytics teams to help navigate growing uncertainty. Now is the time to apply the full force of business intelligence used by analytics teams to help navigate growing uncertainty.” Evan Castle, Sisense Head of BI & Analytics Products.
Management 91
91 
Let's personalize your content