CIO Business Intelligence

NOVEMBER 13, 2024

One is going through the big areas where we have operational services and look at every process to be optimized using artificial intelligence and large language models. But a substantial 23% of respondents say the AI has underperformed expectations as models can prove to be unreliable and projects fail to scale.

IT 141

IT Consulting ROI Cost-Benefit 141

Analytics Vidhya

SEPTEMBER 30, 2022

Introduction Evaluation metrics are used to measure the quality of the model. Selecting an appropriate evaluation metric is important because it can impact your selection of a model or decide whether to put your model into production. This article was published as a part of the Data Science Blogathon.

Metrics 397

Metrics Measurement Data Science Publishing 397

O'Reilly on Data

DECEMBER 12, 2019

Not least is the broadening realization that ML models can fail. And that’s why model debugging, the art and science of understanding and fixing problems in ML models, is so critical to the future of ML. Because all ML models make mistakes, everyone who cares about ML should also care about model debugging. [1]

Machine Learning 276

Machine Learning Modeling Testing Risk Management 276

O'Reilly on Data

MARCH 20, 2019

Apply fair and private models, white-hat and forensic model debugging, and common sense to protect machine learning models from malicious actors. Like many others, I’ve known for some time that machine learning models themselves could pose security risks. This is like a denial-of-service (DOS) attack on your model itself.

Modeling 278

Modeling Machine Learning Predictive Modeling Consulting 278

O'Reilly on Data

MARCH 25, 2025

Throughout this article, well explore real-world examples of LLM application development and then consolidate what weve learned into a set of first principlescovering areas like nondeterminism, evaluation approaches, and iteration cyclesthat can guide your work regardless of which models or frameworks you choose. Which multiagent frameworks?

Testing 168

Testing Data-driven Software Measurement 168

O'Reilly on Data

NOVEMBER 19, 2024

Reasons for using RAG are clear: large language models (LLMs), which are effectively syntax engines, tend to “hallucinate” by inventing answers from pieces of their training data. See the primary sources “ REALM: Retrieval-Augmented Language Model Pre-Training ” by Kelvin Guu, et al., at Facebook—both from 2020.

Unstructured Data 312

Unstructured Data Structured Data Statistics Modeling 312

CIO Business Intelligence

APRIL 8, 2025

In many cases, companies should opt for closed, proprietary AI models that arent connected to the internet, ensuring that critical data remains secure within the enterprise. Yet failing to successfully address risk with an effective risk management program is courting disaster. Cybersecurity is now a multi-front war, Selby says.

Risk Management 131

Risk Management Risk Management Business Objectives 131

O'Reilly on Data

OCTOBER 12, 2021

Kevlin Henney and I were riffing on some ideas about GitHub Copilot , the tool for automatically generating code base on GPT-3’s language model, trained on the body of code that’s in GitHub. This article poses some questions and (perhaps) some answers, without trying to present any conclusions. Things like that.

Testing 363

Testing Measurement Consulting Modeling 363

datapine

JANUARY 24, 2021

Additionally, incorporating a decision support system software can save a lot of company’s time – combining information from raw data, documents, personal knowledge, and business models will provide a solid foundation for solving business problems. That being said, it seems like we’re in the midst of a data analysis crisis.

IT 317

IT Statistics KPI Data-driven 317

CIO Business Intelligence

DECEMBER 9, 2024

This is particularly true with enterprise deployments as the capabilities of existing models, coupled with the complexities of many business workflows, led to slower progress than many expected. Foundation models (FMs) by design are trained on a wide range of data scraped and sourced from multiple public sources.

Enterprise 116

Enterprise Cost-Benefit ROI Measurement 116

CIO Business Intelligence

JANUARY 23, 2024

Regardless of where organizations are in their digital transformation, CIOs must provide their board of directors, executive committees, and employees definitions of successful outcomes and measurable key performance indicators (KPIs). He suggests, “Choose what you measure carefully to achieve the desired results.

Measurement 141

Measurement Digital Transformation KPI Metrics 141

CIO Business Intelligence

JANUARY 14, 2025

By articulating fitness functions automated tests tied to specific quality attributes like reliability, security or performance teams can visualize and measure system qualities that align with business goals. In todays digital-first economy, enterprise architecture must also evolve from a control function to an enablement platform.

Enterprise 119

Enterprise Technology Metrics Measurement 119

CIO Business Intelligence

NOVEMBER 7, 2024

As digital transformation becomes a critical driver of business success, many organizations still measure CIO performance based on traditional IT values rather than transformative outcomes. This creates a disconnect between the strategic role that CIOs are increasingly expected to play and how their success is measured.

Digital Transformation 128

Digital Transformation Metrics Measurement Key Performance Indicator 128

CIO Business Intelligence

MARCH 25, 2025

Using the companys data in LLMs, AI agents, or other generative AI models creates more risk. Build up: Databases that have grown in size, complexity, and usage build up the need to rearchitect the model and architecture to support that growth over time.

Risk 140

Risk Cost-Benefit Data-driven Digital Transformation 140

CIO Business Intelligence

DECEMBER 10, 2024

Set clear, measurable metrics around what you want to improve with generative AI, including the pain points and the opportunities, says Shaown Nandi, director of technology at AWS. That gives CIOs breathing room, but not unlimited tether, to prove the value of their gen AI investments.

Sales 143

Sales Metrics Data-driven Unstructured Data 143

CIO Business Intelligence

OCTOBER 24, 2024

Take for instance large language models (LLMs) for GenAI. Then there’s reinforcement learning, a type of machine learning model that trains algorithms to make effective cybersecurity decisions. IT leaders are placing faith in AI. But when it comes to cybersecurity, AI has become a double-edged sword.

Risk 128

Risk Measurement Optimization Machine Learning 128

CIO Business Intelligence

APRIL 8, 2025

These strategies, such as investing in AI-powered cleansing tools and adopting federated governance models, not only address the current data quality challenges but also pave the way for improved decision-making, operational efficiency and customer satisfaction. Data quality is no longer a back-office concern.

Data Quality 102

Data Quality Data-driven Key Performance Indicator Metadata 102

David Menninger's Analyst Perspectives

JANUARY 15, 2025

ISG Research asserts that by 2027, one-third of enterprises will incorporate comprehensive external measures to enable ML to support AI and predictive analytics and achieve more consistently performative planning models. What I discovered is that the availability of this type of vital information is exceedingly slim.

Predictive Modeling 130

Predictive Modeling Forecasting Predictive Analytics Statistics 130

Analytics Vidhya

FEBRUARY 20, 2024

In a bid to bolster cybersecurity measures, Google has unveiled Magika, an AI-driven file detection tool aimed at identifying malicious files with unprecedented speed and accuracy.

Deep Learning 289

Deep Learning Measurement Modeling Analytics 289

CIO Business Intelligence

NOVEMBER 7, 2024

The government also plans to introduce measures to support businesses, particularly small and medium-sized enterprises (SMEs), in adopting responsible AI management practices through a new self-assessment tool. Meanwhile, the measures could also introduce fresh challenges for businesses, particularly SMEs.

Risk 130

Risk Management Measurement Consulting 130

O'Reilly on Data

JUNE 15, 2023

The world changed on November 30, 2022 as surely as it did on August 12, 1908 when the first Model T left the Ford assembly line. If we want prosocial outcomes, we need to design and report on the metrics that explicitly aim for those outcomes and measure the extent to which they have been achieved.

Metrics 359

Metrics Reporting Measurement Finance 359

CIO Business Intelligence

DECEMBER 30, 2024

Small language models and edge computing Most of the attention this year and last has been on the big language models specifically on ChatGPT in its various permutations, as well as competitors like Anthropics Claude and Metas Llama models. Reasoning also helps us use AI as more of a decision support system, he adds.

ROI 141

ROI Software Modeling Interactive 141

David Menninger's Analyst Perspectives

NOVEMBER 19, 2024

Additionally, while the tools available at the time enabled data teams to respond to quality issues, they did not provide a way to identify quality thresholds or measure improvement, making it difficult to demonstrate to the business the value of time spent remedying data-quality problems. With The company has raised $73.5

Data Quality 130

Data Quality Dashboards Data-driven Software 130

CIO Business Intelligence

OCTOBER 23, 2024

To address this, Gartner has recommended treating AI-driven productivity like a portfolio — balancing operational improvements with high-reward, game-changing initiatives that reshape business models. You must understand the cost components and pricing model options, and you need to know how to reduce these costs and negotiate with vendors.

Enterprise 141

Enterprise Cost-Benefit Broadcasting Risk 141

CIO Business Intelligence

FEBRUARY 4, 2025

Instead of seeing digital as a new paradigm for our business, we over-indexed on digitizing legacy models and processes and modernizing our existing organization. This only fortified traditional models instead of breaking down the walls that separate people and work inside our organizations. And its testing us all over again.

Digital Transformation 144

Digital Transformation Sales Optimization Enterprise 144

CIO Business Intelligence

DECEMBER 20, 2024

Its an offshoot of enterprise architecture that comprises the models, policies, rules, and standards that govern the collection, storage, arrangement, integration, and use of data in organizations. AI and machine learning models. An organizations data architecture is the purview of data architects. Ensure security and access controls.

Data Architecture 122

Data Architecture Management Consulting Internet of Things 122

CIO Business Intelligence

NOVEMBER 18, 2024

As a result, organisations are continually investing in cloud to re-invent existing business models and leapfrog their competitors. What began as a need to navigate complex pricing models to better control costs and gain efficiency has evolved into a focus on demonstrating the value of cloud through Unit Economics.

Measurement 111

Measurement Modeling Forecasting Metrics 111

CIO Business Intelligence

DECEMBER 20, 2024

From the discussions, it is clear that today, the critical focus for CISOs, CIOs, CDOs, and CTOs centers on protecting proprietary AI models from attack and protecting proprietary data from being ingested by public AI models. isnt intentionally or accidentally exfiltrated into a public LLM model?

Risk 105

Risk Measurement Modeling Interactive 105

O'Reilly on Data

MARCH 31, 2020

Instead of writing code with hard-coded algorithms and rules that always behave in a predictable manner, ML engineers collect a large number of examples of input and output pairs and use them as training data for their models. You’re responsible for the design, the product-market fit, and ultimately for getting the product out the door.

Management 363

Management Machine Learning Experimentation Metrics 363

CIO Business Intelligence

SEPTEMBER 17, 2024

CISOs can only know the performance and maturity of their security program by actively measuring it themselves; after all, to measure is to know. However, CISOs aren’t typically measuring their security program proactively or methodically to understand their current security program.

Measurement 111

Measurement Metrics Risk Dashboards 111

O'Reilly on Data

MAY 14, 2020

Experimentation: It’s just not possible to create a product by building, evaluating, and deploying a single model. In reality, many candidate models (frequently hundreds or even thousands) are created during the development process. Modelling: The model is often misconstrued as the most important component of an AI product.

Management 364

Management Experimentation B2B Machine Learning 364

CIO Business Intelligence

NOVEMBER 14, 2024

Online will become increasingly central, with the launch of new collections and models, as well as opening in new markets, transacting in different currencies, and using in-depth analytics to make quick decisions.” It’s a change fundamentally based on digital capabilities.

ROI 119

ROI B2B Business Objectives Digital Transformation 119

CIO Business Intelligence

OCTOBER 22, 2024

By 2028, 40% of large enterprises will deploy AI to manipulate and measure employee mood and behaviors, all in the name of profit. “AI CMOs view GenAI as a tool that can launch both new products and business models. AI is evolving as human use of AI evolves.

Enterprise 135

Enterprise Risk IoT Cost-Benefit 135

CIO Business Intelligence

JANUARY 21, 2025

Deloittes State of Generative AI in the Enterprise reports nearly 70% have moved 30% or fewer of their gen AI experiments into production, and 41% of organizations have struggled to define and measure the impacts of their gen AI efforts. Even this breakdown leaves out data management, engineering, and security functions.

Cost-Benefit 142

Cost-Benefit Data-driven Strategy Marketing 142

CIO Business Intelligence

MARCH 19, 2025

Developers, data architects and data engineers can initiate change at the grassroots level from integrating sustainability metrics into data models to ensuring ESG data integrity and fostering collaboration with sustainability teams. However, embedding ESG into an enterprise data strategy doesnt have to start as a C-suite directive.

IT 59

IT Data Governance Data-driven Metrics 59

CIO Business Intelligence

DECEMBER 18, 2024

Our history is rooted in a traditional distribution model of marketing, selling, and shipping vendor products to our resellers. What were the technical considerations moving from a distribution model to a platform? As a platform company, measurement is crucial to success. This is crucial in a value-driven development model.

Digital Transformation 116

Digital Transformation Metrics Measurement ROI 116

CIO Business Intelligence

JUNE 11, 2024

IT’s mission has transformed — perhaps so should its brand Another approach I recommend is to rebrand IT and recast its mission to modernize its objectives, organizational structure, core competencies, and operating model. One way IT leaders convey this transformed mission is to alter the CIO title.

IT 136

IT Digital Transformation Broadcasting Business Objectives 136