CIO Business Intelligence

OCTOBER 23, 2024

Despite AI’s potential to transform businesses, many senior technology leaders find themselves wrestling with unpredictable expenses, uneven productivity gains, and growing risks as AI adoption scales, Gartner said. CIOs should create proofs of concept that test how costs will scale, not just how the technology works.”

Enterprise 141

Enterprise Cost-Benefit Broadcasting Risk 141

O'Reilly on Data

NOVEMBER 28, 2023

And everyone has opinions about how these language models and art generation programs are going to change the nature of work, usher in the singularity, or perhaps even doom the human race. 16% of respondents working with AI are using open source models. A few have even tried out Bard or Claude, or run LLaMA 1 on their laptop.

Enterprise 342

Enterprise Testing Modeling Reporting 342

O'Reilly on Data

OCTOBER 22, 2020

Taking the time to work this out is like building a mathematical model: if you understand what a company truly does, you don’t just get a better understanding of the present, but you can also predict the future. Since I work in the AI space, people sometimes have a preconceived notion that I’ll only talk about data and models.

Consulting 358

Consulting Risk Cost-Benefit Modeling 358

O'Reilly on Data

JULY 28, 2020

Product Managers are responsible for the successful development, testing, release, and adoption of a product, and for leading the team that implements those milestones. You must detect when the model has become stale, and retrain it as necessary. The Core Responsibilities of the AI Product Manager. The AI Product Development Process.

Marketing 364

Marketing Experimentation Metrics Testing 364

CIO Business Intelligence

DECEMBER 18, 2024

Financial institutions have an unprecedented opportunity to leverage AI/GenAI to expand services, drive massive productivity gains, mitigate risks, and reduce costs. GenAI is also helping to improve risk assessment via predictive analytics.

Forecasting 105

Forecasting Predictive Analytics Risk Finance 105

CIO Business Intelligence

JANUARY 30, 2025

These changes can expose businesses to risks and vulnerabilities such as security breaches, data privacy issues and harm to the companys reputation. It also includes managing the risks, quality and accountability of AI systems and their outcomes. It is easy to see how the detractions can get in the way. Start with: An AI culture.

Testing 93

Testing Metrics Cost-Benefit Modeling 93

Domino Data Lab

AUGUST 6, 2021

All models require testing and auditing throughout their deployment and, because models are continually learning, there is always an element of risk that they will drift from their original standards. As such, model governance needs to be applied to each model for as long as it’s being used.

Machine Learning 145

Machine Learning Modeling Testing Data Science 145

O'Reilly on Data

MAY 27, 2020

Under school district policy, each of Audrey’s eleven- and twelve-year old students is tested at least three times a year to determine his or her Lexile, a number between 200 and 1,700 that reflects how well the student can read. They test each student’s grasp of a particular sentence or paragraph—but not of a whole story.

Risk 361

Risk Testing Reporting Measurement 361

CIO Business Intelligence

AUGUST 21, 2024

According to Gartner, an agent doesn’t have to be an AI model. Starting in 2018, the agency used agents, in the form of Raspberry PI computers running biologically-inspired neural networks and time series models, as the foundation of a cooperative network of sensors. “It Adding smarter AI also adds risk, of course. “At

Risk 136

Risk Insurance Cost-Benefit Software 136

O'Reilly on Data

SEPTEMBER 19, 2023

Stage 2: Machine learning models Hadoop could kind of do ML, thanks to third-party tools. While data scientists were no longer handling Hadoop-sized workloads, they were trying to build predictive models on a different kind of “large” dataset: so-called “unstructured data.” And it was good.

Machine Learning 335

Machine Learning Testing Modeling Cost-Benefit 335

CIO Business Intelligence

NOVEMBER 13, 2024

One is going through the big areas where we have operational services and look at every process to be optimized using artificial intelligence and large language models. But a substantial 23% of respondents say the AI has underperformed expectations as models can prove to be unreliable and projects fail to scale.

IT 141

IT Consulting ROI Cost-Benefit 141

O'Reilly on Data

MARCH 23, 2023

What is it, how does it work, what can it do, and what are the risks of using it? It’s important to understand that ChatGPT is not actually a language model. It’s a convenient user interface built around one specific language model, GPT-3.5, The GPT-series LLMs are also called “foundation models.” GPT-2, 3, 3.5,

IT 344

IT Modeling Testing Risk 344

O'Reilly on Data

MAY 25, 2021

GPT-3 is essentially an auto-complete bot whose underlying Machine Learning (ML) model has been trained on vast quantities of text available on the Internet. I’d like to share my thoughts on GPT-3 in terms of risks and countermeasures, and discuss real examples of how I have interacted with the model to support my learning journey.

Modeling 363

Modeling Marketing Testing Machine Learning 363

Rocket-Powered Data Science

FEBRUARY 15, 2023

While generative AI has been around for several years , the arrival of ChatGPT (a conversational AI tool for all business occasions, built and trained from large language models) has been like a brilliant torch brought into a dark room, illuminating many previously unseen opportunities. So, if you have 1 trillion data points (g.,

Strategy 290

Strategy Experimentation Uncertainty Machine Learning 290

AWS Big Data

JANUARY 9, 2025

This simplifies data modification processes, which is crucial for ingesting and updating large volumes of market and trade data, quickly iterating on backtesting and reprocessing workflows, and maintaining detailed audit trails for risk and compliance requirements. At petabyte scale, Icebergs advantages become clear.

Metadata 111

Metadata Snapshot Cost-Benefit Optimization 111

O'Reilly on Data

JULY 15, 2019

In recent posts, we described requisite foundational technologies needed to sustain machine learning practices within organizations, and specialized tools for model development, model governance, and model operations/testing/monitoring. Note that the emphasis of SR 11-7 is on risk management.). Image by Ben Lorica.

Machine Learning 48

Machine Learning Management Enterprise Risk Management 48

CIO Business Intelligence

MARCH 4, 2025

From AI models that boost sales to robots that slash production costs, advanced technologies are transforming both top-line growth and bottom-line efficiency. The takeaway is clear: embrace deep tech now, or risk being left behind by those who do. Today, that timeline is shrinking dramatically. Thats a remarkably short horizon for ROI.

Technology 118

Technology ROI Cost-Benefit Manufacturing 118

CIO Business Intelligence

JUNE 13, 2024

Not instant perfection The NIPRGPT experiment is an opportunity to conduct real-world testing, measuring generative AI’s computational efficiency, resource utilization, and security compliance to understand its practical applications. It is not training the model, nor are responses refined based on any user inputs.

Testing 119

Testing Experimentation Data Processing Modeling 119

CIO Business Intelligence

DECEMBER 17, 2024

What are the associated risks and costs, including operational, reputational, and competitive? For AI models to succeed, they must be fed high-quality data thats accurate, up-to-date, secure, and complies with privacy regulations such as the Colorado Privacy Act, California Consumer Privacy Act, or General Data Protection Regulation (GDPR).

Experimentation 98

Experimentation ROI Risk Data Quality 98

DataKitchen

SEPTEMBER 20, 2021

The best way to ensure error-free execution of data production is through automated testing and monitoring. The DataKitchen Platform enables data teams to integrate testing and observability into data pipeline orchestrations. Automated tests work 24×7 to ensure that the results of each processing stage are accurate and correct.

Testing 246

Testing Metrics Measurement Dashboards 246

David Menninger's Analyst Perspectives

SEPTEMBER 24, 2024

erroneous results), and an equal amount (32%) mentioned legal risk. AI governance should address a number of issues, including data privacy, bias in data and models, drift in model accuracy, hallucinations and toxicity. Red-teaming is a term used to describe human testing of models for vulnerabilities.

Testing 173

Testing Modeling Enterprise Risk 173

CIO Business Intelligence

FEBRUARY 26, 2025

AI agents are powered by gen AI models but, unlike chatbots, they can handle more complex tasks, work autonomously, and be combined with other AI agents into agentic systems capable of tackling entire workflows, replacing employees or addressing high-level business goals. D&B is not alone in worrying about the risks of AI agents.

IT 141

IT Enterprise Interactive Data Quality 141

CIO Business Intelligence

JANUARY 30, 2025

Chinese AI startup DeepSeek made a big splash last week when it unveiled an open-source version of its reasoning model, DeepSeek-R1, claiming performance superior to OpenAIs o1 generative pre-trained transformer (GPT). That echoes a statement issued by NVIDIA on Monday: DeepSeek is a perfect example of test time scaling.

Modeling 117

Modeling Data-driven Technology Strategy 117

CIO Business Intelligence

JANUARY 19, 2023

Giving up control: rewards outweigh the risks The benefits of greater delegation include an increase in the velocity of decision making and an increased sense of ownership and accountability, says Cisco Sanchez, SVP and CIO at Qualcomm, which has been increasing the pace of delegation. They just need visibility.

Testing 134

Testing Digital Transformation Risk Management 134

CIO Business Intelligence

JANUARY 21, 2025

DevOps teams follow their own practices of using continuous integration and continuous deployment (CI/CD) tools to automatically merge code changes and automate testing steps to deploy changes more frequently and reliably. With this information, teams can ask the AI agent additional questions such as Should I approve the change?

Risk 98

Risk Data-driven Interactive Management 98

CIO Business Intelligence

MARCH 5, 2025

If they decide a project could solve a big enough problem to merit certain risks, they then make sure they understand what type of data will be needed to address the solution. The next thing is to make sure they have an objective way of testing the outcome and measuring success. But we dont ignore the smaller players.

Testing 82

Testing Measurement Technology Experimentation 82

CIO Business Intelligence

DECEMBER 11, 2024

To solve the problem, the company turned to gen AI and decided to use both commercial and open source models. With security, many commercial providers use their customers data to train their models, says Ringdahl. Thats one of the catches of proprietary commercial models, he says. Its possible to opt-out, but there are caveats.

Cost-Benefit 115

Cost-Benefit Modeling Marketing Sales 115

O'Reilly on Data

MAY 14, 2020

Experimentation: It’s just not possible to create a product by building, evaluating, and deploying a single model. In reality, many candidate models (frequently hundreds or even thousands) are created during the development process. Modelling: The model is often misconstrued as the most important component of an AI product.

Management 364

Management Experimentation B2B Machine Learning 364

O'Reilly on Data

JANUARY 17, 2023

Building Models. A common task for a data scientist is to build a predictive model. You’ll try this with a few other algorithms, and their respective tuning parameters–maybe even break out TensorFlow to build a custom neural net along the way–and the winning model will be the one that heads to production.

Machine Learning 363

Machine Learning Predictive Modeling Software Modeling 363

O'Reilly on Data

MARCH 26, 2024

TL;DR LLMs and other GenAI models can reproduce significant chunks of training data. Researchers are finding more and more ways to extract training data from ChatGPT and other models. And the space is moving quickly: SORA , OpenAI’s text-to-video model, is yet to be released and has already taken the world by storm.

Modeling 322

Modeling Machine Learning Risk Advertising 322

CIO Business Intelligence

MARCH 4, 2025

In fact, successful recovery from cyberattacks and other disasters hinges on an approach that integrates business impact assessments (BIA), business continuity planning (BCP), and disaster recovery planning (DRP) including rigorous testing. See also: How resilient CIOs future-proof to mitigate risks.)

Testing 64

Testing Risk Consulting Risk Management 64

CIO Business Intelligence

MARCH 31, 2025

The 2024 Security Priorities study shows that for 72% of IT and security decision makers, their roles have expanded to accommodate new challenges, with Risk management, Securing AI-enabled technology and emerging technologies being added to their plate. Regular engagement with the board and business leaders ensures risk visibility.

Data-driven 71

Data-driven Risk Risk Management Metrics 71

O'Reilly on Data

JUNE 7, 2022

DeepMind’s new model, Gato, has sparked a debate on whether artificial general intelligence (AGI) is nearer–almost at hand–just a matter of scale. Gato is a model that can solve multiple unrelated problems: it can play a large number of different games, label images, chat, operate a robot, and more. If we had AGI, how would we know it?

Modeling 364

Modeling Interactive Optimization Deep Learning 364

CIO Business Intelligence

NOVEMBER 4, 2024

Mark Read, CEO of global advertising giant WPP recently told shareholders: “AI will also offer the ability to develop new business and financial models.” Lead the conversation with the board on risks, pros and cons, and talk like a businessperson. Do not dismiss yourself from being the driver, and reinvent yourself,” Langer advises.

Business Driver 111

Business Driver Advertising Data-driven Modeling 111

O'Reilly on Data

SEPTEMBER 15, 2020

In my book, I introduce the Technical Maturity Model: I define technical maturity as a combination of three factors at a given point of time. Technical competence results in reduced risk and uncertainty. AI initiatives may also require significant considerations for governance, compliance, ethics, cost, and risk.

Advertising 357

Advertising Cost-Benefit ROI Machine Learning 357

DataKitchen

NOVEMBER 20, 2021

A DataOps Engineer can make test data available on demand. We have automated testing and a system for exception reporting, where tests identify issues that need to be addressed. It then autogenerates QC tests based on those rules. Let’s say a data scientist has developed a model that works perfectly with training data.

Testing 246

Testing Machine Learning Data Warehouse Analytics 246

datapine

APRIL 15, 2020

Your Chance: Want to test an agile business intelligence solution? Business intelligence is moving away from the traditional engineering model: analysis, design, construction, testing, and implementation. In the traditional model communication between developers and business users is not a priority. Finalize testing.

Business Intelligence 276

Business Intelligence Analytics Testing Dashboards 276