O'Reilly on Data

OCTOBER 19, 2021

Let’s start by considering the job of a non-ML software engineer: writing traditional software deals with well-defined, narrowly-scoped inputs, which the engineer can exhaustively and cleanly model in the code. Not only is data larger, but models—deep learning models in particular—are much larger than before.

IT 363

IT Testing Experimentation Software 363

DataKitchen

APRIL 13, 2021

Testing and Data Observability. DataOps needs a directed graph-based workflow that contains all the data access, integration, model and visualization steps in the data analytic production process. It orchestrates complex pipelines, toolchains, and tests across teams, locations, and data centers. Testing and Data Observability.

Testing 300

Testing Machine Learning Consulting Data Science 300

DataKitchen

SEPTEMBER 20, 2021

The best way to ensure error-free execution of data production is through automated testing and monitoring. The DataKitchen Platform enables data teams to integrate testing and observability into data pipeline orchestrations. Automated tests work 24×7 to ensure that the results of each processing stage are accurate and correct.

Testing 246

Testing Metrics Measurement Dashboards 246

David Menninger's Analyst Perspectives

NOVEMBER 19, 2024

As a result, many data teams were not as productive as they might be, with time and effort spent on manually troubleshooting data-quality issues and testing data pipelines. The ability to monitor and measure improvements in data quality relies on instrumentation.

Data Quality 130

Data Quality Dashboards Data-driven Software 130

CIO Business Intelligence

NOVEMBER 20, 2023

Measuring developer productivity has long been a Holy Grail of business. In addition, system, team, and individual productivity all need to be measured. The inner loop comprises activities directly related to creating the software product: coding, building, and unit testing. And like the Holy Grail, it has been elusive.

Measurement 126

Measurement Metrics Testing Software 126

CIO Business Intelligence

APRIL 5, 2023

In a joint study with Markus Westner and Tobias Held from the department of computer science and mathematics at the University of Regensburg, the 4C experts examined the topic by focusing on how the IT value proposition is measured, made visible, and communicated. They also tested the concept in a German mechanical engineering company.

Measurement 133

Measurement IT Metrics Consulting 133

O'Reilly on Data

JUNE 18, 2024

Many farmers measure their yield in bags of rice, but what is “a bag of rice”? While RAG is conceptually simple—look up relevant documents and construct a prompt that tells the model to build its response from them—in practice, it’s more complex. Digital Green tests with “Golden QAs,” highly rated sets of questions and answers.

Testing 316

Testing Software Modeling Measurement 316

CIO Business Intelligence

MARCH 25, 2025

Using the companys data in LLMs, AI agents, or other generative AI models creates more risk. Build up: Databases that have grown in size, complexity, and usage build up the need to rearchitect the model and architecture to support that growth over time.

Risk 123

Risk Cost-Benefit Data-driven Digital Transformation 123

O'Reilly on Data

FEBRUARY 15, 2022

But a recent discussion of Google’s new Large Language Models (LLMs), and its claim that one of these models (named Gopher) has demonstrated reading comprehension approaching human performance , has spurred some thoughts about comprehension, ambiguity, intelligence, and will. Ethics is for beings who can make choices.

Testing 357

Testing Modeling Measurement IT 357

CIO Business Intelligence

OCTOBER 23, 2024

To address this, Gartner has recommended treating AI-driven productivity like a portfolio — balancing operational improvements with high-reward, game-changing initiatives that reshape business models. You must understand the cost components and pricing model options, and you need to know how to reduce these costs and negotiate with vendors.

Enterprise 141

Enterprise Cost-Benefit Broadcasting Risk 141

O'Reilly on Data

MAY 27, 2020

Using the new scores, Apgar and her colleagues proved that many infants who initially seemed lifeless could be revived, with success or failure in each case measured by the difference between an Apgar score at one minute after birth, and a second score taken at five minutes. Books, in turn, get matching scores to reflect their difficulty.

Risk 361

Risk Testing Reporting Measurement 361

O'Reilly on Data

MAY 14, 2020

Experimentation: It’s just not possible to create a product by building, evaluating, and deploying a single model. In reality, many candidate models (frequently hundreds or even thousands) are created during the development process. Modelling: The model is often misconstrued as the most important component of an AI product.

Management 364

Management Experimentation B2B Machine Learning 364

Rocket-Powered Data Science

FEBRUARY 15, 2023

While generative AI has been around for several years , the arrival of ChatGPT (a conversational AI tool for all business occasions, built and trained from large language models) has been like a brilliant torch brought into a dark room, illuminating many previously unseen opportunities. So, if you have 1 trillion data points (g.,

Strategy 290

Strategy Experimentation Uncertainty Machine Learning 290

O'Reilly on Data

OCTOBER 13, 2020

Similarly, in “ Building Machine Learning Powered Applications: Going from Idea to Product ,” Emmanuel Ameisen states: “Indeed, exposing a model to users in production comes with a set of challenges that mirrors the ones that come with debugging a model.”. Debugging AI Products.

Management 313

Management Machine Learning Metrics Modeling 313

CIO Business Intelligence

FEBRUARY 4, 2025

Instead of seeing digital as a new paradigm for our business, we over-indexed on digitizing legacy models and processes and modernizing our existing organization. This only fortified traditional models instead of breaking down the walls that separate people and work inside our organizations. And its testing us all over again.

Digital Transformation 144

Digital Transformation Sales Optimization Enterprise 144

CIO Business Intelligence

JANUARY 30, 2025

While there is a lot of effort and content that is now available, it tends to be at a higher level which will require work to be done to create a governance model specifically for your organization. Governance is action and there are many actions an organization can take to create and implement an effective AI governance model.

Testing 93

Testing Metrics Cost-Benefit Modeling 93

CIO Business Intelligence

JUNE 13, 2024

Not instant perfection The NIPRGPT experiment is an opportunity to conduct real-world testing, measuring generative AI’s computational efficiency, resource utilization, and security compliance to understand its practical applications. It is not training the model, nor are responses refined based on any user inputs.

Testing 119

Testing Experimentation Data Processing Modeling 119

CIO Business Intelligence

MARCH 5, 2025

The next thing is to make sure they have an objective way of testing the outcome and measuring success. Large software vendors are used to solving the integration problems that enterprises deal with on a daily basis, says Lee McClendon, chief digital and technology officer at software testing company Tricentis.

Testing 82

Testing Measurement Technology Experimentation 82

DataKitchen

NOVEMBER 29, 2021

Model developers will test for AI bias as part of their pre-deployment testing. Quality test suites will enforce “equity,” like any other performance metric. Continuous testing, monitoring and observability will prevent biased models from deploying or continuing to operate. Companies Commit to Remote.

Testing 245

Testing Data Lake Data Architecture Manufacturing 245

DataKitchen

SEPTEMBER 18, 2024

DataOps introduces agility by advocating for: Measuring data quality early : Data quality leaders should begin measuring and assessing data quality even before perfect standards are in place. Early measurements provide valuable insights that can guide future improvements. Measuring and Refining : DataOps is an iterative process.

Scorecard 177

Scorecard Data Quality Measurement Testing 177

AWS Big Data

DECEMBER 5, 2024

To address this, we used the AWS performance testing framework for Apache Kafka to evaluate the theoretical performance limits. We conducted performance and capacity tests on the test MSK clusters that had the same cluster configurations as our development and production clusters.

Metrics 95

Metrics Dashboards Testing Optimization 95

O'Reilly on Data

SEPTEMBER 15, 2020

In my book, I introduce the Technical Maturity Model: I define technical maturity as a combination of three factors at a given point of time. Technical sophistication: Sophistication measures a team’s ability to use advanced tools and techniques (e.g., PyTorch, TensorFlow, reinforcement learning, self-supervised learning).

Advertising 357

Advertising Cost-Benefit ROI Machine Learning 357

DataRobot Blog

JULY 22, 2022

Using AI-based models increases your organization’s revenue, improves operational efficiency, and enhances client relationships. You need to know where your deployed models are, what they do, the data they use, the results they produce, and who relies upon their results. That requires a good model governance framework.

Modeling 105

Modeling Risk Management Testing Risk 105

DataKitchen

JANUARY 20, 2021

Centralizing analytics helps the organization standardize enterprise-wide measurements and metrics. Develop/execute regression testing . Test data management and other functions provided ‘as a service’ . Central DataOps process measurement function with reports. Agile ticketing/Kanban tools. Deploy to production.

Metrics 243

Metrics Experimentation Measurement Testing 243

O'Reilly on Data

MARCH 23, 2023

It’s important to understand that ChatGPT is not actually a language model. It’s a convenient user interface built around one specific language model, GPT-3.5, is one of a class of language models that are sometimes called “large language models” (LLMs)—though that term isn’t very helpful. with specialized training.

IT 344

IT Modeling Testing Risk 344

O'Reilly on Data

MARCH 7, 2019

Language understanding benefits from every part of the fast-improving ABC of software: AI (freely available deep learning libraries like PyText and language models like BERT ), big data (Hadoop, Spark, and Spark NLP ), and cloud (GPU's on demand and NLP-as-a-service from all the major cloud providers). Azure Text Analytics. Stanford Core NLP.

Deep Learning 275

Deep Learning Testing Machine Learning Modeling 275

DataKitchen

MAY 10, 2021

Some will argue that observability is nothing more than testing and monitoring applications using tests, metrics, logs, and other artifacts. Since 2008, teams working for our founding team and our customers have delivered 100s of millions of data sets, dashboards, and models with almost no errors. Tie tests to alerts.

Testing 214

Testing Manufacturing Data Quality Statistics 214

O'Reilly on Data

OCTOBER 22, 2020

Taking the time to work this out is like building a mathematical model: if you understand what a company truly does, you don’t just get a better understanding of the present, but you can also predict the future. Since I work in the AI space, people sometimes have a preconceived notion that I’ll only talk about data and models.

Consulting 358

Consulting Risk Cost-Benefit Modeling 358

O'Reilly on Data

MARCH 31, 2020

Instead of writing code with hard-coded algorithms and rules that always behave in a predictable manner, ML engineers collect a large number of examples of input and output pairs and use them as training data for their models. This has serious implications for software testing, versioning, deployment, and other core development processes.

Management 363

Management Machine Learning Experimentation Metrics 363

O'Reilly on Data

AUGUST 4, 2022

This kind of humility is likely to deliver more meaningful progress and a more measured understanding of such progress. DeepMind’s Gato is an AI model that can be taught to carry out many different kinds of tasks based on a single transformer neural network. We typically underappreciate how complex such systems are.

Machine Learning 339

Machine Learning Modeling Statistics Software 339

DataKitchen

NOVEMBER 20, 2021

A DataOps Engineer can make test data available on demand. We have automated testing and a system for exception reporting, where tests identify issues that need to be addressed. It then autogenerates QC tests based on those rules. You can track, measure and create graphs and reporting in an automated way.

Testing 246

Testing Machine Learning Data Warehouse Analytics 246

CIO Business Intelligence

NOVEMBER 13, 2024

One is going through the big areas where we have operational services and look at every process to be optimized using artificial intelligence and large language models. But a substantial 23% of respondents say the AI has underperformed expectations as models can prove to be unreliable and projects fail to scale.

IT 141

IT Consulting ROI Cost-Benefit 141

O'Reilly on Data

JUNE 24, 2020

The argument is that some systems are intrinsically difficult to model. You can’t control for, or even measure, several of these factors. Wearing masks as a prophylactic measure isn’t the big cultural leap that it has been in the United States. What does that mean?

Measurement 335

Measurement Testing Interactive Modeling 335

datapine

APRIL 15, 2020

Your Chance: Want to test an agile business intelligence solution? Business intelligence is moving away from the traditional engineering model: analysis, design, construction, testing, and implementation. In the traditional model communication between developers and business users is not a priority. Finalize testing.

Business Intelligence 276

Business Intelligence Analytics Testing Dashboards 276

O'Reilly on Data

JULY 15, 2019

In recent posts, we described requisite foundational technologies needed to sustain machine learning practices within organizations, and specialized tools for model development, model governance, and model operations/testing/monitoring. Sources of model risk. Model risk management. Image by Ben Lorica.

Machine Learning 48

Machine Learning Management Enterprise Risk Management 48

DataKitchen

JANUARY 3, 2022

Business analytic teams have ongoing deliverables – a dashboard, a PowerPoint, or a model that they refresh and renew. Tests that verify and validate data flowing through the data pipelines are executed continuously. An impact review test suite executes before new analytics are deployed. Business Analytic Challenges.

Business Analytics 245

Business Analytics Analytics Testing Dashboards 245

DataKitchen

SEPTEMBER 17, 2021

DataOps produces clear measurement and monitoring of the end-to-end analytics pipelines starting with data sources. Design your data analytics workflows with tests at every stage of processing so that errors are virtually zero in number. In the DataKitchen context, monitoring and functional tests use the same code.

Testing 237

Testing Data Architecture Measurement Visualization 237