O'Reilly on Data

OCTOBER 19, 2021

Let’s start by considering the job of a non-ML software engineer: writing traditional software deals with well-defined, narrowly-scoped inputs, which the engineer can exhaustively and cleanly model in the code. Not only is data larger, but models—deep learning models in particular—are much larger than before.

IT 350

IT Testing Experimentation Software 350

DataKitchen

SEPTEMBER 20, 2021

The best way to ensure error-free execution of data production is through automated testing and monitoring. The DataKitchen Platform enables data teams to integrate testing and observability into data pipeline orchestrations. Automated tests work 24×7 to ensure that the results of each processing stage are accurate and correct.

Testing 246

Testing Metrics Measurement Dashboards 246

CIO Business Intelligence

FEBRUARY 7, 2024

Key AI companies have told the UK government to speed up its safety testing for their systems, raising questions about future government initiatives that too may hinge on technology providers opening up generative AI models to tests before new releases hit the public.

Testing 124

Testing Reporting Risk Modeling 124

CIO Business Intelligence

NOVEMBER 20, 2023

Measuring developer productivity has long been a Holy Grail of business. In addition, system, team, and individual productivity all need to be measured. The inner loop comprises activities directly related to creating the software product: coding, building, and unit testing. And like the Holy Grail, it has been elusive.

Measurement 126

Measurement Metrics Testing Software 126

CIO Business Intelligence

APRIL 5, 2023

In a joint study with Markus Westner and Tobias Held from the department of computer science and mathematics at the University of Regensburg, the 4C experts examined the topic by focusing on how the IT value proposition is measured, made visible, and communicated. They also tested the concept in a German mechanical engineering company.

Measurement 133

Measurement IT Metrics Consulting 133

O'Reilly on Data

JUNE 15, 2023

The world changed on November 30, 2022 as surely as it did on August 12, 1908 when the first Model T left the Ford assembly line. If we want prosocial outcomes, we need to design and report on the metrics that explicitly aim for those outcomes and measure the extent to which they have been achieved.

Metrics 289

Metrics Reporting Measurement Finance 289

O'Reilly on Data

MAY 14, 2020

Experimentation: It’s just not possible to create a product by building, evaluating, and deploying a single model. In reality, many candidate models (frequently hundreds or even thousands) are created during the development process. Modelling: The model is often misconstrued as the most important component of an AI product.

Management 363

Management Experimentation B2B Machine Learning 363

Rocket-Powered Data Science

FEBRUARY 15, 2023

While generative AI has been around for several years , the arrival of ChatGPT (a conversational AI tool for all business occasions, built and trained from large language models) has been like a brilliant torch brought into a dark room, illuminating many previously unseen opportunities. So, if you have 1 trillion data points (g.,

Strategy 290

Strategy Experimentation Uncertainty Machine Learning 290

O'Reilly on Data

MAY 27, 2020

Using the new scores, Apgar and her colleagues proved that many infants who initially seemed lifeless could be revived, with success or failure in each case measured by the difference between an Apgar score at one minute after birth, and a second score taken at five minutes. Books, in turn, get matching scores to reflect their difficulty.

Risk 355

Risk Testing Reporting Measurement 355

CIO Business Intelligence

JUNE 13, 2024

Not instant perfection The NIPRGPT experiment is an opportunity to conduct real-world testing, measuring generative AI’s computational efficiency, resource utilization, and security compliance to understand its practical applications. It is not training the model, nor are responses refined based on any user inputs.

Testing 119

Testing Experimentation Data Processing Modeling 119

DataKitchen

SEPTEMBER 18, 2024

DataOps introduces agility by advocating for: Measuring data quality early : Data quality leaders should begin measuring and assessing data quality even before perfect standards are in place. Early measurements provide valuable insights that can guide future improvements. Measuring and Refining : DataOps is an iterative process.

Scorecard 189

Scorecard Data Quality Measurement Testing 189

CIO Business Intelligence

SEPTEMBER 17, 2024

CISOs can only know the performance and maturity of their security program by actively measuring it themselves; after all, to measure is to know. However, CISOs aren’t typically measuring their security program proactively or methodically to understand their current security program. people, processes, and technology).

Measurement 111

Measurement Metrics Risk Dashboards 111

O'Reilly on Data

FEBRUARY 15, 2022

But a recent discussion of Google’s new Large Language Models (LLMs), and its claim that one of these models (named Gopher) has demonstrated reading comprehension approaching human performance , has spurred some thoughts about comprehension, ambiguity, intelligence, and will. Ethics is for beings who can make choices.

Testing 299

Testing Modeling Measurement IT 299

DataKitchen

NOVEMBER 29, 2021

Model developers will test for AI bias as part of their pre-deployment testing. Quality test suites will enforce “equity,” like any other performance metric. Continuous testing, monitoring and observability will prevent biased models from deploying or continuing to operate. Companies Commit to Remote.

Testing 245

Testing Data Lake Data Architecture Manufacturing 245

DataKitchen

JANUARY 20, 2021

Centralizing analytics helps the organization standardize enterprise-wide measurements and metrics. Develop/execute regression testing . Test data management and other functions provided ‘as a service’ . Central DataOps process measurement function with reports. Agile ticketing/Kanban tools. Deploy to production.

Metrics 243

Metrics Experimentation Measurement Testing 243

DataKitchen

MAY 10, 2021

Some will argue that observability is nothing more than testing and monitoring applications using tests, metrics, logs, and other artifacts. Since 2008, teams working for our founding team and our customers have delivered 100s of millions of data sets, dashboards, and models with almost no errors. Tie tests to alerts.

Testing 214

Testing Manufacturing Data Quality Statistics 214

O'Reilly on Data

JUNE 18, 2024

Many farmers measure their yield in bags of rice, but what is “a bag of rice”? While RAG is conceptually simple—look up relevant documents and construct a prompt that tells the model to build its response from them—in practice, it’s more complex. Digital Green tests with “Golden QAs,” highly rated sets of questions and answers.

Testing 239

Testing Software Modeling Measurement 239

DataKitchen

NOVEMBER 20, 2021

A DataOps Engineer can make test data available on demand. We have automated testing and a system for exception reporting, where tests identify issues that need to be addressed. It then autogenerates QC tests based on those rules. You can track, measure and create graphs and reporting in an automated way.

Testing 246

Testing Machine Learning Data Warehouse Reporting 246

datapine

APRIL 15, 2020

Your Chance: Want to test an agile business intelligence solution? Business intelligence is moving away from the traditional engineering model: analysis, design, construction, testing, and implementation. In the traditional model communication between developers and business users is not a priority. Finalize testing.

Business Intelligence 276

Business Intelligence Analytics Testing Dashboards 276

O'Reilly on Data

OCTOBER 13, 2020

Similarly, in “ Building Machine Learning Powered Applications: Going from Idea to Product ,” Emmanuel Ameisen states: “Indeed, exposing a model to users in production comes with a set of challenges that mirrors the ones that come with debugging a model.”. Debugging AI Products.

Management 250

Management Metrics Machine Learning Modeling 250

O'Reilly on Data

MARCH 31, 2020

Instead of writing code with hard-coded algorithms and rules that always behave in a predictable manner, ML engineers collect a large number of examples of input and output pairs and use them as training data for their models. This has serious implications for software testing, versioning, deployment, and other core development processes.

Management 338

Management Machine Learning Experimentation Metrics 338

DataKitchen

JANUARY 3, 2022

Business analytic teams have ongoing deliverables – a dashboard, a PowerPoint, or a model that they refresh and renew. Tests that verify and validate data flowing through the data pipelines are executed continuously. An impact review test suite executes before new analytics are deployed. Business Analytic Challenges.

Business Analytics 245

Business Analytics Analytics Testing Dashboards 245

CIO Business Intelligence

JANUARY 18, 2024

Recognizing this need, we have created a cutting-edge VSM maturity model. Drawing upon our extensive experience facilitating successful VSM initiatives within large-scale enterprises, this model serves as a compass, offering insights into their current VSM maturity and providing practical direction to advance and reap the additional benefits.

Digital Transformation 119

Digital Transformation Modeling Management Data-driven 119

DataKitchen

SEPTEMBER 21, 2023

In the context of Data in Place, validating data quality automatically with Business Domain Tests is imperative for ensuring the trustworthiness of your data assets. Running these automated tests as part of your DataOps and Data Observability strategy allows for early detection of discrepancies or errors.

Testing 182

Testing Data Quality Predictive Modeling Metrics 182

DataKitchen

SEPTEMBER 17, 2021

DataOps produces clear measurement and monitoring of the end-to-end analytics pipelines starting with data sources. Design your data analytics workflows with tests at every stage of processing so that errors are virtually zero in number. In the DataKitchen context, monitoring and functional tests use the same code.

Testing 237

Testing Data Architecture Measurement Enterprise 237

Corinium

JUNE 6, 2019

Fractal’s recommendation is to take an incremental, test and learn approach to analytics to fully demonstrate the program value before making larger capital investments. A properly set framework will ensure quality, timeliness, scalability, consistency, and industrialization in measuring and driving the return on investment.

Insurance 250

Insurance Analytics Forecasting Deep Learning 250

CIO Business Intelligence

MARCH 6, 2024

What has IT’s role been in the transformation to a SaaS model? We built that end-to-end data model and process from scratch while we ran the old business. We knew we had a unique opportunity to build a new end-to-end architecture with a common AI-powered data model. Today, we’re a $1.6 Today, we’re a $1.6

Modeling 116

Modeling Management IT Metrics 116

O'Reilly on Data

SEPTEMBER 15, 2020

In my book, I introduce the Technical Maturity Model: I define technical maturity as a combination of three factors at a given point of time. Technical sophistication: Sophistication measures a team’s ability to use advanced tools and techniques (e.g., PyTorch, TensorFlow, reinforcement learning, self-supervised learning).

Advertising 299

Advertising Cost-Benefit ROI Machine Learning 299

O'Reilly on Data

MARCH 7, 2019

Language understanding benefits from every part of the fast-improving ABC of software: AI (freely available deep learning libraries like PyText and language models like BERT ), big data (Hadoop, Spark, and Spark NLP ), and cloud (GPU's on demand and NLP-as-a-service from all the major cloud providers). Azure Text Analytics. Stanford Core NLP.

Deep Learning 231

Deep Learning Testing Machine Learning Modeling 231

O'Reilly on Data

OCTOBER 22, 2020

Taking the time to work this out is like building a mathematical model: if you understand what a company truly does, you don’t just get a better understanding of the present, but you can also predict the future. Since I work in the AI space, people sometimes have a preconceived notion that I’ll only talk about data and models.

Consulting 301

Consulting Risk Cost-Benefit Modeling 301

O'Reilly on Data

JUNE 24, 2020

The argument is that some systems are intrinsically difficult to model. You can’t control for, or even measure, several of these factors. Wearing masks as a prophylactic measure isn’t the big cultural leap that it has been in the United States. What does that mean?

Measurement 288

Measurement Testing Interactive Modeling 288

DataKitchen

OCTOBER 28, 2021

Machine learning (ML) models are computer programs that draw inferences from data — usually lots of data. One way to think of ML models is that they instantiate an algorithm (a decision-making procedure often involving math) in software and then, at relatively low cost, deploy it on a large scale. What Is AI Bias?

Testing 130

Testing IT Manufacturing Metrics 130

CIO Business Intelligence

SEPTEMBER 10, 2024

The US Department of Commerce’s Bureau of Industry and Security (BIS) plans to introduce mandatory reporting requirements for developers of advanced AI models and cloud computing providers.

Reporting 131

Reporting Measurement Testing Risk 131

datapine

JUNE 18, 2019

A data-driven finance report is also an effective means of remaining updated with any significant progress or changes in the status of your finances, and help you measure your financial results, cash flow, and financial position. b) Measure Revenue Loss. Metrics used to measure these factors can include: Number of daily transactions.

Reporting 251

Reporting KPI Metrics Finance 251

DataKitchen

NOVEMBER 27, 2024

For example, metrics like the percentage of missing values help measure completeness, while deviations from authoritative sources gauge accuracy. For instance, a bank might use a CDE dashboard to monitor the accuracy of customer credit scores, which are vital for regulatory compliance and effective risk modeling.

Data Quality 219

Data Quality Dashboards Metrics Cost-Benefit 219

O'Reilly on Data

MARCH 23, 2023

It’s important to understand that ChatGPT is not actually a language model. It’s a convenient user interface built around one specific language model, GPT-3.5, is one of a class of language models that are sometimes called “large language models” (LLMs)—though that term isn’t very helpful. with specialized training.

IT 267

IT Modeling Testing Risk 267

O'Reilly on Data

AUGUST 4, 2022

This kind of humility is likely to deliver more meaningful progress and a more measured understanding of such progress. DeepMind’s Gato is an AI model that can be taught to carry out many different kinds of tasks based on a single transformer neural network. We typically underappreciate how complex such systems are.

Machine Learning 268

Machine Learning Modeling Statistics Software 268