O'Reilly on Data

MARCH 25, 2025

What makes LLM applications so different? Whats worse: Inputs are rarely exactly the same. Weve been working with dozens of companies building LLM applications, and weve noticed patterns in what works and what doesnt. FOCUS ON PRINCIPLES, NOT FRAMEWORKS (OR AGENTS) A lot of people ask us: What tools should I use?

Testing 168

Testing Data-driven Software Measurement 168

DataKitchen

DECEMBER 9, 2024

data engineers delivered over 100 lines of code and 1.5 Applying DataOps an Agile, automation-focused approach to data engineering ensured a dynamic feedback loop where data delivery and insights could evolve rapidly. The following diagram shows what the initial infrastructure looked like.

Data Quality 164

Data Quality Data Lake Testing Statistics 164

CIO Business Intelligence

OCTOBER 17, 2024

Organizations and vendors are already rolling out AI coding agents that enable developers to fully automate or offload many tasks, with more pilot programs and proofs-of-concept likely to be launched in 2025, says Philip Walsh, senior principal analyst in Gartner’s software engineering practice. This technology already exists.”

Software 141

Software Testing Experimentation Consulting 141

O'Reilly on Data

MARCH 31, 2020

We won’t go into the mathematics or engineering of modern machine learning here. We won’t go into the mathematics or engineering of modern machine learning here. If you’re already a software product manager (PM), you have a head start on becoming a PM for artificial intelligence (AI) or machine learning (ML).

Management 363

Management Machine Learning Experimentation Metrics 363

CIO Business Intelligence

NOVEMBER 27, 2024

Data quality for AI needs to cover bias detection, infringement prevention, skew detection in data for model features, and noise detection. At worst, it can go in and remove signal from your data, and actually be at cross purposes with what you need.” It’s always relative to what it is you’re using it for.

Enterprise 141

Enterprise Data Quality Structured Data Modeling 141

O'Reilly on Data

OCTOBER 13, 2020

In traditional software engineering, precedent has been established for the transition of responsibility from development teams to maintenance, user operations, and site reliability teams. New features in an existing product often follow a similar progression. New features in an existing product often follow a similar progression.

Management 308

Management Machine Learning Metrics Modeling 308

O'Reilly on Data

FEBRUARY 11, 2020

Data scientists and analysts, data engineers, and the people who manage them comprise 40% of the audience; developers and their managers, about 22%. Analysts and engineers predominate. An additional 7% are data engineers. On top of this, close to 8% manage data scientists or engineers. But what kind of salience?

Data Quality 363

Data Quality Metadata Data Governance Publishing 363

AWS Big Data

APRIL 9, 2025

Built for complex, high-scale catalogs This feature builds on existing keyword and semantic search capabilities in SageMaker Unified Studio and adds an important layer of control for customers managing complex data catalogs with intricate naming conventions. This reduces time-to-insight and makes sure the right metric is used in reporting.

Metadata 111

Metadata Metrics Data-driven Cost-Benefit 111

AWS Big Data

JANUARY 9, 2025

Iceberg offers distinct advantages through its metadata layer over Parquet, such as improved data management, performance optimization, and integration with various query engines. Data management is the foundation of quantitative research. As mentioned earlier, 80% of quantitative research work is attributed to data management tasks.

Metadata 111

Metadata Snapshot Cost-Benefit Optimization 111

Rocket-Powered Data Science

SEPTEMBER 20, 2021

What is a semantic layer? In the early days of web search engines, those engines were primarily keyword search engines. So, I asked my students what results they would expect from such a search engine if I typed the following words into the search box: “How many cows are there in Texas?” Now it is a reality.

Data Science 286

Data Science Forecasting Business Intelligence Sales 286

O'Reilly on Data

JANUARY 9, 2024

That will be a big change for professional programmers—though writing code is a small part of what programmers actually do. When programmers wrote in assembly language, they had to look at the binary 1s and 0s to see exactly what the computer was doing. With C or Python, you can read a program and understand exactly what it does.

Modeling 345

Modeling Software Testing Optimization 345

CIO Business Intelligence

JANUARY 9, 2025

What began with chatbots and simple automation tools is developing into something far more powerful AI systems that are deeply integrated into software architectures and influence everything from backend processes to user interfaces. Instead, users can simply describe what they want to fill a selected area of the image.

Software 128

Software Enterprise Key Performance Indicator Machine Learning 128

O'Reilly on Data

FEBRUARY 18, 2020

This year’s growth in Python usage was buoyed by its increasing popularity among data scientists and machine learning (ML) and artificial intelligence (AI) engineers. This is consistent with what we’ve observed elsewhere : Python has acquired new relevance amid strong interest in AI and ML. Coincidence? Security is surging.

Data-driven 363

Data-driven Software Statistics Marketing 363

CIO Business Intelligence

NOVEMBER 19, 2024

Imagine that you’re a data engineer. The data is spread out across your different storage systems, and you don’t know what is where. These challenges are quite common for the data engineers and data scientists we speak to. What does the next generation of AI workloads need?

Management 116

Management Unstructured Data Deep Learning Metadata 116

O'Reilly on Data

APRIL 2, 2019

Along the way, we described a new job role and title—machine learning engineer —focused on creating data products and making data science work in production, a role that was beginning to emerge in the San Francisco Bay Area two years ago. So, why is this new open source project resonating with data scientists and machine learning engineers?

Machine Learning 270

Machine Learning Modeling Data Science Software 270

AWS Big Data

NOVEMBER 7, 2024

Within this feature, user data is secure and private. In this post, we show you how to enable the Amazon Q generative SQL feature in the Redshift query editor and use the feature to get tailored SQL commands based on your natural language queries. Your data is not shared across accounts.

Metadata 105

Metadata Sales Data Warehouse Optimization 105

CIO Business Intelligence

MARCH 25, 2025

These areas are considerable issues, but what about data, security, culture, and addressing areas where past shortcuts are fast becoming todays liabilities? Another question is: What separates out debt thats fixed opportunistically versus critical debt that could cripple the business?

Risk 140

Risk Cost-Benefit Data-driven Digital Transformation 140

O'Reilly on Data

SEPTEMBER 12, 2023

That statement nicely summarizes what makes software development difficult. gets a few more features, more creep into version 1.2, OS X, which used to trumpet “It just works,” has evolved to “it used to just work”; the most user-centric Unix-like system ever built now staggers under the load of new and poorly thought-out features.

Software 348

Software Management Modeling Metrics 348

O'Reilly on Data

FEBRUARY 19, 2021

Similarly, the data lakehouse, an architecture that features attributes of both the data lake and the data warehouse, gained traction in 2020 and will continue to grow in prominence in 2021. Cloud data warehouse engineering develops as a particular focus as database solutions move more and more to the cloud.

Data Lake 362

Data Lake Machine Learning Data Warehouse Modeling 362

O'Reilly on Data

DECEMBER 12, 2023

If the output of a model can’t be owned by a human, who (or what) is responsible if that output infringes existing copyright? Is an artist’s style copyrightable, and if so, what does that mean? Is an artist’s style copyrightable, and if so, what does that mean? How do we make sense of this?

Modeling 331

Modeling Sales Software Statistics 331

CIO Business Intelligence

APRIL 9, 2025

For those that attended VMware Explore in Las Vegas and Barcelona, there was a new self-paced hands-on lab released exclusively for the attendees to experience Google Cloud VMware Engine while at the events. These refreshed modules capture the latest changes and features, and even let you experience some of the adjacent services.

IT 98

IT Marketing Enterprise Technology 98

O'Reilly on Data

JUNE 11, 2024

Answers is a generative AI-powered feature that aims to answer questions in the flow of learning. What are your specific use cases? What kinds of answers will your users expect? What kind of answers do you want to deliver? At O’Reilly, we’re not just building training materials about AI. Now you can.

Modeling 337

Modeling Experimentation Interactive Data-driven 337

O'Reilly on Data

MAY 21, 2019

Our goal was twofold: (1) find out what tools and platforms people are using, and (2) determine whether or not companies are building the foundational tools needed to sustain their ML initiatives. One of the main questions we asked was: what are you currently building or evaluating? Source: O'Reilly. and managed services in the cloud.

Machine Learning 279

Machine Learning Technology Deep Learning Data Science 279

O'Reilly on Data

MARCH 26, 2024

This seems to be emerging as a feature, not a bug, and hopefully it’s obvious to you why they called their IEEE opinion piece Generative AI Has a Visual Plagiarism Problem. The narrator tells us that, “the Cervantes text and the Menard text are verbally identical, but the second is almost infinitely richer.” Let me explain.

Modeling 317

Modeling Machine Learning Risk Advertising 317

O'Reilly on Data

MARCH 18, 2020

The update sheds light on what AI adoption looks like in the enterprise— hint: deployments are shifting from prototype to production—the popularity of specific techniques and tools, the challenges experienced by adopters, and so on. What is more, almost three-quarters of survey respondents say they work with data in their jobs.

Enterprise 347

Enterprise Deep Learning Data Governance Risk 347

AWS Big Data

MARCH 13, 2025

Governance features including fine-grained access control are built into SageMaker Unified Studio using Amazon SageMaker Catalog to help you meet enterprise security requirements across your entire data estate. Configuring and governing access is also a cumbersome manual process.

Analytics 113

Analytics Data Lake Data Warehouse Data-driven 113

O'Reilly on Data

DECEMBER 10, 2019

Roughly a year ago, we wrote “ What machine learning means for software development.” Up until now, we’ve built systems by carefully and painstakingly telling systems exactly what to do, instruction by instruction. It’s time to evaluate what has happened in the year since we wrote that article. and Matroid.

Software 337

Software Machine Learning Risk Data-driven 337

AWS Big Data

DECEMBER 17, 2024

Amazon Redshift made significant strides in 2024, rolling out over 100 features and enhancements. Figure1: Summary of the features and enhancements in 2024 Lets walk through some of the recent key launches, including the new announcements at AWS re:Invent 2024.

Data Lake 98

Data Lake Data Warehouse Data-driven Optimization 98

AWS Big Data

DECEMBER 20, 2024

Subsequently, Amazon Q also generated the DataFrame-based code for data engineers or more experienced users to use the automatic ETL generated code for scripting purposes. The process involves merging the allevents_pipe and venue_pipe files from the TICKIT dataset. Choose Submit. The workflow is updated with a new filter transform.

Data Integration 105

Data Integration Visualization Data Processing Big Data 105

O'Reilly on Data

JUNE 19, 2019

Organizations now also have more use cases and case studies from which to draw inspiration—no matter what industry or domain you are interested in, chances are there are many interesting ML applications you can learn from. Versioning (of models, feature vectors , data) and the ability to roll out, roll back, or have multiple live versions.

Modeling 256

Modeling Machine Learning Testing Metrics 256

O'Reilly on Data

AUGUST 9, 2022

What is the connection between expertise and ideation? And what kinds of user interfaces will be effective for collaborations between humans and computers, where the computers supply the technique and we supply the ideas? What kinds of creativity does that new technique enable? What would be necessary for another transformation?

Software 291

Software Testing Modeling IT 291

DataKitchen

NOVEMBER 18, 2021

Yet, among all this, one area that hasn’t been studied is the data engineering role. We thought it would be interesting to look at how data engineers are doing under these circumstances. We surveyed 600 data engineers , including 100 managers, to understand how they are faring and feeling about the work that they are doing.

Testing 246

Testing Data Governance Measurement Software 246

CIO Business Intelligence

DECEMBER 18, 2024

the second release of the agentic AI platform, which comes just two months after the first version was released, gets new features and capabilities, such as the option to switch to an updated reasoning engine, new agent skills, and the ability to build agents using natural language. Christened Agentforce 2.0, Agentforce 2.0

Enterprise 98

Enterprise IT Sales Metadata 98

O'Reilly on Data

MARCH 23, 2023

What is it, how does it work, what can it do, and what are the risks of using it? Many of these go slightly (but not very far) beyond your initial expectations: you can ask it to generate a list of terms for search engine optimization, you can ask it to generate a reading list on topics that you’re interested in.

IT 346

IT Modeling Testing Risk 346

O'Reilly on Data

MARCH 20, 2019

A recent flourish of posts and papers has outlined the broader topic, listed attack vectors and vulnerabilities, started to propose defensive solutions, and provided the necessary framework for this post. Forcing your model to make a false prediction for the attacker’s benefit is sometimes called a violation of your model’s “integrity”.)

Modeling 278

Modeling Machine Learning Predictive Modeling Consulting 278

O'Reilly on Data

NOVEMBER 13, 2018

At the recent Strata Data conference we had a series of talks on relevant cultural, organizational, and engineering topics. They share some of the features I list below, including support for multiple ML libraries and frameworks, notebooks, scheduling, and collaboration. Developers have taken notice and are beginning to learn about ML.

Machine Learning 277

Machine Learning Risk Management Statistics 277