O'Reilly on Data

MARCH 20, 2019

Apply fair and private models, white-hat and forensic model debugging, and common sense to protect machine learning models from malicious actors. Like many others, I’ve known for some time that machine learning models themselves could pose security risks. This is like a denial-of-service (DOS) attack on your model itself.

Modeling 275

Modeling Machine Learning Predictive Modeling Consulting 275

O'Reilly on Data

APRIL 2, 2019

Model packaging: companies are using MLflow to incorporate custom logic and dependencies as part of a model’s package abstraction before deploying it to their production environment (example: a recommendation system might be programmed to not display certain images to minors). Model governance. there aren’t enough of them.

Machine Learning 260

Machine Learning Modeling Data Science Software 260

AWS Big Data

JANUARY 9, 2025

Iceberg offers distinct advantages through its metadata layer over Parquet, such as improved data management, performance optimization, and integration with various query engines. Icebergs table format separates data files from metadata files, enabling efficient data modifications without full dataset rewrites.

Metadata 107

Metadata Snapshot Cost-Benefit Optimization 107

Ontotext

JUNE 10, 2021

And yeah, the real-world relationships among the entities represented in the data had to be fudged a bit to fit in the counterintuitive model of tabular data, but, in trade, you get reliability and speed. Not Every Graph is a Knowledge Graph: Schemas and Semantic Metadata Matter. Graph Databases vs Relational Databases.

Metadata 119

Metadata Cost-Benefit OLAP Modeling 119

Rocket-Powered Data Science

MARCH 20, 2023

Datasphere goes beyond the “big three” data usage end-user requirements (ease of discovery, access, and delivery) to include data orchestration (data ops and data transformations) and business data contextualization (semantics, metadata, catalog services). As you would guess, maintaining context relies on metadata.

Data Warehouse 325

Data Warehouse Metadata Digital Transformation Machine Learning 325

CIO Business Intelligence

SEPTEMBER 6, 2024

Central to this is metadata management, a critical component for driving future success AI and ML need large amounts of accurate data for companies to get the most out of the technology. Let’s dive into what that looks like, what workarounds some IT teams use today, and why metadata management is the key to success.

Metadata 116

Metadata Enterprise Management Cost-Benefit 116

AWS Big Data

JULY 29, 2024

In this blog post, we’ll discuss how the metadata layer of Apache Iceberg can be used to make data lakes more efficient. You will learn about an open-source solution that can collect important metrics from the Iceberg metadata layer. This ensures that each change is tracked and reversible, enhancing data governance and auditability.

Metadata 119

Metadata Snapshot Data Lake Metrics 119

O'Reilly on Data

FEBRUARY 11, 2020

These include the basics, such as metadata creation and management, data provenance, data lineage, and other essentials. They’re still struggling with the basics: tagging and labeling data, creating (and managing) metadata, managing unstructured data, etc. They don’t have the resources they need to clean up data quality problems.

Data Quality 362

Data Quality Metadata Data Governance Publishing 362

CIO Business Intelligence

FEBRUARY 27, 2025

According to a study from Rocket Software and Foundry , 76% of IT decision-makers say challenges around accessing mainframe data and contextual metadata are a barrier to mainframe data usage, while 64% view integrating mainframe data with cloud data sources as the primary challenge.

Metadata 98

Metadata Data Lake Cost-Benefit Forecasting 98

AWS Big Data

NOVEMBER 26, 2024

Central to a transactional data lake are open table formats (OTFs) such as Apache Hudi , Apache Iceberg , and Delta Lake , which act as a metadata layer over columnar formats. XTable isn’t a new table format but provides abstractions and tools to translate the metadata associated with existing formats.

Metadata 95

Metadata Data Lake Snapshot Data Warehouse 95

AWS Big Data

NOVEMBER 7, 2024

Amazon Q generative SQL for Amazon Redshift uses generative AI to analyze user intent, query patterns, and schema metadata to identify common SQL query patterns directly within Amazon Redshift, accelerating the query authoring process for users and reducing the time required to derive actionable data insights.

Metadata 97

Metadata Sales Data Warehouse Optimization 97

CIO Business Intelligence

NOVEMBER 19, 2024

You pull an open-source large language model (LLM) to train on your corporate data so that the marketing team can build better assets, and the customer service team can provide customer-facing chatbots. You build your model, but the history and context of the data you used are lost, so there is no way to trace your model back to the source.

Management 116

Management Unstructured Data Deep Learning Metadata 116

O'Reilly on Data

JUNE 14, 2024

Generative AI models are trained on large repositories of information and media. They are then able to take in prompts and produce outputs based on the statistical weights of the pretrained models of those corpora. The newest Answers release is again built with an open source model—in this case, Llama 3.

Metadata 292

Metadata Publishing Data-driven Modeling 292

CIO Business Intelligence

APRIL 8, 2025

These strategies, such as investing in AI-powered cleansing tools and adopting federated governance models, not only address the current data quality challenges but also pave the way for improved decision-making, operational efficiency and customer satisfaction. Data fabric Metadata-rich integration layer across distributed systems.

Data Quality 102

Data Quality Data-driven Key Performance Indicator Metadata 102

AWS Big Data

JANUARY 15, 2025

EUROGATEs data science team aims to create machine learning models that integrate key data sources from various AWS accounts, allowing for training and deployment across different container terminals. From here, the metadata is published to Amazon DataZone by using AWS Glue Data Catalog. This process is shown in the following figure.

IoT 101

IoT Machine Learning Metadata Data-driven 101

AWS Big Data

DECEMBER 4, 2024

This model balances node or domain-level autonomy with enterprise-level oversight, creating a scalable and consistent framework across ANZ. The Institutional Data & AI platform adopts a federated approach to data while centralizing the metadata to facilitate simpler discovery and sharing of data products.

Metadata 94

Metadata Data Governance Data Quality Data-driven 94

CIO Business Intelligence

NOVEMBER 27, 2024

Data quality for AI needs to cover bias detection, infringement prevention, skew detection in data for model features, and noise detection. Not all columns are equal, so you need to prioritize cleaning data features that matter to your model, and your business outcomes. asks Friedman.

Enterprise 141

Enterprise Data Quality Structured Data Modeling 141

O'Reilly on Data

MARCH 22, 2019

When the model for spam detection is systematically wrong, users can correct it. For example, we wouldn’t want real estate agents “correcting” a model to recommend houses based on race or religion; and we could even discuss whether similar behavior would be appropriate for spam detection. It's possible to abuse or to game any solution.

Metadata 275

Metadata Advertising Insurance Modeling 275

Domino Data Lab

JANUARY 11, 2021

Even for experienced developers and data scientists, the process of developing a model could involve stringing together many steps from many packages, in ways that might not be as elegant or efficient as one might like. the experience is still rooted in the same goal: simple efficiency for the whole model development lifecycle.

Modeling 145

Modeling Metrics Data Science Testing 145

Rocket-Powered Data Science

MARCH 10, 2020

And this: perhaps the most powerful node in a graph model for real-world use cases might be “context”. How does one express “context” in a data model? After all, the standard relational model of databases instantiated these types of relationships in its very foundation decades ago: the ERD (Entity-Relationship Diagram).

Metadata 250

Metadata Machine Learning Prescriptive Analytics ROI 250

O'Reilly on Data

DECEMBER 19, 2018

We need to do more than automate model building with autoML; we need to automate tasks at every stage of the data pipeline. There is no GitHub for data, though we are starting to see version control projects for machine learning models, such as DVC. Automation is more than model building. Toward a sustainable ML practice.

Machine Learning 279

Machine Learning Software Metadata Testing 279

Rocket-Powered Data Science

SEPTEMBER 20, 2021

They realized that the search results would probably not provide an answer to my question, but the results would simply list websites that included my words on the page or in the metadata tags: “Texas”, “Cows”, “How”, etc. The semantic layer bridges the gaps between the data cloud, the decision-makers, and the data science modelers.

Data Science 286

Data Science Forecasting Business Intelligence Sales 286

AWS Big Data

NOVEMBER 19, 2024

Solution overview By combining the powerful vector search capabilities of OpenSearch Service with the access control features provided by Amazon Cognito , this solution enables organizations to manage access controls based on custom user attributes and document metadata. If you don’t already have an AWS account, you can create one.

Management 94

Management Metadata Manufacturing Testing 94

Rocket-Powered Data Science

JULY 6, 2021

This is accomplished through tags, annotations, and metadata (TAM). Smart content includes labeled (tagged, annotated) metadata (TAM). The key to success is to start enhancing and augmenting content management systems (CMS) with additional features: semantic content and context. Collect, curate, and catalog (i.e.,

Strategy 267

Strategy Machine Learning Metadata Knowledge Discovery 267

O'Reilly on Data

MARCH 31, 2020

Instead of writing code with hard-coded algorithms and rules that always behave in a predictable manner, ML engineers collect a large number of examples of input and output pairs and use them as training data for their models. The model is produced by code, but it isn’t code; it’s an artifact of the code and the training data.

Management 363

Management Machine Learning Experimentation Metrics 363

CIO Business Intelligence

JANUARY 9, 2025

Generative artificial intelligence ( genAI ) and in particular large language models ( LLMs ) are changing the way companies develop and deliver software. The commodity effect of LLMs over specialized ML models One of the most notable transformations generative AI has brought to IT is the democratization of AI capabilities.

Software 128

Software Enterprise Key Performance Indicator Machine Learning 128

O'Reilly on Data

DECEMBER 12, 2023

If the output of a model can’t be owned by a human, who (or what) is responsible if that output infringes existing copyright? In an article in The New Yorker , Jaron Lanier introduces the idea of data dignity, which implicitly distinguishes between training a model and generating output using a model.

Modeling 335

Modeling Sales Software Statistics 335

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 84

Metadata Data Lake Dashboards Interactive 84

AWS Big Data

OCTOBER 29, 2024

The CDH is used to create, discover, and consume data products through a central metadata catalog, while enforcing permission policies and tightly integrating data engineering, analytics, and machine learning services to streamline the user journey from data to insight.

Data Lake 89

Data Lake Sales Metadata Machine Learning 89

Rocket-Powered Data Science

FEBRUARY 15, 2023

While generative AI has been around for several years , the arrival of ChatGPT (a conversational AI tool for all business occasions, built and trained from large language models) has been like a brilliant torch brought into a dark room, illuminating many previously unseen opportunities. So, if you have 1 trillion data points (g.,

Strategy 290

Strategy Experimentation Uncertainty Machine Learning 290

AWS Big Data

NOVEMBER 19, 2024

Additionally, customers adopting a federated deployment model find it challenging to provide isolated environments for different teams or departments, and at the same time optimize cost. It’s essential to monitor key metrics such as metadata database memory usage, and CPU utilization of the worker/scheduler hybrid container.

Metadata 106

Metadata Cost-Benefit Metrics Optimization 106

O'Reilly on Data

MARCH 18, 2020

Whether it’s controlling for common risk factors—bias in model development, missing or poorly conditioned data, the tendency of models to degrade in production—or instantiating formal processes to promote data governance, adopters will have their work cut out for them as they work to establish reliable AI production lines.

Enterprise 349

Enterprise Deep Learning Data Governance Risk 349

O'Reilly on Data

JANUARY 8, 2019

This is also reflected by the emergence of tools that are specific to machine learning, including data science platforms, data lineage, metadata management and analysis, data governance, and model lifecycle management. Burgeoning IoT technologies.

Machine Learning 267

Machine Learning IoT Internet of Things Data Science 267

AWS Big Data

OCTOBER 21, 2024

As a producer, you can also monetize your data through the subscription model using AWS Data Exchange. To achieve this, they plan to use machine learning (ML) models to extract insights from data. Business analysts enhance the data with business metadata/glossaries and publish the same as data assets or data products.

Sales 105

Sales Data-driven Data Processing Key Performance Indicator 105

O'Reilly on Data

MAY 21, 2019

Companies successfully adopt machine learning either by building on existing data products and services, or by modernizing existing models and algorithms. For example, in a July 2018 survey that drew more than 11,000 respondents, we found strong engagement among companies: 51% stated they already had machine learning models in production.

Machine Learning 279

Machine Learning Technology Deep Learning Data Science 279

AWS Big Data

DECEMBER 17, 2024

We have enhanced data sharing performance with improved metadata handling, resulting in data sharing first query execution that is up to four times faster when the data sharing producers data is being updated. Lakehouse allows you to use preferred analytics engines and AI models of your choice with consistent governance across all your data.

Data Lake 82

Data Lake Data Warehouse Data-driven Optimization 82

O'Reilly on Data

MARCH 23, 2023

It’s important to understand that ChatGPT is not actually a language model. It’s a convenient user interface built around one specific language model, GPT-3.5, is one of a class of language models that are sometimes called “large language models” (LLMs)—though that term isn’t very helpful. with specialized training.

IT 346

IT Modeling Testing Risk 346