O'Reilly on Data

MARCH 20, 2019

Apply fair and private models, white-hat and forensic model debugging, and common sense to protect machine learning models from malicious actors. Like many others, I’ve known for some time that machine learning models themselves could pose security risks. This is like a denial-of-service (DOS) attack on your model itself.

Modeling 278

Modeling Machine Learning Predictive Modeling Consulting 278

AWS Big Data

JANUARY 9, 2025

Iceberg offers distinct advantages through its metadata layer over Parquet, such as improved data management, performance optimization, and integration with various query engines. Icebergs table format separates data files from metadata files, enabling efficient data modifications without full dataset rewrites.

Metadata 111

Metadata Snapshot Cost-Benefit Optimization 111

Ontotext

JUNE 10, 2021

And yeah, the real-world relationships among the entities represented in the data had to be fudged a bit to fit in the counterintuitive model of tabular data, but, in trade, you get reliability and speed. Not Every Graph is a Knowledge Graph: Schemas and Semantic Metadata Matter. Graph Databases vs Relational Databases.

Metadata 119

Metadata Cost-Benefit OLAP Modeling 119

CIO Business Intelligence

SEPTEMBER 6, 2024

Central to this is metadata management, a critical component for driving future success AI and ML need large amounts of accurate data for companies to get the most out of the technology. Let’s dive into what that looks like, what workarounds some IT teams use today, and why metadata management is the key to success.

Metadata 116

Metadata Enterprise Management Cost-Benefit 116

AWS Big Data

JULY 29, 2024

In this blog post, we’ll discuss how the metadata layer of Apache Iceberg can be used to make data lakes more efficient. You will learn about an open-source solution that can collect important metrics from the Iceberg metadata layer. This ensures that each change is tracked and reversible, enhancing data governance and auditability.

Metadata 126

Metadata Snapshot Data Lake Metrics 126

erwin

JANUARY 9, 2020

For decades, data modeling has been the optimal way to design and deploy new relational databases with high-quality data sources and support application development. Today’s data modeling is not your father’s data modeling software. So here’s why data modeling is so critical to data governance.

Data Governance 144

Data Governance Modeling Metadata Unstructured Data 144

AWS Big Data

NOVEMBER 26, 2024

Central to a transactional data lake are open table formats (OTFs) such as Apache Hudi , Apache Iceberg , and Delta Lake , which act as a metadata layer over columnar formats. XTable isn’t a new table format but provides abstractions and tools to translate the metadata associated with existing formats.

Metadata 105

Metadata Data Lake Snapshot Data Warehouse 105

O'Reilly on Data

FEBRUARY 11, 2020

These include the basics, such as metadata creation and management, data provenance, data lineage, and other essentials. They’re still struggling with the basics: tagging and labeling data, creating (and managing) metadata, managing unstructured data, etc. They don’t have the resources they need to clean up data quality problems.

Data Quality 363

Data Quality Metadata Data Governance Publishing 363

CIO Business Intelligence

FEBRUARY 27, 2025

According to a study from Rocket Software and Foundry , 76% of IT decision-makers say challenges around accessing mainframe data and contextual metadata are a barrier to mainframe data usage, while 64% view integrating mainframe data with cloud data sources as the primary challenge.

Metadata 98

Metadata Data Lake Cost-Benefit Forecasting 98

AWS Big Data

NOVEMBER 7, 2024

Amazon Q generative SQL for Amazon Redshift uses generative AI to analyze user intent, query patterns, and schema metadata to identify common SQL query patterns directly within Amazon Redshift, accelerating the query authoring process for users and reducing the time required to derive actionable data insights.

Metadata 105

Metadata Sales Data Warehouse Optimization 105

AWS Big Data

JANUARY 15, 2025

EUROGATEs data science team aims to create machine learning models that integrate key data sources from various AWS accounts, allowing for training and deployment across different container terminals. From here, the metadata is published to Amazon DataZone by using AWS Glue Data Catalog. This process is shown in the following figure.

IoT 110

IoT Machine Learning Metadata Data-driven 110

O'Reilly on Data

JUNE 14, 2024

Generative AI models are trained on large repositories of information and media. They are then able to take in prompts and produce outputs based on the statistical weights of the pretrained models of those corpora. The newest Answers release is again built with an open source model—in this case, Llama 3.

Metadata 299

Metadata Publishing Data-driven Modeling 299

CIO Business Intelligence

NOVEMBER 19, 2024

You pull an open-source large language model (LLM) to train on your corporate data so that the marketing team can build better assets, and the customer service team can provide customer-facing chatbots. You build your model, but the history and context of the data you used are lost, so there is no way to trace your model back to the source.

Management 116

Management Unstructured Data Deep Learning Metadata 116

CIO Business Intelligence

NOVEMBER 27, 2024

Data quality for AI needs to cover bias detection, infringement prevention, skew detection in data for model features, and noise detection. Not all columns are equal, so you need to prioritize cleaning data features that matter to your model, and your business outcomes. asks Friedman.

Enterprise 141

Enterprise Data Quality Structured Data Modeling 141

O'Reilly on Data

MARCH 22, 2019

When the model for spam detection is systematically wrong, users can correct it. For example, we wouldn’t want real estate agents “correcting” a model to recommend houses based on race or religion; and we could even discuss whether similar behavior would be appropriate for spam detection. It's possible to abuse or to game any solution.

Metadata 275

Metadata Advertising Insurance Modeling 275

Domino Data Lab

JANUARY 11, 2021

Even for experienced developers and data scientists, the process of developing a model could involve stringing together many steps from many packages, in ways that might not be as elegant or efficient as one might like. the experience is still rooted in the same goal: simple efficiency for the whole model development lifecycle.

Modeling 145

Modeling Metrics Data Science Testing 145

AWS Big Data

OCTOBER 29, 2024

The CDH is used to create, discover, and consume data products through a central metadata catalog, while enforcing permission policies and tightly integrating data engineering, analytics, and machine learning services to streamline the user journey from data to insight.

Data Lake 104

Data Lake Sales Metadata Machine Learning 104

Rocket-Powered Data Science

MARCH 10, 2020

And this: perhaps the most powerful node in a graph model for real-world use cases might be “context”. How does one express “context” in a data model? After all, the standard relational model of databases instantiated these types of relationships in its very foundation decades ago: the ERD (Entity-Relationship Diagram).

Metadata 250

Metadata Machine Learning Prescriptive Analytics ROI 250

erwin

MARCH 17, 2020

Here are a few examples specific to enterprise architecture and business process modeling, data modeling and data governance. Enterprise Architecture & Business Process Modeling. Data Modeling. erwin Data Modeler can help you find, visualize, design, deploy and standardize high-quality enterprise data assets.

Data Governance 138

Data Governance Enterprise Modeling Metadata 138

AWS Big Data

NOVEMBER 19, 2024

Solution overview By combining the powerful vector search capabilities of OpenSearch Service with the access control features provided by Amazon Cognito , this solution enables organizations to manage access controls based on custom user attributes and document metadata. If you don’t already have an AWS account, you can create one.

Management 98

Management Metadata Manufacturing Testing 98

Rocket-Powered Data Science

JULY 6, 2021

This is accomplished through tags, annotations, and metadata (TAM). Smart content includes labeled (tagged, annotated) metadata (TAM). The key to success is to start enhancing and augmenting content management systems (CMS) with additional features: semantic content and context. Collect, curate, and catalog (i.e.,

Strategy 267

Strategy Machine Learning Metadata Knowledge Discovery 267

O'Reilly on Data

MARCH 31, 2020

Instead of writing code with hard-coded algorithms and rules that always behave in a predictable manner, ML engineers collect a large number of examples of input and output pairs and use them as training data for their models. The model is produced by code, but it isn’t code; it’s an artifact of the code and the training data.

Management 363

Management Machine Learning Experimentation Metrics 363

AWS Big Data

DECEMBER 17, 2024

We have enhanced data sharing performance with improved metadata handling, resulting in data sharing first query execution that is up to four times faster when the data sharing producers data is being updated. Lakehouse allows you to use preferred analytics engines and AI models of your choice with consistent governance across all your data.

Data Lake 95

Data Lake Data Warehouse Data-driven Optimization 95

AWS Big Data

NOVEMBER 19, 2024

Additionally, customers adopting a federated deployment model find it challenging to provide isolated environments for different teams or departments, and at the same time optimize cost. It’s essential to monitor key metrics such as metadata database memory usage, and CPU utilization of the worker/scheduler hybrid container.

Metadata 111

Metadata Cost-Benefit Metrics Optimization 111

O'Reilly on Data

DECEMBER 12, 2023

If the output of a model can’t be owned by a human, who (or what) is responsible if that output infringes existing copyright? In an article in The New Yorker , Jaron Lanier introduces the idea of data dignity, which implicitly distinguishes between training a model and generating output using a model.

Modeling 335

Modeling Sales Software Statistics 335

Cloudera

OCTOBER 23, 2024

In August, we wrote about how in a future where distributed data architectures are inevitable, unifying and managing operational and business metadata is critical to successfully maximizing the value of data, analytics, and AI. It is a critical feature for delivering unified access to data in distributed, multi-engine architectures.

Metadata 84

Metadata Data Lake Dashboards Interactive 84

Rocket-Powered Data Science

FEBRUARY 15, 2023

While generative AI has been around for several years , the arrival of ChatGPT (a conversational AI tool for all business occasions, built and trained from large language models) has been like a brilliant torch brought into a dark room, illuminating many previously unseen opportunities. So, if you have 1 trillion data points (g.,

Strategy 290

Strategy Experimentation Uncertainty Machine Learning 290

O'Reilly on Data

MARCH 18, 2020

Whether it’s controlling for common risk factors—bias in model development, missing or poorly conditioned data, the tendency of models to degrade in production—or instantiating formal processes to promote data governance, adopters will have their work cut out for them as they work to establish reliable AI production lines.

Enterprise 349

Enterprise Deep Learning Data Governance Risk 349

O'Reilly on Data

JANUARY 8, 2019

This is also reflected by the emergence of tools that are specific to machine learning, including data science platforms, data lineage, metadata management and analysis, data governance, and model lifecycle management. Burgeoning IoT technologies.

Machine Learning 272

Machine Learning IoT Internet of Things Data Science 272

AWS Big Data

DECEMBER 4, 2024

They’re taking data they’ve historically used for analytics or business reporting and putting it to work in machine learning (ML) models and AI-powered applications. SageMaker simplifies the discovery, governance, and collaboration for data and AI across your lakehouse, AI models, and applications.

Data Analytics 100

Data Analytics Analytics Data Lake Data Quality 100

O'Reilly on Data

MARCH 23, 2023

It’s important to understand that ChatGPT is not actually a language model. It’s a convenient user interface built around one specific language model, GPT-3.5, is one of a class of language models that are sometimes called “large language models” (LLMs)—though that term isn’t very helpful. with specialized training.

IT 348

IT Modeling Testing Risk 348

erwin

DECEMBER 9, 2024

For instance, Large Language Models (LLMs) are known to ultimately perform better when data is structured. But how can delivering an intelligent data foundation specifically increase your successful outcomes of AI models? One reason would be to counteract our inherent bias as we work to train the data that feeds AI models.

Metadata 52

Metadata Data Quality Sales Modeling 52

O'Reilly on Data

JULY 28, 2020

You must detect when the model has become stale, and retrain it as necessary. The Marketing team built the first model, but because it was from marketing, the model optimized for CTR and lead conversion. Nonetheless, building a superior feature pipeline or model architecture will always be worthwhile.

Marketing 364

Marketing Experimentation Metrics Testing 364

CIO Business Intelligence

SEPTEMBER 19, 2024

This enables companies to directly access key metadata (tags, governance policies, and data quality indicators) from over 100 data sources in Data Cloud, it said. Additional to that, we are also allowing the metadata inside of Alation to be read into these agents.”

Data Governance 123

Data Governance Metadata Unstructured Data Structured Data 123

O'Reilly on Data

JANUARY 28, 2019

An economic model for consent (I consent to let you use my data if you pay me) misses the point: data’s value doesn’t live with the individual. Google's market valuation doesn't derive from the value of my data or yours in isolation, but the added value that comes from aggregating data across billions of searches and other sources.

Insurance 247

Insurance Metadata Data Collection Marketing 247

Smart Data Collective

AUGUST 25, 2020

Some of the benefits are detailed below: Optimizing metadata for greater reach and branding benefits. One of the most overlooked factors is metadata. Metadata is important for numerous reasons. Search engines crawl metadata of image files, videos and other visual creative when they are indexing websites.

Data mining 137

Data mining Metadata Big Data ROI 137

erwin

OCTOBER 8, 2020

But even with the “need for speed” to market, new applications must be modeled and documented for compliance, transparency and stakeholder literacy. With all these diverse metadata sources, it is difficult to understand the complicated web they form much less get a simple visual flow of data lineage and impact analysis.

Data Governance 145

Data Governance Metadata Testing Data Lake 145