O'Reilly on Data

MARCH 20, 2019

Data poisoning attacks. Data poisoning refers to someone systematically changing your training data to manipulate your model’s predictions. Data poisoning attacks have also been called “causative” attacks.) To poison data, an attacker must have access to some or all of your training data.

Modeling 278

Modeling Machine Learning Predictive Modeling Consulting 278

AWS Big Data

FEBRUARY 6, 2023

AWS Glue is a serverless data integration service that makes it simple to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development. MongoDB Atlas is a developer data service from AWS technology partner MongoDB, Inc.

Metadata 98

Metadata Data Lake Machine Learning Big Data 98

AWS Big Data

OCTOBER 21, 2024

Let’s briefly describe the capabilities of the AWS services we referred above: AWS Glue is a fully managed, serverless, and scalable extract, transform, and load (ETL) service that simplifies the process of discovering, preparing, and loading data for analytics. Amazon Athena is used to query, and explore the data.

Sales 115

Sales Data-driven Data Processing Key Performance Indicator 115

AWS Big Data

DECEMBER 4, 2024

SageMaker still includes all the existing ML and AI capabilities you’ve come to know and love for data wrangling, human-in-the-loop data labeling with Amazon SageMaker Ground Truth , experiments, MLOps, Amazon SageMaker HyperPod managed distributed training, and more. Having confidence in your data is key.

Data Analytics 104

Data Analytics Analytics Data Lake Data Quality 104

IBM Big Data Hub

JULY 13, 2023

When we talk about data integrity, we’re referring to the overarching completeness, accuracy, consistency, accessibility, and security of an organization’s data. Together, these factors determine the reliability of the organization’s data. In short, yes.

Data Quality 63

Data Quality Data Integration Metadata Cost-Benefit 63

CIO Business Intelligence

MARCH 24, 2023

The Business Application Research Center (BARC) warns that data governance is a highly complex, ongoing program, not a “big bang initiative,” and it runs the risk of participants losing trust and interest over time. The program must introduce and support standardization of enterprise data.

Data Governance 142

Data Governance Management Metadata Data Quality 142

O'Reilly on Data

DECEMBER 19, 2018

If you suddenly see unexpected patterns in your social data, that may mean adversaries are attempting to poison your data sources. Anomaly detection may have originated in finance, but it is becoming a part of every data scientist’s toolkit. Tim Kraska on “How machine learning will accelerate data management systems”.

Machine Learning 279

Machine Learning Software Metadata Testing 279

DataKitchen

OCTOBER 20, 2023

The Solution: ‘Payload’ Data Journeys Traditional Data Observability usually focuses on a ‘process journey,’ tracking the performance and status of data pipelines. ’ It assigns unique identifiers to each data item—referred to as ‘payloads’—related to each event.

Insurance 173

Insurance Metadata Data-driven Data Quality 173

AWS Big Data

DECEMBER 16, 2024

Third, some services require you to set up and manage compute resources used for federated connectivity, and capabilities like connection testing and data preview arent available in all services. To solve for these challenges, we launched Amazon SageMaker Lakehouse unified data connectivity.

Visualization 93

Visualization Data Processing Testing Publishing 93

erwin

JULY 17, 2019

Your organization won’t be able to take complete advantage of analytics tools to become data-driven unless you establish a foundation for agile and complete data management. You need automated data mapping and cataloging through the integration lifecycle process, inclusive of data at rest and data in motion.

Digital Transformation 98

Digital Transformation Strategy Metadata Data-driven 98

Ontotext

JULY 29, 2021

KGs bring the Semantic Web paradigm to the enterprises, by introducing semantic metadata to drive data management and content management to new levels of efficiency and breaking silos to let them synergize with various forms of knowledge management. Take this restaurant, for example. Enterprise Knowledge Graphs and the Semantic Web.

Enterprise 124

Enterprise Metadata Knowledge Discovery Management 124

AWS Big Data

APRIL 3, 2024

Data engineers use Apache Iceberg because it’s fast, efficient, and reliable at any scale and keeps records of how datasets change over time. Apache Iceberg offers integrations with popular data processing frameworks such as Apache Spark, Apache Flink, Apache Hive, Presto, and more.

Data Lake 127

Data Lake Snapshot Metadata Data Architecture 127

David Menninger's Analyst Perspectives

OCTOBER 24, 2024

Although the terms data fabric and data mesh are often used interchangeably, I previously explained that they are distinct but complementary. The company added a variety of new features with the launch of Denodo Platform 9 in June, designed to address requirements for intelligent applications and AI-driven integration.

Management 130

Management Data-driven Data Governance Data Lake 130

AWS Big Data

FEBRUARY 29, 2024

However, enterprise data generated from siloed sources combined with the lack of a data integration strategy creates challenges for provisioning the data for generative AI applications. Data discoverability Unlike structured data, which is managed in well-defined rows and columns, unstructured data is stored as objects.

Data Governance 124

Data Governance Unstructured Data Metadata Data Lake 124

AWS Big Data

JUNE 7, 2023

For this, Cargotec built an Amazon Simple Storage Service (Amazon S3) data lake and cataloged the data assets in AWS Glue Data Catalog. They chose AWS Glue as their preferred data integration tool due to its serverless nature, low maintenance, ability to control compute resources in advance, and scale when needed.

Metadata 59

Metadata Data Lake Machine Learning Big Data 59

AWS Big Data

AUGUST 31, 2023

Iceberg stores the metadata pointer for all the metadata files. When a SELECT query is reading an Iceberg table, the query engine first goes to the Iceberg catalog, then retrieves the entry of the location of the latest metadata file, as shown in the following diagram.

Data Lake 119

Data Lake Data Warehouse Metadata Data Architecture 119

AWS Big Data

SEPTEMBER 7, 2023

We will partition and format the server access logs with Amazon Web Services (AWS) Glue , a serverless data integration service, to generate a catalog for access logs and create dashboards for insights. They store attributes such as object size, total time, turn-around time, and HTTP referer for log records.

Metadata 116

Metadata Dashboards Metrics Visualization 116

AWS Big Data

JUNE 25, 2024

It provides secure, real-time access to Redshift data without copying, keeping enterprise data in place. This eliminates replication overhead and ensures access to current information, enhancing data integration while maintaining data integrity and efficiency.

Data Lake 122

Data Lake Cost-Benefit Data-driven Data Warehouse 122

Cloudera

OCTOBER 7, 2021

In this blog, I will demonstrate the value of Cloudera DataFlow (CDF) , the edge-to-cloud streaming data platform available on the Cloudera Data Platform (CDP) , as a Data integration and Democratization fabric. Data and Metadata: Data inputs and data outputs produced based on the application logic.

Metadata 124

Metadata Cost-Benefit Enterprise Interactive 124

DataKitchen

SEPTEMBER 21, 2023

.’ It’s not just about playing detective to discover where things went wrong; it’s about proactively monitoring your entire data journey to ensure everything goes right with your data. What is Data in Place? This existing paradigm fails to address the challenges and intricacies of “Data in Use.”

Testing 173

Testing Data Quality Predictive Modeling Metrics 173

AWS Big Data

JUNE 10, 2024

Organizations have multiple Hive data warehouses across EMR clusters, where the metadata gets generated. To address this challenge, organizations can deploy a data mesh using AWS Lake Formation that connects the multiple EMR clusters. An entity can act both as a producer of data assets and as a consumer of data assets.

Data Lake 116

Data Lake Metadata Data Warehouse Data Processing 116

erwin

JANUARY 24, 2019

In most companies, an incredible amount of data flows from multiple sources in a variety of formats and is constantly being moved and federated across a changing system landscape. With an automation framework, data professionals can meet these needs at a fraction of the cost of the traditional manual way. Governing metadata.

Data Governance 71

Data Governance Metadata Data-driven Cost-Benefit 71

Ontotext

JANUARY 10, 2024

And each of these gains requires data integration across business lines and divisions. Limiting growth by (data integration) complexity Most operational IT systems in an enterprise have been developed to serve a single business function and they use the simplest possible model for this. We call this the Bad Data Tax.

Metadata 72

Metadata Slice and Dice Data Integration Enterprise 72

AWS Big Data

AUGUST 15, 2023

Some examples include AWS data analytics services such as AWS Glue for data integration, Amazon QuickSight for business intelligence (BI), as well as third-party software and services from AWS Marketplace. We deploy a Lambda function data source connector to connect AWS with Google Cloud Provider.

Recreation/Entertainment 98

Recreation/Entertainment Unstructured Data Business Intelligence Data-driven 98

AWS Big Data

AUGUST 15, 2024

With Amazon DataZone, individual business units can discover and directly consume these new data assets, gaining insights to a holistic view of the data (360-degree insights) across the organization. The Central IT team manages a unified Redshift data warehouse, handling all data integration, processing, and maintenance.

Data Lake 122

Data Lake Data Warehouse Data Governance Publishing 122

AWS Big Data

SEPTEMBER 11, 2024

AWS Transfer Family seamlessly integrates with other AWS services, automates transfer, and makes sure data is protected with encryption and access controls. Each file arrives as a pair with a tail metadata file in CSV format containing the size and name of the file. 2 GB into the landing zone daily.

Data Architecture 111

Data Architecture Optimization Data Warehouse Metadata 111

Ontotext

JUNE 27, 2019

Ontotext’s GraphDB is an enterprise-ready semantic graph database (also called RDF triplestore as it stores data in RDF triples). It provides the core infrastructure for solutions where modeling agility, data integration, relationship exploration, cross-enterprise data publishing and consumption are critical.

Metadata 40

Metadata Management Enterprise Publishing 40

Ontotext

NOVEMBER 4, 2021

As we’ve said again and again, we believe that knowledge graphs are the next generation tool for helping businesses make critical decisions, based on harmonized knowledge models and data derived from siloed source systems. But these tasks are only part of the story. That covers the two webinars that we wanted to present to you today.

Metadata 98

Metadata Visualization Modeling Enterprise 98

AWS Big Data

MAY 21, 2024

Following the best practices section of the OpenSearch Service Developer Guide, AVB selected an optimal cluster configuration with three dedicated cluster manager nodes and six data nodes, across three Availability Zones , while keeping shard size between 10–30 GiB. This approach allows updates to be executed independently and asynchronously.

Manufacturing 111

Manufacturing Sales Optimization Data Processing 111

Ontotext

JUNE 3, 2020

Fast forward to today and curiosity cabinets have long been replaced by galleries, libraries, archives and museums (the set of institutions often referred to as GLAM). With knowledge graphs , additional facts and figures can be threaded into the collection items and the metadata related to them.

Contextual Data 98

Contextual Data Metadata Digital Transformation Technology 98

AWS Big Data

MARCH 15, 2023

They wanted to develop a simple incremental data processing pipeline without having to update the entire database each time the pipeline ran. The Apache Hudi framework allowed the Infomedia team to maintain a golden reference dataset and capture changes so that the downstream database could be incrementally updated in a short timeframe.

Cost-Benefit 94

Cost-Benefit Data Processing Optimization Data-driven 94

Cloudera

MARCH 23, 2022

Figure 1: Apache Iceberg fits the next generation data architecture by abstracting storage layer from analytics layer while introducing net new capabilities like time-travel and partition evolution. #1: Apache Iceberg enables seamless integration between different streaming and processing engines while maintaining data integrity between them.

Metadata 92

Metadata Data Architecture Machine Learning Cost-Benefit 92

AWS Big Data

JUNE 12, 2023

Acting as a bridge between producer and consumer apps, it enforces the schema, reduces the data footprint in transit, and safeguards against malformed data. AWS Glue is an ideal solution for running stream consumer applications, discovering, extracting, transforming, loading, and integrating data from multiple sources.

Management 98

Management Metadata Internet of Things Testing 98

AWS Big Data

NOVEMBER 29, 2023

For more information about performance improvement capabilities, refer to the list of announcements below. Amazon Redshift ML large language model (LLM) integration Amazon Redshift ML enables customers to create, train, and deploy machine learning models using familiar SQL commands.

Data Warehouse 130

Data Warehouse Analytics Data Lake Machine Learning 130

Cloudera

JUNE 6, 2019

While this approach provides isolation, it creates another significant challenge: duplication of data, metadata, and security policies, or ‘split-brain’ data lake. Now the admins need to synchronize multiple copies of the data and metadata and ensure that users across the many clusters are not viewing stale information.

Metadata 86

Metadata Data Lake Optimization Strategy 86

Ontotext

AUGUST 22, 2023

Context: The Key to Making Data Useful It goes without saying that data without meaning can yield incorrect insights, leading to potentially dangerous decisions. Beyond that, and without a way to visualize, connect, and utilize the data, it’s still just a bunch of random information.

Metadata 52

Metadata Knowledge Discovery Big Data Enterprise 52

Wayne Yaddow

MARCH 5, 2025

The goal is to examine five major methods of verifying and validating data transformations in data pipelines with an eye toward high-quality data deployment. First, we look at how unit and integration tests uncover transformation errors at an early stage. Test harnesses that mimic pipeline steps (e.g.,

Testing 52

Testing Data Transformation Statistics Metadata 52