David Menninger's Analyst Perspectives

SEPTEMBER 28, 2021

Collibra is a data governance software company that offers tools for metadata management and data cataloging. The software enables organizations to find data quickly, identify its source and assure its integrity. Line-of-business workers can use it to create, review and update the organization's policies on different data assets.

Data Governance 349

Data Governance Metadata Software Management 349

AWS Big Data

OCTOBER 14, 2024

Writing SQL queries requires not just remembering the SQL syntax rules, but also knowledge of the tables metadata, which is data about table schemas, relationships among the tables, and possible column values. Although LLMs can generate syntactically correct SQL queries, they still need the table metadata for writing accurate SQL query.

Metadata 103

Metadata Data Lake Modeling Data Warehouse 103

AWS Big Data

NOVEMBER 20, 2024

We’re excited to announce a new feature in Amazon DataZone that offers enhanced metadata governance for your subscription approval process. With this update, domain owners can define and enforce metadata requirements for data consumers when they request access to data assets. Key benefits The feature benefits multiple stakeholders.

Metadata 92

Metadata Data Governance Metrics Marketing 92

O'Reilly on Data

FEBRUARY 11, 2020

These include the basics, such as metadata creation and management, data provenance, data lineage, and other essentials. They’re still struggling with the basics: tagging and labeling data, creating (and managing) metadata, managing unstructured data, etc. They don’t have the resources they need to clean up data quality problems.

Data Quality 309

Data Quality Metadata Data Governance Publishing 309

Cloudera

NOVEMBER 13, 2024

It leverages knowledge graphs to keep track of all the data sources and data flows, using AI to fill the gaps so you have the most comprehensive metadata management solution. Together, Cloudera and Octopai will help reinvent how customers manage their metadata and track lineage across all their data sources.

Metadata 96

Metadata Management Data Governance Data-driven 96

AWS Big Data

NOVEMBER 29, 2023

The Eightfold Talent Intelligence Platform integrates with Amazon Redshift metadata security to implement visibility of data catalog listing of names of databases, schemas, tables, views, stored procedures, and functions in Amazon Redshift. This post discusses restricting listing of data catalog metadata as per the granted permissions.

Metadata 111

Metadata Data Warehouse Analytics Data Analytics 111

Rocket-Powered Data Science

JULY 6, 2021

This is accomplished through tags, annotations, and metadata (TAM). Smart content includes labeled (tagged, annotated) metadata (TAM). The key to success is to start enhancing and augmenting content management systems (CMS) with additional features: semantic content and context. Collect, curate, and catalog (i.e.,

Strategy 267

Strategy Machine Learning Metadata Knowledge Discovery 267

O'Reilly on Data

MARCH 22, 2019

While neither of these is a complete solution, I can imagine a future version of these proposals that standardizes metadata so data routing protocols can determine which flows are appropriate and which aren't. That's work that hasn't been started, but it's work that needed. It's possible to abuse or to game any solution.

Metadata 246

Metadata Advertising Insurance Modeling 246

Rocket-Powered Data Science

MARCH 10, 2020

The training data and feature sets that feed machine learning algorithms can now be immensely enriched with tags, labels, annotations, and metadata that were inferred and/or provided naturally through the transformation of your repository of data into a graph of data.

Metadata 250

Metadata Machine Learning Prescriptive Analytics ROI 250

AWS Big Data

DECEMBER 20, 2024

Solution overview The basic concept of the modernization project is to create metadata-driven frameworks, which are reusable, scalable, and able to respond to the different phases of the modernization process. By reducing the number of files, metadata analysis and integrity phases are reduced, speeding up the migration phase.

Metadata 52

Metadata Data-driven Snapshot Data Lake 52

AWS Big Data

NOVEMBER 26, 2024

Central to a transactional data lake are open table formats (OTFs) such as Apache Hudi , Apache Iceberg , and Delta Lake , which act as a metadata layer over columnar formats. XTable isn’t a new table format but provides abstractions and tools to translate the metadata associated with existing formats.

Metadata 105

Metadata Data Lake Snapshot Cost-Benefit 105

AWS Big Data

MARCH 4, 2024

Apache Iceberg manages these schema changes in a backward-compatible way through its innovative metadata table evolution architecture. With Lake Formation, you can manage fine-grained access control for your data lake data on Amazon S3 and its metadata in the Data Catalog. Iceberg maintains the table state in metadata files.

Snapshot 132

Snapshot Data Lake Metadata Recreation/Entertainment 132

AWS Big Data

OCTOBER 25, 2023

Solution overview Data and metadata discovery is one of the primary requirements in data analytics, where data consumers explore what data is available and in what format, and then consume or query it for analysis. But in the case of unstructured data, metadata discovery is challenging because the raw data isn’t easily readable.

Unstructured Data 133

Unstructured Data Metadata Management Analytics 133

AWS Big Data

NOVEMBER 14, 2024

Under the hood, UniForm generates Iceberg metadata files (including metadata and manifest files) that are required for Iceberg clients to access the underlying data files in Delta Lake tables. Both Delta Lake and Iceberg metadata files reference the same data files. The table is registered in AWS Glue Data Catalog.

Metadata 121

Metadata Data Warehouse Big Data Data Lake 121

CIO Business Intelligence

SEPTEMBER 19, 2024

This enables companies to directly access key metadata (tags, governance policies, and data quality indicators) from over 100 data sources in Data Cloud, it said. Additional to that, we are also allowing the metadata inside of Alation to be read into these agents.”

Data Governance 123

Data Governance Metadata Unstructured Data Structured Data 123

AWS Big Data

MAY 29, 2024

Since its inception, Apache Kafka has depended on Apache Zookeeper for storing and replicating the metadata of Kafka brokers and topics. the Kafka community has adopted KRaft (Apache Kafka on Raft), a consensus protocol, to replace Kafka’s dependency on ZooKeeper for metadata management. For Metadata mode , select KRaft.

Metadata 110

Metadata Cost-Benefit Management Big Data 110

AWS Big Data

NOVEMBER 22, 2024

How RFS works OpenSearch and Elasticsearch snapshots are a directory tree that contains both data and metadata. Metadata files exist in the snapshot to provide details about the snapshot as a whole, the source cluster’s global metadata and settings, each index in the snapshot, and each shard in the snapshot.

Snapshot 96

Snapshot Metadata Recreation/Entertainment Data Processing 96

AWS Big Data

JUNE 17, 2024

Backup and restore architecture The backup and restore strategy involves periodically backing up Amazon MWAA metadata to Amazon Simple Storage Service (Amazon S3) buckets in the primary Region. The pipeline includes a DAG deployed to the DAGs S3 bucket, which performs backup of your Airflow metadata. The steps are as follows: [1.a]

Strategy 111

Strategy Metadata Recreation/Entertainment Metrics 111

AWS Big Data

SEPTEMBER 12, 2024

Along with the Glue Data Catalog’s automated compaction feature, these storage optimizations can help you reduce metadata overhead, control storage costs, and improve query performance. The Glue Data Catalog monitors tables daily, removes snapshots from table metadata, and removes the data files and orphan files that are no longer needed.

Optimization 128

Optimization Snapshot Metadata Metrics 128

AWS Big Data

MARCH 2, 2023

Apache Iceberg is an open table format for very large analytic datasets, which captures metadata information on the state of datasets as they evolve and change over time. Apache Iceberg addresses customer needs by capturing rich metadata information about the dataset at the time the individual data files are created.

Data Lake 136

Data Lake Data Processing Metadata Snapshot 136

AWS Big Data

APRIL 17, 2024

The following diagram illustrates an indexing flow involving a metadata update in OR1 During indexing operations, individual documents are indexed into Lucene and also appended to a write-ahead log also known as a translog. In the event of an infrastructure failure, an OpenSearch domain can end up losing one or more nodes.

Optimization 128

Optimization Snapshot Metadata Cost-Benefit 128

CIO Business Intelligence

DECEMBER 9, 2024

On the storage front, AWS unveiled S3 Table Buckets and the S3 Metadata features. Other updates added to AWS generative AI platform Bedrock included Bedrock Intelligent Prompt Routing, Amazon Kendra GenAI Index, Bedrock Knowledge Bases support for structured data, GraphRAG, and Bedrock Data Automation for unstructured data retrieval.

Metadata 116

Metadata Unstructured Data Data Lake Data-driven 116

AWS Big Data

APRIL 3, 2024

Iceberg tables maintain metadata to abstract large collections of files, providing data management features including time travel, rollback, data compaction, and full schema evolution, reducing management overhead. Snowflake writes Iceberg tables to Amazon S3 and updates metadata automatically with every transaction.

Data Lake 122

Data Lake Snapshot Metadata Data Architecture 122

AWS Big Data

NOVEMBER 13, 2024

In addition, the team aligned on business metadata attributes that would help with data discovery. Business metadata Business metadata helps users understand the context of the data, which can lead to increased trust in the data. This provides consistency of business metadata across the organization.

Metadata 105

Metadata Data Quality Digital Transformation Data-driven 105

AWS Big Data

JUNE 10, 2024

When evolving such a partition definition, the data in the table prior to the change is unaffected, as is its metadata. Only data that is written to the table after the evolution is partitioned with the new definition, and the metadata for this new set of data is kept separately. Old metadata files are kept for history by default.

Data Lake 126

Data Lake Metadata Snapshot Analytics 126

AWS Big Data

OCTOBER 11, 2023

The second streaming data source constitutes metadata information about the call center organization and agents that gets refreshed throughout the day. This data contains metadata information like organization names for their respective organization IDs, agent names, and more. client("s3") S3_BUCKET = ' ' kinesis_client = boto3.client("kinesis")

Management 130

Management Metadata Analytics Dashboards 130

AWS Big Data

APRIL 16, 2024

Running Apache Airflow at scale puts proportionally greater load on the Airflow metadata database, sometimes leading to CPU and memory issues on the underlying Amazon Relational Database Service (Amazon RDS) cluster. A resource-starved metadata database may lead to dropped connections from your workers, failing tasks prematurely.

Metadata 119

Metadata Metrics Testing Management 119

DataKitchen

FEBRUARY 2, 2023

Most data governance tools today start with the slow, waterfall building of metadata with data stewards and then hope to use that metadata to drive code that runs in production. In reality, the ‘active metadata’ is just a written specification for a data developer to write their code.

Metadata 130

Metadata Testing Risk Data Science 130

AWS Big Data

OCTOBER 3, 2023

This means the data files in the data lake aren’t modified during the migration and all Apache Iceberg metadata files (manifests, manifest files, and table metadata files) are generated outside the purview of the data. In this method, the metadata are recreated in an isolated environment and colocated with the existing data files.

Data Lake 122

Data Lake Metadata Snapshot Recreation/Entertainment 122

AWS Big Data

MARCH 22, 2024

Benchmark setup In our testing, we used the 3 TB dataset stored in Amazon S3 in compressed Parquet format and metadata for databases and tables is stored in the AWS Glue Data Catalog. When statistics aren’t available, Amazon EMR and Athena use S3 file metadata to optimize query plans. With Amazon EMR 6.10.0

Metadata 118

Metadata Statistics Broadcasting Optimization 118

AWS Big Data

APRIL 2, 2024

Without the right metadata and documentation, data consumers overlook valuable datasets relevant to their use case or spend more time going back and forth with data producers to understand the data and its relevance for their use case—or worse, misuse the data for a purpose it was not intended for.

Metadata 111

Metadata Metrics Data-driven Contextual Data 111

O'Reilly on Data

JUNE 14, 2024

At the same time, Miso went about an in-depth chunking and metadata-mapping of every book in the O’Reilly catalog to generate enriched vector snippet embeddings of each work.

Metadata 199

Metadata Publishing Modeling Data-driven 199

CIO Business Intelligence

AUGUST 21, 2024

Add context to unstructured content With the help of IDP, modern ECM tools can extract contextual information from unstructured data and use it to generate new metadata and metadata fields. That relieves users from having to fill out such fields themselves to classify documents, which they often don’t do well, if at all.

Insurance 116

Insurance Management Metadata Unstructured Data 116

AWS Big Data

JANUARY 16, 2024

Within Airflow, the metadata database is a core component storing configuration variables, roles, permissions, and DAG run histories. A healthy metadata database is therefore critical for your Airflow environment. The third component is for creating and storing backups of all configurations and metadata that is required to restore.

Strategy 111

Strategy Metadata Metrics Dashboards 111

AWS Big Data

NOVEMBER 3, 2023

The metadata is extracted from each job run, including information like runtime, start time, end time, auto scaling, number of workers, and worker type, and is written to an Amazon DynamoDB table with TTL (time to live) enabled to ensure the table doesn’t grow too large. If the tables don’t exist, Athena creates them.

Metrics 114

Metrics Dashboards Metadata Visualization 114

AWS Big Data

MAY 7, 2024

Figure 1: Flow of actions for self-service analytics around data assets stored in relational databases First, the data producer needs to capture and catalog the technical metadata of the data asset. Second, the data producer needs to consolidate the data asset’s metadata in the business catalog and enrich it with business metadata.

Metadata 104

Metadata Data Lake Data Processing Data-driven 104