AWS Big Data

NOVEMBER 20, 2024

We’re excited to announce a new feature in Amazon DataZone that offers enhanced metadata governance for your subscription approval process. With this update, domain owners can define and enforce metadata requirements for data consumers when they request access to data assets. Key benefits The feature benefits multiple stakeholders.

Metadata 92

Metadata Data Governance Metrics Marketing 92

Cloudera

JANUARY 26, 2024

This will allow a data office to implement access policies over metadata management assets like tags or classifications, business glossaries, and data catalog entities, laying the foundation for comprehensive data access control. First, a set of initial metadata objects are created by the data steward.

Metadata 79

Metadata Data Governance Management Finance 79

Rocket-Powered Data Science

JULY 6, 2021

If you include the title of this blog, you were just presented with 13 examples of heteronyms in the preceding paragraphs. This is accomplished through tags, annotations, and metadata (TAM). Smart content includes labeled (tagged, annotated) metadata (TAM). What you have just experienced is a plethora of heteronyms.

Strategy 267

Strategy Machine Learning Metadata Knowledge Discovery 267

TDAN

MAY 31, 2022

The main goal of creating an enterprise data fabric is not new. It is the ability to deliver the right data at the right time, in the right shape, and to the right data consumer, irrespective of how and where it is stored. Data fabric is the common “net” that stitches integrated data from multiple data […].

Metadata 98

Metadata Enterprise IT Data Architecture 98

erwin

JUNE 26, 2020

Organizations with particularly deep data stores might need a data catalog with advanced capabilities, such as automated metadata harvesting to speed up the data preparation process. Three Types of Metadata in a Data Catalog. The metadata provides information about the asset that makes it easier to locate, understand and evaluate.

Metadata 132

Metadata Cost-Benefit Measurement Data-driven 132

AWS Big Data

MARCH 2, 2023

Apache Iceberg is an open table format for very large analytic datasets, which captures metadata information on the state of datasets as they evolve and change over time. Apache Iceberg addresses customer needs by capturing rich metadata information about the dataset at the time the individual data files are created.

Data Lake 136

Data Lake Data Processing Metadata Snapshot 136

erwin

OCTOBER 8, 2020

With all these diverse metadata sources, it is difficult to understand the complicated web they form much less get a simple visual flow of data lineage and impact analysis. The metadata-driven suite automatically finds, models, ingests, catalogs and governs cloud data assets. Subscribe to the erwin Expert Blog.

Data Governance 145

Data Governance Metadata Testing Data Lake 145

DataKitchen

AUGUST 9, 2021

The domain requires a team that creates/updates/runs the domain, and we can’t forget metadata: catalogs, lineage, test results, processing history, etc., …. At this point in our blog series, we have talked about data mesh from an organizational and technical perspective.

Testing 246

Testing Data Lake Metadata Publishing 246

DataKitchen

AUGUST 3, 2021

Discoverable – users have access to a catalog or metadata management tool which renders the domain discoverable and accessible. We’ll cover some of the potential challenges facing data mesh enterprise architectures in our next blog. Also, the domain must support the attributes that are part of every modern data architecture.

Data Architecture 246

Data Architecture Data Lake Cost-Benefit Data Warehouse 246

Rocket-Powered Data Science

FEBRUARY 15, 2023

These rules are not necessarily “Rocket Science” (despite the name of this blog site), but they are common business sense for most business-disruptive technology implementations in enterprises. The latter is essential for Generative AI implementations.

Strategy 290

Strategy Experimentation Uncertainty Machine Learning 290

Timo Elliott

JANUARY 6, 2022

In a previous blog , I explored the value of dark data and how it can reveal insights that can streamline processes, improve customer experiences, generate more revenue – and maybe even help make the world a better place. Analyze your metadata. The data you’ve collected and saved over the years isn’t free. Use people.

IT 135

IT Metadata Data-driven Data Governance 135

AWS Big Data

OCTOBER 3, 2023

This means the data files in the data lake aren’t modified during the migration and all Apache Iceberg metadata files (manifests, manifest files, and table metadata files) are generated outside the purview of the data. In this method, the metadata are recreated in an isolated environment and colocated with the existing data files.

Data Lake 122

Data Lake Metadata Snapshot Recreation/Entertainment 122

AWS Big Data

NOVEMBER 14, 2024

Under the hood, UniForm generates Iceberg metadata files (including metadata and manifest files) that are required for Iceberg clients to access the underlying data files in Delta Lake tables. Both Delta Lake and Iceberg metadata files reference the same data files. The table is registered in AWS Glue Data Catalog.

Metadata 121

Metadata Data Warehouse Big Data Data Lake 121

erwin

JANUARY 9, 2020

That’s because it’s the best way to visualize metadata , and metadata is now the heart of enterprise data management and data governance/ intelligence efforts. erwin DM 2020 is an essential source of metadata and a critical enabler of data governance and intelligence efforts. erwin Data Modeler: Where the Magic Happens.

Data Governance 144

Data Governance Modeling Metadata Unstructured Data 144

Smart Data Collective

AUGUST 5, 2021

In this blog, we will cover an overview of Delta Lakes , its advantages, and how the above challenges can be overcome by moving to Delta Lake and migrating to Spark 3.0 Along with the ability to implement ACID transactions and scalable metadata handling, Delta Lakes can also unify the streaming and batch data processing”. .

Data Processing 143

Data Processing Metadata Broadcasting Statistics 143

AWS Big Data

NOVEMBER 22, 2024

How RFS works OpenSearch and Elasticsearch snapshots are a directory tree that contains both data and metadata. Metadata files exist in the snapshot to provide details about the snapshot as a whole, the source cluster’s global metadata and settings, each index in the snapshot, and each shard in the snapshot.

Snapshot 96

Snapshot Metadata Recreation/Entertainment Data Processing 96

Cloudera

NOVEMBER 13, 2020

This is part of our series of blog posts on recent enhancements to Impala. Metadata Caching. As Impala’s adoption grew the catalog service started to experience these growing pains, therefore recently we introduced two new features to alleviate the stress, On-demand Metadata and Zero Touch Metadata. More on this below.

Optimization 144

Optimization Metadata Statistics Cost-Benefit 144

AWS Big Data

NOVEMBER 13, 2024

In addition, the team aligned on business metadata attributes that would help with data discovery. Business metadata Business metadata helps users understand the context of the data, which can lead to increased trust in the data. This provides consistency of business metadata across the organization.

Metadata 105

Metadata Data Quality Digital Transformation Data-driven 105

AWS Big Data

MARCH 22, 2024

Benchmark setup In our testing, we used the 3 TB dataset stored in Amazon S3 in compressed Parquet format and metadata for databases and tables is stored in the AWS Glue Data Catalog. When statistics aren’t available, Amazon EMR and Athena use S3 file metadata to optimize query plans. With Amazon EMR 6.10.0

Metadata 118

Metadata Statistics Broadcasting Optimization 118

erwin

DECEMBER 3, 2020

This metadata is ingested into the data catalog, definitions are added within a business glossary, and the searchable repository enables users to understand how data is used and stored. Expanded AI capabilities to enrich metadata scanning and speed the handling of sensitive data for automated GDPR and CCPA compliance programs.

Metadata 141

Metadata Data Governance Data-driven Digital Transformation 141

AWS Big Data

JUNE 10, 2024

In this blog post, we dive into different data aspects and how Cloudinary breaks the two concerns of vendor locking and cost efficient data analytics by using Apache Iceberg, Amazon Simple Storage Service (Amazon S3 ), Amazon Athena , Amazon EMR , and AWS Glue. Each change to a table produces a new metadata file to provide atomicity.

Data Lake 126

Data Lake Metadata Snapshot Analytics 126

Cloudera

FEBRUARY 22, 2022

Companies such as Adobe , Expedia , LinkedIn , Tencent , and Netflix have published blogs about their Apache Iceberg adoption for processing their large scale analytics datasets. . In CDP we enable Iceberg tables side-by-side with the Hive table types, both of which are part of our SDX metadata and security framework.

Snapshot 114

Snapshot Metadata Cost-Benefit Data Architecture 114

erwin

OCTOBER 16, 2020

IDC defines data intelligence as business, technical, relational, and operational metadata that provides transparency of data profiles, classification, quality, location, context, and lineage, providing people, processes, and technology with trustworthy, reliable data. Parts of this blog are excerpted from my keynote on day No.

Internet of Things 145

Internet of Things Data-driven Uncertainty Data Governance 145

DataKitchen

FEBRUARY 2, 2023

Most data governance tools today start with the slow, waterfall building of metadata with data stewards and then hope to use that metadata to drive code that runs in production. In reality, the ‘active metadata’ is just a written specification for a data developer to write their code.

Metadata 130

Metadata Testing Risk Data Science 130

Cloudera

APRIL 18, 2024

In those discussions, it was clear that everyone understood the need to treat data estates more cohesively as a whole—that means bringing more attention to security, data governance, and metadata management, the latter of which has become increasingly popular. Learn more about how Cloudera is accelerating enterprise AI.

Analytics 115

Analytics Metadata Data Strategy Optimization 115

erwin

JANUARY 30, 2020

However, more than 50 percent say they have deployed metadata management, data analytics, and data quality solutions. erwin Named a Leader in Gartner 2019 Metadata Management Magic Quadrant. And close to 50 percent have deployed data catalogs and business glossaries. Most have only data governance operations.

Data Governance 145

Data Governance Metadata Cost-Benefit Digital Transformation 145

Cloudera

JULY 10, 2024

Observability for your most secure data For your most sensitive, protected data, we understand even the metadata and telemetry about your workloads must be kept under close watch, and it must stay within your secured environment. The post Introducing Cloudera Observability Premium appeared first on Cloudera Blog.

Cost-Benefit 105

Cost-Benefit Metadata Optimization Measurement 105

erwin

SEPTEMBER 10, 2020

In a previous blog , I explained that data lineage is basically the history of data, including a data set’s origin, characteristics, quality and movement over time. Data lineage helps answer questions about the origin of data in key performance indicator (KPI) reports, including: How are the report tables and columns defined in the metadata?

Data Governance 131

Data Governance Key Performance Indicator Metadata Digital Transformation 131

Cloudera

FEBRUARY 8, 2024

Overview This blog post describes support for materialized views for the Iceberg table format. Create Iceberg materialized view For the examples in this blog, we will use three tables from the TPC-DS dataset as our base tables: store_sales, customer and date_dim. Both full and incremental rebuild of the materialized view are supported.

Snapshot 96

Snapshot Metadata Cost-Benefit Data Warehouse 96

AWS Big Data

JULY 30, 2024

Under SAML Signing Certificates , select Actions , and then select View IdP Metadata. Under Configure external identity provider , do the following: Under Service provider metadata , choose Download metadata file to download the IAM Identity Center metadata file and save it on your system. Choose the Sign On tab.

Metadata 113

Metadata Dashboards Management Data-driven 113

erwin

JANUARY 14, 2021

Even if you don’t have a formal data intelligence program in place, there is a good possibility your organization has intelligence about its data, because it is difficult for data to exist without some form of associated metadata. Technical metadata is what makes up database schema and table definitions.

Digital Transformation 143

Digital Transformation Metadata Big Data Data-driven 143

AWS Big Data

FEBRUARY 12, 2024

As mentioned in the previous blog, AWS Clean Rooms enables multiple organizations to analyze their data and unlock insights they can act upon, without having to share sensitive, restricted, or proprietary records. When the status changes to SUCCESS, it proceeds to the next step to retrieve the AWS Glue table metadata information.

Publishing 111

Publishing Dashboards Metadata Visualization 111

AWS Big Data

OCTOBER 3, 2023

Iceberg tables store metadata in manifest files. As the number of data files increase, the amount of metadata stored in these manifest files also increases, leading to longer query planning time. The query runtime also increases because it’s proportional to the number of data or metadata file read operations.

Optimization 126

Optimization Snapshot Data Lake Metadata 126

AWS Big Data

NOVEMBER 15, 2023

In this blog post, we’ll explore a solution that streamlines this process by leveraging the capabilities of AWS Glue DataBrew. Extract metadata from the header and remove invalid rows Complete the following steps to extract metadata from the header: Choose Conditions and then choose IF. For Source , choose Value of and Column_1.

Metadata 113

Metadata Sales Data Lake Big Data 113

AWS Big Data

SEPTEMBER 7, 2023

This blog post presents an architecture solution that allows customers to extract key insights from Amazon S3 access logs at scale. AWS Glue Data Catalog stores information as metadata tables, where each table specifies a single data store. With exponential growth in data volume, centralized monitoring becomes challenging.

Metadata 116

Metadata Dashboards Metrics Visualization 116

erwin

JANUARY 17, 2020

Data models provide visualization, create additional metadata and standardize data design across the enterprise. But neither would be possible without data modeling providing the backbone for metadata management and proper data governance. In the modern context, data modeling is a function of data governance. SQL or NoSQL?

Data-driven 138

Data-driven Modeling Metadata Data Governance 138

erwin

MARCH 12, 2020

With metadata-driven automation, many DevOps processes can be automated, adding more “horsepower” to increase their speed and accuracy. Just like with cars, more horsepower in DevOps translates to greater speed. Doing More with Less. Click here to request a free demo of erwin DI.

Metadata 140

Metadata Digital Transformation Data-driven Enterprise 140