article thumbnail

Expand data access through Apache Iceberg using Delta Lake UniForm on AWS

AWS Big Data

Under the hood, UniForm generates Iceberg metadata files (including metadata and manifest files) that are required for Iceberg clients to access the underlying data files in Delta Lake tables. Both Delta Lake and Iceberg metadata files reference the same data files. The table is registered in AWS Glue Data Catalog.

Metadata 122
article thumbnail

Use Amazon Kinesis Data Streams to deliver real-time data to Amazon OpenSearch Service domains with Amazon OpenSearch Ingestion

AWS Big Data

For example, you can use metadata about the Kinesis data stream name to index by data stream ( ${getMetadata("kinesis_stream_name") ), or you can use document fields to index data depending on the CloudWatch log group or other document data ( ${path/to/field/in/document} ).

Metadata 122
Insiders

Sign Up for our Newsletter

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

article thumbnail

Federate to Amazon Redshift Query Editor v2 with Microsoft Entra ID

AWS Big Data

Save the federation metadata XML file You use the federation metadata file to configure the IAM IdP in a later step. In the Single sign-on section , under SAML Certificates , choose Download for Federation Metadata XML. Complete the following steps to download the file: Navigate back to your SAML-based sign-in page.

Sales 105
article thumbnail

Use Amazon OpenSearch Ingestion to migrate to Amazon OpenSearch Serverless

AWS Big Data

Migration of metadata such as security roles and dashboard objects will be covered in another subsequent post. For index , you can leave it as default, which will get the metadata from the source index and write to the same name in the destination as of the sources.

Metadata 122
article thumbnail

Manage Amazon OpenSearch Service Visualizations, Alerts, and More with GitHub and Jenkins

AWS Big Data

In the trust policy, specify that Amazon Elastic Compute Cloud (Amazon EC2) can assume this role: { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "ec2.amazonaws.com" amazonaws.com" }, "Action": "sts:AssumeRole" } ] } Make a note of the role ARN.

article thumbnail

Introducing a new unified data connection experience with Amazon SageMaker Lakehouse unified data connectivity

AWS Big Data

With the ability to browse metadata, you can understand the structure and schema of the data source, identify relevant tables and fields, and discover useful data assets you may not be aware of. This approach simplifies your data journey and helps you meet your security requirements. Choose the created IAM role.

article thumbnail

Becoming a machine learning company means investing in foundational technologies

O'Reilly on Data

Consider deep learning, a specific form of machine learning that resurfaced in 2011/2012 due to record-setting models in speech and computer vision. Metadata and artifacts needed for audits. Use ML to unlock new data types—e.g., images, audio, video. Tackle completely new use cases and applications.