AWS Big Data

OCTOBER 17, 2024

We’re thrilled to announce the launch of the official Amazon OpenSearch Service YouTube channel —a comprehensive resource for anyone looking to master Amazon OpenSearch Service. Amazon OpenSearch Service is a managed service that makes it straightforward to deploy, operate, and scale OpenSearch domains in AWS.

Analytics 115

Analytics Optimization Data-driven Data Architecture 115

AWS Big Data

NOVEMBER 7, 2024

Amazon OpenSearch Service launches a modernized operational analytics experience that can provide comprehensive observability spanning multiple data sources , so that you can gain insights from OpenSearch and other integrated data sources in one place. You can add collaborators by their IAM Amazon Resource Name (ARN) or IDC username.

Dashboards 111

Dashboards Visualization Data-driven Management 111

AWS Big Data

DECEMBER 5, 2024

REA Group, a digital business that specializes in real estate property, solved this problem using Amazon Managed Streaming for Apache Kafka (Amazon MSK) and a data streaming platform called Hydro. REA Group’s team of more than 3,000 people is guided by our purpose: to change the way the world experiences property.

Metrics 97

Metrics Dashboards Testing Optimization 97

AWS Big Data

FEBRUARY 13, 2025

Amazon Managed Streaming for Apache Kafka (Amazon MSK) now offers a new broker type called Express brokers. To learn more about Express brokers, refer to Introducing Express brokers for Amazon MSK to deliver high throughput and faster scaling for your Kafka clusters.

Metrics 67

Metrics Metadata Strategy Management 67

AWS Big Data

NOVEMBER 11, 2024

In this post, we show how to use Amazon Kinesis Data Streams to buffer and aggregate real-time streaming data for delivery into Amazon OpenSearch Service domains and collections using Amazon OpenSearch Ingestion. For the Amazon S3 log use case, see Using an OpenSearch Ingestion pipeline with Amazon S3.

Metadata 122

Metadata Metrics Analytics Data Processing 122

David Menninger's Analyst Perspectives

NOVEMBER 12, 2024

Infor introduced its original AI and machine learning capabilities in 2017 in the form of Coleman, which uses its Infor AI/ML platform built on Amazon’s SageMaker to create predictive and prescriptive analytics. It also offered a chatbot that utilized Amazon Lex.

Finance 130

Finance Prescriptive Analytics Cost-Benefit Manufacturing 130

O'Reilly on Data

NOVEMBER 19, 2024

Large-scale production recommenders, search engines, and other discovery processes also have a long history of leveraging knowledge graphs , such as at Amazon , Alphabet , Microsoft , LinkedIn , eBay , Pinterest , and so on. What is GraphRAG? Graph technologies help reveal nonintuitive connections within data.

Unstructured Data 318

Unstructured Data Structured Data Statistics Modeling 318

AWS Big Data

JULY 8, 2024

Amazon Redshift Serverless is a fully managed, scalable cloud data warehouse that accelerates your time to insights with fast, simple, and secure analytics at scale. Amazon Redshift data sharing allows you to share data within and across organizations, AWS Regions, and even third-party providers, without moving or copying the data.

Data Warehouse 133

Data Warehouse IT Recreation/Entertainment Cost-Benefit 133

datapine

SEPTEMBER 27, 2022

These businesses include eBay, Autotrader, and Amazon. In other words, “Sams Teach Yourself SQL in 10 Minutes” teaches the parts of SQL you need to know: starting with simple data retrieval and quickly going on to more complex topics including the use of SQL joins , subqueries, stored procedures, cursors, triggers, and table constraints.

Business Intelligence 292

Business Intelligence Data Warehouse Data Processing Data mining 292

AWS Big Data

AUGUST 2, 2024

Since the launch of tiered storage for Amazon Managed Streaming for Apache Kafka (Amazon MSK), customers have embraced this feature for its ability to optimize storage costs and improve performance. New messages are initially written to Amazon EBS for fast performance. This frees up space on the EBS volumes for new messages.

Metrics 124

Metrics Testing Cost-Benefit Management 124

O'Reilly on Data

SEPTEMBER 15, 2020

User stakeholders are interested in benefiting from the platform’s functionality: staying up-to-date, quickly finding new people and topics to follow, and engaging with family and friends. Customer stakeholders are the people and companies that advertise on the platform, and are most concerned with ROI on their ad spend.

Advertising 362

Advertising Cost-Benefit ROI Machine Learning 362

AWS Big Data

MARCH 14, 2024

to Amazon Managed Streaming for Apache Kafka (Amazon MSK) running version 2.6.2. Why we migrated to Amazon MSK For us, migrating to Amazon MSK came down to three key decision points: Simplified technical operations – Running Kafka on a self-managed infrastructure was an operational overhead for us.

Management 126

Management Insurance Optimization Strategy 126

CIO Business Intelligence

NOVEMBER 4, 2024

Jacknis advises CIOs to focus on the three reasons why AI is such a hot topic. AI bias has already got organizations such as online retailer Amazon into hot water, and here again, the CIO must play a pivotal role in protecting the business. It’s back to Moore’s Law. We’ve already seen that AI depends on a lot of compute power.

Business Driver 111

Business Driver Advertising Data-driven Modeling 111

AWS Big Data

JUNE 20, 2024

AWS helps SaaS vendors by providing the building blocks needed to implement a streaming application with Amazon Kinesis Data Streams and Amazon Managed Streaming for Apache Kafka (Amazon MSK), and real-time processing applications with Amazon Managed Service for Apache Flink. In particular, we focus on Amazon MSK.

Modeling 115

Modeling Internet of Things Risk Management 115

AWS Big Data

AUGUST 31, 2023

Ingesting a high volume of streaming data has been a defining characteristic of operational analytics workloads with Amazon OpenSearch Service. Many of these workloads involve either self-managed Apache Kafka or Amazon Managed Streaming for Apache Kafka (Amazon MSK) to satisfy their data streaming needs.

Testing 119

Testing Data Processing Dashboards Management 119

O'Reilly on Data

FEBRUARY 11, 2019

The resource examples I’ll cite will be drawn from the upcoming Strata Data conference in San Francisco , where leading companies and speakers will share their learnings on the topics covered in this post. AI and machine learning in the enterprise. Security and Privacy.

Big Data 272

Big Data Technology Machine Learning Deep Learning 272

AWS Big Data

FEBRUARY 21, 2024

Towards the end of 2022, AWS announced the general availability of real-time streaming ingestion to Amazon Redshift for Amazon Kinesis Data Streams and Amazon Managed Streaming for Apache Kafka (Amazon MSK) , eliminating the need to stage streaming data in Amazon Simple Storage Service (Amazon S3) before ingesting it into Amazon Redshift.

Analytics 128

Analytics Data-driven Management Data Integration 128

datapine

SEPTEMBER 16, 2022

A mere Amazon search of this topic returns over 15k items. Though printed in 1983, it remains a classic and a bestseller on Amazon. Boasting near flawless reader reviews on Amazon, this graphically-driven book on data visualization makes an excellent companion when it comes to thriving in the digital age.

Visualization 304

Visualization Dashboards Data-driven Statistics 304

AWS Big Data

MARCH 21, 2024

AWS offers multiple serverless services like Amazon Managed Streaming for Apache Kafka (Amazon MSK), Amazon Data Firehose , Amazon DynamoDB , and AWS Lambda that scale automatically depending on your needs. Create a serverless Kafka cluster on Amazon MSK We use Amazon MSK to ingest real-time telemetry data from modems.

Data Lake 119

Data Lake Management Modeling Optimization 119

AWS Big Data

MAY 29, 2024

Organizations are adopting Apache Kafka and Amazon Managed Streaming for Apache Kafka (Amazon MSK) to capture and analyze data in real time. Since its inception, Apache Kafka has depended on Apache Zookeeper for storing and replicating the metadata of Kafka brokers and topics. Starting from Apache Kafka version 3.3,

Metadata 113

Metadata Cost-Benefit Management Big Data 113

AWS Big Data

SEPTEMBER 22, 2023

In this post, we will describe how and why we decided to migrate from self-managed Kafka to Amazon Managed Streaming for Apache Kafka ( Amazon MSK ). We’ll start with an overview of our self-managed Kafka, why we chose to migrate to Amazon MSK, and ultimately how we did it. in “newkafka.”

Management 119

Management Metrics Cost-Benefit Data Lake 119

O'Reilly on Data

JANUARY 16, 2024

I’m personally interested in this topic since I am a professor who researches human-computer interaction, user experience design, and cognitive science , so AI voice interfaces are fascinating to me. That in turn got me curious about the concept of syndication in the television business, so ChatGPT dived more into this topic.

Interactive 286

Interactive Recreation/Entertainment Visualization IT 286

AWS Big Data

SEPTEMBER 28, 2023

In EDAs, modern event brokers, such as Amazon EventBridge and Apache Kafka, play a key role to publish and subscribe to events. There are two ways to send events from Apache Kafka to EventBridge: the preferred method using Amazon EventBridge Pipes or the EventBridge sink connector for Kafka Connect.

Data-driven 140

Data-driven Metrics Publishing Management 140

AWS Big Data

MAY 3, 2023

In this post, we provide a detailed overview of streaming messages with Amazon Managed Streaming for Apache Kafka (Amazon MSK) and Amazon ElastiCache for Redis , covering technical aspects and design considerations that are essential for achieving optimal results. The following figure shows an example of the data flow at SOCAR.

IoT 111

IoT Internet of Things Data Transformation Management 111

AWS Big Data

NOVEMBER 13, 2023

The AWS Identity and Access Management (IAM) authentication feature in Amazon Managed Streaming for Apache Kafka (Amazon MSK) now supports all programming languages. Both Amazon MSK provisioned and serverless cluster types support the new Amazon MSK IAM expansion to all programming languages.

Testing 111

Testing Management Consulting IT 111

AWS Big Data

DECEMBER 13, 2023

This post showcases how to use streaming ingestion to bring data to Amazon Redshift. It’s simple to set up, and directly ingests streaming data into your data warehouse from Amazon Kinesis Data Streams and Amazon Managed Streaming for Kafka ( Amazon MSK ) without the need to stage in Amazon Simple Storage Service (Amazon S3).

Data Warehouse 116

Data Warehouse Snapshot Data Processing Internet of Things 116

AWS Big Data

SEPTEMBER 5, 2024

Amazon Redshift , a warehousing service, offers a variety of options for ingesting data from diverse sources into its high-performance, scalable environment. This native feature of Amazon Redshift uses massive parallel processing (MPP) to load objects directly from data sources into Redshift tables.

IoT 111

IoT Data Warehouse Cost-Benefit Reporting 111

AWS Big Data

NOVEMBER 9, 2023

dbt enables you to write SQL select statements, and then it manages turning these select statements into tables or views in Amazon Redshift. Queues and topics – Queues and topics come from various integration applications that generate data in real time.

Data Warehouse 126

Data Warehouse Testing Data Quality Reporting 126

DataKitchen

APRIL 11, 2024

The hosted by Christopher Bergh with Gil Benghiat from DataKitchen covered a comprehensive range of topics centered around improving the performance and efficiency of data teams through Agile and DataOps methodologies.

Statistics 154

Statistics Manufacturing Data Processing Data Quality 154

AWS Big Data

SEPTEMBER 13, 2023

To address these issues effectively, we propose using Amazon Managed Streaming for Apache Kafka (Amazon MSK), a fully managed Apache Kafka service that offers a seamless way to ingest and process streaming data. Following the data processing, the streaming job stores data in Amazon S3 and generates a Data Catalog table.

Data Processing 115

Data Processing Management Interactive Metadata 115

AWS Big Data

JULY 1, 2024

Enterprise customers increasingly adopt Amazon OpenSearch Ingestion (OSI) to bring data into Amazon OpenSearch Service for various use cases. These sources can either be on Amazon Elastic Compute Cloud (Amazon EC2) or on-premises environments. Name resolution for data sources – OSI uses an Amazon Route 53 resolver.

Management 116

Management Data Processing Publishing Analytics 116

AWS Big Data

MARCH 29, 2024

In this post, Nexthink shares how Amazon Managed Streaming for Apache Kafka (Amazon MSK) empowered them to achieve massive scale in event processing. With Amazon MSK, Nexthink now seamlessly processes trillions of events per day, reaching over 5 GB per second of aggregated throughput.

Data-driven 111

Data-driven Cost-Benefit Metrics Management 111

AWS Big Data

SEPTEMBER 6, 2023

Amazon’s serverless Apache Kafka offering, Amazon Managed Streaming for Apache Kafka (Amazon MSK) Serverless , is attracting a lot of interest. At the time of writing, the Amazon MSK library for IAM is exclusive to Kafka libraries in Java, creating a challenge for users of other programming languages.

Testing 116

Testing Metadata Cost-Benefit Internet of Things 116

AWS Big Data

JUNE 6, 2023

This post explains how the underlying infrastructure affects Kafka performance when you use Amazon Managed Streaming for Apache Kafka (Amazon MSK) tiered storage. We delve deep into the core components of Amazon MSK tiered storage and address questions such as: How does read and write work in a tiered storage-enabled cluster?

Metadata 104

Metadata Optimization Management Metrics 104

AWS Big Data

SEPTEMBER 19, 2023

Managing configurations for Amazon MSK Connect , a feature of Amazon Managed Streaming for Apache Kafka (Amazon MSK), can become challenging, especially as the number of topics and configurations grows. The challenges lie in the overhead of managing configurations, as well as dealing with patching and upgrades.

Data-driven 111

Data-driven Management Optimization Data Processing 111

AWS Big Data

AUGUST 7, 2024

Amazon Managed Streaming for Apache Kafka (Amazon MSK) is a fully managed Apache Kafka service. For more information about using IAM authentication, refer to Securely process near-real-time data from Amazon MSK Serverless using an AWS Glue streaming ETL job with IAM authentication. Create a Kafka connection in AWS Glue.

Metadata 105

Metadata Visualization Internet of Things Management 105

AWS Big Data

JUNE 12, 2024

In this series, we talk about Swisscom’s journey of automating Amazon Redshift provisioning as part of the Swisscom One Data Platform (ODP) solution using the AWS Cloud Development Kit (AWS CDK), and we provide code snippets and the other useful references. Given this, Swisscom needed a mechanism for AWS Glue to connect to Amazon Redshift.

Data-driven 115

Data-driven Snapshot Optimization Management 115