AWS Big Data

OCTOBER 9, 2024

Customers are migrating their on-premises data warehouse solutions built on databases like Netezza, PostgreSQL, Greenplum, and Teradata to AWS based modern data platforms using services like Amazon Simple Storage Service (Amazon S3) and Amazon Redshift. Use AWS Glue ETL to curate data from the S3 staging bucket to an S3 curated bucket.

Data Quality 107

Data Quality Data Lake Data Warehouse Metrics 107

AWS Big Data

AUGUST 22, 2023

Customers need a cloud-native automated solution to archive historical data from their databases. This post proposes an automated solution by using AWS Glue for automating the PostgreSQL data archiving and restoration process, thereby streamlining the entire procedure. The following diagram illustrates the solution architecture.

Data Processing 98

Data Processing Testing Data Lake Data Integration 98

AWS Big Data

SEPTEMBER 5, 2024

Amazon Redshift zero-ETL integrations can load data from Amazon Aurora MySQL-Compatible Edition , Amazon Relational Database Service (Amazon RDS) for MySQL , Amazon RDS for PostgreSQL , and DynamoDB, with the added ability to perform transformations after loading. Associate the IAM role with your Amazon Redshift cluster.

IoT 111

IoT Data Warehouse Cost-Benefit Reporting 111

Cloudera

DECEMBER 2, 2021

In addition to AKS and the load balancers mentioned above, this includes VNET, Data Lake Storage, PostgreSQL Azure database, and more. By default Azure Data Lake Storage, PostgreSQL Database, and Virtual Machines are accessible over public endpoints. Do enable the Create Private Endpoints option for the PostgreSQL Azure database.

Data Lake 123

Data Lake Data Warehouse Data Processing Interactive 123

AWS Big Data

MARCH 3, 2023

A solution to this problem is to use AWS Database Migration Service (AWS DMS) for migrating historical and real-time transactional data into the data lake. Solution overview The following diagram shows the overall architecture of the solution that we implement in this post.

Data Lake 97

Data Lake Dashboards Metrics Metadata 97

AWS Big Data

MARCH 13, 2024

In this post, we show how to build a RAG extract, transform, and load (ETL) ingestion pipeline to ingest large amounts of data into an Amazon OpenSearch Service cluster and use Amazon Relational Database Service (Amazon RDS) for PostgreSQL with the pgvector extension as a vector data store. Let’s look at the key components in more detail.

Data Processing 130

Data Processing Dashboards Machine Learning Metrics 130

AWS Big Data

APRIL 24, 2025

SageMaker Lakehouse is a unified, open, and secure data lakehouse that now supports ABAC to provide unified access to general purpose Amazon S3 buckets, Amazon S3 Tables , Amazon Redshift data warehouses, and data sources such as Amazon DynamoDB or PostgreSQL. Implementing this solution consists of the following high-level steps.

Sales 82

Sales Data Lake Management Data-driven 82

AWS Big Data

NOVEMBER 28, 2023

To run analytics on their operational data, customers often build solutions that are a combination of a database, a data warehouse, and an extract, transform, and load (ETL) pipeline. ETL is the process data engineers use to combine data from different sources. Figure 2: How to create a zero-ETL integration using Amazon RDS.

Data Warehouse 116

Data Warehouse Data-driven Machine Learning B2B 116

Data Science 101

FEBRUARY 29, 2020

Google Announces Cloud SQL for Microsoft SQL Server Google’s Cloud SQL now supports SQL Server in addition to PostgreSQL and MySQL Google Opens a new Cloud Region Located in Salt Lake City, Utah, it is named us-west3. Azure Sphere for IoT security goes GA This is a comprehensive security solution for IoT.

Data Science 106

Data Science IoT Deep Learning IT 106

CIO Business Intelligence

APRIL 8, 2022

Underpinning each winning project is a diverse suite of products, solutions, and platforms, sourced from an array of vendors large and small. This is the 2022 US CIO 100 Solutions Partners. Please join us in congratulating the Solutions Partners of the 2022 US CIO 100 award winners. The 2022 winners.

Consulting 105

Consulting Technology Enterprise IT 105

AWS Big Data

SEPTEMBER 11, 2024

HPE Aruba Networking is the industry leader in wired, wireless, and network security solutions. Hewlett-Packard acquired Aruba Networks in 2015, making it a wireless networking subsidiary with a wide range of next-generation network access solutions. The following diagram illustrates the solution architecture.

Data Architecture 106

Data Architecture Optimization Data Warehouse Metadata 106

AWS Big Data

AUGUST 27, 2024

Solution overview The solution is designed to help you track the cost of your Spark applications running on EMR on EC2. The proposed solution uses a scheduled AWS Lambda function that operates on a daily basis. The utilization of these AWS services incurs additional costs for implementing this solution.

Metrics 99

Metrics Dashboards Data Lake Optimization 99

AWS Big Data

FEBRUARY 12, 2024

This compiled data is then imported into Aurora PostgreSQL Serverless for operational reporting. Gupshup chose Aurora PostgreSQL as the operational reporting layer due to its anticipated increase in concurrency and cost-effectiveness for queries that retrieve only precalculated metrics.

Analytics 119

Analytics Data Warehouse Snapshot Cost-Benefit 119

AWS Big Data

FEBRUARY 14, 2023

Solution architecture Data lakes are usually organized using separate S3 buckets for three layers of data: the raw layer containing data in its original form, the stage layer containing intermediate processed data optimized for consumption, and the analytics layer containing aggregated data for specific use cases. Choose Create endpoint.

Data Lake 96

Data Lake Statistics Data Architecture Finance 96

AWS Big Data

MAY 7, 2024

This post explains how you can extend the governance capabilities of Amazon DataZone to data assets hosted in relational databases based on MySQL, PostgreSQL, Oracle or SQL Server engines. This solution was built using the AWS Cloud Development Kit (AWS CDK) and was designed to be easy to set up in any AWS environment.

Metadata 97

Metadata Data Lake Data Processing Data-driven 97

Sisense

APRIL 9, 2019

Open source data solutions like the ones we’ll discuss here allow them to do just that: take the software and make it theirs. In this article, we’ll dig into Hadoop, PostgreSQL, Apache Cassandra, and Elasticsearch. PostgreSQL. As the name implies, PostgreSQL is built on and extends the SQL language. Apache Cassandra.

Analytics 75

Analytics Big Data Software IoT 75

Cloudera

MARCH 15, 2023

In this post, I will demonstrate how to use the Cloudera Data Platform (CDP) and its streaming solutions to set up reliable data exchange in modern applications between high-scale microservices, and ensure that the internal state will stay consistent even under the highest load. Introduction Many modern application designs are event-driven.

Data-driven 91

Data-driven Snapshot Publishing Metadata 91

O'Reilly on Data

SEPTEMBER 11, 2018

Previous solutions. For instance, you might want to run an official PostgreSQL database image completely unchanged. The life cycle of data is very different than the life cycle of applications. Upgrading an application is a common occurrence, but data has to live across multiple such upgrades. Until Kubernetes 1.8,

Big Data 233

Big Data Data Processing Statistics Management 233

Insight

AUGUST 20, 2019

PostgreSQL/Redshift PostgreSQL is a popular open source relation database. While NoSQL databases emerged with the advent of big data, relational databases remain widely popular and are still the best solution for many use cases. An accessible introduction into how to setup your own PostgreSQL database can be found here.

Data Warehouse 71

Data Warehouse Big Data Data-driven Data Processing 71

AWS Big Data

FEBRUARY 8, 2024

Solution overview To implement this solution, we complete the following steps: Create an Amazon RDS for PostgreSQL instance. Create and run an AWS Glue job to extract data from the RDS for PostgreSQL DB instance using multiple job bookmark keys. Select the stack you created to deploy the solution and choose Delete.

Consulting 64

Consulting Testing Data Analytics Data Integration 64

AWS Big Data

MAY 30, 2024

Their Actian Vectorwise system was designed to replace Excel plugins and stock screeners but eventually evolved into a much larger and ambitious portfolio analysis solution running multiple API clusters on premises, serving some of the largest financial services firms worldwide. The downside here is over-provisioning.

Data Warehouse 110

Data Warehouse Data Lake Cost-Benefit Structured Data 110

AWS Big Data

JUNE 19, 2023

Solution overview In this scenario, we create a custom SQL direct query dataset to observe unoptimized SQL queries that are generated without dataset parameters, and demonstrate how your current custom SQL queries run if you don’t use dataset parameters. In this example, we use an Amazon RDS for PostgreSQL database.

Optimization 92

Optimization Slice and Dice Dashboards Recreation/Entertainment 92

AWS Big Data

MARCH 15, 2023

Infomedia Ltd (ASX:IFM) is a leading global provider of DaaS and SaaS solutions that empowers the data-driven automotive ecosystem. Infomedia’s solutions help OEMs, NSCs, dealerships and 3rd party partners manage the vehicle and customer lifecycle. This is a guest post co-written with Gowtham Dandu from Infomedia.

Cost-Benefit 83

Cost-Benefit Data Processing Optimization Data-driven 83

Cloudera

AUGUST 10, 2022

Cloudera has a strong track record of providing a comprehensive solution for stream processing. Cloudera Stream Processing (CSP), powered by Apache Flink and Apache Kafka, provides a complete stream management and stateful processing solution. Deploying a new JDBC Sink connector to write data from a Kafka topic to a PostgreSQL table.

Dashboards 91

Dashboards Data-driven Statistics Management 91

datapine

FEBRUARY 17, 2023

White labeling and embedding analytics tools and methodologies enhance interpretive capabilities, solve targeted challenges, address risks, and offer personalized real-time analytical solutions through smart modern real-time dashboard tools. BI solutions offer real-time access to multiple data sources in one centralized location.

Analytics 191

Analytics Cost-Benefit Dashboards Business Intelligence 191

Smart Data Collective

APRIL 19, 2022

These factors are why companies will spend over $12 billion on data analytics for marketing solutions by 2027. As a result, employees who work in sales acquire a lot of useful data from a sales report solution. Excel is one of the tools that can help companies use mined revenue data.

Sales 122

Sales Reporting Data mining Forecasting 122

CIO Business Intelligence

APRIL 21, 2022

The solution? This includes all the basic backend foundations such as PostgreSQL or major products such as Salesforce or Quickbooks. Some of its pre-built solutions are focused on major categories such as consumer e-commerce or travel. The world of marketing is awash in data. Embrace a customer data platform.

Sales 145

Sales Marketing Dashboards Advertising 145

Sisense

MARCH 19, 2019

Choosing the right solution to warehouse your data is just as important as how you collect data for business intelligence. In addition to its significant storage capacity, the data warehousing solution delivers several key benefits that make it an intriguing and possibly ideal choice for business intelligence.

Data Warehouse 94

Data Warehouse Cost-Benefit Business Intelligence Data Processing 94

AWS Big Data

MAY 10, 2023

AnyCompany is a professional services firm that specializes in providing workforce solutions to their customers. AnyCompany currently uses Amazon Redshift as their enterprise data warehouse platform and QuickSight as their BI solution. You can further enhance this solution to render a web-based form when the write-back URL is opened.

Dashboards 96

Dashboards Data Warehouse Visualization Data Processing 96

AWS Big Data

APRIL 30, 2024

You can troubleshoot your jobs by asking Amazon Q Developer to explain errors and propose solutions. He is passionate about distributed computing and using ML/AI for designing and building end-to-end solutions to address customers’ data integration needs. In his spare time, he enjoys spending time with family and friends.

Data Integration 125

Data Integration Data Lake Data Warehouse Software 125

AWS Big Data

NOVEMBER 29, 2023

Eightfold is transforming the world of work by providing solutions that empower organizations to recruit and retain a diverse global workforce. Solution overview Exposing a Redshift endpoint to all of Eightfold’s customers as part of the Talent Lake endeavor involved several design choices that had to be carefully considered.

Metadata 111

Metadata Data Warehouse Analytics Data Analytics 111

AWS Big Data

MARCH 7, 2023

This post provides guidance on how to build scalable analytical solutions for gaming industry use cases using Amazon Redshift Serverless. Solutions should be flexible to scale up and down. Portability to multiple channels – Solutions should be compatible with most of endpoint channels like PC, mobile, and gaming platforms.

Analytics 90

Analytics Data Warehouse Data Lake Metadata 90

AWS Big Data

MAY 1, 2024

To provide a variety of products, services, and solutions that are better suited to customers and society in each region, we have built business processes and systems that are optimized for each region and its market. Technical concepts In this section, we discuss some of the technical concepts of the solution.

Dashboards 124

Dashboards Publishing Data-driven Cost-Benefit 124

CIO Business Intelligence

AUGUST 18, 2022

Moonfare, a private equity firm, is transitioning from a PostgreSQL-based data warehouse on AWS to a Dremio data lakehouse on AWS for business intelligence and predictive analytics. Wiesenfeld has since launched a startup called Kawa to bring similar solutions to other customers, particularly hedge funds.

Data Lake 140

Data Lake Data Warehouse Unstructured Data Business Intelligence 140

AWS Big Data

NOVEMBER 29, 2023

This cloud service was a significant leap from the traditional data warehousing solutions, which were expensive, not elastic, and required significant expertise to tune and operate. At AWS re:Invent, we extended zero-ETL integration to additional sources specifically Aurora PostgreSQL, Dynamo DB, and Amazon RDS MySQL.

Data Warehouse 124

Data Warehouse Analytics Data Lake Machine Learning 124

Smart Data Collective

AUGUST 12, 2022

Your server’s configuration is going to be essential for your cloud solutions. PostgreSQL. Another relational system, the PostgreSQL is in full compliance with top management standards; it even has ACID support. Here are a few things to look for when choosing used servers to support your cloud computing strategy.

Strategy 81

Strategy Optimization Data Processing Management 81

Sisense

JULY 22, 2020

Traditionally all this data was stored on-premises, in servers, using databases that many of us will be familiar with, such as SAP, Microsoft Excel , Oracle , Microsoft SQL Server , IBM DB2 , PostgreSQL , MySQL , Teradata. However, cloud computing has grown rapidly because it offers more flexible, agile, and cost-effective storage solutions.

Slice and Dice 98

Slice and Dice Digital Transformation Data Warehouse Data Lake 98