AWS Big Data

FEBRUARY 15, 2024

You might be modernizing your data architecture using Amazon Redshift to enable access to your data lake and data in your data warehouse, and are looking for a centralized and scalable way to define and manage the data access based on IdP identities. For IAM role , choose a Lake Formation user-defined role.

Management 115

Management Data Lake Sales Data Warehouse 115

AWS Big Data

MARCH 12, 2024

In recent years, data lakes have become a mainstream architecture, and data quality validation is a critical factor to improve the reusability and consistency of the data. In this post, we provide benchmark results of running increasingly complex data quality rulesets over a predefined test dataset.

Data Quality 115

Data Quality Measurement Testing Visualization 115

AWS Big Data

OCTOBER 23, 2023

Modern applications store massive amounts of data on Amazon Simple Storage Service (Amazon S3) data lakes, providing cost-effective and highly durable storage, and allowing you to run analytics and machine learning (ML) from your data lake to generate insights on your data.

Data Lake 124

Data Lake Visualization Optimization Interactive 124

AWS Big Data

JANUARY 30, 2023

Many customers need an ACID transaction (atomic, consistent, isolated, durable) data lake that can log change data capture (CDC) from operational data sources. There is also demand for merging real-time data into batch data. Delta Lake framework provides these two capabilities. Choose Create policy.

Insurance 98

Insurance Data Lake Data-driven Analytics 98

AWS Big Data

APRIL 19, 2023

json ) to DynamoDB (for more information, refer to Write data to a table using the console or AWS CLI ): { "name": "step1.q", sample_data/us_current.csv s3://$s3_bucket_name/covid-19-testing-data/base/source_us_current/; Copy states_current.csv : aws s3 cp./sample_data/states_current.csv sample_oozie_job_name/step1/step1.json

Metadata 98

Metadata Data Lake Testing Consulting 98

AWS Big Data

JULY 25, 2024

AWS Data Pipeline helps customers automate the movement and transformation of data. With Data Pipeline, customers can define data-driven workflows, so that tasks can be dependent on the successful completion of previous tasks. You can visually create, run, and monitor ETL pipelines to load data into your data lakes.

Visualization 115

Visualization Management Data Integration Testing 115

AWS Big Data

JUNE 6, 2023

Founded in 2012, SumUp is the financial partner for more than 4 million small merchants in over 35 markets worldwide, helping them start, run and grow their business. Unless, of course, the rest of their data also resides in the Google Cloud. The Data Science teams also use this data for churn prediction and CLTV modeling.

Analytics 59

Analytics Data Lake Testing Optimization 59

AWS Big Data

AUGUST 27, 2024

His background is in data warehouse/data lake – architecture, development and administration. He is in data and analytical field for over 14 years. Ramesh Raghupathy is a Senior Data Architect with WWCO ProServe at AWS. While not at work, Ramesh enjoys traveling, spending time with family, and yoga.

Metrics 105

Metrics Dashboards Data Lake Optimization 105

AWS Big Data

APRIL 20, 2023

For sales across multiple markets, the product sales data such as orders, transactions, and shipment data is available on Amazon S3 in the data lake. The data engineering team can use Apache Spark with Amazon EMR or AWS Glue to analyze this data in Amazon S3. enableHiveSupport().getOrCreate()

Data Lake 98

Data Lake Data Warehouse Sales Data-driven 98

AWS Big Data

DECEMBER 13, 2023

Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. We deploy Debezium MySQL source Kafka connector on Amazon MSK Connect.

Data Warehouse 116

Data Warehouse Snapshot Data Processing Internet of Things 116

AWS Big Data

JUNE 13, 2023

Test the application Let’s invoke the application you have created to seamlessly sign in to QuickSight using the following URL. Vamsi Bhadriraju is a Data Architect at AWS. He works closely with enterprise customers to build data lakes and analytical applications on the AWS Cloud.

Metadata 52

Metadata Dashboards Business Intelligence Data Lake 52

BizAcuity

MAY 10, 2022

2012: Amazon Redshift, the first of its kind cloud-based data warehouse service comes into existence. Google launches BigQuery, its own data warehousing tool and Microsoft introduces Azure SQL Data Warehouse and Azure Data Lake Store. Data lakes or data lake houses alone cannot solve the efficiency problem.

Data-driven 52

Data-driven IoT Unstructured Data Data Lake 52

Domino Data Lab

APRIL 3, 2019

Somehow, the gravity of the data has a geological effect that forms data lakes. Also, data science workflows begin to create feedback loops from the big data side of the illo above over to the DW side. DG emerges for the big data side of the world, e.g., the Alation launch in 2012.

Machine Learning 84

Machine Learning Data Governance Metadata Data Science 84

Domino Data Lab

AUGUST 8, 2019

Once upon a time, circa 2012-ish, data science conferences were replete with talks about an industry hellbent on loading amazing enormous Big Data into some kind of data lake, and applying all kinds of odd astrophysics-ish approaches…for eventual PROFIT! Or something. Nothing Spreads Like Fear”. No big deal.”.

Data Science 71

Data Science Machine Learning Data Governance Statistics 71

Cloudera

DECEMBER 12, 2018

And so I actually transitioned out of that group and into the Big Data Appliance group at Oracle, but soon realized that if that was what I wanted to keep doing, this up and coming company called Cloudera might be a better place to do it since these new technologies weren’t just a hobby at Cloudera. As you mentioned, Qlik is in there.

Data Warehouse 42

Data Warehouse Marketing Big Data Data-driven 42

AWS Big Data

DECEMBER 18, 2023

Solution overview One of the common functionalities involved in data pipelines is extracting data from multiple data sources and exporting it to a data lake or synchronizing the data to another database. Choose the workflow named ETL_Process. Run the workflow with default input.

Metadata 143

Metadata Visualization Data-driven Data Lake 143

AWS Big Data

APRIL 21, 2025

Without the DATA LOCATION permission, write workloads will fail. Test the access to the table by writing new records to the table as the IAM role. Add SELECT table permissions to the Data-Analyst role in Lake Formation. Test access to the table as the Data-Analyst by running SELECT queries in Athena.

Data Lake 81

Data Lake Metadata Interactive Big Data 81

CIO Business Intelligence

JANUARY 16, 2025

The mega-vendor era By 2020, the basis of competition for what are now referred to as mega-vendors was interoperability, automation and intra-ecosystem participation and unlocking access to data to drive business capabilities, value and manage risk. edge compute data distribution that connect broad, deep PLM eco-systems.

Enterprise 105

Enterprise Digital Transformation Scorecard Interactive 105

AWS Big Data

MAY 7, 2025

When extracting data filter rules for the table in another account, the execution role must have necessary access to the tables in the other account. Use case overview For this post, lets consider a large financial institution that has implemented Lake Formation as its central data lake and entitlement management system.

Dashboards 105

Dashboards Publishing Data-driven Business Intelligence 105