AWS Big Data

JANUARY 15, 2025

Need for a data mesh architecture Because entities in the EUROGATE group generate vast amounts of data from various sourcesacross departments, locations, and technologiesthe traditional centralized data architecture struggles to keep up with the demands for real-time insights, agility, and scalability.

IoT 111

IoT Machine Learning Metadata Data-driven 111

AWS Big Data

FEBRUARY 26, 2025

The Gartner Magic Quadrant evaluates 20 data integration tool vendors based on two axesAbility to Execute and Completeness of Vision. Discover, prepare, and integrate all your data at any scale AWS Glue is a fully managed, serverless data integration service that simplifies data preparation and transformation across diverse data sources.

Data Integration 103

Data Integration Data Lake Data Warehouse Unstructured Data 103

AWS Big Data

SEPTEMBER 13, 2023

The Analytics specialty practice of AWS Professional Services (AWS ProServe) helps customers across the globe with modern data architecture implementations on the AWS Cloud. Of those tables, some are larger (such as in terms of record volume) than others, and some are updated more frequently than others.

Data Lake 111

Data Lake Data Processing Metadata Snapshot 111

CIO Business Intelligence

APRIL 8, 2025

We also examine how centralized, hybrid and decentralized data architectures support scalable, trustworthy ecosystems. As data-centric AI, automated metadata management and privacy-aware data sharing mature, the opportunity to embed data quality into the enterprises core has never been more significant.

Data Quality 101

Data Quality Data-driven Key Performance Indicator Metadata 101

AWS Big Data

JULY 25, 2024

Solution To address the challenge, ATPCO sought inspiration from a modern data mesh architecture. In Amazon DataZone, data owners can publish their data and its business catalog (metadata) to ATPCO’s DataZone domain. Data consumers can then search for relevant data assets using these human-friendly metadata terms.

Data Lake 116

Data Lake Metadata Sales Publishing 116

AWS Big Data

JANUARY 31, 2023

We have collected some of the key talks and solutions on data governance, data mesh, and modern data architecture published and presented in AWS re:Invent 2022, and a few data lake solutions built by customers and AWS Partners for easy reference. Starting with Amazon EMR release 6.7.0,

Data Lake 98

Data Lake Data Governance Data Architecture Machine Learning 98

AWS Big Data

FEBRUARY 26, 2025

Analytics remained one of the key focus areas this year, with significant updates and innovations aimed at helping businesses harness their data more efficiently and accelerate insights. From enhancing data lakes to empowering AI-driven analytics, AWS unveiled new tools and services that are set to shape the future of data and analytics.

Analytics 61

Analytics Data Lake Metadata Data Warehouse 61

AWS Big Data

OCTOBER 10, 2023

Data governance is the process of ensuring the integrity, availability, usability, and security of an organization’s data. Due to the volume, velocity, and variety of data being ingested in data lakes, it can get challenging to develop and maintain policies and procedures to ensure data governance at scale for your data lake.

Data Quality 111

Data Quality Data Governance Data Lake Testing 111

DataKitchen

OCTOBER 11, 2021

First, you must understand the existing challenges of the data team, including the data architecture and end-to-end toolchain. Figure 2: Example data pipeline with DataOps automation. In this project, I automated data extraction from SFTP, the public websites, and the email attachments.

Testing 154

Testing Metadata Dashboards Statistics 154

AWS Big Data

DECEMBER 19, 2024

Delta tables technical metadata is stored in the Data Catalog, which is a native source for creating assets in the Amazon DataZone business catalog. Access control is enforced using AWS Lake Formation , which manages fine-grained access control and data sharing on data lake data.

Data Governance 52

Data Governance Publishing Data-driven Metadata 52

IBM Big Data Hub

APRIL 27, 2022

Data fabric and data mesh are emerging data management concepts that are meant to address the organizational change and complexities of understanding, governing and working with enterprise data in a hybrid multicloud ecosystem. The good news is that both data architecture concepts are complimentary.

Management 79

Management Metadata Data Architecture Data Lake 79

AWS Big Data

AUGUST 2, 2023

The Economic Input-Output Life Cycle Assessment (EIO LCA) method is a spend-based method that combines expenditure data with monetary-based emission factors to estimate the emissions produced. The emission factors are published by the U.S. Environment Protection Agency (EPA) and other peer-reviewed academic and government sources.

Data Lake 97

Data Lake Measurement Visualization Data Architecture 97

AWS Big Data

MAY 30, 2024

Amazon Redshift is a fast, scalable, and fully managed cloud data warehouse that allows you to process and run your complex SQL analytics workloads on structured and semi-structured data. It also helps you securely access your data in operational databases, data lakes, or third-party datasets with minimal movement or copying of data.

Data Warehouse 115

Data Warehouse Data Lake Cost-Benefit Structured Data 115

Cloudera

OCTOBER 7, 2021

Those decentralization efforts appeared under different monikers through time, e.g., data marts versus data warehousing implementations (a popular architectural debate in the era of structured data) then enterprise-wide data lakes versus smaller, typically BU-Specific, “data ponds”.

Metadata 124

Metadata Cost-Benefit Enterprise Interactive 124

Cloudera

NOVEMBER 18, 2020

We also celebrated the first-ever winner of the Data Impact Achievement Award — a new award category that recognizes one customer who has consistently achieved transformation across their business, pursuing a diverse set of use cases and creating a culture of data-driven innovation. . Data Impact Achievement Award.

Internet Publishing and Broadcasting 123

Internet Publishing and Broadcasting Data-driven Broadcasting Digital Transformation 123

AWS Big Data

AUGUST 1, 2024

Success criteria alignment by all stakeholders (producers, consumers, operators, auditors) is key for successful transition to a new Amazon Redshift modern data architecture. The success criteria are the key performance indicators (KPIs) for each component of the data workflow.

Data Warehouse 115

Data Warehouse KPI Optimization Cost-Benefit 115

IBM Big Data Hub

JULY 5, 2022

In today’s world of complex data architectures and emerging technologies, databases can sometimes be undervalued and unrecognized. Back in the 1960s and 70s, vast amounts of data were stored in the world’s new mainframe computers—many of them IBM System/360 machines—and had become a problem. They were expensive.

Data Lake 86

Data Lake Data Warehouse Publishing Structured Data 86

AWS Big Data

SEPTEMBER 21, 2023

Integrating Satori with Amazon Redshift accelerates organizations’ ability to make use of their data to generate business value. This faster time-to-value is achieved by enabling companies to manage data access more efficiently and effectively. Lisa Levy is a Content Specialist at Satori.

Data Warehouse 115

Data Warehouse Interactive Data Architecture Data-driven 115

Cloudera

JANUARY 30, 2024

Add appropriate contextual data (IT/business data), which is critical in AI analysis of manufacturing data. Eliminate data silos. Data from multiple sources must be centralized and stored on a common data lake so that you will have one source of truth across the value chain.

Manufacturing 81

Manufacturing Contextual Data IoT Internet of Things 81

IBM Big Data Hub

JANUARY 18, 2023

In another decade, the internet and mobile started the generate data of unforeseen volume, variety and velocity. It required a different data platform solution. Hence, Data Lake emerged, which handles unstructured and structured data with huge volume. Data discoverability. Data mesh: A mostly new culture.

Data Lake 52

Data Lake Data Warehouse Data-driven Metadata 52

AWS Big Data

JULY 14, 2023

These inputs reinforced the need of a unified data strategy across the FinOps teams. We decided to build a scalable data management product that is based on the best practices of modern data architecture. Our source system and domain teams were mapped as data producers, and they would have ownership of the datasets.

Finance 98

Finance Metadata Big Data Recreation/Entertainment 98

Cloudera

NOVEMBER 27, 2023

Consider a few factors: First, many have been using Kafka as long-term storage and have seen their clusters grow without the same elasticity and accessibility one would expect from a modern data lake. For now, Flink plus Iceberg is the compute plus storage solution for streaming data.

Advertising 84

Advertising Data Lake Data Warehouse ROI 84

AWS Big Data

JUNE 7, 2023

Cargotec captures terabytes of IoT telemetry data from their machinery operated by numerous customers across the globe. This data needs to be ingested into a data lake, transformed, and made available for analytics, machine learning (ML), and visualization. The export process on the source account is a scheduled job.

Metadata 59

Metadata Data Lake Machine Learning Big Data 59

Peter James Thomas

APRIL 9, 2019

Overall the total number of articles and new pages I published exceeded 2017’s figures to claim the second spot behind 2009; our first year in business. This article offers a framework for building momentum in the early stages of a Data Programme. Analytics & Big Data. Draining the Swamp. Convergent Evolution.

Data-driven 49

Data-driven Statistics Data Science Big Data 49

Ontotext

NOVEMBER 16, 2023

Figure 1 Shows the overall idea of a data mesh with the major components: What Is a Data Mesh and How Does It Work? Think of data mesh as an operational mode for organizations with a domain-driven, decentralized data architecture.

Metadata 52

Metadata Data-driven Data Quality Data Architecture 52

Jet Global

OCTOBER 1, 2024

Trino allows users to run ad hoc queries across massive datasets, making real-time decision-making a reality without needing extensive data transformations. This is particularly valuable for teams that require instant answers from their data. Data Lake Analytics: Trino doesn’t just stop at databases.

Dashboards 52

Dashboards Data Lake Reporting Cost-Benefit 52

AWS Big Data

APRIL 28, 2025

When the user interacts with resources within SageMaker Unified Studio, it generates IAM session credentials based on the users effective profile in the specific project context, and then users can use tools such as Amazon Athena or Amazon Redshift to query the relevant data. SageMaker Unified Studio supports Lake Formation hybrid mode.

Metadata 61

Metadata Data Lake Big Data Management 61

AWS Big Data

APRIL 29, 2025

While enabling organization-wide efficiency, the team also applied these principles to the data architecture, making sure that CLEA itself operates frugally. After evaluating various tools, we built a serverless data transformation pipeline using Amazon Athena and dbt. However, our initial data architecture led to challenges.

Data Transformation 54

Data Transformation Cost-Benefit Testing Data Lake 54

AWS Big Data

NOVEMBER 14, 2024

The solution uses the following key services: Amazon API Gateway – API Gateway is a fully managed service that makes it straightforward developers to create, publish, maintain, monitor, and secure APIs at any scale. APIs act as the entry point for applications to access data, business logic, or functionality from your backend services.

Data Lake 105

Data Lake Metadata Testing Data-driven 105