AWS Big Data

DECEMBER 13, 2023

In addition to using native managed AWS services that BMS didn’t need to worry about upgrading, BMS was looking to offer an ETL service to non-technical business users that could visually compose data transformation workflows and seamlessly run them on the AWS Glue Apache Spark-based serverless data integration engine.

Metadata 111

Metadata Data Lake Visualization Data Quality 111

AWS Big Data

OCTOBER 11, 2023

Organizations with legacy, on-premises, near-real-time analytics solutions typically rely on self-managed relational databases as their data store for analytics workloads. Traditionally, such a legacy call center analytics platform would be built on a relational database that stores data from streaming sources.

Management 131

Management Metadata Analytics Dashboards 131

CIO Business Intelligence

AUGUST 3, 2023

When global technology company Lenovo started utilizing data analytics, they helped identify a new market niche for its gaming laptops, and powered remote diagnostics so their customers got the most from their servers and other devices. After moving its expensive, on-premise data lake to the cloud, Comcast created a three-tiered architecture.

Analytics 98

Analytics Data Lake Metadata Cost-Benefit 98

Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

AWS Big Data

AUGUST 26, 2024

Solution overview The following diagram illustrates the solution architecture: The solution uses AWS Glue as an ETL engine to extract data from the source Amazon RDS database. Built-in data transformations then scrub columns containing PII using pre-defined masking functions. Run the crawlers. PII detection and scrubbing.

Visualization 116

Visualization Metadata Data Transformation Testing 116

erwin

MARCH 4, 2021

And when you talk about that question at a high level, he says, you get a very “simple answer,”– which is ‘the only thing we want to have is the right data with the right quality to the right person at the right time at the right cost.’. The Why: Data Governance Drivers. Why should companies care about data governance?

Data Governance 122

Data Governance Digital Transformation Data-driven Cost-Benefit 122

AWS Big Data

MARCH 15, 2023

Infomedia was looking to build a cloud-based data platform to take advantage of highly scalable data storage with flexible and cloud-native processing tools to ingest, transform, and deliver datasets to their SaaS applications. The Parquet format results in improved query performance and cost savings for downstream processing.

Cost-Benefit 94

Cost-Benefit Data Processing Optimization Data-driven 94

Cloudera

OCTOBER 11, 2021

The platform converges data cataloging, data ingestion, data profiling, data tagging, data discovery, and data exploration into a unified platform, driven by metadata. Modak Nabu automates repetitive tasks in the data preparation process and thus accelerates the data preparation by 4x.

Data Lake 95

Data Lake Cost-Benefit Data-driven Dashboards 95

AWS Big Data

DECEMBER 21, 2023

The data volume is in double-digit TBs with steady growth as business and data sources evolve. smava’s Data Platform team faced the challenge to deliver data to stakeholders with different SLAs, while maintaining the flexibility to scale up and down while staying cost-efficient.

Data Lake 115

Data Lake Data Warehouse Data-driven B2B 115

Cloudera

DECEMBER 9, 2022

Existing NiFi users can now bring their NiFi flows and run them in our cloud service by creating DataFlow Deployments that benefit from auto-scaling, one-button NiFi version upgrades, centralized monitoring through KPIs, multi-cloud support, and automation through a powerful command-line interface (CLI). Enabling self-service for developers.

Testing 100

Testing Cost-Benefit Interactive Visualization 100

IBM Big Data Hub

JULY 5, 2023

In the case of Hadoop, one of the more popular data lakes, the promise of implementing such a repository using open-source software and having it all run on commodity hardware meant you could store a lot of data on these systems at a very low cost. But it never co-existed amicably within existing data lake environments.

Data Lake 60

Data Lake Metadata Cost-Benefit Data Warehouse 60

Alation

APRIL 4, 2023

Now, joint users will get an enhanced view into cloud and data transformations , with valuable context to guide smarter usage. Integrating helpful metadata into user workflows gives all people, from data scientists to analysts , the context they need to use data more effectively.

Metadata 52

Metadata Cost-Benefit Data Transformation Predictive Modeling 52

AWS Big Data

NOVEMBER 16, 2023

Data Vault 2.0 allows for the following: Agile data warehouse development Parallel data ingestion A scalable approach to handle multiple data sources even on the same entity A high level of automation Historization Full lineage support However, Data Vault 2.0 Data Vault 2.0

Enterprise 111

Enterprise Data Warehouse Data Lake Optimization 111

IBM Big Data Hub

MARCH 14, 2024

This involves unifying and sharing a single copy of data and metadata across IBM® watsonx.data ™, IBM® Db2 ®, IBM® Db2® Warehouse and IBM® Netezza ®, using native integrations and supporting open formats, all without the need for migration or recataloging.

Cost-Benefit 75

Cost-Benefit Metadata Optimization Management 75

IBM Big Data Hub

OCTOBER 16, 2023

It includes processes that trace and document the origin of data, models and associated metadata and pipelines for audits. It also lets you choose the right engine for the right workload at the right cost, potentially reducing your data warehouse costs by optimizing workloads. Increase trust in AI outcomes.

Risk 70

Risk Modeling Management Metadata 70

AWS Big Data

AUGUST 1, 2023

AWS Glue is a serverless data discovery, load, and transformation service that will prepare data for consumption in BI and AI/ML activities. Solution overview This solution uses Amazon AppFlow to retrieve data from the Jira Cloud. This will enable both the CDC steps and the data transformation steps for the Jira data.

Data Lake 98

Data Lake Data Transformation Data-driven Cost-Benefit 98

Alation

MARCH 22, 2022

So, how can you quickly take advantage of the DataOps opportunity while avoiding the risk and costs of DIY? This platform can be implemented in a cost-effective serverless cloud environment and put to work right away. In essence, Alation is acting as a foundational data fabric that Gartner describes as being required for DataOps.

Metadata 52

Metadata Cost-Benefit Data Quality Data Lake 52

Alation

NOVEMBER 22, 2022

Public sector departments and agencies traditionally collect data so that they can support citizens and deliver services. In today’s analytics-driven society, the public sector can transform this historic information to reduce operational costs and improve public service to better address the needs of a given community.

Data Governance 52

Data Governance Metadata Data-driven Unstructured Data 52

AWS Big Data

OCTOBER 18, 2023

FINRA centralizes all its data in Amazon Simple Storage Service (Amazon S3) with a remote Hive metastore on Amazon Relational Database Service (Amazon RDS) to manage their metadata information. Melody Yang is a Senior Big Data Solutions Architect for Amazon EMR at AWS. or later installed.

Big Data 105

Big Data Data Processing Interactive Testing 105

IBM Big Data Hub

JULY 17, 2023

.” Sean Im, CEO, Samsung SDS America “In the field of generative AI and foundation models, watsonx is a platform that will enable us to meet our customers’ requirements in terms of optimization and security, while allowing them to benefit from the dynamism and innovations of the open-source community.”

Machine Learning 62

Machine Learning Data Warehouse Modeling Cost-Benefit 62

Alation

JANUARY 17, 2023

In actual fact, it isn’t all that confusing at all, and understanding what it means can have huge benefits for your organization. In this article, I will explain the modern data stack in detail, list some benefits, and discuss what the future holds. What Is the Modern Data Stack? Extract, load, Transform (ELT) tools.

Data Warehouse 52

Data Warehouse Cost-Benefit Data Science Data Transformation 52

AWS Big Data

JANUARY 17, 2024

Transaction data lake use case Amazon EMR customers often use Open Table Formats to support their ACID transaction and time travel needs in a data lake. Another popular transaction data lake use case is incremental query. Melody Yang is a Senior Big Data Solution Architect for Amazon EMR at AWS.

Data Lake 116

Data Lake Snapshot Big Data Data-driven 116

Ontotext

FEBRUARY 12, 2024

The data mesh concept will mitigate cognitive overload when building data-driven organizations that require intense technical, domain, and operational knowledge. For many organizations, a centralized data platform will fall short as it gives data teams much less autonomy over managing increasingly diverse and voluminous datasets.

Data-driven 59

Data-driven Data Lake Data Quality Business Objectives 59

AWS Big Data

APRIL 25, 2024

This can be attributed to factors such as inefficient data layout, resulting in excessive data scanning and inefficient use of compute resources. To address this challenge, common practices like partitioning and bucketing can significantly improve query performance and reduce computation costs.

Optimization 72

Optimization Data Lake Cost-Benefit Reporting 72

Jet Global

MAY 1, 2023

Now, Delta managers can get a full understanding of their data for compliance purposes. Additionally, with write-back capabilities, they can clear discrepancies and input data. These benefits provide a 360-degree feedback loop. In this new era, users expect to reap the benefits of analytics in every application that they touch.

Analytics 52

Analytics Cost-Benefit Visualization Dashboards 52

Jet Global

FEBRUARY 23, 2024

This field guide to data mapping will explore how data mapping connects volumes of data for enhanced decision-making. Why Data Mapping is Important Data mapping is a critical element of any data management initiative, such as data integration, data migration, data transformation, data warehousing, or automation.

Data Warehouse 52

Data Warehouse Reporting Data Transformation Visualization 52

AWS Big Data

APRIL 29, 2025

Businesses increasingly require scalable, cost-efficient architectures to process and transform massive datasets. At the BMW Group, our Cloud Efficiency Analytics (CLEA) team has developed a FinOps solution to optimize costs across over 10,000 cloud accounts. However, our initial data architecture led to challenges.

Data Transformation 54

Data Transformation Cost-Benefit Testing Data Lake 54

AWS Big Data

APRIL 28, 2025

The data is stored in Apache Parquet format with AWS Glue Catalog providing metadata management. While this architecture supported NI analytical needs, it lacked the flexibility required for a truly open and adaptable data platform.

Data Lake 69

Data Lake Metadata Cost-Benefit Snapshot 69