AWS Big Data

JANUARY 30, 2025

These include data discovery, modern ETL, cleansing, transforming, and centralized cataloging. We used it for executing long-running scripts, such as for ingesting data from an external API. We used it to define flows that would periodically load data from selected operational systems into our data warehouse.

Data Warehouse 52

Data Warehouse Data Architecture Machine Learning Data Transformation 52

AWS Big Data

JANUARY 6, 2025

With Amazon AppFlow, you can run data flows at nearly any scale and at the frequency you chooseon a schedule, in response to a business event, or on demand. You can configure data transformation capabilities such as filtering and validation to generate rich, ready-to-use data as part of the flow itself, without additional steps.

Analytics 97

Analytics Data Warehouse Metrics Big Data 97

erwin

SEPTEMBER 17, 2020

Replace manual and recurring tasks for fast, reliable data lineage and overall data governance. It’s paramount that organizations understand the benefits of automating end-to-end data lineage. The importance of end-to-end data lineage is widely understood and ignoring it is risky business. Doing Data Lineage Right.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

AWS Big Data

NOVEMBER 9, 2023

Building a data platform involves various approaches, each with its unique blend of complexities and solutions. In this post, we delve into a case study for a retail use case, exploring how the Data Build Tool (dbt) was used effectively within an AWS environment to build a high-performing, efficient, and modern data platform.

Data Warehouse 123

Data Warehouse Testing Data Quality Reporting 123

Cloudera

JUNE 30, 2022

These tools empower analysts and data scientists to easily collaborate on the same data, with their choice of tools and analytic engines. No more lock-in, unnecessary data transformations, or data movement across tools and clouds just to extract insights out of the data.

Data Lake 88

Data Lake Data Warehouse Data Architecture Metadata 88

AWS Big Data

NOVEMBER 15, 2023

It seamlessly consolidates data from various data sources within AWS, including AWS Cost Explorer (and forecasting with Cost Explorer ), AWS Trusted Advisor , and AWS Compute Optimizer. Overview of the BMW Cloud Data Hub At the BMW Group, Cloud Data Hub (CDH) is the central platform for managing company-wide data and data solutions.

Dashboards 110

Dashboards Analytics Metadata Data Warehouse 110

Cloudera

DECEMBER 21, 2021

Packaging Apache Airflow and exposing it as a managed service within CDE alleviates the typical operational management overhead of security and uptime while providing data engineers a job management API to schedule and monitor multi-step pipelines. This also enables sharing other directories with full audit trails. Modernizing pipelines.

Snapshot 118

Snapshot Data-driven Optimization Management 118

Sisense

AUGUST 28, 2020

We live in a world of data: There’s more of it than ever before, in a ceaselessly expanding array of forms and locations. Dealing with Data is your window into the ways data teams are tackling the challenges of this new world to help their companies and their customers thrive. What is data integrity?

Data Integration 101

Data Integration Testing Data Quality Data-driven 101

AWS Big Data

APRIL 19, 2023

This allows data consumers to easily identify new datasets and provides agility and innovation without spending hours doing analysis and research. Background The success of a data-driven organization recognizes data as a key enabler to increase and sustain innovation. It follows what is called a distributed system architecture.

Technology 99

Technology Data-driven Machine Learning Sales 99

AWS Big Data

AUGUST 1, 2024

However, you might face significant challenges when planning for a large-scale data warehouse migration. The following diagram illustrates a scalable migration pattern for extract, transform, and load (ETL) scenario. The success criteria are the key performance indicators (KPIs) for each component of the data workflow.

Data Warehouse 106

Data Warehouse KPI Optimization Cost-Benefit 106

Cloudera

NOVEMBER 29, 2021

Data transforms businesses. That’s where the data lifecycle comes into play. Managing data and its flow, from the edge to the cloud, is one of the most important tasks in the process of gaining data intelligence. . The company needed a modern data architecture to manage the growing traffic effectively. .

Data Lake 77

Data Lake Data Warehouse Data Architecture Reporting 77

AWS Big Data

SEPTEMBER 5, 2024

Amazon Redshift , a warehousing service, offers a variety of options for ingesting data from diverse sources into its high-performance, scalable environment. It uses massively parallel processing (MPP) architecture in Amazon Redshift to read and load large amounts of data in parallel from files or data from supported data sources.

IoT 110

IoT Data Warehouse Cost-Benefit Reporting 110

AWS Big Data

JULY 20, 2023

With data becoming the driving force behind many industries today, having a modern data architecture is pivotal for organizations to be successful. In this post, we describe Orca’s journey building a transactional data lake using Amazon Simple Storage Service (Amazon S3), Apache Iceberg, and AWS Analytics.

Data Lake 97

Data Lake Analytics Snapshot Data Quality 97

AWS Big Data

FEBRUARY 6, 2023

The YARN log analyzer provides four key functionalities: Upload transformed YARN job history logs in CSV format (for example, cluster_yarn_logs_*.csv He also understands how to apply technologies to solve big data problems and build a well-designed data architecture. George Zhao is a Senior Data Architect at AWS ProServe.

Dashboards 90

Dashboards Optimization Data Lake Cost-Benefit 90

AWS Big Data

NOVEMBER 13, 2023

Amazon Redshift is a fully managed data warehousing service that offers both provisioned and serverless options, making it more efficient to run and scale analytics without having to manage your data warehouse. Data sharing also provided the flexibility to independently scale the producer and consumer data warehouses.

Data Warehouse 109

Data Warehouse Analytics Data Lake Data Science 109

AWS Big Data

NOVEMBER 16, 2023

Amazon Redshift is a popular cloud data warehouse, offering a fully managed cloud-based service that seamlessly integrates with an organization’s Amazon Simple Storage Service (Amazon S3) data lake, real-time streams, machine learning (ML) workflows, transactional workflows, and much more—all while providing up to 7.9x Data Vault 2.0

Enterprise 104

Enterprise Data Warehouse Data Lake Optimization 104

Peter James Thomas

MARCH 9, 2020

A Data Maturity Model is simply one way of presenting the outcome of a Data Capability Review; it has the nice feature of also pointing the way to the future. Such a model presents a series of states into which an organisation may fall with respect to its data.

Measurement 67

Measurement Data Strategy Strategy Modeling 67

IBM Big Data Hub

FEBRUARY 9, 2023

In our last blog , we delved into the seven most prevalent data challenges that can be addressed with effective data governance. Today we will share our approach to developing a data governance program to drive data transformation and fuel a data-driven culture.

Data Governance 76

Data Governance Business Objectives Data Quality Measurement 76

Peter James Thomas

OCTOBER 30, 2018

Where they have, I have normally found the people holding these roles to be better informed about data matters than their peers. Prelude… I recently came across an article in Marketing Week with the clickbait-worthy headline of Why the rise of the chief data officer will be short-lived (their choice of capitalisation).

Marketing 51

Marketing Strategy Data Architecture Data Strategy 51

Anmut

MARCH 24, 2022

We could give many answers, but they all centre on the same root cause: most data leaders focus on flashy technology and symptomatic fixes instead of approaching data transformation in a way that addresses the root causes of data problems and leads to tangible results and business success. It doesn’t have to be this way.

ROI 52

ROI Measurement Data-driven Data Transformation 52

Ontotext

MARCH 27, 2024

Both nodes and edges have associated properties modeled as key-values with primitive data types and are single-valued. They also don’t have features for enterprise data management such as schema language, data validation capabilities, interoperable serialization formats, or a proper modeling language. This makes LPGs inflexible.

Modeling 53

Modeling Metadata Data Quality Enterprise 53

IBM Big Data Hub

JANUARY 18, 2023

A read-optimized platform that can integrate data from multiple applications emerged. In another decade, the internet and mobile started the generate data of unforeseen volume, variety and velocity. It required a different data platform solution. Value of the data projects are difficult to realize. It was Datawarehouse.

Data Lake 52

Data Lake Data Warehouse Data-driven Metadata 52

AWS Big Data

FEBRUARY 6, 2023

Use case overview Migrating Hadoop workloads to Amazon EMR accelerates big data analytics modernization, increases productivity, and reduces operational cost. Refactoring coupled compute and storage to a decoupling architecture is a modern data solution. George Zhao is a Senior Data Architect at AWS ProServe.

Cost-Benefit 52

Cost-Benefit Data Lake Dashboards Big Data 52

TDAN

DECEMBER 3, 2019

Everyone’s talking about data. Data is the key to unlocking insight— the secret sauce that will help you get predictive, the fuel for business intelligence. The transformative potential in AI? It relies on data. The good news is that data has never […].

Business Intelligence 72

Business Intelligence Dashboards Reporting Data Architecture 72

AWS Big Data

MARCH 3, 2023

You can then apply transformations and store data in Delta format for managing inserts, updates, and deletes. Amazon EMR Serverless is a serverless option in Amazon EMR that makes it easy for data analysts and engineers to run open-source big data analytics frameworks without configuring, managing, and scaling clusters or servers.

Data Lake 95

Data Lake Dashboards Metrics Metadata 95

Data Virtualization

APRIL 21, 2021

BHP is a global resources company headquartered in Melbourne, Australia. It is among the world’s top producers of major commodities, including iron ore, metallurgical coal, and copper, and has substantial interests in oil and gas. BHP has operations and offices.

IT 52

IT Data-driven Data Architecture Data Transformation 52

Data Virtualization

JANUARY 20, 2021

Learn in 12 minutes: What makes a strong use case for data virtualisation How to come up with a solid Proof of Concept How to prepare your organisation for data virtualisation You’ll have read all about data virtualisation and you’ve.

Data Warehouse 59

Data Warehouse Data Architecture Data Transformation Big Data 59

CIO Business Intelligence

AUGUST 7, 2024

This project represents a transformative initiative designed to address the evolving landscape of cyber threats,” says Kunal Krushev, head of cybersecurity automation and intelligence with the firm’s Corporate IT — Digital Infrastructure Services. “We The initiative brought multiple capabilities to the firm’s security operations.

IT 119

IT Insurance Cost-Benefit Testing 119

CIO Business Intelligence

AUGUST 3, 2023

When global technology company Lenovo started utilizing data analytics, they helped identify a new market niche for its gaming laptops, and powered remote diagnostics so their customers got the most from their servers and other devices.

Analytics 98

Analytics Data Lake Metadata Cost-Benefit 98

AWS Big Data

DECEMBER 21, 2023

The data volume is in double-digit TBs with steady growth as business and data sources evolve. smava’s Data Platform team faced the challenge to deliver data to stakeholders with different SLAs, while maintaining the flexibility to scale up and down while staying cost-efficient. Only the cluster’s storage incurs charges.

Data Lake 109

Data Lake Data Warehouse Data-driven B2B 109

erwin

APRIL 16, 2024

The data mesh framework In the dynamic landscape of data management, the search for agility, scalability, and efficiency has led organizations to explore new, innovative approaches. One such innovation gaining traction is the data mesh framework. This empowers individual teams to own and manage their data.

Metadata 52

Metadata Data Quality Data Governance Modeling 52

CIO Business Intelligence

NOVEMBER 7, 2023

For years, IT and business leaders have been talking about breaking down the data silos that exist within their organizations. Given the importance of sharing information among diverse disciplines in the era of digital transformation, this concept is arguably as important as ever.

Data Warehouse 142

Data Warehouse Digital Transformation Data-driven Reporting 142

Ontotext

FEBRUARY 12, 2024

For large enterprises, data mesh distributes data ownership and reduces dependencies between services. This promotes data autonomy and enables decision-making about data domains without centralized gatekeepers. What does data mesh do that other approaches can’t? What does data mesh do that other approaches can’t?

Data-driven 59

Data-driven Data Lake Data Quality Business Objectives 59

FineReport

MARCH 31, 2024

Furthermore, these tools boast customization options, allowing users to tailor data sources to address areas critical to their business success, thereby generating actionable insights and customizable reports. Best BI Tools for Data Analysts 3.1

Dashboards 52

Dashboards Visualization Data mining Data-driven 52

Birst BI

FEBRUARY 25, 2020

Customers such as Crossmark , DJO Global and others use Birst with Snowflake to deliver the ultimate modern data architecture. Data never leaves Snowflake with Birst’s ability to support the reporting and self-service needs of both centralized IT and decentralized LOB teams. Let’s focus on some customer use cases.

Data Warehouse 52

Data Warehouse Cost-Benefit Data Architecture Enterprise 52

IBM Big Data Hub

JULY 5, 2023

The challenges of a monolithic data lake architecture Data lakes are, at a high level, single repositories of data at scale. Data may be stored in its raw original form or optimized into a different format suitable for consumption by specialized engines.

Data Lake 60

Data Lake Metadata Cost-Benefit Data Warehouse 60

AWS Big Data

APRIL 5, 2023

In 2021, Showpad set forth the vision to use the power of data to unlock innovations and drive business decisions across its organization. In 2021, Showpad decided to take the next step in its data evolution and set forth the vision to power innovation, product decisions, and customer engagement using data-driven insights.

Dashboards 52

Dashboards Reporting Cost-Benefit Visualization 52