AWS Big Data

DECEMBER 4, 2024

With the growing emphasis on data, organizations are constantly seeking more efficient and agile ways to integrate their data, especially from a wide variety of applications. In addition, organizations rely on an increasingly diverse array of digital systems, data fragmentation has become a significant challenge.

Data Integration 112

Data Integration Data Lake Statistics Data-driven 112

AWS Big Data

APRIL 3, 2024

Businesses are constantly evolving, and data leaders are challenged every day to meet new requirements. Customers are using AWS and Snowflake to develop purpose-built data architectures that provide the performance required for modern analytics and artificial intelligence (AI) use cases.

Data Lake 121

Data Lake Snapshot Metadata Data Architecture 121

AWS Big Data

DECEMBER 9, 2024

In todays data-driven world, tracking and analyzing changes over time has become essential. As organizations process vast amounts of data, maintaining an accurate historical record is crucial. History management in data systems is fundamental for compliance, business intelligence, data quality, and time-based analysis.

Snapshot 64

Snapshot Data Warehouse Data Lake Data Quality 64

AWS Big Data

OCTOBER 11, 2023

Organizations with legacy, on-premises, near-real-time analytics solutions typically rely on self-managed relational databases as their data store for analytics workloads. Near-real-time streaming analytics captures the value of operational data and metrics to provide new insights to create business opportunities.

Management 129

Management Metadata Analytics Dashboards 129

AWS Big Data

JUNE 5, 2023

If you also needed to preserve the history of DAG runs, you had to take a backup of your metadata database and then restore that backup on the newly created environment. Amazon MWAA manages the entire upgrade process, from provisioning new Apache Airflow versions to upgrading the metadata database.

Snapshot 96

Snapshot Metadata Testing Data-driven 96

AWS Big Data

JANUARY 8, 2024

We’re living in the age of real-time data and insights, driven by low-latency data streaming applications. The volume of time-sensitive data produced is increasing rapidly, with different formats of data being introduced across new businesses and customer use cases.

Analytics 133

Analytics IoT Data-driven Snapshot 133

Sisense

JANUARY 6, 2020

Best practice blends the application of advanced data models with the experience, intuition and knowledge of sales management, to deeply understand the sales pipeline. For this partnership to work, it requires sales leaders who really care about data and are open to analysts’ advice about how to use the Salesforce data they generate.

Sales 91

Sales Forecasting Snapshot Management 91

AWS Big Data

MAY 10, 2023

In OpenSearch Service, you can deploy data nodes to store your data and respond to indexing and search requests, you can also deploy dedicated cluster manager nodes to manage and orchestrate the cluster. When you use OpenSearch Service, you create indexes to hold your data and specify partitioning and replication for those indexes.

Snapshot 118

Snapshot Testing Metadata Management 118

AWS Big Data

JULY 20, 2023

With data becoming the driving force behind many industries today, having a modern data architecture is pivotal for organizations to be successful. In this post, we describe Orca’s journey building a transactional data lake using Amazon Simple Storage Service (Amazon S3), Apache Iceberg, and AWS Analytics.

Data Lake 98

Data Lake Analytics Snapshot Data Quality 98

AWS Big Data

AUGUST 6, 2024

INSITE applications are in general data intensive. They ingest and transform large volumes of data in different formats and processing patterns (such as batch and near real time) from various sources internal and external to Amazon. To enable and meet these requirements, GTTS built its own data platform.

Cost-Benefit 111

Cost-Benefit Snapshot Metadata Metrics 111

AWS Big Data

FEBRUARY 13, 2023

This is a guest post by Miguel Chin, Data Engineering Manager at OLX Group and David Greenshtein, Specialist Solutions Architect for Analytics, AWS. We live in a data-producing world, and as companies want to become data driven, there is the need to analyze more and more data.

Snapshot 89

Snapshot Data Warehouse Analytics Testing 89

AWS Big Data

SEPTEMBER 13, 2023

The Analytics specialty practice of AWS Professional Services (AWS ProServe) helps customers across the globe with modern data architecture implementations on the AWS Cloud. In this post, we discuss a common use case in relation to operational data processing and the solution we built using Apache Hudi and AWS Glue.

Data Lake 111

Data Lake Data Processing Metadata Snapshot 111

Cloudera

NOVEMBER 15, 2024

In fact, according to the Identity Theft Resource Center (ITRC) Annual Data Breach Report , there were 2,365 cyber attacks in 2023 with more than 300 million victims, and a 72% increase in data breaches since 2021. However, there is a fundamental challenge standing in the way of being successful: data.

Analytics 59

Analytics Metadata Snapshot Data-driven 59

AWS Big Data

FEBRUARY 1, 2023

This data is used in procuring devices’ inventory to meet Amazon customers’ demands. With data volumes exhibiting a double-digit percentage growth rate year on year and the COVID pandemic disrupting global logistics in 2021, it became more critical to scale and generate near-real-time data.

Optimization 52

Optimization Forecasting Data Lake Metadata 52

Cloudera

MARCH 15, 2023

In this post, I will demonstrate how to use the Cloudera Data Platform (CDP) and its streaming solutions to set up reliable data exchange in modern applications between high-scale microservices, and ensure that the internal state will stay consistent even under the highest load.

Data-driven 91

Data-driven Snapshot Publishing Metadata 91

AWS Big Data

JANUARY 17, 2024

This allows you to simplify security and governance over transactional data lakes by providing access controls at table-, column-, and row-level permissions with your Apache Spark jobs. Many large enterprise companies seek to use their transactional data lake to gain insights and improve decision-making.

Data Lake 115

Data Lake Snapshot Big Data Data-driven 115

Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

AWS Big Data

MARCH 25, 2024

FMs are multimodal; they work with different data types such as text, video, audio, and images. Large language models (LLMs) are a type of FM and are pre-trained on vast amounts of text data and typically have application uses such as text generation, intelligent chatbots, or summarization.

Data Lake 118

Data Lake Unstructured Data Management Snapshot 118

Alation

JULY 28, 2022

There was a time when most CIOs would never consider putting their crown jewels — AKA customer data and associated analytics — into the cloud. And what must organizations overcome to succeed at cloud data warehousing ? What Are the Biggest Drivers of Cloud Data Warehousing? The cloud is no longer synonymous with risk.

Data Warehouse 69

Data Warehouse Cost-Benefit Data-driven Data Governance 69

AWS Big Data

JANUARY 27, 2023

To build a data-driven business, it is important to democratize enterprise data assets in a data catalog. With a unified data catalog, you can quickly search datasets and figure out data schema, data format, and location. For metadata read/write, Flink has the catalog interface.

Data Lake 81

Data Lake Metadata Business Analysis Data-driven 81

AWS Big Data

DECEMBER 20, 2024

In this post, we share part of the journey that Jumia took with AWS Professional Services to modernize its data platform that ran under a Hadoop distribution to AWS serverless based solutions. These phases are: data orchestration, data migration, data ingestion, data processing, and data maintenance.

Metadata 119

Metadata Data-driven Snapshot Data Lake 119

Alation

AUGUST 26, 2021

What Is Data Intelligence? Data intelligence is a system to deliver trustworthy, reliable data. It includes intelligence about data, or metadata. IDC coined the term, stating, “data intelligence helps organizations answer six fundamental questions about data.” Why do we have data?

Metadata 64

Metadata Data Governance Dashboards Software 64

Cloudera

NOVEMBER 15, 2024

In fact, according to the Identity Theft Resource Center (ITRC) Annual Data Breach Report , there were 2,365 cyber attacks in 2023 with more than 300 million victims, and a 72% increase in data breaches since 2021. However, there is a fundamental challenge standing in the way of being successful: data.

Analytics 84

Analytics Metadata Snapshot Data-driven 84

AWS Big Data

FEBRUARY 18, 2025

To optimize their security operations, organizations are adopting modern approaches that combine real-time monitoring with scalable data analytics. They are using data lake architectures and Apache Iceberg to efficiently process large volumes of security data while minimizing operational overhead.

Snapshot 78

Snapshot Optimization Data Lake Metadata 78

AWS Big Data

DECEMBER 19, 2024

Data lakes were originally designed to store large volumes of raw, unstructured, or semi-structured data at a low cost, primarily serving big data and analytics use cases. By using features like Icebergs compaction, OTFs streamline maintenance, making it straightforward to manage object and metadata versioning at scale.

Data Lake 98

Data Lake IoT Metadata Testing 98

DataKitchen

JANUARY 23, 2023

What is data lineage? Data lineage traces data’s origin, history, and movement through various processing, storage, and analysis stages. It is used to understand the provenance of data and how it is transformed and to identify potential errors or issues. What about DataOps Observability? How does it compare?

Testing 130

Testing Data Governance Data Quality Data-driven 130