Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

Birst BI

FEBRUARY 25, 2020

Managing large-scale data warehouse systems has been known to be very administrative, costly, and lead to analytic silos. The good news is that Snowflake, the cloud data platform, lowers costs and administrative overhead. The result is a lower total cost of ownership and trusted data and analytics.

Data Warehouse 52

Data Warehouse Cost-Benefit Data Architecture Enterprise 52

AWS Big Data

AUGUST 1, 2024

Large-scale data warehouse migration to the cloud is a complex and challenging endeavor that many organizations undertake to modernize their data infrastructure, enhance data management capabilities, and unlock new business opportunities. This makes sure the new data platform can meet current and future business goals.

Data Warehouse 100

Data Warehouse KPI Optimization Cost-Benefit 100

AWS Big Data

SEPTEMBER 5, 2024

Federated queries allow querying data across Amazon RDS for MySQL and PostgreSQL data sources without the need for extract, transform, and load (ETL) pipelines. If storing operational data in a data warehouse is a requirement, synchronization of tables between operational data stores and Amazon Redshift tables is supported.

IoT 107

IoT Data Warehouse Cost-Benefit Reporting 107

DataKitchen

JULY 27, 2023

Azure Synapse Analytics Pipelines: Azure Synapse Analytics (formerly SQL Data Warehouse) provides data exploration, data preparation, data management, and data warehousing capabilities. It provides data prep, management, and enterprise data warehousing tools. It does the job.

Machine Learning 130

Machine Learning Cost-Benefit Data Transformation Testing 130

AWS Big Data

DECEMBER 21, 2023

The data volume is in double-digit TBs with steady growth as business and data sources evolve. smava’s Data Platform team faced the challenge to deliver data to stakeholders with different SLAs, while maintaining the flexibility to scale up and down while staying cost-efficient.

Data Lake 104

Data Lake Data Warehouse Data-driven B2B 104

CIO Business Intelligence

AUGUST 3, 2023

When global technology company Lenovo started utilizing data analytics, they helped identify a new market niche for its gaming laptops, and powered remote diagnostics so their customers got the most from their servers and other devices. Each of the acquired companies had multiple data sets with different primary keys, says Hepworth. “We

Analytics 98

Analytics Data Lake Metadata Cost-Benefit 98

AWS Big Data

NOVEMBER 13, 2023

Amazon Redshift is a fully managed data warehousing service that offers both provisioned and serverless options, making it more efficient to run and scale analytics without having to manage your data warehouse. These upstream data sources constitute the data producer components.

Data Warehouse 104

Data Warehouse Analytics Data Lake Data Science 104

AWS Big Data

NOVEMBER 16, 2023

Amazon Redshift is a popular cloud data warehouse, offering a fully managed cloud-based service that seamlessly integrates with an organization’s Amazon Simple Storage Service (Amazon S3) data lake, real-time streams, machine learning (ML) workflows, transactional workflows, and much more—all while providing up to 7.9x

Enterprise 98

Enterprise Data Warehouse Data Lake Optimization 98

Cloudera

AUGUST 8, 2022

The general availability covers Iceberg running within some of the key data services in CDP, including Cloudera Data Warehouse ( CDW ), Cloudera Data Engineering ( CDE ), and Cloudera Machine Learning ( CML ). Cloudera Data Engineering (Spark 3) with Airflow enabled. Loading data into Iceberg tables with CDE.

Snapshot 94

Snapshot Data Warehouse Machine Learning Cost-Benefit 94

AWS Big Data

DECEMBER 4, 2024

Inspired by these global trends and driven by its own unique challenges, ANZ’s Institutional Division decided to pivot from viewing data as a byproduct of projects to treating it as a valuable product in its own right. For instance, one enhancement involves integrating cross-functional squads to support data literacy.

Metadata 94

Metadata Data Governance Data Quality Data-driven 94

AWS Big Data

JUNE 19, 2024

Amazon Redshift is a fast, scalable, secure, and fully managed cloud data warehouse that makes it simple and cost-effective to analyze all your data using standard SQL and your existing ETL (extract, transform, and load), business intelligence (BI), and reporting tools. All columns should masked for them.

Data Warehouse 102

Data Warehouse Testing Sales Structured Data 102

Alation

JANUARY 17, 2023

In actual fact, it isn’t all that confusing at all, and understanding what it means can have huge benefits for your organization. In this article, I will explain the modern data stack in detail, list some benefits, and discuss what the future holds. What Is the Modern Data Stack? Extract, load, Transform (ELT) tools.

Data Warehouse 52

Data Warehouse Cost-Benefit Data Science Data Transformation 52

AWS Big Data

MARCH 15, 2023

Infomedia was looking to build a cloud-based data platform to take advantage of highly scalable data storage with flexible and cloud-native processing tools to ingest, transform, and deliver datasets to their SaaS applications. The Parquet format results in improved query performance and cost savings for downstream processing.

Cost-Benefit 73

Cost-Benefit Data Processing Optimization Data-driven 73

Cloudera

JUNE 17, 2022

In the second blog of the Universal Data Distribution blog series , we explored how Cloudera DataFlow for the Public Cloud (CDF-PC) can help you implement use cases like data lakehouse and data warehouse ingest, cybersecurity, and log optimization, as well as IoT and streaming data collection.

Cost-Benefit 94

Cost-Benefit IoT Data Warehouse Manufacturing 94

IBM Big Data Hub

JULY 5, 2023

In the case of Hadoop, one of the more popular data lakes, the promise of implementing such a repository using open-source software and having it all run on commodity hardware meant you could store a lot of data on these systems at a very low cost. But it never co-existed amicably within existing data lake environments.

Data Lake 60

Data Lake Metadata Cost-Benefit Data Warehouse 60

Sisense

MAY 11, 2021

The new architecture requires that data be structured in a dimensional model to optimize for BI capabilities, but it also allows for ad hoc analytics with the flexibility to query clean and raw data. Here at Sisense, we think about this flow in five linear layers: Raw This is our data in its raw form within a data warehouse.

Modeling 84

Modeling Big Data IoT Data Warehouse 84

AWS Big Data

OCTOBER 27, 2023

AWS Glue , a serverless data integration and extract, transform, and load (ETL) service, has revolutionized this process, making it more accessible and efficient. AWS Glue eliminates complexities and costs, allowing organizations to perform data integration tasks in minutes, boosting efficiency.

Analytics 111

Analytics Visualization Data Integration Cost-Benefit 111

Insight

APRIL 8, 2020

Data architects and data modelers who specialize in areas such as schema design, identifying query access patterns and building and maintaining data warehouses. The problem requires use of one or two foundational data structures and details some sort of analysis that we’d like performed on a dataset.

Data Warehouse 96

Data Warehouse Cost-Benefit Software Optimization 96

IBM Big Data Hub

OCTOBER 16, 2023

This type of next-generation data store combines a data lake’s flexibility with a data warehouse’s performance and lets you scale AI workloads no matter where they reside. It helps you streamline data engineering with reduced data pipelines, simplified data transformation and enriched data.

Risk 70

Risk Modeling Management Metadata 70

IBM Big Data Hub

MARCH 14, 2024

This involves unifying and sharing a single copy of data and metadata across IBM® watsonx.data ™, IBM® Db2 ®, IBM® Db2® Warehouse and IBM® Netezza ®, using native integrations and supporting open formats, all without the need for migration or recataloging.

Cost-Benefit 75

Cost-Benefit Metadata Optimization Management 75

AWS Big Data

AUGUST 1, 2023

AWS Glue is a serverless data discovery, load, and transformation service that will prepare data for consumption in BI and AI/ML activities. Solution overview This solution uses Amazon AppFlow to retrieve data from the Jira Cloud. This will enable both the CDC steps and the data transformation steps for the Jira data.

Data Lake 92

Data Lake Data Transformation Data-driven Cost-Benefit 92

IBM Big Data Hub

JULY 17, 2023

.” Sean Im, CEO, Samsung SDS America “In the field of generative AI and foundation models, watsonx is a platform that will enable us to meet our customers’ requirements in terms of optimization and security, while allowing them to benefit from the dynamism and innovations of the open-source community.”

Machine Learning 62

Machine Learning Data Warehouse Modeling Cost-Benefit 62

Sisense

MARCH 14, 2021

Looking at the diagram, we see that Business Intelligence (BI) is a collection of analytical methods applied to big data to surface actionable intelligence by identifying patterns in voluminous data. As we move from right to left in the diagram, from big data to BI, we notice that unstructured data transforms into structured data.

Big Data 52

Big Data IoT Data Warehouse Data-driven 52

AWS Big Data

OCTOBER 18, 2023

Apache Hive is a distributed, fault-tolerant data warehouse system that enables analytics at a massive scale. Spark SQL is an Apache Spark module for structured data processing. Melody Yang is a Senior Big Data Solutions Architect for Amazon EMR at AWS. or later installed.

Big Data 92

Big Data Data Processing Interactive Testing 92

FineReport

APRIL 21, 2024

Industries benefit significantly from embedded analytics, driving operational efficiency and deeper customer understanding. When comparing options, consider scalability, total cost of ownership, and the BI tech stack. What Are the Main Benefits of Embedded BI Tools? Dashboard customization requires CSS and JavaScript skills.

Dashboards 52

Dashboards Visualization Interactive Business Intelligence 52

AWS Big Data

MARCH 3, 2023

Pay per use and linear cost scalability Tricentis’s objective is to pay for the compute and storage used and grow analytics infrastructure with linear cost scalability. To better manage storage costs in the data plane, Tricentis stores all raw and intermediate data in Amazon S3 storage in a compressed format.

Software 52

Software Data Lake Testing Cost-Benefit 52

Ontotext

FEBRUARY 12, 2024

The data mesh concept will mitigate cognitive overload when building data-driven organizations that require intense technical, domain, and operational knowledge. For many organizations, a centralized data platform will fall short as it gives data teams much less autonomy over managing increasingly diverse and voluminous datasets.

Data-driven 59

Data-driven Data Lake Data Quality Business Objectives 59

AWS Big Data

JANUARY 17, 2024

Transaction data lake use case Amazon EMR customers often use Open Table Formats to support their ACID transaction and time travel needs in a data lake. Another popular transaction data lake use case is incremental query. He is deeply passionate about applying ML/DL and big data techniques to solve real-world problems.

Data Lake 107

Data Lake Snapshot Big Data Data-driven 107

Jet Global

MAY 1, 2023

Now, Delta managers can get a full understanding of their data for compliance purposes. Additionally, with write-back capabilities, they can clear discrepancies and input data. These benefits provide a 360-degree feedback loop. In this new era, users expect to reap the benefits of analytics in every application that they touch.

Analytics 52

Analytics Cost-Benefit Visualization Dashboards 52

Jet Global

MAY 9, 2024

The key components of a data pipeline are typically: Data Sources : The origin of the data, such as a relational database , data warehouse, data lake , file, API, or other data store. This can include tasks such as data ingestion, cleansing, filtering, aggregation, or standardization.

Data Lake 52

Data Lake Data Warehouse Business Intelligence Machine Learning 52

Jet Global

FEBRUARY 23, 2024

This field guide to data mapping will explore how data mapping connects volumes of data for enhanced decision-making. Why Data Mapping is Important Data mapping is a critical element of any data management initiative, such as data integration, data migration, data transformation, data warehousing, or automation.

Data Warehouse 52

Data Warehouse Reporting Data Transformation Visualization 52

Jet Global

AUGUST 24, 2023

This optimization leads to improved efficiency, reduced operational costs, and better resource utilization. Mitigated Risk and Data Control: Finance teams can retain sensitive financial data on-premises while leveraging the cloud for less sensitive functions.

Finance 52

Finance Reporting Data Integration Data Warehouse 52

Jet Global

OCTOBER 1, 2024

Trino allows users to run ad hoc queries across massive datasets, making real-time decision-making a reality without needing extensive data transformations. This is particularly valuable for teams that require instant answers from their data. Data Lake Analytics: Trino doesn’t just stop at databases.

Dashboards 52

Dashboards Data Lake Reporting Cost-Benefit 52