Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

IBM Big Data Hub

OCTOBER 24, 2017

If you’re holding an event for the very first time, what helps you gauge its success? At IBM Analytics University, we turned to social media analytics. Here’s a summary of what we learned from the experts and from Watson Analytics for Social Media.

Data Transformation 72

Data Transformation Analytics IT 72

AWS Big Data

MARCH 13, 2024

Using EventBridge integration, filtered positional updates are published to an EventBridge event bus. Amazon Location device position events arrive on the EventBridge default bus with source: ["aws.geo"] and detail-type: ["Location Device Position Event"]. In this model, the Lambda function is invoked for each incoming event.

Analytics 116

Analytics IoT Metadata Internet of Things 116

AWS Big Data

NOVEMBER 22, 2024

In this post, we’ll walk through an example ETL process that uses session reuse to efficiently create, populate, and query temporary staging tables across the full data transformation workflow—all within the same persistent Amazon Redshift database session. Building event-driven applications with Amazon EventBridge and Lambda.

Data Warehouse 100

Data Warehouse Recreation/Entertainment Cost-Benefit Data-driven 100

AWS Big Data

JANUARY 30, 2025

We used the AWS Step Function state machines to define, orchestrate, and execute our data pipelines. Amazon EventBridge We used Amazon EventBridge, the serverless event bus service, to define the event-based rules and schedules that would trigger our AWS Step Functions state machines.

Data Warehouse 52

Data Warehouse Data Architecture Machine Learning Data Transformation 52

Smart Data Collective

OCTOBER 13, 2021

There are countless examples of big data transforming many different industries. There is no disputing the fact that the collection and analysis of massive amounts of unstructured data has been a huge breakthrough. In forecasting future events. Prescriptive analytics.

Visualization 145

Visualization Cost-Benefit Big Data Prescriptive Analytics 145

AWS Big Data

JANUARY 15, 2025

With a unified catalog, enhanced analytics capabilities, and efficient data transformation processes, were laying the groundwork for future growth. Amazon DataZone empowers EUROGATE by setting the stage for long-term operational excellence and scalability.

IoT 107

IoT Machine Learning Metadata Data-driven 107

AWS Big Data

AUGUST 13, 2024

You will load the event data from the SFTP site, join it to the venue data stored on Amazon S3, apply transformations, and store the data in Amazon S3. The event and venue files are from the TICKIT dataset. For Node parents , select Rename Venue data and Rename Event data.

Data Processing 106

Data Processing Visualization Data Lake Data Processing 106

AWS Big Data

OCTOBER 11, 2023

Traditionally, such a legacy call center analytics platform would be built on a relational database that stores data from streaming sources. Data transformations through stored procedures and use of materialized views to curate datasets and generate insights is a known pattern with relational databases.

Management 124

Management Metadata Analytics Dashboards 124

AWS Big Data

APRIL 2, 2024

You can use Amazon Data Firehose to aggregate and deliver log events from your applications and services captured in Amazon CloudWatch Logs to your Amazon Simple Storage Service (Amazon S3) bucket and Splunk destinations, for use cases such as data analytics, security analysis, application troubleshooting etc.

Metadata 105

Metadata Marketing Analytics Data Transformation 105

DataKitchen

OCTOBER 25, 2022

It’s because it’s a hard thing to accomplish when there are so many teams, locales, data sources, pipelines, dependencies, data transformations, models, visualizations, tests, internal customers, and external customers. It’s not just a fear of change. If the first is late finishing, there are problems.

Testing 130

Testing Data-driven Visualization Dashboards 130

AWS Big Data

APRIL 11, 2023

Data ingestion and storage Retail businesses have event-driven data that requires action from downstream processes. It’s critical for an inventory management application to handle the data ingestion and storage for changing demands. The volume and velocity of data can change in the retail industry each day.

Forecasting 91

Forecasting Management IoT Data-driven 91

O'Reilly on Data

JANUARY 17, 2023

If your business relied on a hard-and-fast rule, or a predictable sequence of events, someone was bound to write code to do the work and throw that on a few dozen servers to scale it out. This was the early days of developers spotting those for() loops and if/then constructs in the wild. And it made sense. It does not exist in the code.

Machine Learning 363

Machine Learning Predictive Modeling Software Modeling 363

AWS Big Data

SEPTEMBER 5, 2024

If storing operational data in a data warehouse is a requirement, synchronization of tables between operational data stores and Amazon Redshift tables is supported. In scenarios where data transformation is required, you can use Redshift stored procedures to modify data in Redshift tables.

IoT 110

IoT Data Warehouse Cost-Benefit Reporting 110

datapine

SEPTEMBER 29, 2022

This means there are no unintended data errors, and it corresponds to its appropriate designation (e.g., Here, it all comes down to the data transformation error rate. Data time-to-value: evaluates how long it takes you to gain insights from a data set. date, month, and year).

Data Quality 241

Data Quality Metrics Data-driven Management 241

AWS Big Data

AUGUST 9, 2023

The advent of rapid adoption of serverless data lake architectures—with ever-growing datasets that need to be ingested from a variety of sources, followed by complex data transformation and machine learning (ML) pipelines—can present a challenge. These event changes are also routed to the same SNS topic.

Data Lake 97

Data Lake Metrics Testing Cost-Benefit 97

AWS Big Data

JUNE 6, 2023

Furthermore, it allows for necessary actions to be taken, such as rectifying errors in the data source, refining data transformation processes, and updating data quality rules. An EventBridge rule receives an event notification from the AWS Glue Data Quality evaluations including the results.

Data Quality 89

Data Quality Metrics Data-driven Visualization 89

Cloudera

JUNE 17, 2022

It is widely adopted by network device manufacturers to log event messages from routers, switches, firewalls, load balancers, and other networking equipment. Syslog typically follows an architecture of a syslog client that collects event data from the device and pushes it to a syslog server. .

Cost-Benefit 94

Cost-Benefit IoT Data Warehouse Manufacturing 94

erwin

APRIL 29, 2020

Without data lineage, these functions are irrelevant, so it makes sense for a business to have a clear understanding of where data comes from, who uses it, and how it transforms. Business terms and data policies should be implemented through standardized and documented business rules.

Key Performance Indicator 111

Key Performance Indicator Metadata Data Governance Data Quality 111

Sisense

OCTOBER 15, 2020

One of the main challenges when dealing with streaming data comes from performing stateful transformations for individual events. Unlike a batch processing job that runs within an isolated batch with clear start and end times, a stream processing job runs continuously on each event separately.

Dashboards 98

Dashboards IoT Optimization Internet of Things 98

AWS Big Data

MAY 16, 2023

Different communication infrastructure types such as mesh network and cellular can be used to send load information on a pre-defined schedule or event data in real time to the backend servers residing in the utility UDN (Utility Data Network).

Dashboards 92

Dashboards Statistics Data Collection Business Intelligence 92

AWS Big Data

MARCH 9, 2023

Additionally, there are major rewrites to deliver developer-focused improvements, including static type checking, enhanced runtime validation, strong consistency in call patterns, and optimized event chaining. is modernized by using promises for all actions, so developers can use async and await functions for better event management.

Slice and Dice 94

Slice and Dice Dashboards Analytics Interactive 94

AWS Big Data

APRIL 25, 2023

On many occasions, they need to apply business logic to the data received from the source SaaS platform before pushing it to the target SaaS platform. AnyCompany’s marketing team hosted an event at the Anaheim Convention Center, CA. The marketing team created leads based on the event in Adobe Marketo. Let’s take an example.

Sales 87

Sales Visualization Software Metadata 87

Smart Data Collective

APRIL 29, 2020

The techniques for managing organisational data in a standardised approach that minimises inefficiency. Extraction, Transform, Load (ETL). The extraction of raw data, transforming to a suitable format for business needs, and loading into a data warehouse. Data transformation.

Management 98

Management Data Warehouse Digital Transformation Dashboards 98

Cloudera

DECEMBER 9, 2022

Developers need to onboard new data sources, chain multiple data transformation steps together, and explore data as it travels through the flow. This allows developers to make changes to their processing logic on the fly while running some test data through their flow and validating that their changes work as intended.

Testing 100

Testing Cost-Benefit Interactive Visualization 100

AWS Big Data

JANUARY 6, 2025

Amazon AppFlow is a fully managed integration service that you can use to securely transfer data from software as a service (SaaS) applications, such as Google BigQuery, Salesforce, SAP, HubSpot, and ServiceNow, to Amazon Web Services (AWS) services such as Amazon Simple Storage Service (Amazon S3) and Amazon Redshift, in just a few clicks.

Analytics 97

Analytics Data Warehouse Metrics Big Data 97

AWS Big Data

JULY 6, 2023

It has not been specifically designed for heavy data transformation tasks. You also use AWS Glue to consolidate the files produced by the parallel tasks. Note that Lambda is a general purpose serverless engine.

Machine Learning 142

Machine Learning Visualization IoT Digital Transformation 142

erwin

SEPTEMBER 17, 2020

Business terms and data policies should be implemented through standardized and documented business rules. Compliance with these business rules can be tracked through data lineage, incorporating auditability and validation controls across data transformations and pipelines to generate alerts when there are non-compliant data instances.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

IBM Big Data Hub

NOVEMBER 20, 2023

Today’s healthcare providers use a wide variety of applications and data across a broad ecosystem of partners to manage their daily workflows. Integrating these applications and data is critical to their success, allowing them to deliver patient care efficiently and effectively. What is the FHIR Standard?

Cost-Benefit 87

Cost-Benefit Data-driven Data Transformation Management 87

CIO Business Intelligence

DECEMBER 22, 2022

DataOps in practice To make the most of DataOps, enterprises must evolve their data management strategies to deal with data at scale and in response to real-world events as they happen, according to Dunning and Friedman. They also note DataOps fits well with microservices architectures.

Analytics 130

Analytics Machine Learning Data mining Software 130

CIO Business Intelligence

JULY 5, 2022

What is the difference between business analytics and data analytics? Business analytics is a subset of data analytics. Data analytics is used across disciplines to find trends and solve problems using data mining , data cleansing, data transformation, data modeling, and more.

Business Analytics 136

Business Analytics Prescriptive Analytics Data mining Diagnostic Analytics 136

Cloudera

JUNE 1, 2021

These acquisitions usher in a new era of “ self-service ” by automating complex operations so customers can focus on building great data-driven apps instead of managing infrastructure. Datacoral powers fast and easy data transformations for any type of data via a robust multi-tenant SaaS architecture that runs in AWS.

Uncertainty 144

Uncertainty Cost-Benefit Risk Strategy 144

AWS Big Data

NOVEMBER 9, 2023

In this post, we delve into a case study for a retail use case, exploring how the Data Build Tool (dbt) was used effectively within an AWS environment to build a high-performing, efficient, and modern data platform. It does this by helping teams handle the T in ETL (extract, transform, and load) processes.

Data Warehouse 124

Data Warehouse Testing Data Quality Reporting 124

Cloudera

MARCH 14, 2023

Once a draft has been created or opened, developers use the visual Designer to build their data flow logic and validate it using interactive test sessions. In the Designer, you have the ability to start and stop each step of the data pipeline, resulting in events being queued up in the connections that link the processing steps together.

Testing 84

Testing Publishing Metadata Interactive 84

CIO Business Intelligence

JUNE 29, 2022

But the features in Power BI Premium are now more powerful than the functionality in Azure Analysis Services, so while the service isn’t going away, Microsoft will offer an automated migration tool in the second half of this year for customers who want to move their data models into Power BI instead. Azure Data Factory.

Data Lake 116

Data Lake Analytics Data Warehouse Machine Learning 116

AWS Big Data

OCTOBER 11, 2023

Due to this low complexity, the solution uses AWS serverless services to ingest the data, transform it, and make it available for analytics. The architecture uses AWS Lambda , a serverless, event-driven compute service that lets you run code without provisioning or managing servers.

Visualization 116

Visualization Dashboards Data-driven Gap analysis 116

Cloudera

JULY 21, 2022

Cloudera users can securely connect Rill to a source of event stream data, such as Cloudera DataFlow , model data into Rill’s cloud-based Druid service, and share live operational dashboards within minutes via Rill’s interactive metrics dashboard or any connected BI solution. Cloudera Data Warehouse). Apache Hive.

Metrics 87

Metrics Slice and Dice Data Warehouse Dashboards 87