CIO Business Intelligence

AUGUST 7, 2024

This project represents a transformative initiative designed to address the evolving landscape of cyber threats,” says Kunal Krushev, head of cybersecurity automation and intelligence with the firm’s Corporate IT — Digital Infrastructure Services. “We The initiative brought multiple capabilities to the firm’s security operations.

IT 119

IT Insurance Cost-Benefit Testing 119

Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

AWS Big Data

JANUARY 15, 2025

Their terminal operations rely heavily on seamless data flows and the management of vast volumes of data. Recently, EUROGATE has developed a digital twin for its container terminal Hamburg (CTH), generating millions of data points every second from Internet of Things (IoT)devices attached to its container handling equipment (CHE).

IoT 102

IoT Machine Learning Metadata Data-driven 102

Wayne Yaddow

FEBRUARY 21, 2025

Complex Data TransformationsTest Planning Best Practices Ensuring data accuracy with structured testing and best practices Photo by Taylor Vick on Unsplash Introduction Data transformations and conversions are crucial for data pipelines, enabling organizations to process, integrate, and refine raw data into meaningful insights.

Testing 52

Testing Data Transformation Data Quality Data Integration 52

AWS Big Data

JULY 26, 2023

Many AWS customers have integrated their data across multiple data sources using AWS Glue , a serverless data integration service, in order to make data-driven business decisions. Are there recommended approaches to provisioning components for data integration?

Data Integration 81

Data Integration Snapshot Testing Visualization 81

AWS Big Data

DECEMBER 21, 2023

Movement of data across data lakes, data warehouses, and purpose-built stores is achieved by extract, transform, and load (ETL) processes using data integration services such as AWS Glue. AWS Glue provides both visual and code-based interfaces to make data integration effortless.

Analytics 108

Analytics IT Data Lake Visualization 108

Wayne Yaddow

FEBRUARY 6, 2025

Common challenges and practical mitigation strategies for reliable data transformations. Photo by Mika Baumeister on Unsplash Introduction Data transformations are important processes in data engineering, enabling organizations to structure, enrich, and integrate data for analytics , reporting, and operational decision-making.

Testing 52

Testing Data Transformation Data-driven Manufacturing 52

Wayne Yaddow

FEBRUARY 11, 2025

Managing tests of complex data transformations when automated data testing tools lack important features? Photo by Marvin Meyer on Unsplash Introduction Data transformations are at the core of modern business intelligence, blending and converting disparate datasets into coherent, reliable outputs.

Testing 52

Testing Data Transformation Data Quality Statistics 52

Wayne Yaddow

MARCH 5, 2025

In this post, well see the fundamental procedures, tools, and techniques that data engineers, data scientists, and QA/testing teams use to ensure high-quality data as soon as its deployed. First, we look at how unit and integration tests uncover transformation errors at an early stage.

Testing 52

Testing Data Transformation Statistics Metadata 52

Wayne Yaddow

FEBRUARY 26, 2025

AI is transforming how senior data engineers and data scientists validate data transformations and conversions. Artificial intelligence-based verification approaches aid in the detection of anomalies, the enforcement of data integrity, and the optimization of pipelines for improved efficiency.

Data Transformation 52

Data Transformation Testing Data-driven Data Quality 52

DataKitchen

NOVEMBER 10, 2023

Implement a communication protocol that swiftly informs stakeholders, allowing them to brace for or address the potential impacts of the data change. Building a Culture of Accountability: Encourage a culture where data integrity is everyone’s responsibility.

Data Quality 136

Data Quality Testing Data Lake Data Integration 136

Ontotext

AUGUST 24, 2023

To deal with this issue, GraphDB implements a smart Graph Replace optimization that helps you calculate the internal data and only shows you the newly added and removed statements. The Soft Deletes and Versioning has the benefit of keeping track of the full history of your data, but your repository will become extremely big.

Data Integration 52

Data Integration Modeling Business Objectives Optimization 52

DataKitchen

DECEMBER 20, 2020

Data-driven companies sense change through data analytics. Companies turn to their data organization to provide the analytics that stimulates creative problem-solving. Companies turn to their data organization to provide the analytics that stimulates creative problem-solving. – Leon C. Adapt or face decline.

Data-driven 147

Data-driven Manufacturing Data Architecture Data Analytics 147

AWS Big Data

DECEMBER 16, 2024

Third, some services require you to set up and manage compute resources used for federated connectivity, and capabilities like connection testing and data preview arent available in all services. To solve for these challenges, we launched Amazon SageMaker Lakehouse unified data connectivity. Choose Add data. Choose Save changes.

Visualization 73

Visualization Data Processing Testing Publishing 73

CIO Business Intelligence

APRIL 8, 2025

These issues dont just hinder next-gen analytics and AI; they erode trust, delay transformation and diminish business value. Data quality is no longer a back-office concern. In this article, I am drawing from firsthand experience working with CIOs, CDOs, CTOs and transformation leaders across industries.

Data Quality 59

Data Quality Data-driven Key Performance Indicator Metadata 59

DataKitchen

JULY 27, 2023

We counted ten ‘standard’ ways to transform and set up batch data pipelines in Microsoft Azure. Let’s go through the ten Azure data pipeline tools Azure Data Factory : This cloud-based data integration service allows you to create data-driven workflows for orchestrating and automating data movement and transformation.

Machine Learning 130

Machine Learning Cost-Benefit Data Transformation Testing 130

AWS Big Data

MAY 9, 2023

Hundreds of thousands of customers use AWS Glue , a serverless data integration service, to discover, prepare, and combine data for analytics, machine learning (ML), and application development. AWS Glue for Apache Spark jobs work with your code and configuration of the number of data processing units (DPU). Generally, G.2X

Data Lake 103

Data Lake Cost-Benefit Data Integration Data Transformation 103

AWS Big Data

JANUARY 30, 2025

AWS Glue A data integration service, AWS Glue consolidates major data integration capabilities into a single service. These include data discovery, modern ETL, cleansing, transforming, and centralized cataloging. We used it for executing long-running scripts, such as for ingesting data from an external API.

Data Warehouse 52

Data Warehouse Data Architecture Machine Learning Data Transformation 52

Smart Data Collective

OCTOBER 13, 2021

There are countless examples of big data transforming many different industries. There is no disputing the fact that the collection and analysis of massive amounts of unstructured data has been a huge breakthrough. We would like to talk about data visualization and its role in the big data movement.

Visualization 145

Visualization Cost-Benefit Big Data Prescriptive Analytics 145

IBM Big Data Hub

JANUARY 9, 2024

The emergence of generative AI prompted several prominent companies to restrict its use because of the mishandling of sensitive internal data. Currently, no standardized process exists for overcoming data ingestion’s challenges, but the model’s accuracy depends on it. A popular method is extract, load, transform (ELT).

Enterprise 71

Enterprise Data Integration Data Quality Contextual Data 71

AWS Big Data

DECEMBER 13, 2023

In addition to using native managed AWS services that BMS didn’t need to worry about upgrading, BMS was looking to offer an ETL service to non-technical business users that could visually compose data transformation workflows and seamlessly run them on the AWS Glue Apache Spark-based serverless data integration engine.

Metadata 103

Metadata Data Lake Visualization Data Quality 103

AWS Big Data

OCTOBER 27, 2023

Data integration is the foundation of robust data analytics. It encompasses the discovery, preparation, and composition of data from diverse sources. In the modern data landscape, accessing, integrating, and transforming data from diverse sources is a vital process for data-driven decision-making.

Analytics 112

Analytics Visualization Data Integration Cost-Benefit 112

CIO Business Intelligence

NOVEMBER 7, 2023

For years, IT and business leaders have been talking about breaking down the data silos that exist within their organizations. Given the importance of sharing information among diverse disciplines in the era of digital transformation, this concept is arguably as important as ever.

Data Warehouse 142

Data Warehouse Digital Transformation Data-driven Reporting 142

AWS Big Data

JANUARY 6, 2025

With Amazon AppFlow, you can run data flows at nearly any scale and at the frequency you chooseon a schedule, in response to a business event, or on demand. You can configure data transformation capabilities such as filtering and validation to generate rich, ready-to-use data as part of the flow itself, without additional steps.

Analytics 95

Analytics Data Warehouse Big Data Metrics 95

CIO Business Intelligence

FEBRUARY 16, 2023

The company’s orthodontics business, for instance, makes heavy use of image processing to the point that unstructured data is growing at a pace of roughly 20% to 25% per month. For example, imaging data can be used to show patients how an aligner will change their appearance over time. “It

Unstructured Data 105

Unstructured Data Data Lake Prescriptive Analytics Data Warehouse 105

AWS Big Data

AUGUST 19, 2024

As organizations increasingly rely on data stored across various platforms, such as Snowflake , Amazon Simple Storage Service (Amazon S3), and various software as a service (SaaS) applications, the challenge of bringing these disparate data sources together has never been more pressing.

Analytics 109

Analytics Data-driven Data Integration Data Lake 109

CIO Business Intelligence

JUNE 7, 2022

What is data analytics? Data analytics is a discipline focused on extracting insights from data. It comprises the processes, tools and techniques of data analysis and management, including the collection, organization, and storage of data. What are the four types of data analytics? Data analytics tools.

Data Analytics 135

Data Analytics Diagnostic Analytics Management Analytics 135

CIO Business Intelligence

JANUARY 9, 2023

Customers are increasingly demanding access to real-time data, and freight transportation provider Estes Express Lines is among the rising tide of enterprises overhauling their data operations to deliver it. While the company had a data warehouse, it was primarily used for analysis.

Data Strategy 111

Data Strategy Strategy Data Governance Marketing 111

erwin

APRIL 29, 2020

Data lineage is the journey data takes from its creation through its transformations over time. Tracing the source of data is an arduous task. With all these diverse data sources, and if systems are integrated, it is difficult to understand the complicated data web they form much less get a simple visual flow.

Metadata 111

Metadata Key Performance Indicator Data Governance Data Quality 111

CIO Business Intelligence

JANUARY 24, 2024

Unfortunately, the road to data strategy success is fraught with challenges, so CIOs and other technology leaders need to plan and execute carefully. Here are some data strategy mistakes IT leaders would be wise to avoid. Overlooking these data resources is a big mistake. It will not be something they can ignore.

Data Strategy 136

Data Strategy Strategy Unstructured Data Data Governance 136

AWS Big Data

NOVEMBER 13, 2023

Amazon Redshift is a fully managed data warehousing service that offers both provisioned and serverless options, making it more efficient to run and scale analytics without having to manage your data warehouse. These query patterns and concurrency were unpredictable in nature.

Data Warehouse 105

Data Warehouse Analytics Data Lake Data Science 105

DataKitchen

DECEMBER 9, 2022

ChatGPT> DataOps, or data operations, is a set of practices and technologies that organizations use to improve the speed, quality, and reliability of their data analytics processes. One of the key benefits of DataOps automation is the ability to speed up the development and deployment of data-driven solutions.

Machine Learning 130

Machine Learning Data-driven Optimization Data Analytics 130

AWS Big Data

OCTOBER 5, 2023

In today’s data-driven world, the ability to effortlessly move and analyze data across diverse platforms is essential. Amazon AppFlow , a fully managed data integration service, has been at the forefront of streamlining data transfer between AWS services, software as a service (SaaS) applications, and now Google BigQuery.

Data Warehouse 101

Data Warehouse Machine Learning Data Integration Data-driven 101

Octopai

JUNE 9, 2024

.” The Data Strategy HealthCo, like many forward-thinking organizations, recognized early on that data is not just a valuable asset but a strategic imperative. They put data at the forefront of their business, integrating it into decision-making processes, products, and services.

IT 52

IT Data-driven Predictive Analytics Data Strategy 52

DataKitchen

OCTOBER 25, 2022

Observability is a methodology for providing visibility of every journey that data takes from source to customer value across every tool, environment, data store, team, and customer so that problems are detected and addressed immediately. Data journey observability is the first step in implementing DataOps.

Testing 130

Testing Data-driven Visualization Dashboards 130

Cloudera

JUNE 1, 2021

This means we can double down on our strategy – continuing to win the Hybrid Data Cloud battle in the IT department AND building new, easy-to-use cloud solutions for the line of business. It also means we can complete our business transformation with the systems, processes and people that support a new operating model. .

Uncertainty 144

Uncertainty Cost-Benefit Risk Strategy 144

AWS Big Data

AUGUST 13, 2024

AWS Glue is a serverless data integration service that helps analytics users to discover, prepare, move, and integrate data from multiple sources for analytics, machine learning (ML), and application development. The data will be in the target S3 bucket. The event and venue files are from the TICKIT dataset.

Data Processing 100

Data Processing Visualization Data Lake Data Processing 100