CIO Business Intelligence

AUGUST 9, 2024

At Atlanta’s Hartsfield-Jackson International Airport, an IT pilot has led to a wholesale data journey destined to transform operations at the world’s busiest airport, fueled by machine learning and generative AI. Applying AI to elevate ROI Pruitt and Databricks recently finished a pilot test with Microsoft called Smart Flow.

Data Transformation 126

Data Transformation Machine Learning Data Lake Dashboards 126

AWS Big Data

NOVEMBER 22, 2024

The need for streamlined data transformations As organizations increasingly adopt cloud-based data lakes and warehouses, the demand for efficient data transformation tools has grown. This approach helps in managing storage costs while maintaining the flexibility to analyze historical trends when needed.

Data Lake 100

Data Lake Data Warehouse Cost-Benefit Data Transformation 100

Smart Data Collective

JULY 25, 2019

When it comes to the use of modern big data technologies by hospitals, it is about health care and saving lives. These systems rely heavily on big data to improve efficiency and cost-effectiveness. It is simple and convenient to use outsourcing IT services when you need to get a perfect big data solution. Conclusion.

Big Data 110

Big Data Data Transformation Management Software 110

CIO Business Intelligence

JUNE 13, 2022

Ever increasing demands for transformation. Growing cybersecurity, data privacy threats. According to Evanta’s 2022 CIO Leadership Perspectives study, CIOs’ second top priority within the IT function is around data and analytics, with CIOs seeing advancing organizational use of data as key to reaching enterprise objectives.

IT 144

IT Digital Transformation Internet of Things Strategy 144

Analytics Vidhya

DECEMBER 11, 2023

Introduction Power Query is a powerful data transformation and manipulation tool in PowerBI that allows users to extract, transform, and load data from various sources. It provides a user-friendly interface for performing complex data transformations without the need for coding.

Data Transformation 178

Data Transformation Analytics IT 178

Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

Wayne Yaddow

MARCH 12, 2025

Data quality rules are codified into structured Expectation Suites by Great Expectations instead of relying on ad-hoc scripts or manual checks. The framework ensures that your data transformations comply with rigorous specifications from the moment they are created through every iteration of your pipeline.

Data Transformation 52

Data Transformation Data Quality Testing Data Warehouse 52

Wayne Yaddow

FEBRUARY 4, 2025

Its EssentialVerifying Data Transformations (Part4) Uncovering the leading problems in data transformation workflowsand practical ways to detect and preventthem In Parts 13 of this series of blogs, categories of data transformations were identified as among the top causes of data quality defects in data pipeline workflows.

Data Transformation 52

Data Transformation Testing Data Quality Strategy 52

Wayne Yaddow

FEBRUARY 21, 2025

Complex Data TransformationsTest Planning Best Practices Ensuring data accuracy with structured testing and best practices Photo by Taylor Vick on Unsplash Introduction Data transformations and conversions are crucial for data pipelines, enabling organizations to process, integrate, and refine raw data into meaningful insights.

Testing 52

Testing Data Transformation Data Quality Data Integration 52

Wayne Yaddow

MARCH 5, 2025

In this post, well see the fundamental procedures, tools, and techniques that data engineers, data scientists, and QA/testing teams use to ensure high-quality data as soon as its deployed. First, we look at how unit and integration tests uncover transformation errors at an early stage. using Docker or local runners).

Testing 52

Testing Data Transformation Statistics Metadata 52

Wayne Yaddow

FEBRUARY 11, 2025

Managing tests of complex data transformations when automated data testing tools lack important features? Photo by Marvin Meyer on Unsplash Introduction Data transformations are at the core of modern business intelligence, blending and converting disparate datasets into coherent, reliable outputs.

Testing 52

Testing Data Transformation Data Quality Statistics 52

Wayne Yaddow

FEBRUARY 6, 2025

Common challenges and practical mitigation strategies for reliable data transformations. Photo by Mika Baumeister on Unsplash Introduction Data transformations are important processes in data engineering, enabling organizations to structure, enrich, and integrate data for analytics , reporting, and operational decision-making.

Testing 52

Testing Data Transformation Data-driven Manufacturing 52

CIO Business Intelligence

DECEMBER 9, 2024

Agentic AI shifts the dial NTT DATAs report finds that 95% of organisations agree that the technology is driving a new level of creativity and innovation and agentic AI is a major leap forward in the evolution of GenAI. [2] For now, 51% say this strategic alignment has not been fully achieved, according to NTT DATAs study. [3]

Technology 98

Technology Insurance Interactive Reporting 98

CIO Business Intelligence

FEBRUARY 20, 2024

Manufacturers have long held a data-driven vision for the future of their industry. It’s one where near real-time data flows seamlessly between IT and operational technology (OT) systems. Legacy data management is holding back manufacturing transformation Until now, however, this vision has remained out of reach.

Data Architecture 98

Data Architecture Unstructured Data Manufacturing IT 98

CIO Business Intelligence

JANUARY 8, 2025

The hospital uses on-prem supercomputers to generate much of its research data, and the movement of that data into and out of the public cloud can become expensive. The academic community expects data to be close to its high-performance compute resources, so they struggle with these egress fees pretty regularly, he says.

Data Processing 139

Data Processing Optimization Modeling Enterprise 139

Wayne Yaddow

FEBRUARY 26, 2025

AI is transforming how senior data engineers and data scientists validate data transformations and conversions. Artificial intelligence-based verification approaches aid in the detection of anomalies, the enforcement of data integrity, and the optimization of pipelines for improved efficiency.

Data Transformation 52

Data Transformation Testing Data-driven Data Quality 52

CIO Business Intelligence

JANUARY 12, 2023

Data is critical to success for universities. Data provides insights that support the overall strategy of the university. Data also lies at the heart of creating a secure, Trusted Research Environment to accelerate and improve research. Yet most universities struggle to collect, analyse, and activate their data resources.

Data Strategy 98

Data Strategy Strategy IT Consulting 98

CIO Business Intelligence

FEBRUARY 27, 2025

A high hurdle many enterprises have yet to overcome is accessing mainframe data via the cloud. Data professionals need to access and work with this information for businesses to run efficiently, and to make strategic forecasting decisions through AI-powered data models.

Metadata 98

Metadata Data Lake Cost-Benefit Forecasting 98

Bruno Aziza

AUGUST 18, 2019

Although CIO’s and CDO’s aspire to be on the offensive in using data to drive revenue generation and business growth, it is defensive initiatives that are providing cover for forward-looking transformation ambitions.

Data Transformation 77

Data Transformation IT 77

AWS Big Data

DECEMBER 21, 2023

Movement of data across data lakes, data warehouses, and purpose-built stores is achieved by extract, transform, and load (ETL) processes using data integration services such as AWS Glue. AWS Glue provides both visual and code-based interfaces to make data integration effortless.

Analytics 111

Analytics IT Data Lake Visualization 111

O'Reilly on Data

JANUARY 17, 2023

A common task for a data scientist is to build a predictive model. You know the drill: pull some data, carve it up into features, feed it into one of scikit-learn’s various algorithms. Collectively, your attempts teach you about your data and its relation to the problem you’re trying to solve.

Machine Learning 363

Machine Learning Predictive Modeling Software Modeling 363

IBM Big Data Hub

OCTOBER 24, 2017

If you’re holding an event for the very first time, what helps you gauge its success? At IBM Analytics University, we turned to social media analytics. Here’s a summary of what we learned from the experts and from Watson Analytics for Social Media.

Data Transformation 72

Data Transformation Analytics IT 72

Dataiku

SEPTEMBER 26, 2022

Wouldn’t it be great if data just came to you ready and primed for analysis? Unfortunately, as data often comes from different sources, with different definitions, and without standardization, it nearly always requires some modification to be useful for its target destination.

Data Transformation 52

Data Transformation Modeling IT 52

datapine

SEPTEMBER 29, 2022

What Is Data Quality Management (DQM)? Data quality management is a set of practices that aim at maintaining a high quality of information. It goes all the way from the acquisition of data and the implementation of advanced data processes, to an effective distribution of data.

Data Quality 241

Data Quality Metrics Data-driven Management 241

Peter James Thomas

MAY 10, 2019

But the 5 questions I highlight are as follows: Why does my organisation need to embark on a Data Transformation – what will it achieve for us? Do I have the expertise and experience on hand to scope a Data Transformation and then deliver it? How long will a Data Transformation take and how much will it cost?

Data Transformation 50

Data Transformation Consulting Publishing Management 50

AWS Big Data

JANUARY 15, 2025

Their terminal operations rely heavily on seamless data flows and the management of vast volumes of data. Recently, EUROGATE has developed a digital twin for its container terminal Hamburg (CTH), generating millions of data points every second from Internet of Things (IoT)devices attached to its container handling equipment (CHE).

IoT 110

IoT Machine Learning Metadata Data-driven 110

Alation

OCTOBER 26, 2021

These organizations that build an organization-wide data culture enjoy clear and compelling benefits. In our recent State of Data Culture Report , Alation found that nearly every organization (86%) with a top-tier data culture met or exceeded its revenue targets. Building a Data Culture Within a Finance Department.

Finance 52

Finance Data Transformation Enterprise Metrics 52

AWS Big Data

NOVEMBER 27, 2024

The data in Amazon Redshift is transactionally consistent and updates are automatically and continuously propagated. Together with price-performance, Amazon Redshift offers capabilities such as serverless architecture, machine learning integration within your data warehouse and secure data sharing across the organization.

Data Warehouse 108

Data Warehouse Analytics Testing Sales 108

AWS Big Data

NOVEMBER 22, 2024

Redshift Data API provides a secure HTTP endpoint and integration with AWS SDKs. With Data API session reuse, you can use a single long-lived session at the start of the ETL pipeline and use that persistent context across all ETL phases. In the next step, copy data from Amazon Simple Storage Service (Amazon S3) to the temporary table.

Data Warehouse 104

Data Warehouse Recreation/Entertainment Cost-Benefit Data-driven 104

Analytics Vidhya

MAY 13, 2024

Their ability to capture long-range dependencies and handle sequential data effectively has made them a staple in every AI researcher and practitioner’s toolbox.

Machine Learning 327

Machine Learning Analytics IT Forecasting 327

AWS Big Data

DECEMBER 20, 2024

Your generated jobs can use a variety of data transformations, including filters, projections, unions, joins, and aggregations, giving you the flexibility to handle complex data processing requirements. In this post, we discuss how Amazon Q data integration transforms ETL workflow development.

Data Integration 105

Data Integration Visualization Data Processing Data Lake 105

MLWhiz

AUGUST 26, 2020

Pandas is one of the best data manipulation libraries in recent times. It lets you slice and dice, groupby, join and do any arbitrary data transformation. You can take a look at this post, which talks about handling most of the data manipulation cases using a straightforward, simple, and matter of fact way using Pandas.

Slice and Dice 131

Slice and Dice Data Transformation IT 131

CIO Business Intelligence

NOVEMBER 17, 2022

Data has become a top priority for businesses large and small, and while some companies have already established a digital strategy, many of them are just getting started. However, the ability to drive digital technology transformation is going to be the focus,” says Stephen Van Vreede, resume expert at IT Tech Exec.

Data Governance 128

Data Governance Data Transformation Strategy Reporting 128

DataKitchen

JULY 27, 2023

While working in Azure with our customers, we have noticed several standard Azure tools people use to develop data pipelines and ETL or ELT processes. We counted ten ‘standard’ ways to transform and set up batch data pipelines in Microsoft Azure. You can use it for big data analytics and machine learning workloads.

Machine Learning 130

Machine Learning Cost-Benefit Data Transformation Testing 130

Smart Data Collective

OCTOBER 20, 2021

In this new reality, leveraging processes like ETL (Extract, Transform, Load) or API (Application Programming Interface) alone to handle the data deluge is not enough. As per the TDWI survey, more than a third (nearly 37%) of people has shown dissatisfaction with their ability to access and integrate complex data streams.

Data Integration 133

Data Integration Machine Learning Big Data Statistics 133

datapine

MAY 2, 2023

Table of Contents 1) Benefits Of Big Data In Logistics 2) 10 Big Data In Logistics Use Cases Big data is revolutionizing many fields of business, and logistics analytics is no exception. The complex and ever-evolving nature of logistics makes it an essential use case for big data applications. Did you know?

Big Data 275

Big Data Internet of Things Cost-Benefit Optimization 275