CIO Business Intelligence

OCTOBER 18, 2022

The old stadium, which opened in 1992, provided the business operations team with data, but that data came from disparate sources, many of which were not consistently updated. The new Globe Life Field not only boasts a retractable roof, but it produces data in categories that didn’t even exist in 1992.

Data Transformation 141

Data Transformation Consulting Data Lake Reporting 141

AWS Big Data

NOVEMBER 22, 2024

The need for streamlined data transformations As organizations increasingly adopt cloud-based data lakes and warehouses, the demand for efficient data transformation tools has grown. This approach helps in managing storage costs while maintaining the flexibility to analyze historical trends when needed.

Data Lake 104

Data Lake Data Warehouse Cost-Benefit Data Transformation 104

CIO Business Intelligence

AUGUST 9, 2024

At Atlanta’s Hartsfield-Jackson International Airport, an IT pilot has led to a wholesale data journey destined to transform operations at the world’s busiest airport, fueled by machine learning and generative AI. Applying AI to elevate ROI Pruitt and Databricks recently finished a pilot test with Microsoft called Smart Flow.

Data Transformation 126

Data Transformation Machine Learning Data Lake Dashboards 126

Smart Data Collective

JULY 25, 2019

When it comes to the use of modern big data technologies by hospitals, it is about health care and saving lives. These systems rely heavily on big data to improve efficiency and cost-effectiveness. It is simple and convenient to use outsourcing IT services when you need to get a perfect big data solution. Conclusion.

Big Data 110

Big Data Data Transformation Management Software 110

CIO Business Intelligence

JUNE 13, 2022

Ever increasing demands for transformation. Growing cybersecurity, data privacy threats. According to Evanta’s 2022 CIO Leadership Perspectives study, CIOs’ second top priority within the IT function is around data and analytics, with CIOs seeing advancing organizational use of data as key to reaching enterprise objectives.

IT 144

IT Digital Transformation Internet of Things Strategy 144

Analytics Vidhya

DECEMBER 11, 2023

Introduction Power Query is a powerful data transformation and manipulation tool in PowerBI that allows users to extract, transform, and load data from various sources. It provides a user-friendly interface for performing complex data transformations without the need for coding.

Data Transformation 178

Data Transformation Analytics IT 178

Rocket-Powered Data Science

MARCH 20, 2023

This is where we dispel an old “big data” notion (heard a decade ago) that was expressed like this: “we need our data to run at the speed of business.” Instead, what we really need is for our business to run at the speed of data. Datasphere is not just for data managers.

Data Warehouse 325

Data Warehouse Metadata Digital Transformation Machine Learning 325

AWS Big Data

MAY 2, 2025

This middleware consists of custom code that runs data flows to stitch data transformations, search queries, and AI enrichments in varying combinations tailored to use cases, datasets, and requirements. Ingest flows are created to enrich data as its added to an index. Flows are a pipeline of processor resources.

Machine Learning 88

Machine Learning Visualization Dashboards Metadata 88

Wayne Yaddow

FEBRUARY 4, 2025

Its EssentialVerifying Data Transformations (Part4) Uncovering the leading problems in data transformation workflowsand practical ways to detect and preventthem In Parts 13 of this series of blogs, categories of data transformations were identified as among the top causes of data quality defects in data pipeline workflows.

Data Transformation 52

Data Transformation Testing Data Quality Strategy 52

Wayne Yaddow

FEBRUARY 21, 2025

Complex Data TransformationsTest Planning Best Practices Ensuring data accuracy with structured testing and best practices Photo by Taylor Vick on Unsplash Introduction Data transformations and conversions are crucial for data pipelines, enabling organizations to process, integrate, and refine raw data into meaningful insights.

Testing 52

Testing Data Transformation Data Quality Data Integration 52

Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

Wayne Yaddow

FEBRUARY 11, 2025

Managing tests of complex data transformations when automated data testing tools lack important features? Photo by Marvin Meyer on Unsplash Introduction Data transformations are at the core of modern business intelligence, blending and converting disparate datasets into coherent, reliable outputs.

Testing 52

Testing Data Transformation Data Quality Statistics 52

Wayne Yaddow

MARCH 12, 2025

Data quality rules are codified into structured Expectation Suites by Great Expectations instead of relying on ad-hoc scripts or manual checks. The framework ensures that your data transformations comply with rigorous specifications from the moment they are created through every iteration of your pipeline.

Data Transformation 52

Data Transformation Data Quality Testing Data Warehouse 52

Wayne Yaddow

FEBRUARY 6, 2025

Common challenges and practical mitigation strategies for reliable data transformations. Photo by Mika Baumeister on Unsplash Introduction Data transformations are important processes in data engineering, enabling organizations to structure, enrich, and integrate data for analytics , reporting, and operational decision-making.

Testing 52

Testing Data Transformation Data-driven Manufacturing 52

CIO Business Intelligence

DECEMBER 9, 2024

Agentic AI shifts the dial NTT DATAs report finds that 95% of organisations agree that the technology is driving a new level of creativity and innovation and agentic AI is a major leap forward in the evolution of GenAI. [2] For now, 51% say this strategic alignment has not been fully achieved, according to NTT DATAs study. [3]

Technology 98

Technology Insurance Interactive Reporting 98

CIO Business Intelligence

FEBRUARY 20, 2024

Manufacturers have long held a data-driven vision for the future of their industry. It’s one where near real-time data flows seamlessly between IT and operational technology (OT) systems. Legacy data management is holding back manufacturing transformation Until now, however, this vision has remained out of reach.

Data Architecture 98

Data Architecture Unstructured Data Manufacturing IT 98

AWS Big Data

OCTOBER 30, 2024

Amazon DataZone is a data management service that makes it faster and easier for customers to catalog, discover, share, and govern data stored across AWS, on premises, and from third-party sources. You can now view your project’s subscribed data directly within Tableau and build dashboards.

Analytics 119

Analytics Visualization Data Governance Data-driven 119

CIO Business Intelligence

JANUARY 8, 2025

The hospital uses on-prem supercomputers to generate much of its research data, and the movement of that data into and out of the public cloud can become expensive. The academic community expects data to be close to its high-performance compute resources, so they struggle with these egress fees pretty regularly, he says.

Data Processing 139

Data Processing Optimization Modeling Enterprise 139

Wayne Yaddow

FEBRUARY 26, 2025

AI is transforming how senior data engineers and data scientists validate data transformations and conversions. Artificial intelligence-based verification approaches aid in the detection of anomalies, the enforcement of data integrity, and the optimization of pipelines for improved efficiency.

Data Transformation 52

Data Transformation Testing Data-driven Data Quality 52

Wayne Yaddow

MARCH 5, 2025

In this post, well see the fundamental procedures, tools, and techniques that data engineers, data scientists, and QA/testing teams use to ensure high-quality data as soon as its deployed. First, we look at how unit and integration tests uncover transformation errors at an early stage. using Docker or local runners).

Testing 52

Testing Data Transformation Statistics Metadata 52

CIO Business Intelligence

JANUARY 12, 2023

Data is critical to success for universities. Data provides insights that support the overall strategy of the university. Data also lies at the heart of creating a secure, Trusted Research Environment to accelerate and improve research. Yet most universities struggle to collect, analyse, and activate their data resources.

Data Strategy 98

Data Strategy Strategy IT Consulting 98

CIO Business Intelligence

FEBRUARY 27, 2025

A high hurdle many enterprises have yet to overcome is accessing mainframe data via the cloud. Data professionals need to access and work with this information for businesses to run efficiently, and to make strategic forecasting decisions through AI-powered data models.

Metadata 98

Metadata Data Lake Cost-Benefit Forecasting 98

Bruno Aziza

AUGUST 18, 2019

Although CIO’s and CDO’s aspire to be on the offensive in using data to drive revenue generation and business growth, it is defensive initiatives that are providing cover for forward-looking transformation ambitions.

Data Transformation 77

Data Transformation IT 77

AWS Big Data

DECEMBER 21, 2023

Movement of data across data lakes, data warehouses, and purpose-built stores is achieved by extract, transform, and load (ETL) processes using data integration services such as AWS Glue. AWS Glue provides both visual and code-based interfaces to make data integration effortless.

Analytics 111

Analytics IT Data Lake Visualization 111

O'Reilly on Data

JANUARY 17, 2023

A common task for a data scientist is to build a predictive model. You know the drill: pull some data, carve it up into features, feed it into one of scikit-learn’s various algorithms. Collectively, your attempts teach you about your data and its relation to the problem you’re trying to solve.

Machine Learning 363

Machine Learning Predictive Modeling Software Modeling 363

Dataiku

SEPTEMBER 26, 2022

Wouldn’t it be great if data just came to you ready and primed for analysis? Unfortunately, as data often comes from different sources, with different definitions, and without standardization, it nearly always requires some modification to be useful for its target destination.

Data Transformation 52

Data Transformation Modeling IT 52

datapine

SEPTEMBER 29, 2022

What Is Data Quality Management (DQM)? Data quality management is a set of practices that aim at maintaining a high quality of information. It goes all the way from the acquisition of data and the implementation of advanced data processes, to an effective distribution of data.

Data Quality 241

Data Quality Metrics Data-driven Management 241

CIO Business Intelligence

APRIL 8, 2025

These issues dont just hinder next-gen analytics and AI; they erode trust, delay transformation and diminish business value. Data quality is no longer a back-office concern. In this article, I am drawing from firsthand experience working with CIOs, CDOs, CTOs and transformation leaders across industries.

Data Quality 102

Data Quality Data-driven Key Performance Indicator Metadata 102

AWS Big Data

JANUARY 15, 2025

Their terminal operations rely heavily on seamless data flows and the management of vast volumes of data. Recently, EUROGATE has developed a digital twin for its container terminal Hamburg (CTH), generating millions of data points every second from Internet of Things (IoT)devices attached to its container handling equipment (CHE).

IoT 111

IoT Machine Learning Metadata Data-driven 111

AWS Big Data

NOVEMBER 27, 2024

The data in Amazon Redshift is transactionally consistent and updates are automatically and continuously propagated. Together with price-performance, Amazon Redshift offers capabilities such as serverless architecture, machine learning integration within your data warehouse and secure data sharing across the organization.

Data Warehouse 110

Data Warehouse Analytics Testing Sales 110

AWS Big Data

NOVEMBER 22, 2024

Redshift Data API provides a secure HTTP endpoint and integration with AWS SDKs. With Data API session reuse, you can use a single long-lived session at the start of the ETL pipeline and use that persistent context across all ETL phases. In the next step, copy data from Amazon Simple Storage Service (Amazon S3) to the temporary table.

Data Warehouse 105

Data Warehouse Recreation/Entertainment Cost-Benefit Data-driven 105

AWS Big Data

DECEMBER 20, 2024

Your generated jobs can use a variety of data transformations, including filters, projections, unions, joins, and aggregations, giving you the flexibility to handle complex data processing requirements. In this post, we discuss how Amazon Q data integration transforms ETL workflow development.

Data Integration 105

Data Integration Visualization Data Processing Big Data 105

Analytics Vidhya

MAY 13, 2024

Their ability to capture long-range dependencies and handle sequential data effectively has made them a staple in every AI researcher and practitioner’s toolbox.

Machine Learning 327

Machine Learning Analytics IT Forecasting 327

AWS Big Data

OCTOBER 30, 2024

Amazon DataZone natively integrates with Amazon-specific options like Amazon Athena , Amazon Redshift , and Amazon SageMaker , allowing users to analyze their project governed data. Connect to Tableau Desktop Use the Athena JDBC driver to connect Tableau to Amazon DataZone and visualize your subscribed data.

Visualization 119

Visualization Data Lake Testing Data Governance 119

MLWhiz

AUGUST 26, 2020

Pandas is one of the best data manipulation libraries in recent times. It lets you slice and dice, groupby, join and do any arbitrary data transformation. You can take a look at this post, which talks about handling most of the data manipulation cases using a straightforward, simple, and matter of fact way using Pandas.

Slice and Dice 131

Slice and Dice Data Transformation IT 131

DataKitchen

APRIL 26, 2021

At Workiva, they recognized that they are only as good as their data, so they centered their initial DataOps efforts around lowering errors. GSK’s DataOps journey paralleled their data transformation journey. Smith explained, “We knew how to do data, we’ve done it our whole lives. Others have difficulty collaborating.

Measurement 246

Measurement Metrics Data-driven Dashboards 246