AWS Big Data

OCTOBER 23, 2024

The Airflow REST API facilitates a wide range of use cases, from centralizing and automating administrative tasks to building event-driven, data-aware data pipelines. Event-driven architectures – The enhanced API facilitates seamless integration with external events, enabling the triggering of Airflow DAGs based on these events.

Interactive 101

Interactive Testing Data-driven Data Lake 101

Smart Data Collective

JULY 25, 2019

The healthcare industry is happily embracing big data. Hospitals around the world are finding that data can have a profound impact on their operations. A lot of the emphasis so far has been on the use of big data to better engage with external third-parties, but big data can be equally valuable for managing internal hospital systems.

Big Data 110

Big Data Data Transformation Management Software 110

CIO Business Intelligence

APRIL 8, 2025

Data is the foundation of innovation, agility and competitive advantage in todays digital economy. As technology and business leaders, your strategic initiatives, from AI-powered decision-making to predictive insights and personalized experiences, are all fueled by data. Data quality is no longer a back-office concern.

Data Quality 102

Data Quality Data-driven Key Performance Indicator Metadata 102

Rocket-Powered Data Science

MARCH 20, 2023

We live in a data-rich, insights-rich, and content-rich world. Data collections are the ones and zeroes that encode the actionable insights (patterns, trends, relationships) that we seek to extract from our data through machine learning and data science. Plus, AI can also help find key insights encoded in data.

Data Warehouse 325

Data Warehouse Metadata Digital Transformation Machine Learning 325

Analytics Vidhya

FEBRUARY 19, 2023

Introduction Data pipelines play a critical role in the processing and management of data in modern organizations. A well-designed data pipeline can help organizations extract valuable insights from their data, automate tedious manual processes, and ensure the accuracy of data processing.

Data-driven 271

Data-driven Data Processing Management Analytics 271

AWS Big Data

JANUARY 15, 2025

For container terminal operators, data-driven decision-making and efficient data sharing are vital to optimizing operations and boosting supply chain efficiency. Together, these capabilities enable terminal operators to enhance efficiency and competitiveness in an industry that is increasingly data driven.

IoT 100

IoT Machine Learning Metadata Data-driven 100

AWS Big Data

OCTOBER 30, 2024

Amazon DataZone is a data management service that makes it faster and easier for customers to catalog, discover, share, and govern data stored across AWS, on premises, and from third-party sources. Using Amazon DataZone lets us avoid building and maintaining an in-house platform, allowing our developers to focus on tailored solutions.

Analytics 103

Analytics Visualization Data Governance Data-driven 103

DataKitchen

APRIL 26, 2021

In early April 2021, DataKItchen sat down with Jonathan Hodges, VP Data Management & Analytics, at Workiva ; Chuck Smith, VP of R&D Data Strategy at GlaxoSmithKline (GSK) ; and Chris Bergh, CEO and Head Chef at DataKitchen, to find out about their enterprise DataOps transformation journey, including key successes and lessons learned.

Measurement 246

Measurement Metrics Data-driven Dashboards 246

Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

Wayne Yaddow

FEBRUARY 18, 2025

Selecting the strategies and tools for validating data transformations and data conversions in your data pipelines. Introduction Data transformations and data conversions are crucial to ensure that raw data is organized, processed, and ready for useful analysis.

Testing 52

Testing Data Transformation Data-driven Data Quality 52

Wayne Yaddow

FEBRUARY 26, 2025

AI is transforming how senior data engineers and data scientists validate data transformations and conversions. Artificial intelligence-based verification approaches aid in the detection of anomalies, the enforcement of data integrity, and the optimization of pipelines for improved efficiency.

Data Transformation 52

Data Transformation Testing Data-driven Data Quality 52

O'Reilly on Data

OCTOBER 19, 2021

As with many burgeoning fields and disciplines, we don’t yet have a shared canonical infrastructure stack or best practices for developing and deploying data-intensive applications. Why: Data Makes It Different. Not only is data larger, but models—deep learning models in particular—are much larger than before.

IT 363

IT Testing Experimentation Software 363

Wayne Yaddow

FEBRUARY 6, 2025

Common challenges and practical mitigation strategies for reliable data transformations. Photo by Mika Baumeister on Unsplash Introduction Data transformations are important processes in data engineering, enabling organizations to structure, enrich, and integrate data for analytics , reporting, and operational decision-making.

Testing 52

Testing Data Transformation Data-driven Manufacturing 52

AWS Big Data

DECEMBER 4, 2024

In today’s rapidly evolving financial landscape, data is the bedrock of innovation, enhancing customer and employee experiences and securing a competitive edge. Like many large financial institutions, ANZ Institutional Division operated with siloed data practices and centralized data management teams.

Metadata 93

Metadata Data Governance Data Quality Data-driven 93

Sisense

OCTOBER 30, 2019

A critical part of effectively exploring your data, transforming it into actionable insights, and enhancing decision-making for your business is being empowered to slice and dice your data, and be less dependent on technical resources for new updates. Improved visibility into insights will enable you to get more out of them.

Data-driven 70

Data-driven Reporting Management Slice and Dice 70

DataKitchen

DECEMBER 20, 2020

Data-driven companies sense change through data analytics. Companies turn to their data organization to provide the analytics that stimulates creative problem-solving. The speed at which the data team responds to these requests is critical. The agility of analytics directly relates to data analytics workflows.

Data-driven 147

Data-driven Manufacturing Data Architecture Data Analytics 147

AWS Big Data

OCTOBER 30, 2024

Amazon DataZone now launched authentication supports through the Amazon Athena JDBC driver, allowing data users to seamlessly query their subscribed data lake assets via popular business intelligence (BI) and analytics tools like Tableau, Power BI, Excel, SQL Workbench, DBeaver, and more.

Visualization 115

Visualization Data Lake Testing Data Governance 115

datapine

MAY 2, 2023

Table of Contents 1) Benefits Of Big Data In Logistics 2) 10 Big Data In Logistics Use Cases Big data is revolutionizing many fields of business, and logistics analytics is no exception. The complex and ever-evolving nature of logistics makes it an essential use case for big data applications. Did you know?

Big Data 275

Big Data Internet of Things Cost-Benefit Optimization 275

AWS Big Data

NOVEMBER 22, 2024

Amazon Redshift is a fast, scalable, secure, and fully managed cloud data warehouse that you can use to analyze your data at scale. Redshift Data API provides a secure HTTP endpoint and integration with AWS SDKs. Calls to the Data API are asynchronous.

Data Warehouse 92

Data Warehouse Recreation/Entertainment Cost-Benefit Data-driven 92

Ontotext

FEBRUARY 12, 2024

In the final part of this three-part series, we’ll explore ho w data mesh bolsters performance and helps organizations and data teams work more effectively. Usually, organizations will combine different domain topologies, depending on the trade-offs, and choose to focus on specific aspects of data mesh.

Data-driven 59

Data-driven Data Lake Data Quality Business Objectives 59

Ontotext

MARCH 20, 2024

In today’s data-driven world, businesses are drowning in a sea of information. Traditional data integration methods struggle to bridge these gaps, hampered by high costs, data quality concerns, and inconsistencies. Unleashing the Power of Data Connections Zenia Graph isn’t just another data solution company.

Data-driven 52

Data-driven Strategy Sales Data Integration 52

Alation

OCTOBER 26, 2021

Today’s best-performing organizations embrace data for strategic decision-making. Because of the criticality of the data they deal with, we think that finance teams should lead the enterprise adoption of data and analytics solutions. This is because accurate data is “table stakes” for finance teams.

Finance 52

Finance Data Transformation Enterprise Metrics 52

TDAN

FEBRUARY 21, 2024

In a world increasingly dominated by data, organizations are grappling with the need to effectively manage and harness this valuable asset. At the same time, the data management […]

Data Transformation 59

Data Transformation Digital Transformation Data-driven Management 59

AWS Big Data

NOVEMBER 27, 2024

While customers can perform some basic analysis within their operational or transactional databases, many still need to build custom data pipelines that use batch or streaming jobs to extract, transform, and load (ETL) data into their data warehouse for more comprehensive analysis. or a later version) database.

Data Warehouse 93

Data Warehouse Analytics Testing Modeling 93

DataKitchen

DECEMBER 9, 2022

ChatGPT> DataOps, or data operations, is a set of practices and technologies that organizations use to improve the speed, quality, and reliability of their data analytics processes. The goal of DataOps is to help organizations make better use of their data to drive business decisions and improve outcomes.

Machine Learning 130

Machine Learning Data-driven Optimization Data Analytics 130

DataKitchen

JULY 27, 2023

The Ten Standard Tools To Develop Data Pipelines In Microsoft Azure. While working in Azure with our customers, we have noticed several standard Azure tools people use to develop data pipelines and ETL or ELT processes. We counted ten ‘standard’ ways to transform and set up batch data pipelines in Microsoft Azure.

Machine Learning 130

Machine Learning Cost-Benefit Data Transformation Testing 130

CIO Business Intelligence

OCTOBER 11, 2022

To accelerate growth through innovation, the company is expanding its use of data science and artificial intelligence (AI) across the business to improve patient outcomes. . This initiative alone has generated an explosion in the quantity and complexity of data the company collects, stores, and analyzes for insights. . “We

Machine Learning 145

Machine Learning Data Science Data-driven Testing 145

AWS Big Data

DECEMBER 16, 2024

The need to integrate diverse data sources has grown exponentially, but there are several common challenges when integrating and analyzing data from multiple sources, services, and applications. First, you need to create and maintain independent connections to the same data source for different services.

Visualization 71

Visualization Data Processing Testing Publishing 71

erwin

APRIL 29, 2020

Data lineage is the journey data takes from its creation through its transformations over time. Tracing the source of data is an arduous task. With all these diverse data sources, and if systems are integrated, it is difficult to understand the complicated data web they form much less get a simple visual flow.

Metadata 111

Metadata Key Performance Indicator Data Governance Data Quality 111

erwin

MARCH 4, 2021

Modern data governance is a strategic, ongoing and collaborative practice that enables organizations to discover and track their data, understand what it means within a business context, and maximize its security, quality and value. The What: Data Governance Defined. Data governance has no standard definition.

Data Governance 122

Data Governance Digital Transformation Data-driven Cost-Benefit 122

Cloudera

APRIL 19, 2022

Data holds incredible untapped potential for Australian organisations across industries, regardless of individual business goals, and all organisations are at different points in their data transformation journey with some achieving success faster than others. . More importantly, effective data strategies don’t stand still.

Data-driven 93

Data-driven Data Strategy Big Data Strategy 93

CIO Business Intelligence

NOVEMBER 7, 2023

For years, IT and business leaders have been talking about breaking down the data silos that exist within their organizations. Given the importance of sharing information among diverse disciplines in the era of digital transformation, this concept is arguably as important as ever. What are the challenges and potential rewards?

Data Warehouse 142

Data Warehouse Digital Transformation Data-driven Reporting 142

CIO Business Intelligence

FEBRUARY 20, 2024

Manufacturers have long held a data-driven vision for the future of their industry. It’s one where near real-time data flows seamlessly between IT and operational technology (OT) systems. Legacy data management is holding back manufacturing transformation Until now, however, this vision has remained out of reach.

Data Architecture 98

Data Architecture Unstructured Data Manufacturing IT 98

erwin

MARCH 26, 2020

Organizations are flooded with data, so they’re scrambling to find ways to derive meaningful insights from it – and then act on them to improve the bottom line. In today’s data-driven business, enabling employees to access and understand the data that’s relevant to their roles allows them to use data and put those insights into action.

Data-driven 90

Data-driven Unstructured Data Enterprise Machine Learning 90

erwin

SEPTEMBER 17, 2020

Replace manual and recurring tasks for fast, reliable data lineage and overall data governance. It’s paramount that organizations understand the benefits of automating end-to-end data lineage. The importance of end-to-end data lineage is widely understood and ignoring it is risky business. Doing Data Lineage Right.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

Smart Data Collective

APRIL 29, 2020

As the world is gradually becoming more dependent on data, the services, tools and infrastructure are all the more important for businesses in every sector. Data management has become a fundamental business concern, and especially for businesses that are going through a digital transformation. What is data management?

Management 97

Management Data Warehouse Digital Transformation Dashboards 97

AWS Big Data

APRIL 26, 2023

Additionally, this forecasting system needs to provide data enrichment steps including byproducts, serve as the master data around the semiconductor management, and enable further use cases at the BMW Group. To enable this use case, we used the BMW Group’s cloud-native data platform called the Cloud Data Hub.

Forecasting 129

Forecasting Manufacturing Data Lake Big Data 129