datapine

SEPTEMBER 29, 2022

What Is Data Quality Management (DQM)? Data quality management is a set of practices that aim at maintaining a high quality of information. It goes all the way from the acquisition of data and the implementation of advanced data processes, to an effective distribution of data.

Data Quality 241

Data Quality Metrics Data-driven Management 241

CIO Business Intelligence

FEBRUARY 20, 2024

Manufacturers have long held a data-driven vision for the future of their industry. It’s one where near real-time data flows seamlessly between IT and operational technology (OT) systems. Legacy data management is holding back manufacturing transformation Until now, however, this vision has remained out of reach.

Data Architecture 98

Data Architecture Unstructured Data Manufacturing IT 98

Alation

JUNE 28, 2022

This is where metadata, or the data about data, comes into play. Having a data catalog is the cornerstone of your data governance strategy, but what supports your data catalog? Your metadata management framework provides the underlying structure that makes your data accessible and manageable.

Metadata 52

Metadata Management Data Governance Machine Learning 52

AWS Big Data

DECEMBER 13, 2023

In addition to using native managed AWS services that BMS didn’t need to worry about upgrading, BMS was looking to offer an ETL service to non-technical business users that could visually compose data transformation workflows and seamlessly run them on the AWS Glue Apache Spark-based serverless data integration engine.

Metadata 101

Metadata Data Lake Visualization Data Transformation 101

AWS Big Data

OCTOBER 30, 2024

Amazon DataZone natively integrates with Amazon-specific options like Amazon Athena , Amazon Redshift , and Amazon SageMaker , allowing users to analyze their project governed data. Publish data assets – As the data producer from the retail team, you must ingest individual data assets into Amazon DataZone.

Visualization 116

Visualization Data Lake Testing Data Governance 116

Wayne Yaddow

MARCH 5, 2025

In this post, well see the fundamental procedures, tools, and techniques that data engineers, data scientists, and QA/testing teams use to ensure high-quality data as soon as its deployed. First, we look at how unit and integration tests uncover transformation errors at an early stage. using Docker or local runners).

Testing 52

Testing Data Transformation Statistics Metadata 52

Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

Alation

OCTOBER 18, 2022

Data analysts and engineers use dbt to transform, test, and document data in the cloud data warehouse. Yet every dbt transformation contains vital metadata that is not captured – until now. Data Transformation in the Modern Data Stack. How did the data transform exactly?

Metadata 52

Metadata Metrics Recreation/Entertainment Data Quality 52

AWS Big Data

OCTOBER 11, 2023

The lift and shift migration approach is limited in its ability to transform businesses because it relies on outdated, legacy technologies and architectures that limit flexibility and slow down productivity. It shows a call center streaming data source that sends the latest call center feed in every 15 seconds.

Management 118

Management Metadata Analytics Dashboards 118

erwin

SEPTEMBER 17, 2020

Replace manual and recurring tasks for fast, reliable data lineage and overall data governance. It’s paramount that organizations understand the benefits of automating end-to-end data lineage. The importance of end-to-end data lineage is widely understood and ignoring it is risky business. Doing Data Lineage Right.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

erwin

APRIL 29, 2020

Data lineage is the journey data takes from its creation through its transformations over time. Tracing the source of data is an arduous task. With all these diverse data sources, and if systems are integrated, it is difficult to understand the complicated data web they form much less get a simple visual flow.

Data Governance 111

Data Governance Metadata Key Performance Indicator Data Quality 111

AWS Big Data

NOVEMBER 15, 2023

It seamlessly consolidates data from various data sources within AWS, including AWS Cost Explorer (and forecasting with Cost Explorer ), AWS Trusted Advisor , and AWS Compute Optimizer. Overview of the BMW Cloud Data Hub At the BMW Group, Cloud Data Hub (CDH) is the central platform for managing company-wide data and data solutions.

Analytics 107

Analytics Dashboards Metadata Data Warehouse 107

CIO Business Intelligence

AUGUST 3, 2023

When global technology company Lenovo started utilizing data analytics, they helped identify a new market niche for its gaming laptops, and powered remote diagnostics so their customers got the most from their servers and other devices.

Analytics 98

Analytics Data Lake Metadata Cost-Benefit 98

AWS Big Data

OCTOBER 23, 2024

It’s a set of HTTP endpoints to perform operations such as invoking Directed Acyclic Graphs (DAGs), checking task statuses, retrieving metadata about workflows, managing connections and variables, and even initiating dataset-related events, without directly accessing the Airflow web interface or command line tools.

Interactive 101

Interactive Testing Data-driven Data Lake 101

AWS Big Data

AUGUST 26, 2024

You will learn how to prepare a multi-account environment to access the databases from AWS Glue, and how to model an ETL data flow that automatically masks PII as part of the transfer process, so that no sensitive information will be copied to the target database in its original form. 16 10.2.10.0/24 24 AWS Glue account 10.1.0.0/16

Visualization 107

Visualization Metadata Data Transformation Testing 107

Smart Data Collective

OCTOBER 13, 2021

There are countless examples of big data transforming many different industries. There is no disputing the fact that the collection and analysis of massive amounts of unstructured data has been a huge breakthrough. We would like to talk about data visualization and its role in the big data movement.

Visualization 145

Visualization Cost-Benefit Big Data Prescriptive Analytics 145

AWS Big Data

DECEMBER 16, 2024

With the ability to browse metadata, you can understand the structure and schema of the data source, identify relevant tables and fields, and discover useful data assets you may not be aware of. Choose Add data. Prerequisites Before you begin, make sure you have the followings: An AWS account. Choose Save changes.

Visualization 72

Visualization Data Processing Testing Publishing 72

Alation

OCTOBER 26, 2021

These organizations that build an organization-wide data culture enjoy clear and compelling benefits. In our recent State of Data Culture Report , Alation found that nearly every organization (86%) with a top-tier data culture met or exceeded its revenue targets. Building a Data Culture Within a Finance Department.

Finance 52

Finance Data Transformation Enterprise Metrics 52

CIO Business Intelligence

APRIL 30, 2024

At Tableau Conference 2024 in San Diego today, Tableau announced new AI features for Tableau Pulse and Einstein Copilot for Tableau, along with several platform improvements aimed at democratizing data insights. This is really empowering everyone to be a data expert,” Maxon said. “It Metric Goals.

Analytics 96

Analytics Metrics Visualization Dashboards 96

erwin

MARCH 4, 2021

Modern data governance is a strategic, ongoing and collaborative practice that enables organizations to discover and track their data, understand what it means within a business context, and maximize its security, quality and value. The What: Data Governance Defined. Data governance has no standard definition.

Data Governance 122

Data Governance Digital Transformation Data-driven Cost-Benefit 122

Octopai

NOVEMBER 23, 2020

The LIBOR transition is, first and foremost, a data challenge – and it’s one with a concrete deadline that companies can’t afford to miss. In fact, the LIBOR transition program marks one of the largest data transformation obstacles ever seen in financial services. Like any data project, it boils down to the details.

Metadata 72

Metadata Enterprise Business Intelligence Data Governance 72

Cloudera

OCTOBER 7, 2022

We’re excited to announce the general availability of the open source adapters for dbt for all the engines in CDP — Apache Hive , Apache Impala , and Apache Spark, with added support for Apache Livy and Cloudera Data Engineering. The Open Data Lakehouse . Cloudera builds dbt adaptors for all engines in the open data lakehouse.

Data Warehouse 81

Data Warehouse Data Transformation Machine Learning Data Lake 81

Octopai

JUNE 9, 2024

.” The Data Strategy HealthCo, like many forward-thinking organizations, recognized early on that data is not just a valuable asset but a strategic imperative. They put data at the forefront of their business, integrating it into decision-making processes, products, and services. The lack of trust in data created inertia.

IT 52

IT Data-driven Predictive Analytics Data Strategy 52

Cloudera

JUNE 30, 2022

These tools empower analysts and data scientists to easily collaborate on the same data, with their choice of tools and analytic engines. No more lock-in, unnecessary data transformations, or data movement across tools and clouds just to extract insights out of the data.

Data Lake 88

Data Lake Data Warehouse Data Architecture Metadata 88

Octopai

NOVEMBER 13, 2022

The modern data stack is a data management system built out of cloud-based data systems. A given modern data stack will usually include components for data ingestion from your data sources, data transformation, data storage, data analysis and reporting.

Enterprise 97

Enterprise Data Warehouse Reporting Metadata 97

Cloudera

OCTOBER 11, 2021

The platform converges data cataloging, data ingestion, data profiling, data tagging, data discovery, and data exploration into a unified platform, driven by metadata. Modak Nabu automates repetitive tasks in the data preparation process and thus accelerates the data preparation by 4x.

Data Lake 95

Data Lake Cost-Benefit Data-driven Dashboards 95

AWS Big Data

DECEMBER 5, 2023

This frees up our local computer space, greatly automates the survey cleaning and analysis step, and allows our clients to easily access the data results. .” — Harman Singh Dhodi, Analyst at HR&A Advisors, Inc. A combination of Amazon Redshift Spectrum and COPY commands are used to ingest the survey data stored as CSV files.

Measurement 94

Measurement Dashboards Data Warehouse Analytics 94

IBM Big Data Hub

JANUARY 9, 2024

The emergence of generative AI prompted several prominent companies to restrict its use because of the mishandling of sensitive internal data. Currently, no standardized process exists for overcoming data ingestion’s challenges, but the model’s accuracy depends on it. Increased variance: Variance measures consistency.

Enterprise 71

Enterprise Data Integration Data Quality Contextual Data 71

Alation

APRIL 4, 2023

Now, joint users will get an enhanced view into cloud and data transformations , with valuable context to guide smarter usage. Integrating helpful metadata into user workflows gives all people, from data scientists to analysts , the context they need to use data more effectively. How was it used in the past?

Metadata 52

Metadata Cost-Benefit Data Transformation Predictive Modeling 52

AWS Big Data

JANUARY 27, 2023

To build a data-driven business, it is important to democratize enterprise data assets in a data catalog. With a unified data catalog, you can quickly search datasets and figure out data schema, data format, and location. For metadata read/write, Flink has the catalog interface.

Data Lake 67

Data Lake Metadata Business Analysis Data-driven 67

AWS Big Data

APRIL 12, 2023

The solution uses the TPC-DS dataset and unmodified data schema and table relationships, but derives queries from TPC-DS to support the SparkSQL test cases. Metadata store – We use Spark’s in-memory data catalog to store metadata for TPC-DS databases and tables— spark.sql.catalogImplementation is set to the default value in-memory.

Testing 86

Testing Big Data Metadata Optimization 86

Ontotext

NOVEMBER 18, 2021

You step onto the market, and if you don’t keep your data, there’s no knowing where you might be swept off to. [1]. Picture this – you start with the perfect use case for your data analytics product. Nowadays, data analytics doesn’t exist on its own. You make a great pitch and you sell well. Sounds unlikely.

Visualization 98

Visualization Reporting Metadata Enterprise 98

Alation

AUGUST 11, 2022

DataOps sprung up to connect data sources to data consumers. The data warehouse and analytical data stores moved to the cloud and disaggregated into the data mesh. We chatted about industry trends, why decentralization has become a hot topic in the data world, and how metadata drives many data-centric use cases.

Metadata 52

Metadata Data Warehouse Data Quality Data Lake 52

IBM Big Data Hub

OCTOBER 16, 2023

It includes processes that trace and document the origin of data, models and associated metadata and pipelines for audits. Curated foundation models, such as those created by IBM or Microsoft, help enterprises scale and accelerate the use and impact of the most advanced AI capabilities using trusted data.

Risk 70

Risk Modeling Management Metadata 70

AWS Big Data

NOVEMBER 29, 2023

dbt is an open source, SQL-first templating engine that allows you to write repeatable and extensible data transforms in Python and SQL. dbt is predominantly used by data warehouses (such as Amazon Redshift ) customers who are looking to keep their data transform logic separate from storage and engine.

Data Lake 117

Data Lake Management Metrics Data Warehouse 117

AWS Big Data

MARCH 13, 2024

The solution consists of the following interfaces: IoT or mobile application – A mobile application or an Internet of Things (IoT) device allows the tracking of a company vehicle while it is in use and transmits its current location securely to the data ingestion layer in AWS. The ingestion approach is not in scope of this post. Choose Run.

Analytics 110

Analytics IoT Metadata Internet of Things 110

IBM Big Data Hub

JULY 5, 2023

The challenges of a monolithic data lake architecture Data lakes are, at a high level, single repositories of data at scale. Data may be stored in its raw original form or optimized into a different format suitable for consumption by specialized engines. Data governance remains an unexplored frontier for this technology.

Data Lake 60

Data Lake Metadata Cost-Benefit Data Warehouse 60