Octopai

MARCH 31, 2021

Look for the Metadata. In order to perform accurate data lineage mapping, every process in the system that transforms or touches the data must be recorded. This metadata (read: data about your data) is key to tracking your data. Data Lineage by Tagging or Self-Contained Data Lineage.

Metadata 133

Metadata Data Transformation Business Intelligence Reporting 133

AWS Big Data

DECEMBER 13, 2023

In collaboration with AWS, BMS identified a business need to migrate and modernize their custom extract, transform, and load (ETL) platform to a native AWS solution to reduce complexities, resources, and investment to upgrade when new Spark, Python, or AWS Glue versions are released.

Metadata 111

Metadata Data Lake Visualization Data Quality 111

AWS Big Data

DECEMBER 16, 2024

For each service, you need to learn the supported authorization and authentication methods, data access APIs, and framework to onboard and test data sources. This fragmented, repetitive, and error-prone experience for data connectivity is a significant obstacle to data integration, analysis, and machine learning (ML) initiatives.

Visualization 92

Visualization Data Processing Testing Publishing 92

Wayne Yaddow

MARCH 14, 2025

How dbt Core aids data teams test, validate, and monitor complex data transformations and conversions Photo by NASA on Unsplash Introduction dbt Core, an open-source framework for developing, testing, and documenting SQL-based data transformations, has become a must-have tool for modern data teams as the complexity of data pipelines grows.

Data Transformation 52

Data Transformation Testing Unstructured Data Data Quality 52

Wayne Yaddow

FEBRUARY 18, 2025

Selecting the strategies and tools for validating data transformations and data conversions in your data pipelines. Introduction Data transformations and data conversions are crucial to ensure that raw data is organized, processed, and ready for useful analysis.

Testing 52

Testing Data Transformation Data-driven Data Quality 52

AWS Big Data

NOVEMBER 15, 2023

It seamlessly consolidates data from various data sources within AWS, including AWS Cost Explorer (and forecasting with Cost Explorer ), AWS Trusted Advisor , and AWS Compute Optimizer. Overview of the BMW Cloud Data Hub At the BMW Group, Cloud Data Hub (CDH) is the central platform for managing company-wide data and data solutions.

Dashboards 111

Dashboards Analytics Metadata Data Warehouse 111

erwin

SEPTEMBER 17, 2020

Regulatory compliance places greater transparency demands on firms when it comes to tracing and auditing data. Business terms and data policies should be implemented through standardized and documented business rules. Some data needs to be backed up daily while some types of data demand weekly or monthly backups.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

AWS Big Data

AUGUST 26, 2024

AWS Glue Studio visual editor provides a low-code graphic environment to build, run, and monitor extract, transform, and load (ETL) scripts. There’s no infrastructure to manage, so you can focus on rapidly building compliant data flows between key systems. An AWS Identity and Access Management (IAM) role is used for AWS Glue.

Visualization 116

Visualization Metadata Data Transformation Testing 116

IBM Big Data Hub

OCTOBER 16, 2023

AI governance refers to the practice of directing, managing and monitoring an organization’s AI activities. It includes processes that trace and document the origin of data, models and associated metadata and pipelines for audits.

Risk 70

Risk Modeling Management Metadata 70

erwin

APRIL 29, 2020

An understanding of the data’s origins and history helps answer questions about the origin of data in a Key Performance Indicator (KPI) reports, including: How the report tables and columns are defined in the metadata? Who are the data owners? What are the transformation rules? Data Governance.

Key Performance Indicator 111

Key Performance Indicator Metadata Data Governance Data Quality 111

Smart Data Collective

OCTOBER 13, 2021

There are countless examples of big data transforming many different industries. There is no disputing the fact that the collection and analysis of massive amounts of unstructured data has been a huge breakthrough. How does Data Virtualization manage data quality requirements?

Visualization 145

Visualization Cost-Benefit Big Data Prescriptive Analytics 145

CIO Business Intelligence

FEBRUARY 20, 2024

Where all data – structured, semi-structured, and unstructured – is sourced, unified, and exploited in automated processes, AI tools and by highly skilled, but over-stretched, employees. Legacy data management is holding back manufacturing transformation Until now, however, this vision has remained out of reach.

Data Architecture 98

Data Architecture Unstructured Data Manufacturing IT 98

Alation

OCTOBER 26, 2021

Building a Data Culture Within a Finance Department. Our finance users tell us that their first exposure to the Alation Data Catalog often comes soon after the launch of organization-wide data transformation efforts. After all, finance is one of the greatest consumers of data within a business. Don’t overthink it.

Finance 52

Finance Data Transformation Enterprise Metrics 52

TDAN

AUGUST 15, 2023

Nearly every data leader I talk to is in the midst of a data transformation. As businesses look for ways to increase sales, improve customer experience, and stay ahead of the competition, they are realizing that data is their competitive advantage and the key to achieving their goals. And it’s no surprise, really.

Data Transformation 52

Data Transformation Sales Metadata Data Governance 52

AWS Big Data

APRIL 2, 2024

You can see the decompressed data has metadata information such as logGroup , logStream , and subscriptionFilters , and the actual data is included within the message field under logEvents (the following example shows an example of CloudTrail events in the CloudWatch Logs). You can connect with Ranjit on LinkedIn.

Metadata 111

Metadata Marketing Analytics Data Transformation 111

erwin

MARCH 4, 2021

However, while digital transformation and other data-driven initiatives are desired outcomes, few organizations know what data they have or where it is, and they struggle to integrate known data in various formats and numerous systems – especially if they don’t have a way to automate those processes.

Data Governance 122

Data Governance Digital Transformation Data-driven Cost-Benefit 122

Octopai

NOVEMBER 23, 2020

Naturally, this will produce a massive growth in data management needs. Particularly in financial services, each company’s data team will have significant work to do to keep up with these developments and this is on top of the regulatory challenges that they already face on a daily basis.

Metadata 72

Metadata Enterprise Business Intelligence Data Governance 72

erwin

JANUARY 11, 2019

Organizations have spent a lot of time and money trying to harmonize data across diverse platforms , including cleansing, uploading metadata, converting code, defining business glossaries, tracking data transformations and so on. Creating a High-Quality Data Pipeline.

Data Governance 40

Data Governance Risk Metadata Management 40

Cloudera

OCTOBER 7, 2022

We’re excited to announce the general availability of the open source adapters for dbt for all the engines in CDP — Apache Hive , Apache Impala , and Apache Spark, with added support for Apache Livy and Cloudera Data Engineering. The Open Data Lakehouse . Cloudera builds dbt adaptors for all engines in the open data lakehouse.

Data Warehouse 81

Data Warehouse Data Transformation Machine Learning Data Lake 81

AWS Big Data

DECEMBER 5, 2023

A combination of Amazon Redshift Spectrum and COPY commands are used to ingest the survey data stored as CSV files. For the files with unknown structures, AWS Glue crawlers are used to extract metadata and create table definitions in the Data Catalog.

Measurement 105

Measurement Dashboards Data Warehouse Analytics 105

Cloudera

JUNE 30, 2022

These tools empower analysts and data scientists to easily collaborate on the same data, with their choice of tools and analytic engines. No more lock-in, unnecessary data transformations, or data movement across tools and clouds just to extract insights out of the data.

Data Lake 88

Data Lake Data Warehouse Data Architecture Metadata 88

AWS Big Data

APRIL 12, 2023

EMR on EKS simplifies your infrastructure management, maximizes resource utilization, and reduces your cost. We have been continually improving the Spark performance in each Amazon EMR release to further shorten job runtime and optimize users’ spending on their Amazon EMR big data workloads. test: EMR release – EMR 6.10.0

Testing 104

Testing Big Data Metadata Optimization 104

AWS Big Data

MARCH 9, 2023

Amazon QuickSight is a fully managed, cloud-native business intelligence (BI) service that makes it easy to connect to your data, create interactive dashboards and reports, and share these with tens of thousands of users, either within QuickSight or embedded in your application or website. The QuickSight SDK v2.0

Slice and Dice 98

Slice and Dice Dashboards Analytics Interactive 98

AWS Big Data

MARCH 13, 2024

This post shows how you can use Amazon Location, EventBridge, Lambda, Amazon Data Firehose , and Amazon S3 to build a location-aware data pipeline, and use this data to drive meaningful insights using AWS Glue and Athena. Overview of solution This is a fully serverless solution for location-based asset management.

Analytics 122

Analytics IoT Metadata Internet of Things 122

AWS Big Data

JULY 20, 2023

Moreover, running advanced analytics and ML on disparate data sources proved challenging. To overcome these issues, Orca decided to build a data lake. By decoupling storage and compute, data lakes promote cost-effective storage and processing of big data. This ensures that the data is suitable for training purposes.

Data Lake 98

Data Lake Analytics Snapshot Data Quality 98

Alation

AUGUST 11, 2022

We chatted about industry trends, why decentralization has become a hot topic in the data world, and how metadata drives many data-centric use cases. But, through it all, Mohan says it’s critical to view everything through the same lens: gaining business value from data. Data fabric is a technology architecture.

Metadata 52

Metadata Data Warehouse Data Quality Data Lake 52

AWS Big Data

NOVEMBER 16, 2023

Amazon Redshift is a popular cloud data warehouse, offering a fully managed cloud-based service that seamlessly integrates with an organization’s Amazon Simple Storage Service (Amazon S3) data lake, real-time streams, machine learning (ML) workflows, transactional workflows, and much more—all while providing up to 7.9x

Enterprise 111

Enterprise Data Warehouse Data Lake Optimization 111

Cloudera

DECEMBER 9, 2022

Developers need to onboard new data sources, chain multiple data transformation steps together, and explore data as it travels through the flow. What if there was a way to not require developers to manage their own Apache NiFi installation without putting that burden on platform administrators?

Testing 100

Testing Cost-Benefit Interactive Visualization 100

Octopai

JANUARY 19, 2021

This is done by visualizing the Azure Data Factory pipelines’ full column-level with source-to-target traceability through different data transformations at the most detailed level. Octopai can fully map the BI landscape and trace metadata movement in a mixed environment including complex multi-vendor landscapes.

Metadata 52

Metadata ROI Machine Learning Data Quality 52

AWS Big Data

OCTOBER 11, 2023

Due to this low complexity, the solution uses AWS serverless services to ingest the data, transform it, and make it available for analytics. The architecture uses AWS Lambda , a serverless, event-driven compute service that lets you run code without provisioning or managing servers. On the Datasets page, choose New data set.

Visualization 119

Visualization Dashboards Data-driven Gap analysis 119

Octopai

AUGUST 11, 2024

It guides data professionals through the complexities of data management, much like a reliable navigation app guides drivers through the complexities of road travel. Pros and Cons As a global leader in automated data flow lineage, Octopai brings extensive experience to the deepest levels of data management.

Metadata 40

Metadata Visualization Data Quality Data Governance 40

AWS Big Data

APRIL 19, 2023

Data mesh is a new approach to data management. Companies across industries are using a data mesh to decentralize data management to improve data agility and get value from data. A modern data architecture is critical in order to become a data-driven organization.

Technology 105

Technology Data-driven Machine Learning Sales 105

IBM Big Data Hub

MARCH 14, 2024

Businesses face significant hurdles when preparing data for artificial intelligence (AI) applications. The existence of data silos and duplication, alongside apprehensions regarding data quality, presents a multifaceted environment for organizations to manage.

Cost-Benefit 75

Cost-Benefit Metadata Optimization Management 75

Alation

MARCH 22, 2022

In fact, DataOps (and its underlying foundation, Data Fabric ), has ranked as a top area of inquiry and interest with analysts over the past twelve months. DataOps requires an array of technology to automate the design, development, deployment, and management of data delivery, with governance sprinkled on for good measure.

Metadata 52

Metadata Cost-Benefit Data Quality Data Lake 52

AWS Big Data

OCTOBER 18, 2023

To look for fraud, market manipulation, insider trading, and abuse, FINRA’s technology group has developed a robust set of big data tools in the AWS Cloud to support these activities. InstanceId[]' Go to the Amazon EC2 console and connect to the master node through the Session Manager. or later installed. Instances[].InstanceId[]'

Big Data 105

Big Data Data Processing Interactive Testing 105

AWS Big Data

FEBRUARY 6, 2023

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) is a managed orchestration service for Apache Airflow that makes it simple to set up and operate end-to-end data pipelines in the cloud at scale. format(S3_BUCKET_NAME), 's3://{}/data/aggregated/green'.format(S3_BUCKET_NAME),

Management 98

Management Interactive Publishing Metadata 98

Octopai

JUNE 9, 2024

These data products were intended to enhance patient outcomes, streamline hospital operations, and provide actionable insights for decision-making. This strategic choice justified further investment into their data team, infrastructure, management, and science. The complexity of their data ecosystem became a major obstacle.

IT 52

IT Data-driven Predictive Analytics Data Strategy 52