AWS Big Data

DECEMBER 13, 2023

In addition to using native managed AWS services that BMS didn’t need to worry about upgrading, BMS was looking to offer an ETL service to non-technical business users that could visually compose data transformation workflows and seamlessly run them on the AWS Glue Apache Spark-based serverless data integration engine.

Metadata 107

Metadata Data Lake Visualization Data Quality 107

Alation

JUNE 28, 2022

This is where metadata, or the data about data, comes into play. Having a data catalog is the cornerstone of your data governance strategy, but what supports your data catalog? Your metadata management framework provides the underlying structure that makes your data accessible and manageable.

Metadata 52

Metadata Management Data Governance Machine Learning 52

Wayne Yaddow

MARCH 5, 2025

In this post, well see the fundamental procedures, tools, and techniques that data engineers, data scientists, and QA/testing teams use to ensure high-quality data as soon as its deployed. First, we look at how unit and integration tests uncover transformation errors at an early stage. using Docker or local runners).

Testing 52

Testing Data Transformation Statistics Metadata 52

AWS Big Data

OCTOBER 11, 2023

The lift and shift migration approach is limited in its ability to transform businesses because it relies on outdated, legacy technologies and architectures that limit flexibility and slow down productivity. It shows a call center streaming data source that sends the latest call center feed in every 15 seconds.

Management 124

Management Metadata Analytics Dashboards 124

Alation

OCTOBER 18, 2022

Data analysts and engineers use dbt to transform, test, and document data in the cloud data warehouse. Yet every dbt transformation contains vital metadata that is not captured – until now. Data Transformation in the Modern Data Stack. How did the data transform exactly?

Metadata 52

Metadata Metrics Recreation/Entertainment Data Quality 52

erwin

SEPTEMBER 17, 2020

Replace manual and recurring tasks for fast, reliable data lineage and overall data governance. It’s paramount that organizations understand the benefits of automating end-to-end data lineage. The importance of end-to-end data lineage is widely understood and ignoring it is risky business. Doing Data Lineage Right.

Cost-Benefit 143

Cost-Benefit Data Governance Metadata Reporting 143

erwin

APRIL 29, 2020

Data lineage is the journey data takes from its creation through its transformations over time. Tracing the source of data is an arduous task. With all these diverse data sources, and if systems are integrated, it is difficult to understand the complicated data web they form much less get a simple visual flow.

Key Performance Indicator 111

Key Performance Indicator Metadata Data Governance Data Quality 111

AWS Big Data

OCTOBER 23, 2024

It’s a set of HTTP endpoints to perform operations such as invoking Directed Acyclic Graphs (DAGs), checking task statuses, retrieving metadata about workflows, managing connections and variables, and even initiating dataset-related events, without directly accessing the Airflow web interface or command line tools.

Interactive 108

Interactive Testing Data-driven Data Lake 108

AWS Big Data

NOVEMBER 15, 2023

It seamlessly consolidates data from various data sources within AWS, including AWS Cost Explorer (and forecasting with Cost Explorer ), AWS Trusted Advisor , and AWS Compute Optimizer. Overview of the BMW Cloud Data Hub At the BMW Group, Cloud Data Hub (CDH) is the central platform for managing company-wide data and data solutions.

Dashboards 110

Dashboards Analytics Metadata Data Warehouse 110

AWS Big Data

AUGUST 26, 2024

You will learn how to prepare a multi-account environment to access the databases from AWS Glue, and how to model an ETL data flow that automatically masks PII as part of the transfer process, so that no sensitive information will be copied to the target database in its original form. 16 10.2.10.0/24 24 AWS Glue account 10.1.0.0/16

Visualization 113

Visualization Metadata Data Transformation Testing 113

AWS Big Data

DECEMBER 16, 2024

With the ability to browse metadata, you can understand the structure and schema of the data source, identify relevant tables and fields, and discover useful data assets you may not be aware of. Choose Add data. Prerequisites Before you begin, make sure you have the followings: An AWS account. Choose Save changes.

Visualization 79

Visualization Data Processing Testing Publishing 79

Smart Data Collective

OCTOBER 13, 2021

There are countless examples of big data transforming many different industries. There is no disputing the fact that the collection and analysis of massive amounts of unstructured data has been a huge breakthrough. We would like to talk about data visualization and its role in the big data movement.

Visualization 145

Visualization Cost-Benefit Big Data Prescriptive Analytics 145

Alation

OCTOBER 26, 2021

These organizations that build an organization-wide data culture enjoy clear and compelling benefits. In our recent State of Data Culture Report , Alation found that nearly every organization (86%) with a top-tier data culture met or exceeded its revenue targets. Building a Data Culture Within a Finance Department.

Finance 52

Finance Data Transformation Enterprise Metrics 52

erwin

MARCH 4, 2021

Modern data governance is a strategic, ongoing and collaborative practice that enables organizations to discover and track their data, understand what it means within a business context, and maximize its security, quality and value. The What: Data Governance Defined. Data governance has no standard definition.

Data Governance 122

Data Governance Digital Transformation Data-driven Cost-Benefit 122

Octopai

NOVEMBER 23, 2020

The LIBOR transition is, first and foremost, a data challenge – and it’s one with a concrete deadline that companies can’t afford to miss. In fact, the LIBOR transition program marks one of the largest data transformation obstacles ever seen in financial services. Like any data project, it boils down to the details.

Metadata 72

Metadata Enterprise Business Intelligence Data Governance 72

Cloudera

OCTOBER 7, 2022

We’re excited to announce the general availability of the open source adapters for dbt for all the engines in CDP — Apache Hive , Apache Impala , and Apache Spark, with added support for Apache Livy and Cloudera Data Engineering. The Open Data Lakehouse . Cloudera builds dbt adaptors for all engines in the open data lakehouse.

Data Warehouse 81

Data Warehouse Data Transformation Machine Learning Data Lake 81

Octopai

JUNE 9, 2024

.” The Data Strategy HealthCo, like many forward-thinking organizations, recognized early on that data is not just a valuable asset but a strategic imperative. They put data at the forefront of their business, integrating it into decision-making processes, products, and services. The lack of trust in data created inertia.

IT 52

IT Data-driven Predictive Analytics Data Strategy 52

Cloudera

JUNE 30, 2022

These tools empower analysts and data scientists to easily collaborate on the same data, with their choice of tools and analytic engines. No more lock-in, unnecessary data transformations, or data movement across tools and clouds just to extract insights out of the data.

Data Lake 88

Data Lake Data Warehouse Data Architecture Metadata 88

AWS Big Data

DECEMBER 5, 2023

This frees up our local computer space, greatly automates the survey cleaning and analysis step, and allows our clients to easily access the data results. .” — Harman Singh Dhodi, Analyst at HR&A Advisors, Inc. A combination of Amazon Redshift Spectrum and COPY commands are used to ingest the survey data stored as CSV files.

Measurement 100

Measurement Dashboards Data Warehouse Analytics 100

AWS Big Data

APRIL 12, 2023

The solution uses the TPC-DS dataset and unmodified data schema and table relationships, but derives queries from TPC-DS to support the SparkSQL test cases. Metadata store – We use Spark’s in-memory data catalog to store metadata for TPC-DS databases and tables— spark.sql.catalogImplementation is set to the default value in-memory.

Testing 93

Testing Big Data Metadata Optimization 93

IBM Big Data Hub

JANUARY 9, 2024

The emergence of generative AI prompted several prominent companies to restrict its use because of the mishandling of sensitive internal data. Currently, no standardized process exists for overcoming data ingestion’s challenges, but the model’s accuracy depends on it. Increased variance: Variance measures consistency.

Enterprise 71

Enterprise Data Integration Data Quality Contextual Data 71

Alation

APRIL 4, 2023

Now, joint users will get an enhanced view into cloud and data transformations , with valuable context to guide smarter usage. Integrating helpful metadata into user workflows gives all people, from data scientists to analysts , the context they need to use data more effectively. How was it used in the past?

Metadata 52

Metadata Cost-Benefit Data Transformation Predictive Modeling 52

AWS Big Data

MARCH 13, 2024

The solution consists of the following interfaces: IoT or mobile application – A mobile application or an Internet of Things (IoT) device allows the tracking of a company vehicle while it is in use and transmits its current location securely to the data ingestion layer in AWS. The ingestion approach is not in scope of this post. Choose Run.

Analytics 116

Analytics IoT Metadata Internet of Things 116

Ontotext

NOVEMBER 18, 2021

You step onto the market, and if you don’t keep your data, there’s no knowing where you might be swept off to. [1]. Picture this – you start with the perfect use case for your data analytics product. Nowadays, data analytics doesn’t exist on its own. You make a great pitch and you sell well. Sounds unlikely.

Visualization 98

Visualization Reporting Metadata Enterprise 98

AWS Big Data

MARCH 9, 2023

Amazon QuickSight is a fully managed, cloud-native business intelligence (BI) service that makes it easy to connect to your data, create interactive dashboards and reports, and share these with tens of thousands of users, either within QuickSight or embedded in your application or website.

Slice and Dice 94

Slice and Dice Dashboards Analytics Interactive 94

Alation

AUGUST 11, 2022

DataOps sprung up to connect data sources to data consumers. The data warehouse and analytical data stores moved to the cloud and disaggregated into the data mesh. We chatted about industry trends, why decentralization has become a hot topic in the data world, and how metadata drives many data-centric use cases.

Metadata 52

Metadata Data Warehouse Data Quality Data Lake 52

IBM Big Data Hub

OCTOBER 16, 2023

It includes processes that trace and document the origin of data, models and associated metadata and pipelines for audits. Curated foundation models, such as those created by IBM or Microsoft, help enterprises scale and accelerate the use and impact of the most advanced AI capabilities using trusted data.

Risk 70

Risk Modeling Management Metadata 70

AWS Big Data

OCTOBER 11, 2023

Due to this low complexity, the solution uses AWS serverless services to ingest the data, transform it, and make it available for analytics. AWS Glue is a serverless data integration service that makes it easier to discover, prepare, move, and integrate data from multiple sources for analytics, ML, and application development.

Visualization 116

Visualization Dashboards Data-driven Gap analysis 116

Octopai

JANUARY 19, 2021

Octopai is the first BI Intelligence Platform in the Industry to Support Azure Data Factory, Providing Full Lineage of Advanced BI Tools. This is done by visualizing the Azure Data Factory pipelines’ full column-level with source-to-target traceability through different data transformations at the most detailed level.

Metadata 52

Metadata ROI Machine Learning Data Quality 52

AWS Big Data

NOVEMBER 16, 2023

Amazon Redshift is a popular cloud data warehouse, offering a fully managed cloud-based service that seamlessly integrates with an organization’s Amazon Simple Storage Service (Amazon S3) data lake, real-time streams, machine learning (ML) workflows, transactional workflows, and much more—all while providing up to 7.9x Data Vault 2.0

Enterprise 105

Enterprise Data Warehouse Data Lake Optimization 105

Octopai

AUGUST 11, 2024

Just as a navigation app provides a detailed map of roads, guiding you from your starting point to your destination while highlighting every turn and intersection, data flow lineage offers a comprehensive view of data movement and transformations throughout its lifecycle. Open Source Data Lineage Tools 1.

Metadata 40

Metadata Visualization Data Quality Data Governance 40

Cloudera

DECEMBER 9, 2022

Cloudera has been providing enterprise support for Apache NiFi since 2015, helping hundreds of organizations take control of their data movement pipelines on premises and in the public cloud. Developers need to onboard new data sources, chain multiple data transformation steps together, and explore data as it travels through the flow.

Testing 100

Testing Cost-Benefit Interactive Visualization 100

AWS Big Data

JULY 20, 2023

The Orca Platform is powered by a state-of-the-art anomaly detection system that uses cutting-edge ML algorithms and big data capabilities to detect potential security threats and alert customers in real time, ensuring maximum security for their cloud environment. Why did Orca build a data lake? Why did Orca choose Apache Iceberg?

Data Lake 97

Data Lake Analytics Snapshot Data Quality 97

Cloudera

MARCH 14, 2023

We just announced the general availability of Cloudera DataFlow Designer , bringing self-service data flow development to all CDP Public Cloud customers. In this blog post we will put these capabilities in context and dive deeper into how the built-in, end-to-end data flow life cycle enables self-service data pipeline development.

Testing 84

Testing Publishing Metadata Interactive 84

AWS Big Data

APRIL 19, 2023

This allows data consumers to easily identify new datasets and provides agility and innovation without spending hours doing analysis and research. Background The success of a data-driven organization recognizes data as a key enabler to increase and sustain innovation. It follows what is called a distributed system architecture.

Technology 100

Technology Data-driven Machine Learning Sales 100

erwin

JANUARY 11, 2019

Organizations have spent a lot of time and money trying to harmonize data across diverse platforms , including cleansing, uploading metadata, converting code, defining business glossaries, tracking data transformations and so on. So questions linger about whether transformed data can be trusted.

Data Governance 40

Data Governance Risk Metadata Management 40

Alation

MARCH 22, 2022

As the latest iteration in this pursuit of high-quality data sharing, DataOps combines a range of disciplines. It synthesizes all we’ve learned about agile, data quality , and ETL/ELT. Simply put, IDF standardizes data engineering processes. How the IDF Supports a Smarter Data Pipeline.

Metadata 52

Metadata Cost-Benefit Data Quality Data Lake 52