AWS Big Data

OCTOBER 11, 2023

Due to this low complexity, the solution uses AWS serverless services to ingest the data, transform it, and make it available for analytics. The data ingestion process copies the machine-readable files from the hospitals, validates the data, and keeps the validated files available for analysis.

Visualization 119

Visualization Dashboards Data-driven Gap analysis 119

Alation

AUGUST 11, 2022

We chatted about industry trends, why decentralization has become a hot topic in the data world, and how metadata drives many data-centric use cases. But, through it all, Mohan says it’s critical to view everything through the same lens: gaining business value from data. Data fabric is a technology architecture.

Metadata 52

Metadata Data Warehouse Data Quality Data Lake 52

erwin

JANUARY 11, 2019

Organizations have spent a lot of time and money trying to harmonize data across diverse platforms , including cleansing, uploading metadata, converting code, defining business glossaries, tracking data transformations and so on. And there’s control of that landscape to facilitate insight and collaboration and limit risk.

Data Governance 40

Data Governance Risk Metadata Management 40

Ontotext

MARCH 27, 2024

For example, GPS, social media, cell phone handoffs are modeled as graphs while data catalogs, data lineage and MDM tools leverage knowledge graphs for linking metadata with semantics. RDF is used extensively for data publishing and data interchange and is based on W3C and other industry standards.

Modeling 53

Modeling Metadata Data Quality Enterprise 53

AWS Big Data

JULY 20, 2023

Specifically, the system uses Amazon SageMaker Processing jobs to process the data stored in the data lake, employing the AWS SDK for Pandas (previously known as AWS Wrangler) for various data transformation operations, including cleaning, normalization, and feature engineering.

Data Lake 98

Data Lake Analytics Snapshot Data Quality 98

Octopai

JUNE 9, 2024

By embracing automated, multidimensional data lineage, HealthCo was able to maintain a consistent and reliable data environment across their hybrid systems. Solving the data lineage problem directly supported their data products by ensuring data integrity and reliability. This is where Octopai excels.

IT 52

IT Data-driven Predictive Analytics Data Strategy 52

Alation

JANUARY 17, 2023

What if, experts asked, you could load raw data into a warehouse, and then empower people to transform it for their own unique needs? Today, data integration platforms like Rivery do just that. By pushing the T to the last step in the process, such products have revolutionized how data is understood and analyzed.

Data Warehouse 52

Data Warehouse Cost-Benefit Data Science Data Transformation 52

AWS Big Data

NOVEMBER 29, 2023

dbt is an open source, SQL-first templating engine that allows you to write repeatable and extensible data transforms in Python and SQL. dbt is predominantly used by data warehouses (such as Amazon Redshift ) customers who are looking to keep their data transform logic separate from storage and engine.

Data Lake 122

Data Lake Management Metrics Data Warehouse 122

AWS Big Data

MARCH 15, 2023

The API retrieves data at runtime from an Amazon Aurora PostgreSQL-Compatible Edition database for end-user consumption. To populate the database, the Infomedia team developed a data pipeline using Amazon Simple Storage Service (Amazon S3) for data storage, AWS Glue for data transformations, and Apache Hudi for CDC and record-level updates.

Cost-Benefit 94

Cost-Benefit Data Processing Optimization Data-driven 94

Octopai

NOVEMBER 13, 2022

The modern data stack is a data management system built out of cloud-based data systems. A given modern data stack will usually include components for data ingestion from your data sources, data transformation, data storage, data analysis and reporting.

Enterprise 97

Enterprise Data Warehouse Reporting Metadata 97

Cloudera

NOVEMBER 22, 2021

Rise in polyglot data movement because of the explosion in data availability and the increased need for complex data transformations (due to, e.g., different data formats used by different processing frameworks or proprietary applications). As a result, alternative data integration technologies (e.g.,

Data Processing 94

Data Processing Data Warehouse Enterprise Visualization 94

BI-Survey

MARCH 6, 2025

While efficiency is a priority, data quality and security remain non-negotiable. Developing and maintaining data transformation pipelines are among the first tasks to be targeted for automation. However, caution is advised since accuracy, timeliness, and other aspects of data quality depend on the quality of data pipelines.

Data Warehouse 52

Data Warehouse Metadata Unstructured Data Data-driven 52

AWS Big Data

DECEMBER 2, 2024

More companies have realized there is an opportunity to integrate, enhance, and present this SaaS data to improve internal operations and gain valuable insights on their data. From there, they can perform meaningful analytics, gain valuable insights, and optionally push enriched data back to external SaaS platforms.

Data Lake 105

Data Lake Testing Data Integration Metadata 105

AWS Big Data

JANUARY 29, 2025

To avoid this situation, Oktank aims to decouple compute from storage, allowing them to scale down compute nodes and repurpose them for other workloads without compromising data integrity and accessibility. The Data Catalog is a centralized metadata repository for all your data assets across various data sources.

Big Data 91

Big Data Data Analytics Analytics Interactive 91

DataKitchen

JANUARY 23, 2023

To capture a more complete picture of the data’s journey, it is important to have a DataOps Observability system in place. Data lineage is static and often lags by weeks or months. Data lineage is often considered static because it is typically based on snapshots of data and metadata taken at a specific time.

Testing 130

Testing Data Quality Data Governance Data-driven 130

Jet Global

MAY 1, 2023

Requirement Multi-Source Data Blending Data from multiple sources is compiled and the output is a single view, metric, or visualization. Data Transformation and Enrichment Data can be enriched for analysis. addresses).

Analytics 52

Analytics Cost-Benefit Visualization Dashboards 52

Jet Global

FEBRUARY 23, 2024

Data mapping is essential for integration, migration, and transformation of different data sets; it allows you to improve your data quality by preventing duplications and redundancies in your data fields. Data mapping is important for several reasons.

Data Warehouse 52

Data Warehouse Reporting Data Transformation Visualization 52

Jet Global

MARCH 14, 2024

Jet streamlines many aspects of data administration, greatly improving data solutions built on Microsoft Fabric. It enhances analytics capabilities, streamlines migration, and enhances data integration. Through Jet’s integration with Fabric, your organization can better handle, process, and use your data.

Analytics 52

Analytics Management Reporting Data Quality 52