CIO Business Intelligence

MARCH 24, 2023

Whereas data governance is about the roles, responsibilities, and processes for ensuring accountability for and ownership of data assets, DAMA defines data management as “an overarching term that describes the processes used to plan, specify, enable, create, acquire, maintain, use, archive, retrieve, control, and purge data.”

Data Governance 142

Data Governance Management Metadata Data Quality 142

AWS Big Data

APRIL 3, 2024

For instructions, refer to Amazon DataZone quickstart with AWS Glue data. You also need to define and run a ruleset against your data, which is a set of data quality rules in AWS Glue Data Quality. The ruleset contains 27 individual rules (one of them failing), so the overall data quality score is 96%.

Data Quality 93

Data Quality Visualization Metadata Metrics 93

AWS Big Data

JUNE 6, 2023

These are six main steps in the data pipeline: Amazon EventBridge triggers an AWS Lambda function when the event pattern for AWS Glue Data Quality matches the defined rule. For more information, refer to Working with Query Results, Output Files, and Query History. For S3 path , enter the S3 path to your data source. (

Data Quality 71

Data Quality Metrics Visualization Dashboards 71

AWS Big Data

JUNE 29, 2023

Tracking such user queries as part of the centralized governance of the data warehouse helps stakeholders understand potential risks and take prompt action to mitigate them following the operational excellence pillar of the AWS Data Analytics Lens. Test the filter by selecting the actual log stream.

Data Warehouse 60

Data Warehouse Dashboards Testing Visualization 60

AWS Big Data

SEPTEMBER 11, 2024

This post describes how HPE Aruba automated their Supply Chain management pipeline, and re-architected and deployed their data solution by adopting a modern data architecture on AWS. The new solution has helped Aruba integrate data from multiple sources, along with optimizing their cost, performance, and scalability.

Data Architecture 86

Data Architecture Optimization Data Warehouse Metadata 86

AWS Big Data

JUNE 12, 2024

Swisscom’s Data, Analytics, and AI division is building a One Data Platform (ODP) solution that will enable every Swisscom employee, process, and product to benefit from the massive value of Swisscom’s data. The following high-level architecture diagram shows ODP with different layers of the modern data architecture.

Data Architecture 96

Data Architecture Cost-Benefit Experimentation Data-driven 96

AWS Big Data

AUGUST 1, 2024

Success criteria alignment by all stakeholders (producers, consumers, operators, auditors) is key for successful transition to a new Amazon Redshift modern data architecture. The success criteria are the key performance indicators (KPIs) for each component of the data workflow.

Data Warehouse 89

Data Warehouse KPI Optimization Cost-Benefit 89

AWS Big Data

FEBRUARY 7, 2024

Refer to How can I access OpenSearch Dashboards from outside of a VPC using Amazon Cognito authentication for a detailed evaluation of the available options and the corresponding pros and cons. For more information, refer to the AWS CDK v2 Developer Guide. For instructions, refer to Creating a public hosted zone.

Dashboards 85

Dashboards Data Processing Metadata Consulting 85

DataKitchen

OCTOBER 11, 2021

First, you must understand the existing challenges of the data team, including the data architecture and end-to-end toolchain. The biggest challenge is broken data pipelines due to highly manual processes. Figure 1 shows a manually executed data analytics pipeline.

Testing 152

Testing Metadata Dashboards Statistics 152

AWS Big Data

OCTOBER 18, 2023

Many customers migrate their data warehousing workloads to Amazon Redshift and benefit from the rich capabilities it offers, such as the following: Amazon Redshift seamlessly integrates with broader data, analytics, and AI or machine learning (ML) services on AWS , enabling you to choose the right tool for the right job.

Analytics 99

Analytics Data Warehouse Dashboards Testing 99

AWS Big Data

MARCH 28, 2023

Test SCD Type 2 implementation With the infrastructure in place, you’re ready to test out the overall solution design and query historical records from the employee dataset. This post is designed to be implemented for a real customer use case, where you get full snapshot data on a daily basis.

Data Lake 72

Data Lake Testing Snapshot Sales 72

AWS Big Data

MAY 30, 2024

It also helps you securely access your data in operational databases, data lakes, or third-party datasets with minimal movement or copying of data. Tens of thousands of customers use Amazon Redshift to process large amounts of data, modernize their data analytics workloads, and provide insights for their business users.

Data Warehouse 89

Data Warehouse Data Lake Cost-Benefit Structured Data 89

AWS Big Data

NOVEMBER 13, 2023

The downstream consumers consist of business intelligence (BI) tools, with multiple data science and data analytics teams having their own WLM queues with appropriate priority values. Consequently, there was a fivefold rise in data integrations and a fivefold increase in ad hoc queries submitted to the Redshift cluster.

Data Warehouse 95

Data Warehouse Data Lake Analytics Data Science 95

AWS Big Data

SEPTEMBER 7, 2023

They store attributes such as object size, total time, turn-around time, and HTTP referer for log records. In our test bed with 470 GB (1,544,692 objects) of access logs, large Spark drivers using AWS Glue’s G.8X Ray job run details: Please feel free to download the script and test this solution in your development environment.

Metadata 103

Metadata Dashboards Metrics Visualization 103

AWS Big Data

JUNE 9, 2023

A modern data architecture enables companies to ingest virtually any type of data through automated pipelines into a data lake, which provides highly durable and cost-effective object storage at petabyte or exabyte scale. This post is not intended to provide detailed technical guidance (e.g.

Data Lake 117

Data Lake Metadata Optimization Statistics 117

DataKitchen

JANUARY 23, 2023

DataOps Observability includes monitoring and testing the data pipeline, data quality, data testing, and alerting. Data testing is an essential aspect of DataOps Observability; it helps to ensure that data is accurate, complete, and consistent with its specifications, documentation, and end-user requirements.

Testing 130

Testing Data Governance Data Quality Data-driven 130

Jet Global

MAY 1, 2023

that gathers data from many sources. Third-party data might include industry benchmarks, data feeds (such as weather and social media), and/or anonymized customer data. Four Approaches to Data Analytics The world of data analytics is constantly and quickly changing. It’s all about context.

Analytics 52

Analytics Cost-Benefit Visualization Dashboards 52