CIO Business Intelligence

APRIL 25, 2024

“The challenge that a lot of our customers have is that requires you to copy that data, store it in Salesforce; you have to create a place to store it; you have to create an object or field in which to store it; and then you have to maintain that pipeline of data synchronization and make sure that data is updated,” Carlson said.

Data Integration 119

Data Integration Data Lake Data Warehouse Metadata 119

AWS Big Data

AUGUST 15, 2024

Unlocking the true value of data often gets impeded by siloed information. Traditional data management—wherein each business unit ingests raw data in separate data lakes or warehouses—hinders visibility and cross-functional analysis. Amazon DataZone natively supports data sharing for Amazon Redshift data assets.

Data Lake 109

Data Lake Data Warehouse Data Governance Publishing 109

AWS Big Data

JANUARY 15, 2025

In addition to real-time analytics and visualization, the data needs to be shared for long-term data analytics and machine learning applications. To achieve this, EUROGATE designed an architecture that uses Amazon DataZone to publish specific digital twin data sets, enabling access to them with SageMaker in a separate AWS account.

IoT 101

IoT Machine Learning Metadata Data-driven 101

AWS Big Data

JUNE 10, 2024

Use cases for Hive metastore federation for Amazon EMR Hive metastore federation for Amazon EMR is applicable to the following use cases: Governance of Amazon EMR-based data lakes – Producers generate data within their AWS accounts using an Amazon EMR-based data lake supported by EMRFS on Amazon Simple Storage Service (Amazon S3)and HBase.

Data Lake 96

Data Lake Metadata Data Warehouse Data Processing 96

CIO Business Intelligence

APRIL 8, 2025

These strategies, such as investing in AI-powered cleansing tools and adopting federated governance models, not only address the current data quality challenges but also pave the way for improved decision-making, operational efficiency and customer satisfaction. When financial data is inconsistent, reporting becomes unreliable.

Data Quality 102

Data Quality Data-driven Key Performance Indicator Metadata 102

AWS Big Data

JULY 20, 2023

With data becoming the driving force behind many industries today, having a modern data architecture is pivotal for organizations to be successful. In this post, we describe Orca’s journey building a transactional data lake using Amazon Simple Storage Service (Amazon S3), Apache Iceberg, and AWS Analytics.

Data Lake 94

Data Lake Analytics Snapshot Data Quality 94

AWS Big Data

JANUARY 9, 2025

These features allow efficient data corrections, gap-filling in time series, and historical data updates without disrupting ongoing analyses or compromising data integrity. Unlike direct Amazon S3 access, Iceberg supports these operations on petabyte-scale data lakes without requiring complex custom code.

Metadata 107

Metadata Snapshot Cost-Benefit Optimization 107

AWS Big Data

JULY 29, 2024

In the era of big data, data lakes have emerged as a cornerstone for storing vast amounts of raw data in its native format. They support structured, semi-structured, and unstructured data, offering a flexible and scalable environment for data ingestion from multiple sources.

Metadata 119

Metadata Snapshot Data Lake Metrics 119

AWS Big Data

OCTOBER 21, 2024

However, enterprises often encounter challenges with data silos, insufficient access controls, poor governance, and quality issues. Embracing data as a product is the key to address these challenges and foster a data-driven culture. To achieve this, they plan to use machine learning (ML) models to extract insights from data.

Sales 105

Sales Data-driven Data Processing Key Performance Indicator 105

erwin

AUGUST 31, 2023

Q: Is data modeling cool again? In today’s fast-paced digital landscape, data reigns supreme. The data-driven enterprise relies on accurate, accessible, and actionable information to make strategic decisions and drive innovation. A: It always was and is getting cooler!!

Data-driven 98

Data-driven Modeling Enterprise Structured Data 98

CIO Business Intelligence

AUGUST 9, 2024

The original proof of concept was to have one data repository ingesting data from 11 sources, including flat files and data stored via APIs on premises and in the cloud, Pruitt says. There are a lot of variables that determine what should go into the data lake and what will probably stay on premise,” Pruitt says.

Data Transformation 126

Data Transformation Machine Learning Data Lake Dashboards 126

AWS Big Data

NOVEMBER 29, 2023

We have seen a strong customer demand to expand its scope to cloud-based data lakes because data lakes are increasingly the enterprise solution for large-scale data initiatives due to their power and capabilities. The team uses dbt-glue to build a transformed gold model optimized for business intelligence (BI).

Data Lake 117

Data Lake Management Metrics Data Warehouse 117

erwin

OCTOBER 8, 2020

But even with the “need for speed” to market, new applications must be modeled and documented for compliance, transparency and stakeholder literacy. The metadata-driven suite automatically finds, models, ingests, catalogs and governs cloud data assets. Subscribe to the erwin Expert Blog.

Data Governance 145

Data Governance Metadata Testing Data Lake 145

CIO Business Intelligence

AUGUST 3, 2022

Modern delivery is product (rather than project) management , agile development, small cross-functional teams that co-create , and continuous integration and delivery all with a new financial model that funds “value” not “projects.”. This model allows us to pivot from a data-defensive to a data-offensive position.”.

Finance 145

Finance IoT Digital Transformation Sales 145

CIO Business Intelligence

APRIL 25, 2022

The data lakehouse is a relatively new data architecture concept, first championed by Cloudera, which offers both storage and analytics capabilities as part of the same solution, in contrast to the concepts for data lake and data warehouse which, respectively, store data in native format, and structured data, often in SQL format.

Recreation/Entertainment 119

Recreation/Entertainment Data Lake Data Warehouse Unstructured Data 119

AWS Big Data

FEBRUARY 26, 2025

Analytics remained one of the key focus areas this year, with significant updates and innovations aimed at helping businesses harness their data more efficiently and accelerate insights. From enhancing data lakes to empowering AI-driven analytics, AWS unveiled new tools and services that are set to shape the future of data and analytics.

Analytics 59

Analytics Data Lake Metadata Data Warehouse 59

CIO Business Intelligence

APRIL 29, 2022

This would be straightforward task were it not for the fact that, during the digital-era, there has been an explosion of data – collected and stored everywhere – much of it poorly governed, ill-understood, and irrelevant. Many organisations focus too heavily on fine tuning their computational models in their pursuit of ‘quick-wins.’

Data Governance 145

Data Governance IT Risk Data Lake 145

CIO Business Intelligence

FEBRUARY 20, 2024

Recent research by McGuide Research Services for Avanade found 91% of organisations in the sector believe they need to shift to an AI-first operating model within the next 12 months, while 87% of employees feel generative AI tools will make them more efficient, and more innovative. This requires skillsets that firms may not have in-house.

Cost-Benefit 115

Cost-Benefit Data Lake Data Warehouse Data Processing 115

AWS Big Data

NOVEMBER 29, 2023

Zero-ETL integration also enables you to load and analyze data from multiple operational database clusters in a new or existing Amazon Redshift instance to derive holistic insights across many applications. Learn more about the zero-ETL integrations, data lake performance enhancements, and other announcements below.

Data Warehouse 113

Data Warehouse Analytics Data Lake Machine Learning 113

AWS Big Data

NOVEMBER 29, 2024

Customers often want to augment and enrich SAP source data with other non-SAP source data. Such analytic use cases can be enabled by building a data warehouse or data lake. Customers can now use the AWS Glue SAP OData connector to extract data from SAP.

Visualization 91

Visualization Data Processing Data-driven Cost-Benefit 91

AWS Big Data

JULY 19, 2023

You can structure your data, measure business processes, and get valuable insights quickly can be done by using a dimensional model. Amazon Redshift provides built-in features to accelerate the process of modeling, orchestrating, and reporting from a dimensional model. Declare the grain of your data.

Modeling 76

Modeling Sales Data Warehouse Snapshot 76

AWS Big Data

NOVEMBER 29, 2023

This includes tools to help you customize your foundation models, and new services and features to build a strong data foundation to fuel your generative AI applications. Customizing foundation models The need for data is quite obvious if you are building your own foundation models (FMs).

Data Warehouse 116

Data Warehouse Data Lake Dashboards Data Integration 116

erwin

JULY 17, 2019

Part Two of the Digital Transformation Journey … In our last blog on driving digital transformation , we explored how enterprise architecture (EA) and business process (BP) modeling are pivotal factors in a viable digital transformation strategy. Digital Transformation Strategy: Smarter Data.

Digital Transformation 98

Digital Transformation Strategy Metadata Data-driven 98

AWS Big Data

NOVEMBER 10, 2023

As a result of utilizing the Amazon Redshift integration for Apache Spark, developer productivity increased by a factor of 10, feature generation pipelines were streamlined, and data duplication reduced to zero. These tables are then joined with tables from the Enterprise Data Lake (EDL) at runtime. cast("string")).dropDuplicates())

Data Processing 105

Data Processing Data Lake Data Warehouse Optimization 105

AWS Big Data

FEBRUARY 29, 2024

Data is your generative AI differentiator, and a successful generative AI implementation depends on a robust data strategy incorporating a comprehensive data governance approach. Data governance is a critical building block across all these approaches, and we see two emerging areas of focus.

Data Governance 110

Data Governance Unstructured Data Metadata Data Lake 110

BI-Survey

AUGUST 21, 2020

The primary modernization approach is data warehouse/ETL automation, which helps promote broad usage of the data warehouse but can only partially improve efficiency in data management processes. However, an automation approach alone is of limited usefulness when data management processes are inefficient.

Data Warehouse 98

Data Warehouse Data Lake Data Governance Data Architecture 98

AWS Big Data

MAY 3, 2023

In today’s data-driven business environment, organizations face the challenge of efficiently preparing and transforming large amounts of data for analytics and data science purposes. Businesses need to build data warehouses and data lakes based on operational data.

Data Lake 92

Data Lake Data Warehouse Data-driven Optimization 92

AWS Big Data

AUGUST 19, 2024

As organizations increasingly rely on data stored across various platforms, such as Snowflake , Amazon Simple Storage Service (Amazon S3), and various software as a service (SaaS) applications, the challenge of bringing these disparate data sources together has never been more pressing.

Analytics 108

Analytics Data-driven Data Integration Data Lake 108

BI-Survey

APRIL 27, 2023

Compared with laggards, a higher portion of best-in-class companies adopt the data vault, embrace its standards, and intend to expand their use of it. They plan to expand their use of this modeling technique and methodology. The lakehouse, data fabric, and data mesh have 8-12% usage each.

Modeling 52

Modeling Data Warehouse Data Quality Business Driver 52

CIO Business Intelligence

FEBRUARY 16, 2023

Hence the drive to provide ML as a service to the Data & Tech team’s internal customers. All they would have to do is just build their model and run with it,” he says. That step, primarily undertaken by developers and data architects, established data governance and data integration.

Unstructured Data 105

Unstructured Data Data Lake Prescriptive Analytics Data Warehouse 105

Sisense

JULY 22, 2020

The trend has been towards using cloud-based applications and tools for different functions, such as Salesforce for sales, Marketo for marketing automation, and large-scale data storage like AWS or data lakes such as Amazon S3 , Hadoop and Microsoft Azure. Sisense provides instant access to your cloud data warehouses.

Slice and Dice 98

Slice and Dice Digital Transformation Data Warehouse Data Lake 98

AWS Big Data

DECEMBER 21, 2023

As the volume and complexity of analytics workloads continue to grow, customers are looking for more efficient and cost-effective ways to ingest and analyse data. AWS Glue provides both visual and code-based interfaces to make data integration effortless.

Analytics 108

Analytics IT Data Lake Visualization 108

CIO Business Intelligence

MARCH 18, 2024

But when it comes to getting the most value out of hybrid cloud, one of the most crucial capabilities required is data replication and synchronization—what enables businesses to efficiently capture data changes and unify various data stores while ensuring low latency, high availability, and data integrity.

Cost-Benefit 98

Cost-Benefit Data Lake Machine Learning Data Integration 98

CIO Business Intelligence

OCTOBER 1, 2022

It requires taking data from equipment sensors, applying advanced analytics to derive descriptive and predictive insights, and automating corrective actions. The end-to-end process requires several steps, including data integration and algorithm development, training, and deployment.

Manufacturing 145

Manufacturing Digital Transformation IoT Internet of Things 145

AWS Big Data

DECEMBER 18, 2023

Amazon Redshift empowers users to extract powerful insights by securely and cost-effectively analyzing data across data warehouses, operational databases, data lakes, third-party data stores, and streaming sources using zero-ETL approaches.

Data Warehouse 109

Data Warehouse Machine Learning Data-driven Data Lake 109

DataKitchen

DECEMBER 9, 2022

DataOps involves close collaboration between data scientists, IT professionals, and business stakeholders, and it often involves the use of automation and other technologies to streamline data-related tasks. One of the key benefits of DataOps is the ability to accelerate the development and deployment of data-driven solutions.

Machine Learning 130

Machine Learning Data-driven Optimization Data Analytics 130

AWS Big Data

SEPTEMBER 12, 2024

In many organizations, the focus is on large language models (LLMs), and foundation models (FMs) more broadly. This is just the tip of the iceberg, because what enables you to obtain differential value from generative AI is your data. In essence, you have to enrich the generative AI models with your differentiated data.

Management 86

Management Analytics Data Lake Interactive 86