AWS Big Data

JUNE 6, 2023

We are excited to announce the General Availability of AWS Glue Data Quality. Our journey started by working backward from our customers who create, manage, and operate data lakes and data warehouses for analytics and machine learning. It takes days for data engineers to identify and implement data quality rules.

Data Quality 97

Data Quality Statistics Data Lake Visualization 97

AWS Big Data

JULY 20, 2023

With data becoming the driving force behind many industries today, having a modern data architecture is pivotal for organizations to be successful. In this post, we describe Orca’s journey building a transactional data lake using Amazon Simple Storage Service (Amazon S3), Apache Iceberg, and AWS Analytics.

Data Lake 93

Data Lake Analytics Snapshot Data Quality 93

AWS Big Data

JANUARY 15, 2025

In addition to real-time analytics and visualization, the data needs to be shared for long-term data analytics and machine learning applications. To achieve this, EUROGATE designed an architecture that uses Amazon DataZone to publish specific digital twin data sets, enabling access to them with SageMaker in a separate AWS account.

IoT 100

IoT Machine Learning Metadata Data-driven 100

DataKitchen

AUGUST 3, 2021

First-generation – expensive, proprietary enterprise data warehouse and business intelligence platforms maintained by a specialized team drowning in technical debt. Second-generation – gigantic, complex data lake maintained by a specialized team drowning in technical debt.

Data Architecture 246

Data Architecture Data Lake Cost-Benefit Data Warehouse 246

AWS Big Data

APRIL 8, 2025

In modern data architectures, Apache Iceberg has emerged as a popular table format for data lakes, offering key features including ACID transactions and concurrent write support. When combined with well-timed maintenance operations, these patterns help build resilient data pipelines that can handle concurrent writes reliably.

Snapshot 116

Snapshot Management Metadata Big Data 116

Jet Global

JULY 2, 2019

To address the flood of data and the needs of enterprise businesses to store, sort, and analyze that data, a new storage solution has evolved: the data lake. What’s in a Data Lake? Data warehouses do a great job of standardizing data from disparate sources for analysis. Taking a Dip.

Data Lake 73

Data Lake Data Warehouse Big Data Machine Learning 73

AWS Big Data

DECEMBER 4, 2024

Domain ownership recognizes that the teams generating the data have the deepest understanding of it and are therefore best suited to manage, govern, and share it effectively. This principle makes sure data accountability remains close to the source, fostering higher data quality and relevance.

Metadata 93

Metadata Data Governance Data Quality Data-driven 93

CIO Business Intelligence

MARCH 9, 2022

However, they do contain effective data management, organization, and integrity capabilities. As a result, users can easily find what they need, and organizations avoid the operational and cost burdens of storing unneeded or duplicate data copies. On the other hand, they don’t support transactions or enforce data quality.

Data Lake 119

Data Lake Unstructured Data Data Warehouse Big Data 119

AWS Big Data

JUNE 6, 2023

Poor-quality data can lead to incorrect insights, bad decisions, and lost opportunities. AWS Glue Data Quality measures and monitors the quality of your dataset. It supports both data quality at rest and data quality in AWS Glue extract, transform, and load (ETL) pipelines.

Data Quality 75

Data Quality Data Lake Visualization Data-driven 75

AWS Big Data

JULY 29, 2024

In the era of big data, data lakes have emerged as a cornerstone for storing vast amounts of raw data in its native format. They support structured, semi-structured, and unstructured data, offering a flexible and scalable environment for data ingestion from multiple sources.

Metadata 118

Metadata Snapshot Data Lake Metrics 118

IBM Big Data Hub

JANUARY 5, 2023

Poor data quality is one of the top barriers faced by organizations aspiring to be more data-driven. Ill-timed business decisions and misinformed business processes, missed revenue opportunities, failed business initiatives and complex data systems can all stem from data quality issues.

Data Architecture 52

Data Architecture Data Quality Strategy Data Lake 52

erwin

AUGUST 31, 2023

Q: Is data modeling cool again? In today’s fast-paced digital landscape, data reigns supreme. The data-driven enterprise relies on accurate, accessible, and actionable information to make strategic decisions and drive innovation. A: It always was and is getting cooler!!

Data-driven 98

Data-driven Modeling Enterprise Structured Data 98

AWS Big Data

FEBRUARY 29, 2024

Data is your generative AI differentiator, and a successful generative AI implementation depends on a robust data strategy incorporating a comprehensive data governance approach. Data governance is a critical building block across all these approaches, and we see two emerging areas of focus.

Data Governance 109

Data Governance Unstructured Data Metadata Data Lake 109

IBM Big Data Hub

MAY 9, 2023

Data must be laboriously collected, curated, and labeled with task-specific annotations to train AI models. Building a model requires specialized, hard-to-find skills — and each new task requires repeating the process. ” These large models have lowered the cost and labor involved in automation.

Enterprise 128

Enterprise Technology Modeling Cost-Benefit 128

AWS Big Data

FEBRUARY 27, 2024

The following are the key components of the Bluestone Data Platform: Data mesh architecture – Bluestone adopted a data mesh architecture, a paradigm that distributes data ownership across different business units. This enables data-driven decision-making across the organization.

Data-driven 106

Data-driven Data Lake Data Quality Data Governance 106

CIO Business Intelligence

APRIL 29, 2022

This would be straightforward task were it not for the fact that, during the digital-era, there has been an explosion of data – collected and stored everywhere – much of it poorly governed, ill-understood, and irrelevant. Many organisations focus too heavily on fine tuning their computational models in their pursuit of ‘quick-wins.’

Data Governance 145

Data Governance IT Data Lake Risk 145

datapine

APRIL 15, 2020

Business intelligence is moving away from the traditional engineering model: analysis, design, construction, testing, and implementation. In the traditional model communication between developers and business users is not a priority. This is also known as model storming, one of the practices in agile analytics development.

Business Intelligence 276

Business Intelligence Analytics Testing Dashboards 276

AWS Big Data

MARCH 1, 2023

Finding similar columns in a data lake has important applications in data cleaning and annotation, schema matching, data discovery, and analytics across multiple data sources. The workflow begins with an AWS Glue job that converts the CSV files into Apache Parquet data format.

Data Lake 95

Data Lake Deep Learning Interactive Machine Learning 95

BI-Survey

APRIL 27, 2023

While most continue to struggle with data quality issues and cumbersome manual processes, best-in-class companies are making improvements with commercial automation tools. The data vault has strong adherents among best-in-class companies, even though its usage lags the alternative approaches of third-normal-form and star schema.

Modeling 52

Modeling Data Warehouse Data Quality Business Driver 52

AWS Big Data

MARCH 1, 2024

One of the bank’s key challenges related to strict cybersecurity requirements is to implement field level encryption for personally identifiable information (PII), Payment Card Industry (PCI), and data that is classified as high privacy risk (HPR). Only users with required permissions are allowed to access data in clear text.

Data Lake 113

Data Lake Data Warehouse Management Risk 113

erwin

JULY 17, 2019

Part Two of the Digital Transformation Journey … In our last blog on driving digital transformation , we explored how enterprise architecture (EA) and business process (BP) modeling are pivotal factors in a viable digital transformation strategy. With automation, data quality is systemically assured.

Digital Transformation 98

Digital Transformation Strategy Metadata Data-driven 98

CIO Business Intelligence

AUGUST 9, 2024

Privacy protection The first step in AI and gen AI projects is always to get the right data. “In In cases where privacy is essential, we try to anonymize as much as possible and then move on to training the model,” says University of Florence technologist Vincenzo Laveglia. “A A balance between privacy and utility is needed.

Risk 138

Risk Measurement Data Lake Data Collection 138

CIO Business Intelligence

OCTOBER 2, 2024

According to Kari Briski, VP of AI models, software, and services at Nvidia, successfully implementing gen AI hinges on effective data management and evaluating how different models work together to serve a specific use case. But some IT leaders are getting it right because they focus on three key aspects.

Management 141

Management Data Governance Cost-Benefit Structured Data 141

AWS Big Data

JANUARY 31, 2023

Data governance is increasingly top-of-mind for customers as they recognize data as one of their most important assets. Effective data governance enables better decision-making by improving data quality, reducing data management costs, and ensuring secure access to data for stakeholders.

Data Lake 89

Data Lake Data Governance Data Architecture Machine Learning 89

AWS Big Data

FEBRUARY 13, 2025

Amazon SageMaker Unified Studio (preview) provides a unified experience for using data, analytics, and AI capabilities. You can use familiar AWS services for model development, generative AI, data processing, and analyticsall within a single, governed environment. To use Amazon Bedrock FMs, grant access to base models.

Data Analytics 72

Data Analytics Analytics Modeling Management 72

DataKitchen

OCTOBER 11, 2021

Figure 2: Example data pipeline with DataOps automation. In this project, I automated data extraction from SFTP, the public websites, and the email attachments. The automated orchestration published the data to an AWS S3 Data Lake. All the code, Talend job, and the BI report are version controlled using Git.

Testing 152

Testing Metadata Dashboards Statistics 152

CIO Business Intelligence

FEBRUARY 16, 2023

Hence the drive to provide ML as a service to the Data & Tech team’s internal customers. All they would have to do is just build their model and run with it,” he says. That step, primarily undertaken by developers and data architects, established data governance and data integration. The offensive side?

Unstructured Data 105

Unstructured Data Data Lake Prescriptive Analytics Data Warehouse 105

Alation

AUGUST 4, 2022

Aligning business goals to data usage is a key foundation. There is no digital transformation without business-aligned data control,” Capgemini Executive Steve Jones adds. “The architectural model of big data is fundamental to this.”. Everything comes down to what organizations do with data.

Big Data 95

Big Data Digital Transformation Data Lake Data-driven 95

Timo Elliott

DECEMBER 14, 2022

Most innovation platforms make you rip the data out of your existing applications and move it to some another environment—a data warehouse, or data lake, or data lake house or data cloud—before you can do any innovation. Business Context. Business Content.

Data Lake 105

Data Lake Recreation/Entertainment Data Warehouse Metadata 105

AWS Big Data

JULY 25, 2024

Solution To address the challenge, ATPCO sought inspiration from a modern data mesh architecture. Implementation Now, we’ll walk through how ATPCO implemented their solution to solve the challenges of analysts discovering, getting access to, and using data quickly to help their airline customers. Select Create environment profile.

Data Lake 105

Data Lake Metadata Sales Publishing 105

Cloudera

JUNE 13, 2024

After countless open-source innovations ushered in the Big Data era, including the first commercial distribution of HDFS (Apache Hadoop Distributed File System), commonly referred to as Hadoop, the two companies joined forces, giving birth to an entire ecosystem of technology and tech companies.

Big Data 103

Big Data Machine Learning Contextual Data Data Lake 103

Cloudera

MARCH 28, 2024

At a time when AI is exploding in popularity and finding its way into nearly every facet of business operations, data has arguably never been more valuable. More recently, that value has been made clear by the emergence of AI-powered technologies like generative AI (GenAI) and the use of Large Language Models (LLMs).

Data Architecture 94

Data Architecture Strategy Data Lake Data-driven 94

CIO Business Intelligence

NOVEMBER 28, 2023

With each game release and update, the amount of unstructured data being processed grows exponentially, Konoval says. This volume of data poses serious challenges in terms of storage and efficient processing,” he says. To address this problem RetroStyle Games invested in data lakes. Quality is job one.

Unstructured Data 130

Unstructured Data Data-driven Visualization Data Quality 130

AWS Big Data

MAY 1, 2024

Foundation When the teams don’t have a robust enough skillset, it can require more time to model and process data, and cause longer latency and lower data quality. To address this, the foundation role provides an already processed dataset as a generic data model for data commonly use cases used by many consumers.

Dashboards 122

Dashboards Publishing Data-driven Cost-Benefit 122

Domino Data Lab

JUNE 15, 2021

Over the next decade, the companies that will beat competitors will be “model-driven” businesses. These companies often undertake large data science efforts in order to shift from “data-driven” to “model-driven” operations, and to provide model-underpinned insights to the business. anomaly detection).

Modeling 69

Modeling Data Science Data-driven Data Warehouse 69

CIO Business Intelligence

APRIL 24, 2024

In fact, AMA collects a huge amount of structured and unstructured data from bins, collection vehicles, facilities, and user reports, and until now, this data has remained disconnected, managed by disparate systems and interfaces, through Excel spreadsheets.

Digital Transformation 105

Digital Transformation Business Intelligence Unstructured Data Data Lake 105

DataKitchen

NOVEMBER 18, 2022

In addition to the tracking of relationships and quality metrics, DataOps Observability journeys allow users to establish baselines?concrete concrete expectations for run schedules, run durations, data quality, and upstream and downstream dependencies. And she’ll know when newer data will arrive.

Testing 130

Testing Statistics Measurement Dashboards 130