AWS Big Data

JUNE 6, 2023

We are excited to announce the General Availability of AWS Glue Data Quality. Our journey started by working backward from our customers who create, manage, and operate data lakes and data warehouses for analytics and machine learning. It takes days for data engineers to identify and implement data quality rules.

Data Quality 98

Data Quality Statistics Data Lake Visualization 98

AWS Big Data

DECEMBER 4, 2024

With this new functionality, customers can create up-to-date replicas of their data from applications such as Salesforce, ServiceNow, and Zendesk in an Amazon SageMaker Lakehouse and Amazon Redshift. SageMaker Lakehouse gives you the flexibility to access and query your data in-place with all Apache Iceberg compatible tools and engines.

Data Integration 114

Data Integration Data Lake Statistics Data-driven 114

AWS Big Data

APRIL 8, 2025

In modern data architectures, Apache Iceberg has emerged as a popular table format for data lakes, offering key features including ACID transactions and concurrent write support. We will also cover the pattern with automatic compaction through AWS Glue Data Catalog table optimization.

Snapshot 137

Snapshot Management Metadata Big Data 137

AWS Big Data

JANUARY 15, 2025

For container terminal operators, data-driven decision-making and efficient data sharing are vital to optimizing operations and boosting supply chain efficiency. The data science and AI teams are able to explore and use new data sources as they become available through Amazon DataZone. She can reached via LinkedIn.

IoT 111

IoT Machine Learning Metadata Data-driven 111

AWS Big Data

DECEMBER 4, 2024

These formats, exemplified by Apache Iceberg, Apache Hudi, and Delta Lake, addresses persistent challenges in traditional data lake structures by offering an advanced combination of flexibility, performance, and governance capabilities. In this post, we highlight notable updates on Iceberg, Hudi, and Delta Lake in AWS Glue 5.0.

Snapshot 84

Snapshot Metadata Data Lake Optimization 84

CIO Business Intelligence

OCTOBER 23, 2024

For the first time, we’re consolidating data to create real-time dashboards for revenue forecasting, resource optimization, and labor utilization. We pulled these people together, and defined use cases we could all agree were the best to demonstrate our new data capability. How is the new platform helping?

Enterprise 119

Enterprise Dashboards KPI Data Lake 119

AWS Big Data

JUNE 6, 2023

You can use AWS Glue to create, run, and monitor data integration and ETL (extract, transform, and load) pipelines and catalog your assets across multiple data stores. Hundreds of thousands of customers use data lakes for analytics and ML to make data-driven business decisions.

Data Quality 98

Data Quality Data-driven Data Lake Metrics 98

Jet Global

JULY 2, 2019

To address the flood of data and the needs of enterprise businesses to store, sort, and analyze that data, a new storage solution has evolved: the data lake. What’s in a Data Lake? Data warehouses do a great job of standardizing data from disparate sources for analysis. Taking a Dip.

Data Lake 73

Data Lake Data Warehouse Big Data Machine Learning 73

AWS Big Data

JULY 29, 2024

In the era of big data, data lakes have emerged as a cornerstone for storing vast amounts of raw data in its native format. They support structured, semi-structured, and unstructured data, offering a flexible and scalable environment for data ingestion from multiple sources.

Metadata 126

Metadata Snapshot Data Lake Metrics 126

CIO Business Intelligence

MARCH 9, 2022

However, they do contain effective data management, organization, and integrity capabilities. As a result, users can easily find what they need, and organizations avoid the operational and cost burdens of storing unneeded or duplicate data copies. On the other hand, they don’t support transactions or enforce data quality.

Data Lake 119

Data Lake Unstructured Data Data Warehouse Big Data 119

AWS Big Data

DECEMBER 4, 2024

Domain ownership recognizes that the teams generating the data have the deepest understanding of it and are therefore best suited to manage, govern, and share it effectively. This principle makes sure data accountability remains close to the source, fostering higher data quality and relevance.

Metadata 105

Metadata Data Governance Data Quality Data-driven 105

CIO Business Intelligence

SEPTEMBER 20, 2023

On the agribusiness side we source, purchase, and process agricultural commodities and offer a diverse portfolio of products including grains, soybean meal, blended feed ingredients, and top-quality oils for the food industry to add value to the commodities our customers desire. The data can also help us enrich our commodity products.

Data Lake 135

Data Lake Data-driven Dashboards Risk 135

AWS Big Data

JANUARY 18, 2024

AWS Lake Formation and the AWS Glue Data Catalog form an integral part of a data governance solution for data lakes built on Amazon Simple Storage Service (Amazon S3) with multiple AWS analytics services integrating with them. The Data Catalog views feature is available in preview , announced at re:Invent 2023.

Data Lake 119

Data Lake Metadata Data Governance Statistics 119

AWS Big Data

JULY 13, 2023

Data quality for account and customer data – Altron wanted to enable data quality and data governance best practices. Goals – Lay the foundation for a data platform that can be used in the future by internal and external stakeholders.

Optimization 98

Optimization B2B Data Quality Sales 98

IBM Big Data Hub

JANUARY 5, 2023

Poor data quality is one of the top barriers faced by organizations aspiring to be more data-driven. Ill-timed business decisions and misinformed business processes, missed revenue opportunities, failed business initiatives and complex data systems can all stem from data quality issues.

Data Architecture 52

Data Architecture Data Quality Strategy Data Lake 52

Timo Elliott

JANUARY 3, 2022

It’s stored in corporate data warehouses, data lakes, and a myriad of other locations – and while some of it is put to good use, it’s estimated that around 73% of this data remains unexplored. Improving data quality. Unexamined and unused data is often of poor quality. Data augmentation.

IT 143

IT Unstructured Data Data Quality Machine Learning 143

AWS Big Data

AUGUST 1, 2024

Migrating to Amazon Redshift offers organizations the potential for improved price-performance, enhanced data processing, faster query response times, and better integration with technologies such as machine learning (ML) and artificial intelligence (AI).

Data Warehouse 115

Data Warehouse KPI Optimization Cost-Benefit 115

AWS Big Data

DECEMBER 9, 2024

As organizations process vast amounts of data, maintaining an accurate historical record is crucial. History management in data systems is fundamental for compliance, business intelligence, data quality, and time-based analysis. Hes passionate about helping customers use Apache Iceberg for their data lakes on AWS.

Snapshot 64

Snapshot Data Warehouse Data Lake Data Quality 64

AWS Big Data

FEBRUARY 1, 2023

With data volumes exhibiting a double-digit percentage growth rate year on year and the COVID pandemic disrupting global logistics in 2021, it became more critical to scale and generate near-real-time data. This introduces the need for both polling and pushing the data to access and analyze in near-real time.

Optimization 52

Optimization Forecasting Data Lake Metadata 52

IBM Big Data Hub

MAY 9, 2023

Data: the foundation of your foundation model Data quality matters. An AI model trained on biased or toxic data will naturally tend to produce biased or toxic outputs. When objectionable data is identified, we remove it, retrain the model, and repeat. Data curation is a task that’s never truly finished.

Enterprise 128

Enterprise Technology Modeling Cost-Benefit 128

CIO Business Intelligence

APRIL 29, 2022

This would be straightforward task were it not for the fact that, during the digital-era, there has been an explosion of data – collected and stored everywhere – much of it poorly governed, ill-understood, and irrelevant. Further, data management activities don’t end once the AI model has been developed. Addressing the Challenge.

Data Governance 145

Data Governance IT Data Lake Risk 145

Timo Elliott

DECEMBER 14, 2022

The future is enabled by technology, but it’s not about the technical infrastructures: it’s about optimizing end-to-end processes, business capabilities, and business ecosystems. You lose the roots: the metadata, the hierarchies, the security, the business context of the data. So how do organizations do that? Business Context.

Data Lake 105

Data Lake Recreation/Entertainment Data Warehouse Metadata 105

CIO Business Intelligence

AUGUST 9, 2024

Twenty-nine percent of 644 executives at companies in the US, Germany, and the UK said they were already using gen AI, and it was more widespread than other AI-related technologies, such as optimization algorithms, rule-based systems, natural language processing, and other types of ML.

Risk 138

Risk Measurement Data Lake Data Collection 138

AWS Big Data

MAY 1, 2024

To provide a variety of products, services, and solutions that are better suited to customers and society in each region, we have built business processes and systems that are optimized for each region and its market. Here, the foundation role takes the lead in compiling the knowledge of domain experts and making data suitable for analysis.

Dashboards 124

Dashboards Publishing Data-driven Cost-Benefit 124

Cloudera

JUNE 13, 2024

After countless open-source innovations ushered in the Big Data era, including the first commercial distribution of HDFS (Apache Hadoop Distributed File System), commonly referred to as Hadoop, the two companies joined forces, giving birth to an entire ecosystem of technology and tech companies.

Big Data 103

Big Data Machine Learning Contextual Data Data Lake 103

CIO Business Intelligence

MAY 21, 2024

In the annual Porsche Carrera Cup Brasil, data is essential to keep drivers safe and sustain optimal performance of race cars. Until recently, getting at and analyzing that essential data was a laborious affair that could take hours, and only once the race was over.

Broadcasting 127

Broadcasting Recreation/Entertainment Manufacturing Data Lake 127

CIO Business Intelligence

FEBRUARY 16, 2023

Selling the value of data transformation Iyengar and his team are 18 months into a three- to five-year journey that started by building out the data layer — corralling data sources such as ERP, CRM, and legacy databases into data warehouses for structured data and data lakes for unstructured data.

Unstructured Data 106

Unstructured Data Data Lake Prescriptive Analytics Data Warehouse 106

CIO Business Intelligence

MARCH 22, 2022

Which type(s) of storage consolidation you use depends on the data you generate and collect. . One option is a data lake—on-premises or in the cloud—that stores unprocessed data in any type of format, structured or unstructured, and can be queried in aggregate. Just starting out with analytics?

Analytics 115

Analytics Key Performance Indicator Data Warehouse Data-driven 115

Cloudera

MARCH 28, 2024

And of those organizations working on some stage of AI adoption, a few of the top benefits included increased productivity (35%), enhanced operational efficiency (33%), improved customer experience (33%), and optimized supply chain and logistics (33%). The benefits are clear, and there’s plenty of potential that comes with AI adoption.

Data Architecture 94

Data Architecture Strategy Data Lake Data-driven 94

AWS Big Data

MARCH 7, 2023

Flexible and easy to use – The solutions should provide less restrictive, easy-to-access, and ready-to-use data. They should also provide optimal performance with low or no tuning. A data hub contains data at multiple levels of granularity and is often not integrated. Data repositories represent the hub.

Analytics 96

Analytics Data Warehouse Data Lake Metadata 96

DataKitchen

DECEMBER 9, 2022

Observability in DataOps refers to the ability to monitor and understand the performance and behavior of data-related systems and processes, and to use that information to improve the quality and speed of data-driven decision making. Overall, DataOps observability is an essential component of modern data-driven organizations.

Machine Learning 130

Machine Learning Data-driven Optimization Data Analytics 130

DataKitchen

NOVEMBER 18, 2022

In addition to the tracking of relationships and quality metrics, DataOps Observability journeys allow users to establish baselines?concrete concrete expectations for run schedules, run durations, data quality, and upstream and downstream dependencies. And she’ll know when newer data will arrive.

Testing 130

Testing Statistics Measurement Metrics 130

AWS Big Data

MARCH 26, 2024

A Gartner Marketing survey found only 14% of organizations have successfully implemented a C360 solution, due to lack of consensus on what a 360-degree view means, challenges with data quality, and lack of cross-functional governance structure for customer data.

Data Strategy 98

Data Strategy Strategy Data Warehouse Prescriptive Analytics 98

AWS Big Data

DECEMBER 19, 2024

Data has become an invaluable asset for businesses, offering critical insights to drive strategic decision-making and operational optimization. Delta tables technical metadata is stored in the Data Catalog, which is a native source for creating assets in the Amazon DataZone business catalog.

Data Governance 52

Data Governance Publishing Data-driven Metadata 52

AWS Big Data

NOVEMBER 16, 2023

Amazon Redshift is a popular cloud data warehouse, offering a fully managed cloud-based service that seamlessly integrates with an organization’s Amazon Simple Storage Service (Amazon S3) data lake, real-time streams, machine learning (ML) workflows, transactional workflows, and much more—all while providing up to 7.9x

Enterprise 111

Enterprise Data Warehouse Snapshot Cost-Benefit 111

CIO Business Intelligence

SEPTEMBER 8, 2022

In Foundry’s 2022 Data & Analytics Study , 88% of IT decision-makers agree that data collection and analysis have the potential to fundamentally change their business models over the next three years. The ability to pivot quickly to address rapidly changing customer or market demands is driving the need for real-time data.

Cost-Benefit 122

Cost-Benefit Internet of Things Data-driven Data Lake 122

Cloudera

SEPTEMBER 9, 2022

analyst Sumit Pal, in “Exploring Lakehouse Architecture and Use Cases,” published January 11, 2022: “Data lakehouses integrate and unify the capabilities of data warehouses and data lakes, aiming to support AI, BI, ML, and data engineering on a single platform.” According to Gartner, Inc.

Metadata 102

Metadata Machine Learning Unstructured Data Data Lake 102