AWS Big Data

NOVEMBER 14, 2024

The landscape of big data management has been transformed by the rising popularity of open table formats such as Apache Iceberg, Apache Hudi, and Linux Foundation Delta Lake. These formats, designed to address the limitations of traditional data storage systems, have become essential in modern data architectures.

Metadata 104

Metadata Data Warehouse Big Data Data Lake 104

TDAN

MAY 31, 2022

The main goal of creating an enterprise data fabric is not new. It is the ability to deliver the right data at the right time, in the right shape, and to the right data consumer, irrespective of how and where it is stored. Data fabric is the common “net” that stitches integrated data from multiple data […].

Metadata 98

Metadata Enterprise IT Data Architecture 98

Data Virtualization

MAY 19, 2023

Reading Time: 11 minutes The post Data Strategies for Getting Greater Business Value from Distributed Data appeared first on Data Management Blog - Data Integration and Modern Data Management Articles, Analysis and Information.

Data Strategy 52

Data Strategy Strategy Data Integration Management 52

AWS Big Data

JUNE 10, 2024

A modern data strategy redefines and enables sharing data across the enterprise and allows for both reading and writing of a singular instance of the data using an open table format. When evolving such a partition definition, the data in the table prior to the change is unaffected, as is its metadata.

Data Lake 121

Data Lake Metadata Snapshot Analytics 121

AWS Big Data

OCTOBER 25, 2023

But most important of all, the assumed dormant value in the unstructured data is a question mark, which can only be answered after these sophisticated techniques have been applied. Therefore, there is a need to being able to analyze and extract value from the data economically and flexibly. The solution integrates data in three tiers.

Unstructured Data 128

Unstructured Data Metadata Management Analytics 128

Alation

MAY 31, 2023

This means that specialized roles such as data architects, which focus on modernizing data architecture to help meet business goals, are increasingly important to support data governance. What is a data architect? Their broad range of responsibilities include: Design and implement data architecture.

Data Governance 52

Data Governance Metadata Data Architecture Data Quality 52

Cloudera

JUNE 7, 2022

In our very own Enterprise Data Maturity research surveying over 3,000 IT and senior business leaders, we found that 40% of organizations are currently running hybrid but mostly on-premises, and 36% of respondents expect to shift to hybrid multi-cloud in the next 18 months. Where data flows, ideas follow.

IT 111

IT Data Architecture Unstructured Data Big Data 111

CIO Business Intelligence

JUNE 23, 2022

In our very own Enterprise Data Maturity research surveying over 3,000 IT and senior business leaders, we found that 40% of organizations are currently running hybrid but mostly on-premises, and 36% of respondents expect to shift to hybrid multi-cloud in the next 18 months. Where data flows, ideas follow. Jonathan Takiff / IDG.

IT 88

IT Data Architecture Unstructured Data Big Data 88

Jen Stirrup

MARCH 29, 2023

What does a sound, intelligent data foundation give you? It can give business-oriented data strategy for business leaders to help drive better business decisions and ROI. It can also increase productivity by enabling the business to find the data they need when the business teams need it.

Data Warehouse 121

Data Warehouse Metadata Data Integration Business Intelligence 121

AWS Big Data

APRIL 26, 2023

Having an accurate and up-to-date inventory of all technical assets helps an organization ensure it can keep track of all its resources with metadata information such as their assigned oners, last updated date, used by whom, how frequently and more. This is a guest blog post co-written with Corey Johnson from Huron.

Metadata 52

Metadata Dashboards Visualization Consulting 52

Cloudera

JANUARY 5, 2023

What is certain is that having an enterprise data strategy aligned to the organization’s cloud strategy and business priorities will help the organization drive greater business value by improving operational efficiencies and unlocking new revenue streams. Find out more about CDP for modern data architectures here.

Cost-Benefit 52

Cost-Benefit Business Objectives Machine Learning Data Architecture 52

AWS Big Data

APRIL 19, 2023

Independent data products often only have value if you can connect them, join them, and correlate them to create a higher order data product that creates additional insights. A modern data architecture is critical in order to become a data-driven organization.

Technology 92

Technology Data-driven Machine Learning Sales 92

AWS Big Data

SEPTEMBER 13, 2023

The Analytics specialty practice of AWS Professional Services (AWS ProServe) helps customers across the globe with modern data architecture implementations on the AWS Cloud. The File Manager Lambda function consumes those messages, parses the metadata, and inserts the metadata to the DynamoDB table odpf_file_tracker.

Data Lake 102

Data Lake Data Processing Metadata Snapshot 102

AWS Big Data

MARCH 10, 2023

They enable transactions on top of data lakes and can simplify data storage, management, ingestion, and processing. These transactional data lakes combine features from both the data lake and the data warehouse. One important aspect to a successful data strategy for any organization is data governance.

Data Lake 115

Data Lake Sales Data Warehouse Snapshot 115

TDAN

DECEMBER 3, 2019

I said I thought it affected all of them pretty profoundly, but perhaps the Metadata wedge the most. Recently, I was giving a presentation and someone asked me which segment of “the DAMA wheel” did I think semantics most affected. I thought I’d spend a bit of time to reflect on the question and answer […].

Metadata 88

Metadata Data Architecture IT Data Strategy 88

Alation

APRIL 19, 2021

With a good plan and a modern data catalog, you can minimize the time and cost of cloud migration. Source: Webinar with data expert Ibby Rahmani: Emerging Trends in Data Architecture: What’s the Next Big Thing? Alation & Global Data Strategy). Data Strategy Drives Cloud Strategy.

Cost-Benefit 111

Cost-Benefit Strategy Enterprise Data Strategy 111

AWS Big Data

APRIL 3, 2024

If the asset has AWS Glue Data Quality enabled, you can now quickly visualize the data quality score directly in the catalog search pane. By selecting the corresponding asset, you can understand its content through the readme, glossary terms , and technical and business metadata.

Data Quality 102

Data Quality Visualization Metadata Metrics 102

Alation

DECEMBER 14, 2021

AWS is supporting green initiatives : Sessions covered new innovative advancements, including open data initiatives that provide new paths to space exploration. AWS provides end-to-end data strategy for ML use cases to address world challenges in biotech (pandemic), climate change, driverless cars, healthcare, and more.

Data Lake 52

Data Lake Data Warehouse Machine Learning Cost-Benefit 52

Sisense

JULY 6, 2020

First off, this involves defining workflows for every business process within the enterprise: the what, how, why, who, when, and where aspects of data. These regulations, ultimately, ensure key business values: data consistency, quality, and trustworthiness.

Enterprise 98

Enterprise Management Data Architecture Data-driven 98

AWS Big Data

JULY 16, 2024

Priority 2 logs, such as operating system security logs, firewall, identity provider (IdP), email metadata, and AWS CloudTrail , are ingested into Amazon OpenSearch Service to enable the following capabilities. She currently serves as the Global Head of Cyber Data Management at Zurich Group.

Insurance 112

Insurance Management Cost-Benefit Optimization 112

AWS Big Data

OCTOBER 30, 2024

This is part two of a three-part series where we show how to build a data lake on AWS using a modern data architecture. This post shows how to load data from a legacy database (SQL Server) into a transactional data lake ( Apache Iceberg ) using AWS Glue.

Data Lake 100

Data Lake Data Processing Optimization Machine Learning 100

Cloudera

AUGUST 21, 2020

The governance aspect is perhaps even more important and businesses need to be able to understand where the data comes from. Data lineage, personally identifiable information or PPI and metadata all fall under a broad data governance banner which is critically important in terms of what needs to be protected and mapped out.

Enterprise 62

Enterprise Data Lake Strategy Metadata 62

TDAN

JUNE 6, 2023

You would think that after knocking around in semantics and knowledge graphs for over two decades I’d have had a pretty good idea about Knowledge Management, but it turns out I didn’t. I think in the rare event the term came up I internally conflated it with Knowledge Graphs and moved on. The first tap […]

Management 97

Management IT Data Architecture Metadata 97

Data Virtualization

JUNE 19, 2024

Reading Time: 5 minutes The data landscape has evolved and become more complex as organizations recognize the need to leverage data and analytics. Generative artificial intelligence has further put pressure on organizations to manage this complexity. At TDWI, we see companies collecting traditional structured.

Data Integration 59

Data Integration Management Analytics Data Architecture 59

TDAN

APRIL 5, 2022

Data Governance is defined as the execution and enforcement of authority over the management of data and data-related assets.1 1 The terms “Data Mesh” and “Data Fabric” are the most recent examples of names being given to something that describes techniques to help organizations manage their data.

Data Governance 98

Data Governance Management Data Architecture Metadata 98

TDAN

AUGUST 17, 2021

It has been an incredible run. I hope it is just “see you soon” rather than “goodbye.” With this column, DAMA International’s streak of quarterly columns since mid-2001 is coming to an end. The columns have featured the activities and incredible work of DAMA International over the past two decades. Thank you, DAMA, and I […].

IT 105

IT Data Architecture Metadata Data Strategy 105

TDAN

JUNE 21, 2022

Twenty-five years ago today, I published the first issue of The Data Administration Newsletter. It only took a few months to recognize that there was an audience for an “online” publication focused on data administration. […].

Publishing 78

Publishing Data Quality Data Architecture IT 78

TDAN

SEPTEMBER 16, 2020

Differences in Terminology and Capability Building on the terms and concepts introduced in Part I of this white paper, Part II digs deeper into the difference in the meaning of some key terms used in both Property Graphs and Knowledge Graphs, including LABELS, TYPES, and PROPERTIES. Key terms such as these actually mean very different […].

Data Architecture 92

Data Architecture Metadata Data Strategy Strategy 92

TDAN

DECEMBER 6, 2022

The FAIR principles for data sets are gaining traction, especially in the pharmaceutical industry in Europe. In a world of exponential data growth and ever-increasing silo-ization, the FAIR principles are needed more than ever. FAIR stands for: Findable, Accessible, Interoperable and Reusable.

Data Architecture 52

Data Architecture Metadata Data Strategy Strategy 52

TDAN

SEPTEMBER 15, 2020

In the world of data, automation plays a well-honed role in rapidly developing modern data estates. Before we proceed any further, let’s establish an understanding about the purpose of a corporate data estate. A data estate is the technical architecture and enterprise infrastructure that enables organizations to […].

Enterprise 85

Enterprise Data Architecture Metadata Data Strategy 85

TDAN

MAY 31, 2022

In my September 2020 Data is Risky Business column for TDAN.com, I wrote about the strategy lessons for data that we can learn from the writings of a 16thcentury Japanese swordsman. To summarize: – We need to study different disciplines to understand how they relate to each other.

Strategy 52

Strategy Data Architecture Data Strategy Metadata 52

TDAN

JULY 19, 2022

In reference to the prior column on enterprise data management and high level lego framework, this column reviews in detail the foundational layer of Organization Mission, Level 1.

Strategy 52

Strategy Enterprise Management Modeling 52

TDAN

MARCH 31, 2020

Executive Summary It seems obvious enough that companies, government agencies and non-profits would benefit from a common language. Without it, coordinating work is more difficult, computers “don’t talk,” and basic questions such as “how many customers do we have?” yield differing answers, making it more difficult to run the business.

IT 84

IT Data Architecture Metadata Data Strategy 84

TDAN

SEPTEMBER 15, 2020

Remember the Sears catalog? Hundreds of pages of clothes quickly flipped past to get to the toys! Who didn’t spend countless hours trying to find the perfect suggested gift for your next birthday or holiday? In an Amazon and eBay world, thinking about old-school catalogs seems so quaint. Opening a paper volume to manually flip […].

Data Architecture 69

Data Architecture Metadata Data Strategy Data Governance 69

TDAN

DECEMBER 3, 2019

Wherever we go, we are overwhelmed by MORE: more sales, more discounts, more fun, more excitement, more features – the list goes on and on! What humans seem to be far less attuned to is reducing what we don’t need. Drive around any suburban neighborhood and see the many cars parked outside their garages! Believe […].

Sales 78

Sales Data Architecture Data Governance Metadata 78

TDAN

MAY 31, 2022

One very influential factor that can potentially undermine your data and document strategies is the natural and emotional reactions of people when things change. It is common to take great care in the selection and implementation of new technology.

Testing 52

Testing Interactive Strategy Software 52

Data Virtualization

NOVEMBER 17, 2022

Reading Time: 5 minutes Today, many applications call themselves “data catalogs.” The idea seems, on the face of it, easy to understand: a data catalog is simply a centralized inventory of the data assets within an organization. Data catalogs also seek to be the.

Data Integration 59

Data Integration Management Data Lake IT 59