AWS Big Data

FEBRUARY 26, 2025

S3 Tables integration with the AWS Glue Data Catalog is in preview, allowing you to stream, query, and visualize dataincluding Amazon S3 Metadata tablesusing AWS analytics services such as Amazon Data Firehose , Amazon Athena , Amazon Redshift, Amazon EMR, and Amazon QuickSight. With AWS Glue 5.0,

Analytics 61

Analytics Data Lake Metadata Data Warehouse 61

Jen Stirrup

MARCH 29, 2023

What does a sound, intelligent data foundation give you? It can give business-oriented data strategy for business leaders to help drive better business decisions and ROI. It can also increase productivity by enabling the business to find the data they need when the business teams need it.

Data Warehouse 121

Data Warehouse Metadata Data Integration Business Intelligence 121

CIO Business Intelligence

OCTOBER 13, 2023

According to Bob Lambert , analytics delivery lead at Anthem and former director of CapTech Consulting, important data architect skills include: A foundation in systems development: Data architects must understand the system development life cycle, project management approaches, and requirements, design, and test techniques.

Data Architecture 132

Data Architecture Data Warehouse Statistics Visualization 132

TDAN

JUNE 30, 2020

You may already have a formal Data Governance program in place. Or … you are presently going through the process of trying to convince your Senior Leadership or stakeholders that a formal Data Governance program is necessary. Maybe you are going through the process of convincing the stakeholders that Data […].

Data Governance 145

Data Governance Data Quality Metadata Data Strategy 145

Cloudera

JUNE 7, 2022

The cause is hybrid data – the massive amounts of data created everywhere businesses operate – in clouds, on-prem, and at the edge. Only a fraction of data created is actually stored and managed, with analysts estimating it to be between 4 – 6 ZB in 2020. Where data flows, ideas follow.

IT 112

IT Data Architecture Unstructured Data Big Data 112

AWS Big Data

DECEMBER 19, 2024

From establishing an enterprise-wide data inventory and improving data discoverability, to enabling decentralized data sharing and governance, Amazon DataZone has been a game changer for HEMA. The integration of Databricks Delta tables into Amazon DataZone is done using the AWS Glue Data Catalog.

Data Governance 52

Data Governance Publishing Data-driven Metadata 52

DataKitchen

JUNE 24, 2021

Agility is absolutely the cornerstone of what DataOps presents in the build and in the run aspects of our data products.”. Before we jump into a methodology or even a data strategy-based approach, what are we trying to accomplish? Bergh added, “ DataOps is part of the data fabric. Be the provider of choice.

Metrics 211

Metrics ROI Measurement Cost-Benefit 211

CIO Business Intelligence

MARCH 21, 2022

Data scientists are often engaged in long-term research and prediction, while data analysts seek to support business leaders in making tactical decisions through reporting and ad hoc queries aimed at describing the current state of reality for their organizations based on present and historical data.

Unstructured Data 137

Unstructured Data Data Analytics Analytics Data Science 137

AWS Big Data

APRIL 19, 2023

The data science algorithm Valentine is an effective tool for this. Valentine is presented in the paper Valentine: Evaluating Matching Techniques for Dataset Discovery (2021, Koutras et al.). This solution solves the interoperability and linkage problem for data products. We focus on the former.

Technology 105

Technology Data-driven Machine Learning Sales 105

Ontotext

SEPTEMBER 15, 2020

The Data Fabric paradigm combines design principles and methodologies for building efficient, flexible and reliable data management ecosystems. Knowledge Graphs are the Warp and Weft of a Data Fabric. To implement any Data Fabric approach, it is essential to be able to understand the context of data.

Metadata 69

Metadata Knowledge Discovery Data Quality Strategy 69

CIO Business Intelligence

JUNE 23, 2022

The cause is hybrid data – the massive amounts of data created everywhere businesses operate – in clouds, on-prem, and at the edge. Only a fraction of data created is actually stored and managed, with analysts estimating it to be between 4 – 6 ZB in 2020. Clearly, hybrid data presents a massive opportunity and a tough challenge.

IT 97

IT Data Architecture Unstructured Data Big Data 97

AWS Big Data

APRIL 3, 2024

Today, we are pleased to announce that Amazon DataZone is now able to present data quality information for data assets. If the asset has AWS Glue Data Quality enabled, you can now quickly visualize the data quality score directly in the catalog search pane.

Data Quality 119

Data Quality Visualization Metadata Metrics 119

AWS Big Data

SEPTEMBER 13, 2023

The File Manager Lambda function consumes those messages, parses the metadata, and inserts the metadata to the DynamoDB table odpf_file_tracker. We use the following terminology when discussing File Processor: Refresh cadence – This represents the data ingestion frequency (for example, 10 minutes).

Data Lake 111

Data Lake Data Processing Metadata Snapshot 111

Alation

OCTOBER 5, 2021

Yet, so many companies today are still failing miserably in implementing data strategy and governance protocols. Why is your data governance strategy failing? So, why is YOUR data governance strategy failing? Common data governance challenges. Top 3 Roadblocks to Successful Data Governance.

Data Governance 96

Data Governance Strategy Data Quality Metrics 96

Domino Data Lab

APRIL 3, 2019

This month’s article features updates from one of the early data conferences of the year, Strata Data Conference – which was held just last week in San Francisco. In particular, here’s my Strata SF talk “Overview of Data Governance” presented in article form. Some of the finer points seem amorphous at best.

Machine Learning 84

Machine Learning Data Governance Metadata Data Science 84

AWS Big Data

MAY 4, 2023

Data scientists have access to the Jupyter notebook hosted on SageMaker. The OpenSearch Service domain stores metadata on the datasets connected at the Regions. Notebook users can query this service to retrieve details such as the correct Region of Dask workers without needing to know the data’s Regional location beforehand.

Data Processing 122

Data Processing Metadata Informatics Interactive 122

Octopai

JANUARY 31, 2021

The particular episode we recommend looks at how WeWork struggled with understanding their data lineage so they created a metadata repository to increase visibility. Agile Data. Another podcast we think is worth a listen is Agile Data. Techcopedia follows the latest trends in data and provides comprehensive tutorials.

Data Governance 83

Data Governance Data Processing Data Quality Metadata 83

Alation

MAY 10, 2021

Folks who work with data face these challenges every day. A data catalog helps people find, understand, trust, and govern data. The catalog gathers metadata, (or data about data), to add context to every asset. In phase one, an enterprise must create a data strategy , which will inform later plans.

Strategy 80

Strategy Enterprise Data Strategy Data Governance 80

Alation

JUNE 7, 2022

The key to that innovation is data. Yet Fifth Third’s vast data environment presents a number of challenges. A small group of data leaders faced an explosion in both the need and demand for data — and a lack of the structure to support it. Maturing our data strategy helps to accelerate our value to the customer.”.

Metadata 52

Metadata Data Strategy Strategy Data Governance 52

Alation

AUGUST 11, 2022

We chatted about industry trends, why decentralization has become a hot topic in the data world, and how metadata drives many data-centric use cases. But, through it all, Mohan says it’s critical to view everything through the same lens: gaining business value from data. Data fabric is a technology architecture.

Metadata 52

Metadata Data Warehouse Data Quality Data Lake 52

Alation

MARCH 28, 2023

With data becoming more prevalent in every industry, organisations have to determine how to not only manage it but also drive value from it. The MoD identify three key issues: firstly, that ‘Defence data operates in contractual, technical and behavioural silos’. Culturally, defence lacks recognition over the importance of data’.

Metadata 52

Metadata Data Strategy Data Quality Strategy 52

Alation

NOVEMBER 22, 2022

Effective data governance for the public sector enables entities to ensure data quality, enhance security, protect privacy, and meet compliance requirements. With so much focus on compliance, democratizing data for self-service analytics can present a challenge. Balance Defensive And Offensive Data Strategy.

Data Governance 52

Data Governance Metadata Data-driven Unstructured Data 52

AWS Big Data

AUGUST 14, 2023

At the same time, unstructured approaches to data mesh management that don’t have a vision for what types of products should exist and how to ensure they are developed are at high risk of creating the same effect through simple neglect. Acts as chair of, and appoints members to, the data council.

Management 98

Management Risk Measurement Optimization 98

AWS Big Data

MARCH 9, 2023

SCD2 metadata – rec_eff_dt and rec_exp_dt indicate the state of the record. The value in rec_exp_dt will be set as ‘9999-12-31’ for presently active records. Register source tables in the AWS Glue Data Catalog We use an AWS Glue crawler to infer metadata from delimited data files like the CSV files used in this post.

Slice and Dice 98

Slice and Dice Data Warehouse Metrics Metadata 98

AWS Big Data

JANUARY 31, 2023

In this post, we are excited to summarize the features that the AWS Glue Data Catalog, AWS Glue crawler, and Lake Formation teams delivered in 2022. Whether you are a data platform builder, data engineer, data scientist, or any technology leader interested in data lake solutions, this post is for you.

Data Lake 98

Data Lake Data Governance Data Architecture Machine Learning 98

Alation

AUGUST 11, 2021

The three of us talked migration strategy and the best way to move to the Snowflake Data Cloud. As Vice President of Data Governance at TMIC, Anthony has robust experience leading cloud migration as part of a larger data strategy. Creating an environment better suited for data governance.

Data Governance 52

Data Governance Insurance Metadata Strategy 52

Alation

APRIL 12, 2022

Between 2010 and 2018 the number of CDOs present in Fortune 1500 companies increased nearly 8-fold. Today, the modern CDO drives the data strategy for the entire organization. The CDO’s Role in Driving a Data Strategy. Beyond the concrete implementation of a data strategy, CDOs often have to foster a data culture.

Data Quality 52

Data Quality Data Governance Metrics Data Strategy 52

IBM Big Data Hub

MARCH 14, 2024

Businesses face significant hurdles when preparing data for artificial intelligence (AI) applications. The existence of data silos and duplication, alongside apprehensions regarding data quality, presents a multifaceted environment for organizations to manage.

Cost-Benefit 75

Cost-Benefit Metadata Optimization Management 75

TDAN

AUGUST 31, 2021

Can the responsibilities for vocabulary ownership and data ownership by business stakeholders be separate? I have listened to many presentations and read many articles about data governance (or data stewardship if you prefer), but I have never come across anyone saying they can and should be. Should they be?

Data Governance 52

Data Governance IT Metadata Data Strategy 52

TDAN

JANUARY 5, 2021

Several years ago, I wrote an article called the Data Governance Bill of “Rights.” I also speak often about my Bill of “Rights” in many of my webinars and presentations. Please notice that I put the word “rights” in quotations. By rights, I do not mean human rights, or the freedoms to claim equality based […].

Data Governance 52

Data Governance Metadata Data Strategy Strategy 52

TDAN

DECEMBER 20, 2022

Effective data governance for the public sector enables entities to ensure data quality, enhance security, protect privacy, and meet compliance requirements. With so much focus on compliance, democratizing data for self-service analytics can present a challenge.

Data Governance 52

Data Governance Data Quality Analytics Metadata 52

TDAN

MAY 14, 2019

To own data or not to own data, that is the question. This question comes up often when I am speaking with clients or groups of people during my Data Governance webinars and conference presentations.

Data Governance 51

Data Governance Data Quality Metadata Data Strategy 51

TDAN

MARCH 5, 2019

Getting business and leadership support for data governance programs – and building a data culture on that buy-in – remains a significant challenge in many organizations. The results of the new survey were presented at a Collibra event […].

Data Governance 40

Data Governance Metadata Data Strategy Strategy 40

CIO Business Intelligence

JULY 13, 2022

Data makes the most ambitious and even idealistic goals—like making the world a better place—possible. This is intrinsically worthwhile, but it has now been codified as part of the Federal Data Strategy and its stated mission to “fully leverage the value of federal data for mission, service, and the public good.”

Data-driven 98

Data-driven Metadata Optimization Machine Learning 98

Domino Data Lab

OCTOBER 9, 2019

This was alongside keynotes by: Rebecca Williams from OMB at the Whitehouse—who helped develop the US federal data strategy and year-1 action plan —check out her slides for the “Federal Data Strategy” keynote. Monica Youngman, director of data stewardship—check out her slides for the “Data Archiving at NOAA” keynote.

Deep Learning 69

Deep Learning Metadata Machine Learning Data Science 69

Cloudera

JUNE 24, 2022

Data makes the most ambitious and even idealistic goals —like making the world a better place — possible. This is intrinsically worthwhile, but it has now been codified as part of the Federal Data Strategy and its stated mission to “fully leverage the value of federal data for mission, service, and the public good.”

Data-driven 76

Data-driven Metadata Optimization Machine Learning 76

AWS Big Data

NOVEMBER 13, 2024

The first section of this post discusses how we aligned the technical design of the data solution with the data strategy of Volkswagen Autoeuropa. Next, we detail the governance guardrails of the Volkswagen Autoeuropa data solution. These data products belonged to data domains such as production, finance, and logistics.

Metadata 105

Metadata Data Quality Digital Transformation Data-driven 105