O'Reilly on Data

MAY 21, 2019

Consider deep learning, a specific form of machine learning that resurfaced in 2011/2012 due to record-setting models in speech and computer vision. Metadata and artifacts needed for audits. The technologies I’ve alluded to above—data governance, data lineage, model governance—are all going to be useful for helping manage these risks.

Machine Learning 279

Machine Learning Technology Deep Learning Data Science 279

AWS Big Data

APRIL 19, 2023

Instead, we can use automation to speed up the process of migration and reduce heavy lifting tasks, costs, and risks. We split the solution into two primary components: generating Spark job metadata and running the SQL on Amazon EMR. Generate Spark SQL metadata Our batch job consists of Hive steps scheduled to run sequentially.

Metadata 87

Metadata Data Lake Testing Consulting 87

AWS Big Data

AUGUST 15, 2024

Eliminating dependency on business units – Redshift Spectrum uses a metadata layer to directly query the data residing in S3 data lakes, eliminating the need for data copying or relying on individual business units to initiate the copy jobs. There are no duplicate data products created by business units or the Central IT team.

Data Lake 108

Data Lake Data Warehouse Data Governance Publishing 108

AWS Big Data

MARCH 1, 2024

One of the bank’s key challenges related to strict cybersecurity requirements is to implement field level encryption for personally identifiable information (PII), Payment Card Industry (PCI), and data that is classified as high privacy risk (HPR). Only users with required permissions are allowed to access data in clear text.

Data Lake 113

Data Lake Data Warehouse Management Risk 113

Domino Data Lab

APRIL 3, 2019

That’s a lot of priorities – especially when you group together closely related items such as data lineage and metadata management which rank nearby. Also, while surveying the literature two key drivers stood out: Risk management is the thin-edge-of-the-wedge ?for Allows metadata repositories to share and exchange.

Machine Learning 84

Machine Learning Data Governance Metadata Data Science 84

AWS Big Data

APRIL 3, 2024

As data is refreshed and updated, changes can happen through upstream processes that put it at risk of not maintaining the intended quality. By selecting the corresponding asset, you can understand its content through the readme, glossary terms , and technical and business metadata.

Data Quality 103

Data Quality Visualization Metadata Metrics 103

AWS Big Data

JANUARY 22, 2024

Data as a product Treating data as a product entails three key components: the data itself, the metadata, and the associated code and infrastructure. For orchestration, they use the AWS Cloud Development Kit (AWS CDK) for infrastructure as code (IaC) and AWS Glue Data Catalogs for metadata management.

Data-driven 99

Data-driven Advertising Metadata Data Architecture 99