O'Reilly on Data

DECEMBER 19, 2018

We won’t be writing code to optimize scheduling in a manufacturing plant; we’ll be training ML algorithms to find optimum performance based on historical data. have a large body of tools to choose from: IDEs, CI/CD tools, automated testing tools, and so on. If humans are no longer needed to write enterprise applications, what do we do?

Machine Learning 279

Machine Learning Software Metadata Testing 279

AWS Big Data

NOVEMBER 17, 2023

Starting today, the Athena SQL engine uses a cost-based optimizer (CBO), a new feature that uses table and column statistics stored in the AWS Glue Data Catalog as part of the table’s metadata. Let’s discuss some of the cost-based optimization techniques that contributed to improved query performance.

Optimization 124

Optimization Statistics Metadata Data Lake 124

O'Reilly on Data

JUNE 19, 2019

In a previous post , we noted some key attributes that distinguish a machine learning project: Unlike traditional software where the goal is to meet a functional specification, in ML the goal is to optimize a metric. A catalog or a database that lists models, including when they were tested, trained, and deployed.

Modeling 257

Modeling Machine Learning Testing Metrics 257

CIO Business Intelligence

APRIL 8, 2025

As data-centric AI, automated metadata management and privacy-aware data sharing mature, the opportunity to embed data quality into the enterprises core has never been more significant. Data fabric Metadata-rich integration layer across distributed systems. Implementation complexity, relies on robust metadata management.

Data Quality 102

Data Quality Data-driven Key Performance Indicator Metadata 102

AWS Big Data

DECEMBER 5, 2024

As the use of Hydro grows within REA, it’s crucial to perform capacity planning to meet user demands while maintaining optimal performance and cost-efficiency. To address this, we used the AWS performance testing framework for Apache Kafka to evaluate the theoretical performance limits.

Metrics 78

Metrics Dashboards Testing Optimization 78

AWS Big Data

DECEMBER 17, 2024

First query response times for dashboard queries have significantly improved by optimizing code execution and reducing compilation overhead. We have enhanced autonomics algorithms to generate and implement smarter and quicker optimal data layout recommendations for distribution and sort keys, further optimizing performance.

Data Lake 81

Data Lake Data Warehouse Data-driven Optimization 81

erwin

JANUARY 9, 2020

For decades, data modeling has been the optimal way to design and deploy new relational databases with high-quality data sources and support application development. That’s because it’s the best way to visualize metadata , and metadata is now the heart of enterprise data management and data governance/ intelligence efforts.

Data Governance 144

Data Governance Modeling Metadata Unstructured Data 144

O'Reilly on Data

APRIL 2, 2019

Recall the following key attributes of a machine learning project: Unlike traditional software where the goal is to meet a functional specification , in ML the goal is to optimize a metric. A catalog or a database that lists models, including when they were tested, trained, and deployed.

Machine Learning 271

Machine Learning Modeling Data Science Software 271

AWS Big Data

JUNE 10, 2024

Cloudinary is a cloud-based media management platform that provides a comprehensive set of tools and services for managing, optimizing, and delivering images, videos, and other media assets on websites and mobile applications. This concept makes Iceberg extremely versatile.

Data Lake 121

Data Lake Metadata Snapshot Analytics 121

erwin

OCTOBER 8, 2020

With all these diverse metadata sources, it is difficult to understand the complicated web they form much less get a simple visual flow of data lineage and impact analysis. The metadata-driven suite automatically finds, models, ingests, catalogs and governs cloud data assets. GDPR, CCPA, HIPAA, SOX, PIC DSS).

Data Governance 145

Data Governance Metadata Testing Data Lake 145

AWS Big Data

NOVEMBER 20, 2024

We’re excited to announce a new feature in Amazon DataZone that offers enhanced metadata governance for your subscription approval process. With this update, domain owners can define and enforce metadata requirements for data consumers when they request access to data assets. Key benefits The feature benefits multiple stakeholders.

Metadata 75

Metadata Data Governance Metrics Marketing 75

DataKitchen

AUGUST 16, 2024

Data Pipeline Observability: Optimizes pipelines by monitoring data quality, detecting issues, tracing data lineage, and identifying anomalies using live and historical metadata. This capability includes monitoring, logging, and business-rule detection.

Marketing 124

Marketing Data Quality Testing Metadata 124

DataKitchen

JANUARY 5, 2021

Data Governance/Catalog (Metadata management) Workflow – Alation, Collibra, Wikis. Tools influence their optimal iteration cycle time, e.g., months/weeks/days. Observability – Testing inputs, outputs, and business logic at each stage of the data analytics pipeline. Tools determine their approach to solving problems.

Testing 147

Testing Data Governance Metadata Visualization 147

AWS Big Data

NOVEMBER 19, 2024

You can now test the newly created application by running the following command: npm run dev By default, the application is available on port 5173 on your local machine. Unfiltered Table Metadata This tab displays the response of the AWS Glue API GetUnfilteredTableMetadata policies for the selected table.

Data Processing 80

Data Processing Metadata Publishing Testing 80

AWS Big Data

FEBRUARY 26, 2025

S3 Tables are specifically optimized for analytics workloads, resulting in up to 3 times faster query throughput and up to 10 times higher transactions per second compared to self-managed tables. These metadata tables are stored in S3 Tables, the new S3 storage offering optimized for tabular data.

Analytics 59

Analytics Data Lake Metadata Data Warehouse 59

AWS Big Data

MARCH 2, 2023

Apache Iceberg is an open table format for very large analytic datasets, which captures metadata information on the state of datasets as they evolve and change over time. Apache Iceberg addresses customer needs by capturing rich metadata information about the dataset at the time the individual data files are created.

Data Lake 127

Data Lake Data Processing Metadata Snapshot 127

DataKitchen

SEPTEMBER 21, 2023

L1 is usually the raw, unprocessed data ingested directly from various sources; L2 is an intermediate layer featuring data that has undergone some form of transformation or cleaning; and L3 contains highly processed, optimized, and typically ready for analytics and decision-making processes.

Testing 169

Testing Data Quality Predictive Modeling Metrics 169

datapine

JANUARY 31, 2022

Everything is being tested, and then the campaigns that succeed get more money put into them, while the others aren’t repeated. This methodology of “test, look at the data, adjust” is at the heart and soul of business intelligence. Your Chance: Want to try a professional BI analytics software? Let’s see it with a real-world example.

Business Intelligence 214

Business Intelligence Analytics Cost-Benefit ROI 214

AWS Big Data

APRIL 16, 2024

Running Apache Airflow at scale puts proportionally greater load on the Airflow metadata database, sometimes leading to CPU and memory issues on the underlying Amazon Relational Database Service (Amazon RDS) cluster. A resource-starved metadata database may lead to dropped connections from your workers, failing tasks prematurely.

Metadata 110

Metadata Metrics Testing Management 110

O'Reilly on Data

MARCH 23, 2023

Many of these go slightly (but not very far) beyond your initial expectations: you can ask it to generate a list of terms for search engine optimization, you can ask it to generate a reading list on topics that you’re interested in. It was not optimized to provide correct responses. It has helped to write a book.

IT 346

IT Modeling Testing Risk 346

O'Reilly on Data

MAY 21, 2019

A catalog or a database that lists models, including when they were tested, trained, and deployed. Metadata and artifacts needed for audits. In particular, auditing and testing machine learning systems will rely on many of the tools I’ve described above. There are real, not just theoretical, risks and considerations.

Machine Learning 279

Machine Learning Technology Deep Learning Data Science 279

AWS Big Data

OCTOBER 1, 2024

Over the last year, Amazon Redshift added several performance optimizations for data lake queries across multiple areas of query engine such as rewrite, planning, scan execution and consuming AWS Glue Data Catalog column statistics. Performance was tested on a Redshift serverless data warehouse with 128 RPU.

Data Lake 105

Data Lake Statistics Broadcasting Optimization 105

AWS Big Data

AUGUST 8, 2024

We dive into the various optimization techniques AppsFlyer employed, such as partition projection, sorting, parallel query runs, and the use of query result reuse. Additionally, we discuss the thorough testing, monitoring, and rollout process that resulted in a successful transition to the new Athena architecture.

Interactive 117

Interactive Metadata Optimization Testing 117

AWS Big Data

SEPTEMBER 11, 2024

Each file arrives as a pair with a tail metadata file in CSV format containing the size and name of the file. This metadata file is later used to read source file names during processing into the staging layer. These files follow the same naming pattern, with a daily system-generated timestamp appended to each file name.

Data Architecture 96

Data Architecture Optimization Data Warehouse Metadata 96

AWS Big Data

OCTOBER 24, 2024

With its scalability, reliability, and ease of use, Amazon OpenSearch Service helps businesses optimize data-driven decisions and improve operational efficiency. es.amazonaws.com' # e.g. my-test-domain.us-east-1.es.amazonaws.com, Jenkins retrieves JSON files from the GitHub repository and performs validation.

Visualization 91

Visualization Management Data Processing Testing 91

AWS Big Data

MARCH 22, 2024

When you use Trino on Amazon EMR or Athena, you get the latest open source community innovations along with proprietary, AWS developed optimizations. and Athena engine version 2, AWS has been developing query plan and engine behavior optimizations that improve query performance on Trino. Starting from Amazon EMR 6.8.0

Metadata 104

Metadata Statistics Broadcasting Optimization 104

Cloudera

JANUARY 15, 2021

Pushing down column predicate filters to Kudu allows for optimized execution by skipping reading column values for filtered out rows and reducing network IO between a client, like the distributed query engine Apache Impala, and Kudu. The following test was performed on a 6 node cluster with CDP Runtime 7.1.5. CDP Runtime 7.1.5

Optimization 72

Optimization Broadcasting Testing Metadata 72

Occam's Razor

APRIL 8, 2013

Sometimes, we escape the clutches of this sub optimal existence and do pick good metrics or engage in simple A/B testing. You're choosing only one metric because you want to optimize it. Testing out a new feature. Identify, hypothesize, test, react. But it is not routine. So, how do we fix this problem?

Metrics 157

Metrics KPI Analytics Key Performance Indicator 157

Wayne Yaddow

FEBRUARY 18, 2025

However, these two processes are essentially distinct, and their testing needs differ in manyways. As enterprises extend their data pipelines, high-quality, automated testing for both transformations and conversions is critical to assuring data integrity, performance, and compliance across many platforms.

Testing 52

Testing Data Transformation Data-driven Data Quality 52

Cloudera

FEBRUARY 22, 2022

By optimizing the various CDP Data Services, including CDW, CDE, and Cloudera Machine Learning (CML) with Iceberg, Cloudera customers can define and manipulate datasets with SQL commands, build complex data pipelines using features like Time Travel operations, and deploy machine learning models built from Iceberg tables. What’s Next.

Snapshot 110

Snapshot Metadata Cost-Benefit Data Architecture 110

AWS Big Data

JUNE 21, 2024

The Amazon EMR runtime for Apache Spark is a performance-optimized runtime that is 100% API compatible with open source Apache Spark. Amazon EMR on EC2 , Amazon EMR Serverless , Amazon EMR on Amazon EKS , and Amazon EMR on AWS Outposts all use this optimized runtime, which is 4.5 times faster than Apache Spark 3.5.1 and EMR 7.1.

Cost-Benefit 107

Cost-Benefit Testing Optimization Statistics 107

Cloudera

SEPTEMBER 28, 2020

They have dev, test, and production clusters running critical workloads and want to upgrade their clusters to CDP Private Cloud Base. Customer Environment: The customer has three environments: development, test, and production. Test and QA. Test and QA. Let’s take a look at one customer’s upgrade journey. Background: .

Testing 132

Testing Metadata Risk Data Science 132

AWS Big Data

JUNE 9, 2023

Despite these capabilities, data lakes are not databases, and object storage does not provide support for ACID processing semantics, which you may require to effectively optimize and manage your data at scale across hundreds or thousands of users using a multitude of different technologies.

Data Lake 125

Data Lake Metadata Statistics Optimization 125

CIO Business Intelligence

OCTOBER 13, 2023

They understand data modeling, including conceptualization and database optimization, and demonstrate a commitment to continuing education. According to Dataversity , good data architects have a solid understanding of the cloud, databases, and the applications and programs used by those databases.

Data Architecture 132

Data Architecture Data Warehouse Statistics Visualization 132

Domino Data Lab

JULY 2, 2019

In other words, using metadata about data science work to generate code. SQL optimization provides helpful analogies, given how SQL queries get translated into query graphs internally , then the real smarts of a SQL engine work over that graph. On deck this time ’round the Moon: program synthesis. SQL and Spark.

Metadata 105

Metadata Data Science Machine Learning Data-driven 105

datapine

SEPTEMBER 29, 2022

It involves: Reviewing data in detail Comparing and contrasting the data to its own metadata Running statistical models Data quality reports. Also known as data validation, integrity refers to the structural testing of data to ensure that the data complies with procedures. Your Chance: Want to test a professional analytics software?

Data Quality 241

Data Quality Metrics Data-driven Management 241

Ontotext

JANUARY 10, 2025

Our goal is to test whether GenAI can handle diverse domains effectively and determine if its a viable tool for domain-specific graph-building tasks. We also experimented with prompt optimization tools, however these experiments did not yield promising results.

Informatics 59

Informatics Modeling Metadata Experimentation 59