Cloudera

OCTOBER 19, 2021

Airflow has been adopted by many Cloudera Data Platform (CDP) customers in the public cloud as the next generation orchestration service to setup and operationalize complex data pipelines. Multiple steps comprise the overall pipeline, which are stored as pipeline definition files in the CDE resource of the job.

Data Transformation 118

Data Transformation Interactive Machine Learning Testing 118

AWS Big Data

JULY 26, 2023

To grow the power of data at scale for the long term, it’s highly recommended to design an end-to-end development lifecycle for your data integration pipelines. The following are common asks from our customers: Is it possible to develop and test AWS Glue data integration jobs on my local laptop?

Data Integration 72

Data Integration Snapshot Testing Visualization 72

Jen Stirrup

SEPTEMBER 30, 2021

This initial phase focuses on understanding the business value-add from a business perspective, then translating this knowledge into a data mining problem definition. For example, data can be filtered so that the investigation can be focused more specifically. Using R means that we could test, build and assess our model using R.

Business Intelligence 135

Business Intelligence Data mining Machine Learning Testing 135

AWS Big Data

DECEMBER 13, 2023

In addition to using native managed AWS services that BMS didn’t need to worry about upgrading, BMS was looking to offer an ETL service to non-technical business users that could visually compose data transformation workflows and seamlessly run them on the AWS Glue Apache Spark-based serverless data integration engine.

Metadata 94

Metadata Data Lake Visualization Data Transformation 94

AWS Big Data

FEBRUARY 6, 2023

We then discuss a solution to further enhance end-to-end observability by modifying the task definitions that make up the data pipeline. For this post, we focus on task definitions for two services: AWS Glue and Amazon EMR­, however the same method can be applied across different services.

Management 86

Management Interactive Metadata Publishing 86

AWS Big Data

OCTOBER 14, 2024

Prompt with no metadata For the first test, we used a basic prompt containing just the SQL generating instructions and no table metadata. Prompt definition: Human: You are an Amazon Athena query expert whose output is a valid sql query. Prompt definition: Human: You are an Amazon Athena query expert whose output is a valid sql query.

Metadata 82

Metadata Data Lake Modeling Data Warehouse 82

Alation

OCTOBER 18, 2022

Data analysts and engineers use dbt to transform, test, and document data in the cloud data warehouse. Yet every dbt transformation contains vital metadata that is not captured – until now. Data Transformation in the Modern Data Stack. Improve data analysis accuracy.

Metadata 52

Metadata Metrics Recreation/Entertainment Data Quality 52

AWS Big Data

NOVEMBER 15, 2023

It works as a bundle for resources that are bound to a specific staging environment and Region to store data on Amazon Simple Storage Service (Amazon S3), which is renowned for its industry-leading scalability, data availability, security, and performance. Data providers and consumers are the two fundamental users of a CDH dataset.

Dashboards 101

Dashboards Analytics Metadata Data Warehouse 101

AWS Big Data

AUGUST 26, 2024

Duplicating data from a production database to a lower or lateral environment and masking personally identifiable information (PII) to comply with regulations enables development, testing, and reporting without impacting critical systems or exposing sensitive customer data. This saves time over manually defining schemas.

Visualization 100

Visualization Metadata Data Transformation Testing 100

O'Reilly on Data

JANUARY 17, 2023

Upload your data, click through a workflow, walk away. If you’re a professional data scientist, you already have the knowledge and skills to test these models. Get your results in a few hours. Why would you want autoML to build models for you? It buys time and breathing room. It does not exist in the code.

Machine Learning 332

Machine Learning Predictive Modeling Software Modeling 332

Alation

FEBRUARY 20, 2020

In perhaps a preview of things to come next year, we decided to test how a Data Catalog might work with Tableau on the same data. You can check out a self service data prep flow from catalog to viz in this recorded version here. Rita Sallam Introduces the Data Prep Rodeo. To Coming Home, Home on the Range.

Scorecard 52

Scorecard ROI Data-driven Visualization 52

AWS Big Data

MARCH 13, 2024

You can also use the data transformation feature of Data Firehose to invoke a Lambda function to perform data transformation in batches. This solution includes a Lambda function that continuously updates the Amazon Location tracker with simulated location data from fictitious journeys.

Analytics 102

Analytics IoT Metadata Internet of Things 102

Alation

JANUARY 17, 2023

You should look for a data warehouse that is scalable, flexible, and efficient. Popular cloud data warehouses today include Snowflake, Databricks, and BigQuery. If your organization is large, you definitely need to look for robustness. Good data warehouses should be reliable. An example of a data science tool is Dataiku.

Data Warehouse 52

Data Warehouse Cost-Benefit Data Transformation Data Science 52

erwin

JANUARY 11, 2019

Organizations have spent a lot of time and money trying to harmonize data across diverse platforms , including cleansing, uploading metadata, converting code, defining business glossaries, tracking data transformations and so on. So questions linger about whether transformed data can be trusted.

Data Governance 40

Data Governance Risk Metadata Management 40

Ontotext

APRIL 4, 2019

We apply Artificial Intelligence techniques to understand the value locked in this data so we can extract knowledge that can benefit people. Milena Yankova : There are two definitions of Artificial Intelligence. One area where computers will definitely not replace us is our creativity. This is one of the main diagnostic tests.

Recreation/Entertainment 76

Recreation/Entertainment Testing Enterprise Knowledge Discovery 76

IBM Big Data Hub

JANUARY 18, 2023

For these workloads, data lake vendors usually recommend extracting data into flat files to be used solely for model training and testing purposes. This adds an additional ETL step, making the data even more stale. Data lakehouse was created to solve these problems. It is narrower in focus than data fabric.

Data Lake 52

Data Lake Data Warehouse Data-driven Metadata 52

AWS Big Data

JUNE 29, 2023

In this post, we discuss why AWS recommends moving from Kinesis Data Analytics for SQL Applications to Amazon Kinesis Data Analytics for Apache Flink to take advantage of Apache Flink’s advanced streaming capabilities. View the stream data. Transform and enrich the data. Manipulate the data with Python.

Data Analytics 78

Data Analytics Analytics IoT Data Lake 78

Domino Data Lab

AUGUST 20, 2019

The problem is that a new unique identifier of a test example won’t be anywhere in the tree. Feature extraction means moving from low-level features that are unsuitable for learning—practically speaking, we get poor testing results—to higher-level features which are useful for learning. Separate out a hold-out test set.

Testing 68

Testing Modeling Interactive Measurement 68

DataRobot Blog

APRIL 4, 2022

Data Requirements. As with any type of statistical model, data is key and GIGO (“Garbage In, Garbage Out”) principle definitely applies. Historical data needs to be accurate, consistent and, most importantly, available. to find the one that maximizes model fit.

Machine Learning 52

Machine Learning Sales Measurement ROI 52

Occam's Razor

OCTOBER 19, 2010

An expansive view of what it means to use data online, both from the type of data perspective and the kind of desired impact perspective. Given the definition above, I am a firm believer in Multiplicity. If you have evolved to a stage that you need behavior targeting then get Omniture Test and Target or Sitespect.

Analytics 136

Analytics Testing Measurement Optimization 136

AWS Big Data

APRIL 25, 2024

Alternatively, you can use AWS Glue for Apache Spark, which provides built-in support for bucketing configurations during the data transformation process. AWS Glue allows you to define bucketing parameters, such as the number of buckets and the columns to bucket on, providing an optimized data layout for efficient querying with Athena.

Optimization 72

Optimization Data Lake Cost-Benefit Reporting 72

AWS Big Data

NOVEMBER 29, 2023

dbt is an open source, SQL-first templating engine that allows you to write repeatable and extensible data transforms in Python and SQL. dbt is predominantly used by data warehouses (such as Amazon Redshift ) customers who are looking to keep their data transform logic separate from storage and engine.

Data Lake 110

Data Lake Management Metrics Data Warehouse 110

AWS Big Data

OCTOBER 30, 2024

With this launch of JDBC connectivity, Amazon DataZone expands its support for data users, including analysts and scientists, allowing them to work in their preferred environments—whether it’s SQL Workbench, Domino, or Amazon-native solutions—while ensuring secure, governed access within Amazon DataZone. Choose Test connection.

Visualization 110

Visualization Data Lake Testing Data Governance 110

AWS Big Data

OCTOBER 30, 2024

Amazon DataZone recently announced the expansion of data analysis and visualization options for your project-subscribed data within Amazon DataZone using the Amazon Athena JDBC driver. Joel has led data transformation projects on fraud analytics, claims automation, and Master Data Management.

Analytics 96

Analytics Visualization Data Governance Data-driven 96

DataKitchen

APRIL 26, 2021

GSK had been pursuing DataOps capabilities such as automation, containerization, automated testing and monitoring, and reusability, for several years. DataOps provides the “continuous delivery equivalent for Machine Learning and enables teams to manage the complexities around continuous training, A/B testing, and deploying without downtime.

Measurement 246

Measurement Metrics Data-driven Dashboards 246

AWS Big Data

DECEMBER 21, 2023

The data products from the Business Vault and Data Mart stages are now available for consumers. smava decided to use Tableau for business intelligence, data visualization, and further analytics. The data transformations are managed with dbt to simplify the workflow governance and team collaboration.

Data Lake 96

Data Lake Data Warehouse B2B Data-driven 96

Jet Global

FEBRUARY 23, 2024

The quick and dirty definition of data mapping is the process of connecting different types of data from various data sources. Data mapping is a crucial step in data modeling and can help organizations achieve their business goals by enabling data integration, migration, transformation, and quality.

Data Warehouse 52

Data Warehouse Reporting Data Transformation Sales 52

Jet Global

MAY 1, 2023

Introduction Why should I read the definitive guide to embedded analytics? The Definitive Guide to Embedded Analytics is designed to answer any and all questions you have about the topic. It is now most definitely a need-to-have. Strategic Objective Create a complete, user-friendly view of the data by preparing it for analysis.

Analytics 52

Analytics Cost-Benefit Visualization Dashboards 52