AWS Big Data

MAY 4, 2023

The sheer volume of data captured daily continues to grow, calling for platforms and solutions to evolve. Services such as Amazon Simple Storage Service (Amazon S3) offer a scalable solution that adapts yet remains cost-effective for growing datasets. This solution was inspired by work with a key AWS customer, the UK Met Office.

Data Processing 122

Data Processing Metadata Informatics Interactive 122

CIO Business Intelligence

MAY 5, 2022

Jupyter Notebooks. Jupyter Notebooks let readers do more than absorb. Jupyter Notebooks let readers do more than absorb. Today, the standard Jupyter Notebook supports more than 40 programming languages, and it’s common to find R, Julia, or even Java or C within them. Jupyter Notebooks don’t just run themselves.

Data Science 130

Data Science Analytics Machine Learning Optimization 130

AWS Big Data

JUNE 26, 2023

Overview of solution In this post, we go through the various steps to apply ML-based fuzzy matching to harmonize customer data across two different datasets for auto and property insurance. The following diagram shows our solution architecture. Prerequisites To follow along with this walkthrough, you must have an AWS account.

Insurance 98

Insurance Visualization Data Lake Metrics 98

AWS Big Data

SEPTEMBER 20, 2023

AWS Glue interactive sessions offer a powerful way to iteratively explore datasets and fine-tune transformations using Jupyter-compatible notebooks. Solution overview You can quickly provision new interactive sessions directly from your notebook without needing to interact with the AWS Command Line Interface (AWS CLI) or the console.

Interactive 111

Interactive Visualization Measurement Data Architecture 111

Cloudera

NOVEMBER 17, 2021

In recognition of the diverse workload that data scientists face, Cloudera’s library of Applied ML Prototypes (AMPs) provide Data Scientists with pre-built reference examples and end-to-end solutions, using some of the most cutting edge ML methods, for a variety of common data science projects. AutoML with TPOT.

Machine Learning 124

Machine Learning Visualization Data Science Metrics 124

AWS Big Data

SEPTEMBER 20, 2023

This technology is enabled by the use of notebook IDEs, such as the AWS Glue Studio notebook, Amazon SageMaker Studio , or your own Jupyter notebooks. Users can run AWS Glue interactive sessions by using both AWS Glue Studio notebooks via the AWS Glue console, as well as Jupyter notebooks that run on their local machine.

Interactive 119

Interactive Management Reporting IT 119

AWS Big Data

JUNE 15, 2023

Amazon EMR , with its open-source Hadoop modules and support for Apache Spark and Jupyter and JupyterLab notebooks, is a good choice to solve this multi-cloud data access problem. Overview of solution Amazon EMR inherently includes Apache Hadoop at its core and integrates other related open-source modules.

Big Data 105

Big Data Data Lake Management Testing 105

O'Reilly on Data

FEBRUARY 18, 2020

Kubernetes has emerged as the de facto solution for orchestrating services and microservices in cloud native design patterns. relational database,” “Oracle database solutions,” “Hive,” “database administration,” “data models,” “Spark”—declined in usage, year-over-year, in 2019.

Data-driven 363

Data-driven Software Statistics Marketing 363

CIO Business Intelligence

MAY 12, 2022

Composite AI mixes statistics and machine learning; industry-specific solutions. SageMaker is a full-service platform with data preparation tools such as the Data Wrangler, a nice presentation layer built out of Jupyter notebooks, and an automated option called Autopilot. On premises or in SAP cloud. Per user, per month. Free tier.

Predictive Analytics 137

Predictive Analytics Analytics Statistics Machine Learning 137

AWS Big Data

JUNE 18, 2024

This blog post provides a step-by-step guide for building a multimodal search solution using OpenSearch Service. Multimodal search solution architecture We will provide the steps required to set up multimodal search using OpenSearch Service. The following image depicts the solution architecture. OpenSearch version is 2.13

Dashboards 119

Dashboards Metadata Modeling Visualization 119

AWS Big Data

OCTOBER 3, 2023

Open table formats, such as Apache Iceberg , provide a solution to this issue. In this post, we show you how you can convert existing data in an Amazon S3 data lake in Apache Parquet format to Apache Iceberg format to support transactions on the data using Jupyter Notebook based interactive sessions over AWS Glue 4.0. Choose ETL Jobs.

Data Lake 122

Data Lake Metadata Snapshot Recreation/Entertainment 122

AWS Big Data

MARCH 17, 2023

In this post, we present a solution that uses Uber’s Hexagonal Hierarchical Spatial Index (H3) to divide the globe into equally-sized hexagons. Solution overview The solution extends Athena’s built-in geospatial capabilities by creating a UDF powered by AWS Lambda. Open the notebook instance by choosing Jupyter or JupyterLab.

Visualization 105

Visualization Machine Learning Consulting Data Warehouse 105

AWS Big Data

JANUARY 17, 2024

Solution overview In this post, we demonstrate how to implement FGAC on Apache Hudi tables using Amazon EMR on Amazon Elastic Compute Cloud (Amazon EC2) integrated with Lake Formation. The following diagram illustrates the solution architecture. For example, the users only can access data rows that belong to their country.

Data Lake 116

Data Lake Snapshot Big Data Data-driven 116

AWS Big Data

JULY 6, 2023

Solution overview To achieve our goal, we use parallel Lambda functions. Deployment of this solution: In this post, we provide step-by-step instructions to deploy each part of the architecture manually. The following screenshot illustrates running the preceding code in a Jupyter notebook.

Machine Learning 145

Machine Learning Visualization IoT Digital Transformation 145

CIO Business Intelligence

OCTOBER 7, 2022

BI developers, with an average salary of $83,091, work with databases and software to develop and fine-tune IT solutions. The cert demonstrates that you are up-to-date with BI technologies and are knowledgeable about best practices, solutions, and emerging trends.

Business Intelligence 145

Business Intelligence Data Warehouse Visualization Dashboards 145

Domino Data Lab

JUNE 2, 2019

It’s that practice of engineering at Netflix which I find so interesting – really since Ariel Tseitlin led Cloud Solutions there. For example, while enjoying dinner the evening before Rev, a close friend questioned the use of Jupyter beyond exploratory data science. Jupyter fits brilliantly for that purpose. — Randi R.

Data Science 110

Data Science Data-driven Machine Learning Modeling 110

Domino Data Lab

JANUARY 11, 2021

Domino provides data scientists with the ability to run code either in workspace environments that provide IDEs such as Jupyter, R Studio or VS Code or conversely to create a job that runs a particular piece of code. About Snowflake. About Domino Data Lab. Domino Data Lab is the system-of-record for enterprise data science teams.

Data Science 103

Data Science Recreation/Entertainment Data Warehouse Publishing 103

Ontotext

DECEMBER 2, 2021

In our previous blog posts of the series, we talked about how to ingest data from different sources into GraphDB , validate it and infer new knowledge from the extant facts as well as how to adapt and scale our basic solution. With Ontotext’s products , LAZY is firmly on the way towards finding a solution.

Visualization 98

Visualization Interactive Dashboards Enterprise 98

CIO Business Intelligence

APRIL 22, 2022

While that may involve one-off projects, more typically data science teams seek to identify key data assets that can be turned into data pipelines that feed maintainable tools and solutions. Examples include credit card fraud monitoring solutions used by banks, or tools used to optimize the placement of wind turbines in wind farms.

Data Science 140

Data Science Statistics Machine Learning Visualization 140

Alation

AUGUST 9, 2021

Today, Blackstone’s data analytics stack includes Fivetran, Snowflake, Sigma, Jupyter Notebooks, Alation, and various other tools. Pologruto solved this problem by giving people the option to analyze data in the tool of their choice: Jupyter Notebooks for data scientists and more technical teams, and Sigma for business teams.

Analytics 98

Analytics Data-driven Data Analytics Contextual Data 98

DataRobot

MAY 11, 2021

Founded in 2016 by the creator of Apache Zeppelin, Zepl provides a self-service data science notebook solution for advanced data scientists to do exploratory, code-centric work in Python, R, and Scala. It supports both Zeppelin and Jupyter notebooks — new or imported. The Perfect Complement. DataRobot + Zepl. Sign Up Now.

Data Science 98

Data Science Cost-Benefit Data Processing Machine Learning 98

AWS Big Data

APRIL 20, 2023

A Jupyter notebook to run using Amazon EMR Studio using Amazon EMR on an EC2 cluster A PySpark script to run using Amazon EMR Studio and Amazon EMR Serverless After the stack creation is complete, choose the stack name redshift-spark and navigate to the Outputs We utilize these output values later in this post.

Data Lake 98

Data Lake Data Warehouse Sales Data-driven 98

AWS Big Data

FEBRUARY 7, 2025

This example provides a solution for enterprises looking to enhance their AI capabilities. Solution overview The following diagram illustrates the solution architecture. In an actual solution, you would encapsulate the code in classes and pass the values where needed. Use these scripts as examples to pull from.

Data Processing 69

Data Processing Dashboards Modeling Statistics 69

AWS Big Data

FEBRUARY 1, 2024

Solution overview In the following sections, we first introduce the Common Crawl dataset and how to explore and filter the data we need. SageMaker JumpStart provides a set of solutions for the most common use cases that can be deployed with just a few clicks. The following diagram illustrates the architecture of this solution.

Metadata 122

Metadata Modeling Data Processing Unstructured Data 122

CIO Business Intelligence

NOVEMBER 4, 2022

Then the data is consumed by SaaS-based computational tools, but it still sits within our organization and sits within the controls of our cloud-based solutions.” From a language perspective, scientists use Python and Jupyter Notebooks. Much of Regeneron’s data, of course, is confidential.

Data Lake 124

Data Lake IT Experimentation Data-driven 124

AWS Big Data

MAY 28, 2024

Solution overview Our solution demonstrates how financial analysts can use generative artificial intelligence (AI) to adapt their investment recommendations based on financial reports and earnings transcripts with RAG to use LLMs to generate factual content. This makes RAG adaptive for situations where facts could evolve over time.

Unstructured Data 118

Unstructured Data Data Warehouse Structured Data Testing 118

Jen Stirrup

MAY 5, 2020

You can use Eclipse technology, Jupyter and Zeppelin. In the Data Science world, Jupyter is very common. Apache Spark is an open source software solution until allows you to work on data that’s held in memory. We can also use it with Jupyter Notebooks as well. So, let’s take a look!

Machine Learning 89

Machine Learning Data Science Data Lake Big Data 89

DataKitchen

OCTOBER 25, 2022

Your “simple” pipeline involves a toolchain that features Fivetran, DBT, SQL, a Jupyter notebook, and Tableau. While quite valuable, these solutions all produce lagging indicators. For more information about DataKitchen's Observability solution, contact us!

Testing 130

Testing Data-driven Visualization Dashboards 130

Domino Data Lab

APRIL 29, 2021

If we can crack the nut of enabling a wider workforce to build AI solutions, we can start to realize the promise of data science. In the future it will be as easy to train on hundreds of GPUs as it is to train on a Jupyter notebook in a managed workspace. An important part of these foundational layers is Keras Tuner and AutoKeras.

Data Science 134

Data Science Deep Learning Modeling Forecasting 134

FineReport

JULY 28, 2021

By observing and analyzing data, we can develop more accurate theories and formulate more effective solutions. Because sending Jupyter Notebook or Python scripts to non-technical people will undoubtedly cause some problems, which prevents them from being widely promoted in the traditional business of large enterprises.

Business Intelligence 98

Business Intelligence Data Science Data Analytics Analytics 98

AWS Big Data

SEPTEMBER 13, 2024

This post proposes a solution to this challenge by introducing the Batch Processing Gateway (BPG) , a centralized gateway that automates job management and routing in multi-cluster environments. Solution overview Martin Fowler describes a gateway as an object that encapsulates access to an external system or resource.

Management 111

Management Snapshot Cost-Benefit Testing 111

AWS Big Data

AUGUST 27, 2024

Solution overview The solution is designed to help you track the cost of your Spark applications running on EMR on EC2. The proposed solution uses a scheduled AWS Lambda function that operates on a daily basis. The utilization of these AWS services incurs additional costs for implementing this solution.

Metrics 105

Metrics Dashboards Data Lake Optimization 105

Domino Data Lab

JANUARY 6, 2019

We do not and cannot have a “one size fits all” solution for data science training. NASA persistently misspells Jupyter. In terms of teaching and learning data science, Project Jupyter is probably the biggest news over the past decade – even though Jupyter’s origins go back to 2001! Translated: MOOCs are no panacea.

Data Science 56

Data Science Machine Learning Visualization Reporting 56

Ontotext

NOVEMBER 18, 2021

Through this series of blog posts, we’ll discuss how to best scale and branch out an analytics solution using a knowledge graph technology stack. Our main weapons when beating that beast will be GraphDB , Ontotext Platform , Kafka , Elasticsearch , Kibana and Jupyter.

Visualization 98

Visualization Reporting Metadata Enterprise 98

Domino Data Lab

OCTOBER 23, 2019

Often it is easy to collect data manually, that is, downloading from a web site and cleaning it up manually in Excel, Jupyter Notebook, or RStudio. Install a few packages that we will use for this chapter: i.e., Pandas, Jupyter. To inspect the data, start a Jupyter Notebook using the command: jupyter notebook.

Statistics 59

Statistics Machine Learning Testing Modeling 59

IBM Big Data Hub

NOVEMBER 29, 2023

” No-code and low-code solutions for time series data exploration IBM introduced Downer to the realm where no-code and low-code solutions could build predictive models to provide faster insights. Speed and scalability : Downer efficiently adapted models to diverse data signals using tools like the Jupyter Notebook.

Predictive Modeling 63

Predictive Modeling Data Science Data-driven Forecasting 63

Sisense

AUGUST 22, 2019

Even if your company is already using them, they are often used through Jupyter Notebooks or R Studio. This means that every exercise is a complex challenge of data engineering, and even when the work is done the results are removed from your visualization and reporting solutions.

Analytics 73

Analytics Business Intelligence Visualization Predictive Analytics 73