Book Image

Apache Spark Deep Learning Cookbook

By : Ahmed Sherif, Amrith Ravindra
Book Image

Apache Spark Deep Learning Cookbook

By: Ahmed Sherif, Amrith Ravindra

Overview of this book

Organizations these days need to integrate popular big data tools such as Apache Spark with highly efficient deep learning libraries if they’re looking to gain faster and more powerful insights from their data. With this book, you’ll discover over 80 recipes to help you train fast, enterprise-grade, deep learning models on Apache Spark. Each recipe addresses a specific problem, and offers a proven, best-practice solution to difficulties encountered while implementing various deep learning algorithms in a distributed environment. The book follows a systematic approach, featuring a balance of theory and tips with best practice solutions to assist you with training different types of neural networks such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). You’ll also have access to code written in TensorFlow and Keras that you can run on Spark to solve a variety of deep learning problems in computer vision and natural language processing (NLP), or tweak to tackle other problems encountered in deep learning. By the end of this book, you'll have the skills you need to train and deploy state-of-the-art deep learning models on Apache Spark.

Related Content you might be interested in

Current Title:

Small book image
Apache Spark Deep Learning Cookbook
Ahmed Sherif | Amrith Ravindra
Jul, 2018 | 474 pages
Small book image
Machine Learning with Apache Spark Quick Start Guide
Jillur Quddus
Dec, 2018 | 240 pages
Small book image
Deep Learning for Natural Language Processing
Shubhangi Hora | Monicah Wambugu | Karthiek Reddy Bokka | Tanuj Jain
Jun, 2019 | 372 pages
Small book image
Hands-On Deep Learning with R
Rodger Devine | Michael Pawlus
Apr, 2020 | 330 pages
Table of Contents (21 chapters)
Title Page
Title Page
Copyright and Credits
Copyright and Credits
Packt Upsell
Packt Upsell
Foreword
Foreword
Contributors
Contributors
Preface
Preface
Free Chapter
1
Setting Up Spark for Deep Learning Development
Setting Up Spark for Deep Learning Development
Introduction
Downloading an Ubuntu Desktop image
Installing and configuring Ubuntu with VMWare Fusion on macOS
Installing and configuring Ubuntu with Oracle VirtualBox on Windows
Installing and configuring Ubuntu Desktop for Google Cloud Platform
Installing and configuring Spark and prerequisites on Ubuntu Desktop
Integrating Jupyter notebooks with Spark
Starting and configuring a Spark cluster
Stopping a Spark cluster
2
Creating a Neural Network in Spark
Creating a Neural Network in Spark
Introduction
Creating a dataframe in PySpark
Manipulating columns in a PySpark dataframe
Converting a PySpark dataframe to an array
Visualizing an array in a scatterplot
Setting up weights and biases for input into the neural network
Normalizing the input data for the neural network
Validating array for optimal neural network performance
Setting up the activation function with sigmoid
Creating the sigmoid derivative function
Calculating the cost function in a neural network
Predicting gender based on height and weight
Visualizing prediction scores
3
Pain Points of Convolutional Neural Networks
Pain Points of Convolutional Neural Networks
Introduction
Pain Point #1: Importing MNIST images
Pain Point #2: Visualizing MNIST images
Pain Point #3: Exporting MNIST images as files
Pain Point #4: Augmenting MNIST images
Pain Point #5: Utilizing alternate sources for trained images
Pain Point #6: Prioritizing high-level libraries for CNNs
4
Pain Points of Recurrent Neural Networks
Pain Points of Recurrent Neural Networks
Introduction
Introduction to feedforward networks
Sequential workings of RNNs
Pain point #1 – The vanishing gradient problem
Pain point #2 – The exploding gradient problem
Sequential working of LSTMs
5
Predicting Fire Department Calls with Spark ML
Predicting Fire Department Calls with Spark ML
Introduction
Downloading the San Francisco fire department calls dataset
Identifying the target variable of the logistic regression model
Preparing feature variables for the logistic regression model
Applying the logistic regression model
Evaluating the accuracy of the logistic regression model
6
Using LSTMs in Generative Networks
Using LSTMs in Generative Networks
Introduction
Downloading novels/books that will be used as input text
Preparing and cleansing data
Tokenizing sentences
Training and saving the LSTM model
Generating similar text using the model
7
Natural Language Processing with TF-IDF
Natural Language Processing with TF-IDF
Introduction
Downloading the therapy bot session text dataset
Analyzing the therapy bot session dataset
Visualizing word counts in the dataset
Calculating sentiment analysis of text
Removing stop words from the text
Training the TF-IDF model
Evaluating TF-IDF model performance
Comparing model performance to a baseline score
8
Real Estate Value Prediction Using XGBoost
Real Estate Value Prediction Using XGBoost
Downloading the King County House sales dataset
Performing exploratory analysis and visualization
Plotting correlation between price and other features
Predicting the price of a house
9
Predicting Apple Stock Market Cost with LSTM
Predicting Apple Stock Market Cost with LSTM
Downloading stock market data for Apple
Exploring and visualizing stock market data for Apple
Preparing stock data for model performance
Building the LSTM model
Evaluating the model
10
Face Recognition Using Deep Convolutional Networks
Face Recognition Using Deep Convolutional Networks
Introduction
Downloading and loading the MIT-CBCL dataset into the memory
Plotting and visualizing images from the directory
Preprocessing images
Model building, training, and analysis
11
Creating and Visualizing Word Vectors Using Word2Vec
Creating and Visualizing Word Vectors Using Word2Vec
Introduction
Acquiring data
Importing the necessary libraries
Preparing the data
Building and training the model
Visualizing further 
Analyzing further
12
Creating a Movie Recommendation Engine with Keras
Creating a Movie Recommendation Engine with Keras
Introduction
Downloading MovieLens datasets
Manipulating and merging the MovieLens datasets
Exploring the MovieLens datasets
Preparing dataset for the deep learning pipeline
Applying the deep learning model with Keras
Evaluating the recommendation engine's accuracy
13
Image Classification with TensorFlow on Spark
Image Classification with TensorFlow on Spark
Introduction
Downloading 30 images each of Messi and Ronaldo
Configuring PySpark installation with deep learning packages
Loading images on to PySpark dataframes
Understanding transfer learning
Creating a pipeline for image classification training
Evaluating model performance
Fine-tuning model parameters
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think
Index
Index

Installing and configuring Ubuntu with VMWare Fusion on macOS

This section will focus on building a virtual machine using an Ubuntu operating system with VMWare Fusion.

Getting ready

A previous installation of VMWare Fusion is required on your system. If you do not currently have this, you can download a trial version from the following website:

https://www.vmware.com/products/fusion/fusion-evaluation.html

How to do it...

Follow the steps in the recipe to configure Ubuntu with VMWare Fusion on macOS:

  1. Once VMWare Fusion is up and running, click on the + button on the upper-left-hand side to begin the configuration process and select New..., as seen in the following screenshot:

  1. Once the selection has been made, select the option to Install from Disk or Image, as seen in the following screenshot:

  1. Select the operating system's iso file that was downloaded from the Ubuntu Desktop website, as seen in the following screenshot:

  1. The next step will ask whether you want to choose Linux Easy Install. It is recommended to do so, as well as to incorporate a Display Name/Password combination for the Ubuntu environment, as seen in the following screenshot:

  1. The configuration process is almost complete. A Virtual Machine Summary is displayed with the option to Customize Settings to increase the Memory and Hard Disk, as seen in the following screenshot:

  1. Anywhere from 20 to 40 GB hard disk space is sufficient for the virtual machine; however, bumping up the memory to either 2 GB or even 4 GB will assist with the performance of the virtual machine when executing Spark code in later chapters. Update the memory by selecting Processors and Memory under the Settings of the virtual machine and increasing the Memory to the desired amount, as seen in the following screenshot:

How it works...

The setup allows for manual configuration of the settings necessary to get Ubuntu Desktop up and running successfully on VMWare Fusion. The memory and hard drive storage can be increased or decreased based on the needs and availability of the host machine.

There's more...

All that is remaining is to fire up the virtual machine for the first time, which initiates the installation process of the system onto the virtual machine. Once all the setup is complete and the user has logged in, the Ubuntu virtual machine should be available for development, as seen in the following screenshot:

See also

Aside from VMWare Fusion, there is also another product that offers similar functionality on a Mac. It is called Parallels Desktop for Mac. To learn more about VMWare and Parallels, and decide which program is a better fit for your development, visit the following websites:

Previous Section
End of Section 4
Next Section