Book Image

Apache Spark Deep Learning Cookbook

By : Ahmed Sherif, Amrith Ravindra
Book Image

Apache Spark Deep Learning Cookbook

By: Ahmed Sherif, Amrith Ravindra

Overview of this book

Organizations these days need to integrate popular big data tools such as Apache Spark with highly efficient deep learning libraries if they’re looking to gain faster and more powerful insights from their data. With this book, you’ll discover over 80 recipes to help you train fast, enterprise-grade, deep learning models on Apache Spark. Each recipe addresses a specific problem, and offers a proven, best-practice solution to difficulties encountered while implementing various deep learning algorithms in a distributed environment. The book follows a systematic approach, featuring a balance of theory and tips with best practice solutions to assist you with training different types of neural networks such as convolutional neural networks (CNNs) and recurrent neural networks (RNNs). You’ll also have access to code written in TensorFlow and Keras that you can run on Spark to solve a variety of deep learning problems in computer vision and natural language processing (NLP), or tweak to tackle other problems encountered in deep learning. By the end of this book, you'll have the skills you need to train and deploy state-of-the-art deep learning models on Apache Spark.

Related Content you might be interested in

Current Title:

Small book image
Apache Spark Deep Learning Cookbook
Ahmed Sherif | Amrith Ravindra
Jul, 2018 | 474 pages
Small book image
Machine Learning with Apache Spark Quick Start Guide
Jillur Quddus
Dec, 2018 | 240 pages
Small book image
Deep Learning for Natural Language Processing
Shubhangi Hora | Karthiek Reddy Bokka | Tanuj Jain | Monicah Wambugu
Jun, 2019 | 372 pages
Small book image
Hands-On Deep Learning with R
Rodger Devine | Michael Pawlus
Apr, 2020 | 330 pages
Table of Contents (21 chapters)
Title Page
Title Page
Copyright and Credits
Copyright and Credits
Packt Upsell
Packt Upsell
Foreword
Foreword
Contributors
Contributors
Preface
Preface
Free Chapter
1
Setting Up Spark for Deep Learning Development
Setting Up Spark for Deep Learning Development
Introduction
Downloading an Ubuntu Desktop image
Installing and configuring Ubuntu with VMWare Fusion on macOS
Installing and configuring Ubuntu with Oracle VirtualBox on Windows
Installing and configuring Ubuntu Desktop for Google Cloud Platform
Installing and configuring Spark and prerequisites on Ubuntu Desktop
Integrating Jupyter notebooks with Spark
Starting and configuring a Spark cluster
Stopping a Spark cluster
2
Creating a Neural Network in Spark
Creating a Neural Network in Spark
Introduction
Creating a dataframe in PySpark
Manipulating columns in a PySpark dataframe
Converting a PySpark dataframe to an array
Visualizing an array in a scatterplot
Setting up weights and biases for input into the neural network
Normalizing the input data for the neural network
Validating array for optimal neural network performance
Setting up the activation function with sigmoid
Creating the sigmoid derivative function
Calculating the cost function in a neural network
Predicting gender based on height and weight
Visualizing prediction scores
3
Pain Points of Convolutional Neural Networks
Pain Points of Convolutional Neural Networks
Introduction
Pain Point #1: Importing MNIST images
Pain Point #2: Visualizing MNIST images
Pain Point #3: Exporting MNIST images as files
Pain Point #4: Augmenting MNIST images
Pain Point #5: Utilizing alternate sources for trained images
Pain Point #6: Prioritizing high-level libraries for CNNs
4
Pain Points of Recurrent Neural Networks
Pain Points of Recurrent Neural Networks
Introduction
Introduction to feedforward networks
Sequential workings of RNNs
Pain point #1 – The vanishing gradient problem
Pain point #2 – The exploding gradient problem
Sequential working of LSTMs
5
Predicting Fire Department Calls with Spark ML
Predicting Fire Department Calls with Spark ML
Introduction
Downloading the San Francisco fire department calls dataset
Identifying the target variable of the logistic regression model
Preparing feature variables for the logistic regression model
Applying the logistic regression model
Evaluating the accuracy of the logistic regression model
6
Using LSTMs in Generative Networks
Using LSTMs in Generative Networks
Introduction
Downloading novels/books that will be used as input text
Preparing and cleansing data
Tokenizing sentences
Training and saving the LSTM model
Generating similar text using the model
7
Natural Language Processing with TF-IDF
Natural Language Processing with TF-IDF
Introduction
Downloading the therapy bot session text dataset
Analyzing the therapy bot session dataset
Visualizing word counts in the dataset
Calculating sentiment analysis of text
Removing stop words from the text
Training the TF-IDF model
Evaluating TF-IDF model performance
Comparing model performance to a baseline score
8
Real Estate Value Prediction Using XGBoost
Real Estate Value Prediction Using XGBoost
Downloading the King County House sales dataset
Performing exploratory analysis and visualization
Plotting correlation between price and other features
Predicting the price of a house
9
Predicting Apple Stock Market Cost with LSTM
Predicting Apple Stock Market Cost with LSTM
Downloading stock market data for Apple
Exploring and visualizing stock market data for Apple
Preparing stock data for model performance
Building the LSTM model
Evaluating the model
10
Face Recognition Using Deep Convolutional Networks
Face Recognition Using Deep Convolutional Networks
Introduction
Downloading and loading the MIT-CBCL dataset into the memory
Plotting and visualizing images from the directory
Preprocessing images
Model building, training, and analysis
11
Creating and Visualizing Word Vectors Using Word2Vec
Creating and Visualizing Word Vectors Using Word2Vec
Introduction
Acquiring data
Importing the necessary libraries
Preparing the data
Building and training the model
Visualizing further 
Analyzing further
12
Creating a Movie Recommendation Engine with Keras
Creating a Movie Recommendation Engine with Keras
Introduction
Downloading MovieLens datasets
Manipulating and merging the MovieLens datasets
Exploring the MovieLens datasets
Preparing dataset for the deep learning pipeline
Applying the deep learning model with Keras
Evaluating the recommendation engine's accuracy
13
Image Classification with TensorFlow on Spark
Image Classification with TensorFlow on Spark
Introduction
Downloading 30 images each of Messi and Ronaldo
Configuring PySpark installation with deep learning packages
Loading images on to PySpark dataframes
Understanding transfer learning
Creating a pipeline for image classification training
Evaluating model performance
Fine-tuning model parameters
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think
Index
Index

Integrating Jupyter notebooks with Spark

When learning Python for the first time, it is useful to use Jupyter notebooks as an interactive developing environment (IDE). This is one of the main reasons why Anaconda is so powerful. It fully integrates all of the dependencies between Python and Jupyter notebooks. The same can be done with PySpark and Jupyter notebooks. While Spark is written in Scala, PySpark allows for the translation of code to occur within Python instead.

Getting ready

Most of the work in this section will just require accessing the .bashrc script from the terminal.

How to do it...

PySpark is not configured to work within Jupyter notebooks by default, but a slight tweak of the .bashrc script can remedy this issue. We will walk through these steps in this section:

  1. Access the .bashrc script by executing the following command:
$ nano .bashrc
  1. Scrolling all the way to the end of the script should reveal the last command modified, which should be the PATH set by Anaconda during the installation earlier in the previous section. The PATH should appear as seen in the following:
# added by Anaconda3 4.4.0 installer
export PATH="/home/asherif844/anaconda3/bin:$PATH"
  1. Underneath, the PATH added by the Anaconda installer can include a custom function that helps communicate the Spark installation with the Jupyter notebook installation from Anaconda3. For the purposes of this chapter and remaining chapters, we will name that function sparknotebook. The configuration should appear as the following for sparknotebook():
function sparknotebook()
{
export SPARK_HOME=/home/asherif844/spark-2.2.0-bin-hadoop2.7
export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS="notebook"
$SPARK_HOME/bin/pyspark
}
  1. The updated .bashrc script should look like the following once saved:
  1. Save and exit from the .bashrc file. It is recommended to communicate that the .bashrc file has been updated by executing the following command and restarting the terminal application:
$ source .bashrc

How it works...

Our goal in this section is to integrate Spark directly into a Jupyter notebook so that we are not doing our development at the terminal and instead utilizing the benefits of developing within a notebook. This section explains how the Spark integration within a Jupyter notebook takes place.

  1. We will create a command function, sparknotebook, that we can call from the terminal to open up a Spark session through Jupyter notebooks from the Anaconda installation. This requires two settings to be set in the .bashrc file:
    1. PySpark Python be set to python 3
    2. PySpark driver for python to be set to Jupyter
  2. The sparknotebook function can now be accessed directly from the terminal by executing the following command:
$ sparknotebook
  1. The function should then initiate a brand new Jupyter notebook session through the default web browser. A new Python script within Jupyter notebooks with a .ipynb extension can be created by clicking on the New button on the right-hand side and by selecting Python 3 under Notebook: as seen in the following screenshot:

  1. Once again, just as was done at the terminal level for Spark, a simple script of sc will be executed within the notebook to confirm that Spark is up and running through Jupyter:
  1. Ideally, the Version, Master, and AppName should be identical to the earlier output when sc was executed at the terminal. If this is the case, then PySpark has been successfully installed and configured to work with Jupyter notebooks.

There's more...

It is important to note that if we were to call a Jupyter notebook through the terminal without specifying sparknotebook, our Spark session will never be initiated and we will receive an error when executing the SparkContext script.

We can access a traditional Jupyter notebook by executing the following at the terminal:

jupyter-notebook

Once we start the notebook, we can try and execute the same script for sc.master as we did previously, but this time we will receive the following error:

See also

There are many managed offerings online of companies offering Spark through a notebook interface where the installation and configuration of Spark with a notebook have already been managed for you. These are the following:

Previous Section
End of Section 8
Next Section