Book Image

Apache Spark for Data Science Cookbook

By : Padma Priya Chitturi
Book Image

Apache Spark for Data Science Cookbook

By: Padma Priya Chitturi

Overview of this book

Spark has emerged as the most promising big data analytics engine for data science professionals. The true power and value of Apache Spark lies in its ability to execute data science tasks with speed and accuracy. Spark’s selling point is that it combines ETL, batch analytics, real-time stream analysis, machine learning, graph processing, and visualizations. It lets you tackle the complexities that come with raw unstructured data sets with ease. This guide will get you comfortable and confident performing data science tasks with Spark. You will learn about implementations including distributed deep learning, numerical computing, and scalable machine learning. You will be shown effective solutions to problematic concepts in data science using Spark’s data science libraries such as MLLib, Pandas, NumPy, SciPy, and more. These simple and efficient recipes will show you how to implement algorithms and optimize your work.

Related Content you might be interested in

Current Title:

Small book image
Apache Spark for Data Science Cookbook
Padma Priya Chitturi
Dec, 2016 | 392 pages
No titles found
Table of Contents (17 chapters)
Apache Spark for Data Science Cookbook
Apache Spark for Data Science Cookbook
Credits
Credits
About the Author
About the Author
About the Reviewer
About the Reviewer
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Big Data Analytics with Spark
Big Data Analytics with Spark
Introduction
Initializing SparkContext
Working with Spark's Python and Scala shells
Building standalone applications
Working with the Spark programming model
Working with pair RDDs
Persisting RDDs
Loading and saving data
Creating broadcast variables and accumulators
Submitting applications to a cluster
Working with DataFrames
Working with Spark Streaming
2
Tricky Statistics with Spark
Tricky Statistics with Spark
Introduction
Variable identification
Sampling data
Summary and descriptive statistics
Generating frequency tables
Installing Pandas on Linux
Installing Pandas from source
Using IPython with PySpark
Creating Pandas DataFrames over Spark
Splitting, slicing, sorting, filtering, and grouping DataFrames over Spark
Implementing co-variance and correlation using Pandas
Concatenating and merging operations over DataFrames
Complex operations over DataFrames
Sparkling Pandas
3
Data Analysis with Spark
Data Analysis with Spark
Introduction
Univariate analysis
Bivariate analysis
Missing value treatment
Outlier detection
Use case - analyzing the MovieLens dataset
Use case - analyzing the Uber dataset
4
Clustering, Classification, and Regression
Clustering, Classification, and Regression
Introduction
Supervised learning
Unsupervised learning
Applying regression analysis for sales data
Variable identification
Data exploration
Feature engineering
Applying linear regression
Applying logistic regression on bank marketing data
Variable identification
Data exploration
Feature engineering
Applying logistic regression
Real-time intrusion detection using streaming k-means
Variable identification
Simulating real-time data
Applying streaming k-means
5
Working with Spark MLlib
Working with Spark MLlib
Introduction
Working with Spark ML pipelines
Implementing Naive Bayes' classification
Implementing decision trees
Building a recommendation system
Implementing logistic regression using Spark ML pipelines
6
NLP with Spark
NLP with Spark
Introduction
Installing NLTK on Linux
Installing Anaconda on Linux
Anaconda for cluster management
POS tagging with PySpark on an Anaconda cluster
NER with IPython over Spark
Implementing openNLP - chunker over Spark
Implementing openNLP - sentence detector over Spark
Implementing stanford NLP - lemmatization over Spark
Implementing sentiment analysis using stanford NLP over Spark
7
Working with Sparkling Water - H2O
Working with Sparkling Water - H2O
Introduction
Features
Working with H2O on Spark
Implementing k-means using H2O over Spark
Implementing spam detection with Sparkling Water
Deep learning with airlines and weather data
Implementing a crime detection application
Running SVM with H2O over Spark
8
Data Visualization with Spark
Data Visualization with Spark
Introduction
Visualization using Zeppelin
Installing Zeppelin
Customizing Zeppelin's server and websocket port
Visualizing data on HDFS - parameterizing inputs
Running custom functions
Adding external dependencies to Zeppelin
Pointing to an external Spark Cluster
Creating scatter plots with Bokeh-Scala
Creating a time series MultiPlot with Bokeh-Scala
Creating plots with the lightning visualization server
Visualize machine learning models with Databricks notebook
9
Deep Learning on Spark
Deep Learning on Spark
Introduction
Installing CaffeOnSpark
Working with CaffeOnSpark
Running a feed-forward neural network with DeepLearning 4j over Spark
Running an RBM with DeepLearning4j over Spark
Running a CNN for learning MNIST with DeepLearning4j over Spark
Installing TensorFlow
Working with Spark TensorFlow
10
Working with SparkR
Working with SparkR
Introduction
Installing R
Interactive analysis with the SparkR shell
Creating a SparkR standalone application from RStudio
Creating SparkR DataFrames
SparkR DataFrame operations
Applying user-defined functions in SparkR
Running SQL queries from SparkR and caching DataFrames
Machine learning with SparkR

Missing value treatment

Missing data in the training dataset can reduce the fitness of a model or can lead to a biased model because we have not analyzed the behavior and relationship with other variables correctly. This could also lead to wrong predictions or classifications. The reasons for the occurrence of the missing values could be that while extracting data from multiple sources, there is a possible chance to have missing data. Hence, using some hashing procedure ensures that the data extraction is correct. The errors that occur at the time of data collection are tougher to correct as the values might miss at random and the missing values might also depend on the unobserved predictors.

Getting ready

To step through this recipe, you will need a running Spark cluster in any one of the modes, that is, local, standalone, YARN, or Mesos. For installing Spark on a standalone cluster, please refer http://spark.apache.org/docs/latest/spark-standalone.html. Also, include the Spark MLlib package...

Previous Section
End of Section 5
Next Section