Spark Cookbook

Spark Cookbook

By : Rishi Yadav

Buy this Book

Spark Cookbook

By: Rishi Yadav

Buy this Book

Overview of this book

Spark Cookbook

Credits

About the Author

About the Reviewers

www.PacktPub.com

Preface

Free Chapter

Getting Started with Apache Spark

Introduction

Installing Spark from binaries

Building the Spark source code with Maven

Launching Spark on Amazon EC2

Deploying on a cluster in standalone mode

Deploying on a cluster with Mesos

Deploying on a cluster with YARN

Using Tachyon as an off-heap storage layer

Developing Applications with Spark

Introduction

Exploring the Spark shell

Developing Spark applications in Eclipse with Maven

Developing Spark applications in Eclipse with SBT

Developing a Spark application in IntelliJ IDEA with Maven

Developing a Spark application in IntelliJ IDEA with SBT

External Data Sources

Introduction

Loading data from the local filesystem

Loading data from HDFS

Loading data from HDFS using a custom InputFormat

Loading data from Amazon S3

Loading data from Apache Cassandra

Loading data from relational databases

Spark SQL

Introduction

Understanding the Catalyst optimizer

Creating HiveContext

Inferring schema using case classes

Programmatically specifying the schema

Loading and saving data using the Parquet format

Loading and saving data using the JSON format

Loading and saving data from relational databases

Loading and saving data from an arbitrary source

Spark Streaming

Introduction

Word count using Streaming

Streaming Twitter data

Streaming using Kafka

Getting Started with Machine Learning Using MLlib

Introduction

Creating vectors

Creating a labeled point

Creating matrices

Calculating summary statistics

Calculating correlation

Doing hypothesis testing

Creating machine learning pipelines using ML

Supervised Learning with MLlib – Regression

Introduction

Using linear regression

Understanding cost function

Doing linear regression with lasso

Doing ridge regression

Supervised Learning with MLlib – Classification

Introduction

Doing classification using logistic regression

Doing binary classification using SVM

Doing classification using decision trees

Doing classification using Random Forests

Doing classification using Gradient Boosted Trees

Doing classification with Naïve Bayes

Unsupervised Learning with MLlib

Introduction

Clustering using k-means

Dimensionality reduction with principal component analysis

Dimensionality reduction with singular value decomposition

Recommender Systems

Introduction

Collaborative filtering using explicit feedback

Collaborative filtering using implicit feedback

Graph Processing Using GraphX

Introduction

Fundamental operations on graphs

Using PageRank

Finding connected components

Performing neighborhood aggregation

Optimizations and Performance Tuning

Introduction

Optimizing memory

Using compression to improve performance

Using serialization to improve performance

Optimizing garbage collection

Optimizing the level of parallelism

Understanding the future of optimization – project Tungsten

Index

Customer Reviews

5 star

4 star

3 star

2 star

1 star

Calculating summary statistics

Summary statistics is used to summarize observations to get a collective sense of the data. The summary includes the following:

Central tendency of data—mean, mode, median
Spread of data—variance, standard deviation
Boundary conditions—min, max

This recipe covers how to produce summary statistics.

How to do it…

Start the Spark shell:
```
$ spark-shell
```

Import the matrix-related classes:

scala> import org.apache.spark.mllib.linalg.{Vectors,Vector}
scala> import org.apache.spark.mllib.stat.Statistics

Create a personRDD as RDD of vectors:

scala> val personRDD = sc.parallelize(List(Vectors.dense(150,60,25), Vectors.dense(300,80,40)))

Compute the column summary statistics:

scala> val summary = Statistics.colStats(personRDD)

Print the mean of this summary:
```
scala> print(summary.mean)
```
Print the variance:
```
scala> print(summary.variance)
```
Print the non-zero values in each column:
```
scala> print(summary.numNonzeros)
```
Print the sample size:
```
scala> print(summary...
```

Spark Cookbook

By : Rishi Yadav

Spark Cookbook

By: Rishi Yadav

Overview of this book

Related Content you might be interested in

Current Title:

Spark Cookbook

Calculating summary statistics

How to do it…