Book Image

Scala Data Analysis Cookbook

By : Arun Manivannan
Book Image

Scala Data Analysis Cookbook

By: Arun Manivannan

Overview of this book

This book will introduce you to the most popular Scala tools, libraries, and frameworks through practical recipes around loading, manipulating, and preparing your data. It will also help you explore and make sense of your data using stunning and insightfulvisualizations, and machine learning toolkits. Starting with introductory recipes on utilizing the Breeze and Spark libraries, get to grips withhow to import data from a host of possible sources and how to pre-process numerical, string, and date data. Next, you’ll get an understanding of concepts that will help you visualize data using the Apache Zeppelin and Bokeh bindings in Scala, enabling exploratory data analysis. iscover how to program quintessential machine learning algorithms using Spark ML library. Work through steps to scale your machine learning models and deploy them into a standalone cluster, EC2, YARN, and Mesos. Finally dip into the powerful options presented by Spark Streaming, and machine learning for streaming data, as well as utilizing Spark GraphX.

Related Content you might be interested in

Current Title:

Small book image
Scala Data Analysis Cookbook
Arun Manivannan
Oct, 2015 | 254 pages
No titles found
Table of Contents (14 chapters)
Scala Data Analysis Cookbook
Scala Data Analysis Cookbook
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Getting Started with Breeze
Getting Started with Breeze
Introduction
Getting Breeze – the linear algebra library
Working with vectors
Working with matrices
Vectors and matrices with randomly distributed values
Reading and writing CSV files
2
Getting Started with Apache Spark DataFrames
Getting Started with Apache Spark DataFrames
Introduction
Getting Apache Spark
Creating a DataFrame from CSV
Manipulating DataFrames
Creating a DataFrame from Scala case classes
3
Loading and Preparing Data – DataFrame
Loading and Preparing Data – DataFrame
Introduction
Loading more than 22 features into classes
Loading JSON into DataFrames
Storing data as Parquet files
Using the Avro data model in Parquet
Loading from RDBMS
Preparing data in Dataframes
4
Data Visualization
Data Visualization
Introduction
Visualizing using Zeppelin
Creating scatter plots with Bokeh-Scala
Creating a time series MultiPlot with Bokeh-Scala
5
Learning from Data
Learning from Data
Introduction
Supervised and unsupervised learning
Gradient descent
Predicting continuous values using linear regression
Binary classification using LogisticRegression and SVM
Binary classification using LogisticRegression with Pipeline API
Clustering using K-means
Feature reduction using principal component analysis
6
Scaling Up
Scaling Up
Introduction
Building the Uber JAR
Submitting jobs to the Spark cluster (local)
Running the Spark Standalone cluster on EC2
Running the Spark Job on Mesos (local)
Running the Spark Job on YARN (local)
7
Going Further
Going Further
Introduction
Using Spark Streaming to subscribe to a Twitter stream
Using Spark as an ETL tool
Using StreamingLogisticRegression to classify a Twitter stream using Kafka as a training stream
Using GraphX to analyze Twitter data
Index
Index

Building the Uber JAR

The first step for deploying our Spark application on a cluster is to bundle it into a single Uber JAR, also known as the assembly JAR. In this recipe, we'll be looking at how to use the SBT assembly plugin to generate the assembly JAR. We'll be using this assembly JAR in subsequent recipes when we run Spark in distributed mode. We could alternatively set dependent JARs using the spark.driver.extraClassPath property (https://spark.apache.org/docs/1.3.1/configuration.html#runtime-environment). However, for a large number of dependent JARs, this is inconvenient.

How to do it...

The goal of building the assembly JAR is to build a single, Fat JAR that contains all dependencies and our Spark application. Refer to the following screenshot, which shows the innards of an assembly JAR. You can see not only the application's files in the JAR, but also all the packages and files of the dependent libraries:

The assembly JAR can easily be built in SBT using the SBT assembly plugin...

Previous Section
End of Section 3
Next Section