Book Image

Mastering Scala Machine Learning

By : Alex Kozlov
Book Image

Mastering Scala Machine Learning

By: Alex Kozlov

Overview of this book

Since the advent of object-oriented programming, new technologies related to Big Data are constantly popping up on the market. One such technology is Scala, which is considered to be a successor to Java in the area of Big Data by many, like Java was to C/C++ in the area of distributed programing. This book aims to take your knowledge to next level and help you impart that knowledge to build advanced applications such as social media mining, intelligent news portals, and more. After a quick refresher on functional programming concepts using REPL, you will see some practical examples of setting up the development environment and tinkering with data. We will then explore working with Spark and MLlib using k-means and decision trees. Most of the data that we produce today is unstructured and raw, and you will learn to tackle this type of data with advanced topics such as regression, classification, integration, and working with graph algorithms. Finally, you will discover at how to use Scala to perform complex concept analysis, to monitor model performance, and to build a model repository. By the end of this book, you will have gained expertise in performing Scala machine learning and will be able to build complex machine learning projects using Scala.

Related Content you might be interested in

Current Title:

Small book image
Mastering Scala Machine Learning
Alex Kozlov
Jun, 2016 | 310 pages
No titles found
Table of Contents (17 chapters)
Mastering Scala Machine Learning
Mastering Scala Machine Learning
Credits
Credits
About the Author
About the Author
Acknowlegement
Acknowlegement
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Exploratory Data Analysis
Exploratory Data Analysis
Getting started with Scala
Distinct values of a categorical field
Summarization of a numeric field
Basic, stratified, and consistent sampling
Working with Scala and Spark Notebooks
Basic correlations
Summary
2
Data Pipelines and Modeling
Data Pipelines and Modeling
Influence diagrams
Sequential trials and dealing with risk
Exploration and exploitation
Unknown unknowns
Basic components of a data-driven system
Optimization and interactivity
Summary
3
Working with Spark and MLlib
Working with Spark and MLlib
Setting up Spark
Understanding Spark architecture
Applications
ML libraries
Spark performance tuning
Running Hadoop HDFS
Summary
4
Supervised and Unsupervised Learning
Supervised and Unsupervised Learning
Records and supervised learning
Unsupervised learning
Problem dimensionality
Summary
5
Regression and Classification
Regression and Classification
What regression stands for?
Continuous space and metrics
Linear regression
Logistic regression
Regularization
Multivariate regression
Heteroscedasticity
Regression trees
Classification metrics
Multiclass problems
Perceptron
Generalization error and overfitting
Summary
6
Working with Unstructured Data
Working with Unstructured Data
Nested data
Other serialization formats
Hive and Impala
Sessionization
Working with traits
Working with pattern matching
Other uses of unstructured data
Probabilistic structures
Projections
Summary
7
Working with Graph Algorithms
Working with Graph Algorithms
A quick introduction to graphs
SBT
Graph for Scala
GraphX
Summary
8
Integrating Scala with R and Python
Integrating Scala with R and Python
Integrating with R
Integrating with Python
Summary
9
NLP in Scala
NLP in Scala
Text analysis pipeline
MLlib algorithms in Spark
Segmentation, annotation, and chunking
POS tagging
Using word2vec to find word relationships
Summary
10
Advanced Model Monitoring
Advanced Model Monitoring
System monitoring
Summary
Index
Index

GraphX

While graph for Scala may be considered a DSL for graph operations and querying, one should go to GraphX for scalability. GraphX is build on top of a powerful Spark framework. As an example of Spark/GraphX operations, I'll use the CMU Enron e-mail dataset (about 2 GB). The actual semantic analysis of the e-mail content is not going to be important to us until the next chapters. The dataset can be downloaded from the CMU site. It has e-mail from mailboxes of 150 users, primarily Enron managers, and about 517,401 e-mails between them. The e-mails may be considered as an indication of a relation (edge) between two people: Each email is an edge between a source (From:) and a destination (To:) vertices.

Since GraphX requires the data in RDD format, I'll have to do some preprocessing. Luckily, it is extremely easy with Scala—this is why Scala is the perfect language for semi-structured data. Here is the code:

package org.akozlov.chapter07

import scala.io.Source

import scala.util.hashing...
Previous Section
End of Section 5
Next Section