Book Image

Spark for Data Science

By : Srinivas Duvvuri, Bikramaditya Singhal
Book Image

Spark for Data Science

By: Srinivas Duvvuri, Bikramaditya Singhal

Overview of this book

This is the era of Big Data. The words ‘Big Data’ implies big innovation and enables a competitive advantage for businesses. Apache Spark was designed to perform Big Data analytics at scale, and so Spark is equipped with the necessary algorithms and supports multiple programming languages. Whether you are a technologist, a data scientist, or a beginner to Big Data analytics, this book will provide you with all the skills necessary to perform statistical data analysis, data visualization, predictive modeling, and build scalable data products or solutions using Python, Scala, and R. With ample case studies and real-world examples, Spark for Data Science will help you ensure the successful execution of your data science projects.

Related Content you might be interested in

Current Title:

Small book image
Spark for Data Science
Srinivas Duvvuri | Bikramaditya Singhal
Sep, 2016 | 344 pages
No titles found
Table of Contents (18 chapters)
Spark for Data Science
Spark for Data Science
Credits
Credits
Foreword
Foreword
About the Authors
About the Authors
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Big Data and Data Science – An Introduction
Big Data and Data Science – An Introduction
Big data overview
Challenges with big data analytics
Evolution of big data analytics
Spark for data analytics
The Spark stack
Summary
References
2
The Spark Programming Model
The Spark Programming Model
The programming paradigm
The Spark engine
The RDD API
RDD operations
Summary
References
3
Introduction to DataFrames
Introduction to DataFrames
Why DataFrames?
Spark SQL
The DataFrame API
Creating DataFrames
DataFrame operations
Summary
References
4
Unified Data Access
Unified Data Access
Data abstractions in Apache Spark
Datasets
Spark SQL
Structured Streaming
Continuous applications
Summary
References
5
Data Analysis on Spark
Data Analysis on Spark
Data analytics life cycle
Data acquisition
Data preparation
Basics of statistics
Descriptive statistics
Inferential statistics
Summary
References
6
Machine Learning
Machine Learning
Introduction
MLlib and the Pipeline API
Introduction to machine learning
Regression methods
Classification methods
Linear Support Vector Machines (SVM)
Training an SVM
Decision trees
Ensembles
Multilayer perceptron classifier
Clustering techniques
Summary
References
7
Extending Spark with SparkR
Extending Spark with SparkR
SparkR basics
Advantages and limitations
Programming with SparkR
SparkR DataFrames
Machine learning
Summary
References
8
Analyzing Unstructured Data
Analyzing Unstructured Data
Sources of unstructured data
Processing unstructured data
Text classification
Text clustering
Dimensionality reduction
Singular Value Decomposition
Summary
References:
9
Visualizing Big Data
Visualizing Big Data
Why visualize data?
Data visualization tools
Data visualization techniques
Summary
References
10
Putting It All Together
Putting It All Together
A quick recap
Introducing a case study
The business problem
Data acquisition and data cleansing
Developing the hypothesis
Data exploration
Data preparation
Model building
Data visualization
Communicating the results to business users
Summary
References
11
Building Data Science Applications
Building Data Science Applications
Scope of development
The Scala advantage
Spark development status
The big data trends
Summary
References

The Spark engine

To program with Spark, a basic understanding of Spark components is needed. In this section, some of the important Spark components along with their execution mechanism will be explained so that developers and data scientists can write programs and build applications.

Before getting into the details, we suggest you take a look at the following diagram so that the descriptions of the Spark gears are more comprehensible as you read further:

Driver program

The Spark shell is an example of a driver program. A driver program is a process that executes in the JVM and runs the user's main function on it. It has a SparkContext object which is a connection to the underlying cluster manager. A Spark application is initiated when the driver starts and it completes when the driver stops. The driver, through an instance of SparkContext, coordinates all processes within a Spark application.

Primarily, an RDD lineage Directed Acyclic Graph (DAG) is built on the driver side with data sources...

Previous Section
End of Section 3
Next Section