Book Image

Apache Spark 2 for Beginners

By : Rajanarayanan Thottuvaikkatumana
Book Image

Apache Spark 2 for Beginners

By: Rajanarayanan Thottuvaikkatumana

Overview of this book

<p>Spark is one of the most widely-used large-scale data processing engines and runs extremely fast. It is a framework that has tools that are equally useful for application developers as well as data scientists.</p> <p>This book starts with the fundamentals of Spark 2 and covers the core data processing framework and API, installation, and application development setup. Then the Spark programming model is introduced through real-world examples followed by Spark SQL programming with DataFrames. An introduction to SparkR is covered next. Later, we cover the charting and plotting features of Python in conjunction with Spark data processing. After that, we take a look at Spark's stream processing, machine learning, and graph processing libraries. The last chapter combines all the skills you learned from the preceding chapters to develop a real-world Spark application.</p> <p>By the end of this book, you will have all the knowledge you need to develop efficient large-scale applications using Apache Spark.</p>

Related Content you might be interested in

Current Title:

Small book image
Apache Spark 2 for Beginners
Rajanarayanan Thottuvaikkatumana
Oct, 2016 | 332 pages
No titles found
Table of Contents (15 chapters)
Apache Spark 2 for Beginners
Apache Spark 2 for Beginners
Credits
Credits
About the Author
About the Author
About the Reviewer
About the Reviewer
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Spark Fundamentals
Spark Fundamentals
An overview of Apache Hadoop
Understanding Apache Spark
Installing Spark on your machines
References
Summary
2
Spark Programming Model
Spark Programming Model
Functional programming with Spark
Understanding Spark RDD
Data transformations and actions with RDDs
Monitoring with Spark
The basics of programming with Spark
Creating RDDs from files
Understanding the Spark library stack
Reference
Summary
3
Spark SQL
Spark SQL
Understanding the structure of data
Why Spark SQL?
Anatomy of Spark SQL
DataFrame programming
Understanding Aggregations in Spark SQL
Understanding multi-datasource joining with SparkSQL
Introducing datasets
Understanding Data Catalogs
References
Summary
4
Spark Programming with R
Spark Programming with R
The need for SparkR
Basics of the R language
DataFrames in R and Spark
Spark DataFrame programming with R
Understanding aggregations in Spark R
Understanding multi-datasource joins with SparkR
References
Summary
5
Spark Data Analysis with Python
Spark Data Analysis with Python
Charting and plotting libraries
Setting up a dataset
Data analysis use cases
Charts and plots
References
Summary
6
Spark Stream Processing
Spark Stream Processing
Data stream processing
Micro batch data processing
A log event processor
Windowed data processing
More processing options
Kafka stream processing
Spark Streaming jobs in production
References
Summary
7
Spark Machine Learning
Spark Machine Learning
Understanding machine learning
Why Spark for machine learning?
Wine quality prediction
Model persistence
Wine classification
Spam filtering
Feature algorithms
Finding synonyms
References
Summary
8
Spark Graph Processing
Spark Graph Processing
Understanding graphs and their usage
The Spark GraphX library
Tennis tournament analysis
Applying the PageRank algorithm
Connected component algorithm
Understanding GraphFrames
Understanding GraphFrames queries
References
Summary
9
Designing Spark Applications
Designing Spark Applications
Lambda Architecture
Microblogging with Lambda Architecture
Implementing Lambda Architecture
Working with Spark applications
Coding style
Setting up the source code
Understanding data ingestion
Generating purposed views and queries
Understanding custom data processes
References
Summary

Model persistence

Spark 2.0 comes with the ability to save and load machine learning models across programming languages with ease. In other words, you can create a machine learning model in Scala and load it in Python. This allows us to create a model in one system, save it, copy it, and use it in other systems. Continuing with the same Scala REPL prompt, try the following statements:


	scala> // Assuming that the model definition line "val model = 
    lr.fit(trainingDF)" is still in context
	scala> import org.apache.spark.ml.regression.LinearRegressionModel
	
      import org.apache.spark.ml.regression.LinearRegressionModel
    
	scala> model.save("wineLRModelPath")
	scala> val newModel = LinearRegressionModel.load("wineLRModelPath")
	
      newModel: org.apache.spark.ml.regression.LinearRegressionModel = 
      linReg_6a880215ab96

Now the loaded model can be used for testing or prediction, just like the original model. Continuing with the same Python REPL prompt, try...

Previous Section
End of Section 5
Next Section