Book Image

Frank Kane's Taming Big Data with Apache Spark and Python

By : Frank Kane
Book Image

Frank Kane's Taming Big Data with Apache Spark and Python

By: Frank Kane

Overview of this book

Frank Kane’s Taming Big Data with Apache Spark and Python is your companion to learning Apache Spark in a hands-on manner. Frank will start you off by teaching you how to set up Spark on a single system or on a cluster, and you’ll soon move on to analyzing large data sets using Spark RDD, and developing and running effective Spark jobs quickly using Python. Apache Spark has emerged as the next big thing in the Big Data domain – quickly rising from an ascending technology to an established superstar in just a matter of years. Spark allows you to quickly extract actionable insights from large amounts of data, on a real-time basis, making it an essential tool in many modern businesses. Frank has packed this book with over 15 interactive, fun-filled examples relevant to the real world, and he will empower you to understand the Spark ecosystem and implement production-grade real-time Spark projects with ease.

Related Content you might be interested in

Current Title:

Small book image
Frank Kane's Taming Big Data with Apache Spark and Python
Frank Kane
Jun, 2017 | 296 pages
Small book image
Hands-On Data Science and Python Machine Learning
Frank Kane
Jul, 2017 | 420 pages
Small book image
Learning Apache Spark 2
Muhammad Asif Abbasi
Mar, 2017 | 356 pages
Small book image
Apache Spark Quick Start Guide
Shrey Mehrotra | Akash Grade
Jan, 2019 | 154 pages
Table of Contents (13 chapters)
Title Page
Title Page
Credits
Credits
About the Author
About the Author
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Getting Started with Spark
Getting Started with Spark
Getting set up - installing Python, a JDK, and Spark and its dependencies
Installing the MovieLens movie rating dataset
Run your first Spark program - the ratings histogram example
Summary
2
Spark Basics and Spark Examples
Spark Basics and Spark Examples
What is Spark?
The Resilient Distributed Dataset (RDD)
Ratings histogram walk-through
Key/value RDDs and the average friends by age example
Running the average friends by age example
Filtering RDDs and the minimum temperature by location example
Running the minimum temperature example and modifying it for maximums
Running the maximum temperature by location example
Counting word occurrences using flatmap()
Improving the word-count script with regular expressions
Sorting the word count results
Find the total amount spent by customer
Check your results and sort them by the total amount spent
Check your sorted implementation and results against mine
Summary
3
Advanced Examples of Spark Programs
Advanced Examples of Spark Programs
Finding the most popular movie
Using broadcast variables to display movie names instead of ID numbers
Finding the most popular superhero in a social graph
Running the script - discover who the most popular superhero is
Superhero degrees of separation - introducing the breadth-first search algorithm
Accumulators and implementing BFS in Spark
Superhero degrees of separation - review the code and run it
Item-based collaborative filtering in Spark, cache(), and persist()
Running the similar-movies script using Spark's cluster manager
Improving the quality of the similar movies example
Summary
4
Running Spark on a Cluster
Running Spark on a Cluster
Introducing Elastic MapReduce
Setting up our Amazon Web Services / Elastic MapReduce account and PuTTY
Partitioning
Creating similar movies from one million ratings - part 1
Creating similar movies from one million ratings - part 2
Creating similar movies from one million ratings – part 3
Troubleshooting Spark on a cluster
More troubleshooting and managing dependencies
Summary
5
SparkSQL, DataFrames, and DataSets
SparkSQL, DataFrames, and DataSets
Introducing SparkSQL
Executing SQL commands and SQL-style functions on a DataFrame
Using DataFrames instead of RDDs
Summary
6
Other Spark Technologies and Libraries
Other Spark Technologies and Libraries
Introducing MLlib
Using MLlib to produce movie recommendations
Analyzing the ALS recommendations results
Using DataFrames with MLlib
Spark Streaming and GraphX
Summary
7
Where to Go From Here? – Learning More About Spark and Data Science
Where to Go From Here? – Learning More About Spark and Data Science

Accumulators and implementing BFS in Spark

Now that we have the concept of breadth-first-search under our belt and we understand how that can be used to find the degrees of separation between superheroes, let's apply that and actually write some Spark code to make it happen. So how do we turn breadth-first search into a Spark problem? This will make a lot more sense if that explanation of how BFS works is still fresh in your head. If it's not, it might be a good idea to go back and re-read the previous section; it will really help a lot if you understand the theory.

Convert the input file into structured data

The first thing we need to do is actually convert our data file or input file into something that looks like the nodes that we described in the BFS algorithm in the previous section, Superhero degrees of separation - introducing breadth-first search.

We're starting off, for example, with a line of input that looks like the one shown here that says hero ID 5983 appeared with heroes 1165...

Previous Section
End of Section 7
Next Section