Book Image

Frank Kane's Taming Big Data with Apache Spark and Python

By : Frank Kane
Book Image

Frank Kane's Taming Big Data with Apache Spark and Python

By: Frank Kane

Overview of this book

Frank Kane’s Taming Big Data with Apache Spark and Python is your companion to learning Apache Spark in a hands-on manner. Frank will start you off by teaching you how to set up Spark on a single system or on a cluster, and you’ll soon move on to analyzing large data sets using Spark RDD, and developing and running effective Spark jobs quickly using Python. Apache Spark has emerged as the next big thing in the Big Data domain – quickly rising from an ascending technology to an established superstar in just a matter of years. Spark allows you to quickly extract actionable insights from large amounts of data, on a real-time basis, making it an essential tool in many modern businesses. Frank has packed this book with over 15 interactive, fun-filled examples relevant to the real world, and he will empower you to understand the Spark ecosystem and implement production-grade real-time Spark projects with ease.

Related Content you might be interested in

Current Title:

Small book image
Frank Kane's Taming Big Data with Apache Spark and Python
Frank Kane
Jun, 2017 | 296 pages
Small book image
Hands-On Data Science and Python Machine Learning
Frank Kane
Jul, 2017 | 420 pages
Small book image
Learning Apache Spark 2
Muhammad Asif Abbasi
Mar, 2017 | 356 pages
Small book image
Apache Spark Quick Start Guide
Shrey Mehrotra | Akash Grade
Jan, 2019 | 154 pages
Table of Contents (13 chapters)
Title Page
Title Page
Credits
Credits
About the Author
About the Author
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Getting Started with Spark
Getting Started with Spark
Getting set up - installing Python, a JDK, and Spark and its dependencies
Installing the MovieLens movie rating dataset
Run your first Spark program - the ratings histogram example
Summary
2
Spark Basics and Spark Examples
Spark Basics and Spark Examples
What is Spark?
The Resilient Distributed Dataset (RDD)
Ratings histogram walk-through
Key/value RDDs and the average friends by age example
Running the average friends by age example
Filtering RDDs and the minimum temperature by location example
Running the minimum temperature example and modifying it for maximums
Running the maximum temperature by location example
Counting word occurrences using flatmap()
Improving the word-count script with regular expressions
Sorting the word count results
Find the total amount spent by customer
Check your results and sort them by the total amount spent
Check your sorted implementation and results against mine
Summary
3
Advanced Examples of Spark Programs
Advanced Examples of Spark Programs
Finding the most popular movie
Using broadcast variables to display movie names instead of ID numbers
Finding the most popular superhero in a social graph
Running the script - discover who the most popular superhero is
Superhero degrees of separation - introducing the breadth-first search algorithm
Accumulators and implementing BFS in Spark
Superhero degrees of separation - review the code and run it
Item-based collaborative filtering in Spark, cache(), and persist()
Running the similar-movies script using Spark's cluster manager
Improving the quality of the similar movies example
Summary
4
Running Spark on a Cluster
Running Spark on a Cluster
Introducing Elastic MapReduce
Setting up our Amazon Web Services / Elastic MapReduce account and PuTTY
Partitioning
Creating similar movies from one million ratings - part 1
Creating similar movies from one million ratings - part 2
Creating similar movies from one million ratings – part 3
Troubleshooting Spark on a cluster
More troubleshooting and managing dependencies
Summary
5
SparkSQL, DataFrames, and DataSets
SparkSQL, DataFrames, and DataSets
Introducing SparkSQL
Executing SQL commands and SQL-style functions on a DataFrame
Using DataFrames instead of RDDs
Summary
6
Other Spark Technologies and Libraries
Other Spark Technologies and Libraries
Introducing MLlib
Using MLlib to produce movie recommendations
Analyzing the ALS recommendations results
Using DataFrames with MLlib
Spark Streaming and GraphX
Summary
7
Where to Go From Here? – Learning More About Spark and Data Science
Where to Go From Here? – Learning More About Spark and Data Science

Analyzing the ALS recommendations results

Open up Command Prompt and type spark-submit movie-recommendations-als.py with user 0:

That user 0 is my Star Wars fan that doesn't like Gone With The Wind.

Off it goes, using all the cores that I have. It should finish quite quickly. For such a fancy algorithm, that came back creepily fast, almost suspiciously so:

So, for my fictitious user who loves Star Wars and The Empire Strikes Back, but hated Gone With The Wind, the number one recommendations it produced was something called Love in the Afternoon and Roommates. What? What is this stuff? That's crazy. Lost in Space, okay, I can go with that, but the rest of this just doesn't make sense. What's worse is if I run it again, I'll actually get different results! Now, it could be that the algorithm is taking some shortcuts and randomly sampling things to save time, but even so, that's not good news.

Let's see what we get if we run it again. We get a totally different set of results:

There's something...

Previous Section
End of Section 4
Next Section