Book Image

Frank Kane's Taming Big Data with Apache Spark and Python

By : Frank Kane
Book Image

Frank Kane's Taming Big Data with Apache Spark and Python

By: Frank Kane

Overview of this book

Frank Kane’s Taming Big Data with Apache Spark and Python is your companion to learning Apache Spark in a hands-on manner. Frank will start you off by teaching you how to set up Spark on a single system or on a cluster, and you’ll soon move on to analyzing large data sets using Spark RDD, and developing and running effective Spark jobs quickly using Python. Apache Spark has emerged as the next big thing in the Big Data domain – quickly rising from an ascending technology to an established superstar in just a matter of years. Spark allows you to quickly extract actionable insights from large amounts of data, on a real-time basis, making it an essential tool in many modern businesses. Frank has packed this book with over 15 interactive, fun-filled examples relevant to the real world, and he will empower you to understand the Spark ecosystem and implement production-grade real-time Spark projects with ease.

Related Content you might be interested in

Current Title:

Small book image
Frank Kane's Taming Big Data with Apache Spark and Python
Frank Kane
Jun, 2017 | 296 pages
Small book image
Hands-On Data Science and Python Machine Learning
Frank Kane
Jul, 2017 | 420 pages
Small book image
Learning Apache Spark 2
Muhammad Asif Abbasi
Mar, 2017 | 356 pages
Small book image
Apache Spark Quick Start Guide
Shrey Mehrotra | Akash Grade
Jan, 2019 | 154 pages
Table of Contents (13 chapters)
Title Page
Title Page
Credits
Credits
About the Author
About the Author
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Getting Started with Spark
Getting Started with Spark
Getting set up - installing Python, a JDK, and Spark and its dependencies
Installing the MovieLens movie rating dataset
Run your first Spark program - the ratings histogram example
Summary
2
Spark Basics and Spark Examples
Spark Basics and Spark Examples
What is Spark?
The Resilient Distributed Dataset (RDD)
Ratings histogram walk-through
Key/value RDDs and the average friends by age example
Running the average friends by age example
Filtering RDDs and the minimum temperature by location example
Running the minimum temperature example and modifying it for maximums
Running the maximum temperature by location example
Counting word occurrences using flatmap()
Improving the word-count script with regular expressions
Sorting the word count results
Find the total amount spent by customer
Check your results and sort them by the total amount spent
Check your sorted implementation and results against mine
Summary
3
Advanced Examples of Spark Programs
Advanced Examples of Spark Programs
Finding the most popular movie
Using broadcast variables to display movie names instead of ID numbers
Finding the most popular superhero in a social graph
Running the script - discover who the most popular superhero is
Superhero degrees of separation - introducing the breadth-first search algorithm
Accumulators and implementing BFS in Spark
Superhero degrees of separation - review the code and run it
Item-based collaborative filtering in Spark, cache(), and persist()
Running the similar-movies script using Spark's cluster manager
Improving the quality of the similar movies example
Summary
4
Running Spark on a Cluster
Running Spark on a Cluster
Introducing Elastic MapReduce
Setting up our Amazon Web Services / Elastic MapReduce account and PuTTY
Partitioning
Creating similar movies from one million ratings - part 1
Creating similar movies from one million ratings - part 2
Creating similar movies from one million ratings – part 3
Troubleshooting Spark on a cluster
More troubleshooting and managing dependencies
Summary
5
SparkSQL, DataFrames, and DataSets
SparkSQL, DataFrames, and DataSets
Introducing SparkSQL
Executing SQL commands and SQL-style functions on a DataFrame
Using DataFrames instead of RDDs
Summary
6
Other Spark Technologies and Libraries
Other Spark Technologies and Libraries
Introducing MLlib
Using MLlib to produce movie recommendations
Analyzing the ALS recommendations results
Using DataFrames with MLlib
Spark Streaming and GraphX
Summary
7
Where to Go From Here? – Learning More About Spark and Data Science
Where to Go From Here? – Learning More About Spark and Data Science

Ratings histogram walk-through

Remember the RatingsHistogram code that we ran for your first Spark program? Well, let's take a closer look at that and figure out what's actually going on under the hood with it. Understanding concepts is all well and good, but nothing beats looking at some real examples. Let's go back to the RatingsHistogram example that we started off with in this book. We'll break it down and understand exactly what it's doing under the hood and how it's using our RDDs to actually get the results for the RatingsHistogram data.

Understanding the code

The first couple of lines are just boilerplate stuff. One thing you'll see in every Python Spark script is the import statement to import SparkConf and SparkContext from the pyspark library that Spark includes. You will, at a minimum, need those two objects:

from pyspark import SparkConf, SparkContext 
import collections

SparkContext, as we talked about earlier, is the fundamental starting point that the Spark framework gives you...

Previous Section
End of Section 4
Next Section