Book Image

Frank Kane's Taming Big Data with Apache Spark and Python

By : Frank Kane
Book Image

Frank Kane's Taming Big Data with Apache Spark and Python

By: Frank Kane

Overview of this book

Frank Kane’s Taming Big Data with Apache Spark and Python is your companion to learning Apache Spark in a hands-on manner. Frank will start you off by teaching you how to set up Spark on a single system or on a cluster, and you’ll soon move on to analyzing large data sets using Spark RDD, and developing and running effective Spark jobs quickly using Python. Apache Spark has emerged as the next big thing in the Big Data domain – quickly rising from an ascending technology to an established superstar in just a matter of years. Spark allows you to quickly extract actionable insights from large amounts of data, on a real-time basis, making it an essential tool in many modern businesses. Frank has packed this book with over 15 interactive, fun-filled examples relevant to the real world, and he will empower you to understand the Spark ecosystem and implement production-grade real-time Spark projects with ease.

Related Content you might be interested in

Current Title:

Small book image
Frank Kane's Taming Big Data with Apache Spark and Python
Frank Kane
Jun, 2017 | 296 pages
Small book image
Hands-On Data Science and Python Machine Learning
Frank Kane
Jul, 2017 | 420 pages
Small book image
Learning Apache Spark 2
Muhammad Asif Abbasi
Mar, 2017 | 356 pages
Small book image
Apache Spark Quick Start Guide
Shrey Mehrotra | Akash Grade
Jan, 2019 | 154 pages
Table of Contents (13 chapters)
Title Page
Title Page
Credits
Credits
About the Author
About the Author
www.PacktPub.com
www.PacktPub.com
Customer Feedback
Customer Feedback
Preface
Preface
Free Chapter
1
Getting Started with Spark
Getting Started with Spark
Getting set up - installing Python, a JDK, and Spark and its dependencies
Installing the MovieLens movie rating dataset
Run your first Spark program - the ratings histogram example
Summary
2
Spark Basics and Spark Examples
Spark Basics and Spark Examples
What is Spark?
The Resilient Distributed Dataset (RDD)
Ratings histogram walk-through
Key/value RDDs and the average friends by age example
Running the average friends by age example
Filtering RDDs and the minimum temperature by location example
Running the minimum temperature example and modifying it for maximums
Running the maximum temperature by location example
Counting word occurrences using flatmap()
Improving the word-count script with regular expressions
Sorting the word count results
Find the total amount spent by customer
Check your results and sort them by the total amount spent
Check your sorted implementation and results against mine
Summary
3
Advanced Examples of Spark Programs
Advanced Examples of Spark Programs
Finding the most popular movie
Using broadcast variables to display movie names instead of ID numbers
Finding the most popular superhero in a social graph
Running the script - discover who the most popular superhero is
Superhero degrees of separation - introducing the breadth-first search algorithm
Accumulators and implementing BFS in Spark
Superhero degrees of separation - review the code and run it
Item-based collaborative filtering in Spark, cache(), and persist()
Running the similar-movies script using Spark's cluster manager
Improving the quality of the similar movies example
Summary
4
Running Spark on a Cluster
Running Spark on a Cluster
Introducing Elastic MapReduce
Setting up our Amazon Web Services / Elastic MapReduce account and PuTTY
Partitioning
Creating similar movies from one million ratings - part 1
Creating similar movies from one million ratings - part 2
Creating similar movies from one million ratings – part 3
Troubleshooting Spark on a cluster
More troubleshooting and managing dependencies
Summary
5
SparkSQL, DataFrames, and DataSets
SparkSQL, DataFrames, and DataSets
Introducing SparkSQL
Executing SQL commands and SQL-style functions on a DataFrame
Using DataFrames instead of RDDs
Summary
6
Other Spark Technologies and Libraries
Other Spark Technologies and Libraries
Introducing MLlib
Using MLlib to produce movie recommendations
Analyzing the ALS recommendations results
Using DataFrames with MLlib
Spark Streaming and GraphX
Summary
7
Where to Go From Here? – Learning More About Spark and Data Science
Where to Go From Here? – Learning More About Spark and Data Science

Creating similar movies from one million ratings - part 2

Now it's time to run our similarities script on a Spark cluster in the cloud on Elastic MapReduce. This is a pretty big deal, it's kind of the culmination of the whole course here, so let's kick it off and see what happens.

Our strategy

Before we actually run our script on a Spark cluster using Amazon's Elastic MapReduce service, let's talk about some of the basic strategies that we're going to use to do that.

Specifying memory per executor

Like we talked about earlier, we're going to use the default empty SparkConf in the driver script. That way we'll use the defaults that Elastic MapReduce sets up for us, and that will automatically tell Spark that it should be running on top of EMR's Hadoop cluster manager. Then it will automatically know what the layout of the cluster is, who the master is, how many client machines I have, who they are, how many executors they have, and so on. Now, when we're actually running this, we're going to...

Previous Section
End of Section 6
Next Section