Other Spark Technologies and Libraries | Frank Kane's Taming Big Data with Apache Spark and Python

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Frank Kane's Taming Big Data with Apache Spark and Python

By : Frank Kane

3.8 (11)

Frank Kane's Taming Big Data with Apache Spark and Python

3.8 (11)

By: Frank Kane

Overview of this book

Frank Kane’s Taming Big Data with Apache Spark and Python is your companion to learning Apache Spark in a hands-on manner. Frank will start you off by teaching you how to set up Spark on a single system or on a cluster, and you’ll soon move on to analyzing large data sets using Spark RDD, and developing and running effective Spark jobs quickly using Python. Apache Spark has emerged as the next big thing in the Big Data domain – quickly rising from an ascending technology to an established superstar in just a matter of years. Spark allows you to quickly extract actionable insights from large amounts of data, on a real-time basis, making it an essential tool in many modern businesses. Frank has packed this book with over 15 interactive, fun-filled examples relevant to the real world, and he will empower you to understand the Spark ecosystem and implement production-grade real-time Spark projects with ease.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

Getting Started with Spark

Getting Started with Spark

Getting set up - installing Python, a JDK, and Spark and its dependencies

Installing the MovieLens movie rating dataset

Run your first Spark program - the ratings histogram example

Summary

Spark Basics and Spark Examples

Spark Basics and Spark Examples

What is Spark?

The Resilient Distributed Dataset (RDD)

Ratings histogram walk-through

Key/value RDDs and the average friends by age example

Running the average friends by age example

Filtering RDDs and the minimum temperature by location example

Running the minimum temperature example and modifying it for maximums

Running the maximum temperature by location example

Counting word occurrences using flatmap()

Improving the word-count script with regular expressions

Sorting the word count results

Find the total amount spent by customer

Check your results and sort them by the total amount spent

Check your sorted implementation and results against mine

Summary

Advanced Examples of Spark Programs

Advanced Examples of Spark Programs

Finding the most popular movie

Using broadcast variables to display movie names instead of ID numbers

Finding the most popular superhero in a social graph

Running the script - discover who the most popular superhero is

Superhero degrees of separation - introducing the breadth-first search algorithm

Accumulators and implementing BFS in Spark

Superhero degrees of separation - review the code and run it

Item-based collaborative filtering in Spark, cache(), and persist()

Running the similar-movies script using Spark's cluster manager

Improving the quality of the similar movies example

Summary

Running Spark on a Cluster

Running Spark on a Cluster

Introducing Elastic MapReduce

Setting up our Amazon Web Services / Elastic MapReduce account and PuTTY

Partitioning

Creating similar movies from one million ratings - part 1

Creating similar movies from one million ratings - part 2

Creating similar movies from one million ratings – part 3

Troubleshooting Spark on a cluster

More troubleshooting and managing dependencies

Summary

SparkSQL, DataFrames, and DataSets

SparkSQL, DataFrames, and DataSets

Introducing SparkSQL

Executing SQL commands and SQL-style functions on a DataFrame

Using DataFrames instead of RDDs

Summary

Other Spark Technologies and Libraries

Other Spark Technologies and Libraries

Introducing MLlib

Using MLlib to produce movie recommendations

Analyzing the ALS recommendations results

Using DataFrames with MLlib

Spark Streaming and GraphX

Summary

Where to Go From Here? – Learning More About Spark and Data Science

Where to Go From Here? – Learning More About Spark and Data Science

Analyzing the ALS recommendations results

Open up Command Prompt and type spark-submit movie-recommendations-als.py with user 0:

That user 0 is my Star Wars fan that doesn't like Gone With The Wind.

Off it goes, using all the cores that I have. It should finish quite quickly. For such a fancy algorithm, that came back creepily fast, almost suspiciously so:

So, for my fictitious user who loves Star WarsÂ and The Empire Strikes Back, but hated Gone With The Wind, the number one recommendations it produced was something called Love in the AfternoonÂ and Roommates. What? What is this stuff? That's crazy. Lost in Space, okay, I can go with that, but the rest of this just doesn't make sense. What's worse is if I run it again, I'll actually get different results! Now, it could be that the algorithm is taking some shortcuts and randomly sampling things to save time, but even so, that's not good news.

Let's see what we get if we run it again. We get a totally different set of results:

There's something...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Frank Kane's Taming Big Data with Apache Spark and Python

Search

Your notes and bookmarks