Spark Basics and Spark Examples | Frank Kane's Taming Big Data with Apache Spark and Python

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Frank Kane's Taming Big Data with Apache Spark and Python

By : Frank Kane

3.8 (11)

Frank Kane's Taming Big Data with Apache Spark and Python

3.8 (11)

By: Frank Kane

Overview of this book

Frank Kane’s Taming Big Data with Apache Spark and Python is your companion to learning Apache Spark in a hands-on manner. Frank will start you off by teaching you how to set up Spark on a single system or on a cluster, and you’ll soon move on to analyzing large data sets using Spark RDD, and developing and running effective Spark jobs quickly using Python. Apache Spark has emerged as the next big thing in the Big Data domain – quickly rising from an ascending technology to an established superstar in just a matter of years. Spark allows you to quickly extract actionable insights from large amounts of data, on a real-time basis, making it an essential tool in many modern businesses. Frank has packed this book with over 15 interactive, fun-filled examples relevant to the real world, and he will empower you to understand the Spark ecosystem and implement production-grade real-time Spark projects with ease.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

Getting Started with Spark

Getting Started with Spark

Getting set up - installing Python, a JDK, and Spark and its dependencies

Installing the MovieLens movie rating dataset

Run your first Spark program - the ratings histogram example

Summary

Spark Basics and Spark Examples

Spark Basics and Spark Examples

What is Spark?

The Resilient Distributed Dataset (RDD)

Ratings histogram walk-through

Key/value RDDs and the average friends by age example

Running the average friends by age example

Filtering RDDs and the minimum temperature by location example

Running the minimum temperature example and modifying it for maximums

Running the maximum temperature by location example

Counting word occurrences using flatmap()

Improving the word-count script with regular expressions

Sorting the word count results

Find the total amount spent by customer

Check your results and sort them by the total amount spent

Check your sorted implementation and results against mine

Summary

Advanced Examples of Spark Programs

Advanced Examples of Spark Programs

Finding the most popular movie

Using broadcast variables to display movie names instead of ID numbers

Finding the most popular superhero in a social graph

Running the script - discover who the most popular superhero is

Superhero degrees of separation - introducing the breadth-first search algorithm

Accumulators and implementing BFS in Spark

Superhero degrees of separation - review the code and run it

Item-based collaborative filtering in Spark, cache(), and persist()

Running the similar-movies script using Spark's cluster manager

Improving the quality of the similar movies example

Summary

Running Spark on a Cluster

Running Spark on a Cluster

Introducing Elastic MapReduce

Setting up our Amazon Web Services / Elastic MapReduce account and PuTTY

Partitioning

Creating similar movies from one million ratings - part 1

Creating similar movies from one million ratings - part 2

Creating similar movies from one million ratings – part 3

Troubleshooting Spark on a cluster

More troubleshooting and managing dependencies

Summary

SparkSQL, DataFrames, and DataSets

SparkSQL, DataFrames, and DataSets

Introducing SparkSQL

Executing SQL commands and SQL-style functions on a DataFrame

Using DataFrames instead of RDDs

Summary

Other Spark Technologies and Libraries

Other Spark Technologies and Libraries

Introducing MLlib

Using MLlib to produce movie recommendations

Analyzing the ALS recommendations results

Using DataFrames with MLlib

Spark Streaming and GraphX

Summary

Where to Go From Here? – Learning More About Spark and Data Science

Where to Go From Here? – Learning More About Spark and Data Science

Running the average friends by age example

Okay, let's make it real, let's actually get some real code and some real data and analyze the average number of friends by age in our fabricated dataset here, and see what we come up with.

At this point, you should go to the download package for this book, if you haven't already, and download two things: one is the friends-by-age Python script, and the other is the fakefriends.csv file, which is my randomly generated data that's completely fictitious, but useful for illustration. So go take care of that now. When you're done, move it into your C:\SparkCourse folder or wherever you're installing stuff for this course. At this point in the course, your SparkCourse folder should look like this:

At this moment, we need friends-by-age.py and fakefriends.csv, so let's double-click on the friends-by-age.py script, and Enthought Canopy or your Python environment of choice should come up. Here we have it:

Examining the script

So let's just review again what...

Visually different images

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Frank Kane's Taming Big Data with Apache Spark and Python

Search

Your notes and bookmarks