SparkSQL, DataFrames, and DataSets | Frank Kane's Taming Big Data with Apache Spark and Python

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents
Feedback & Rating

Frank Kane's Taming Big Data with Apache Spark and Python

By : Frank Kane

3.8 (11)

Frank Kane's Taming Big Data with Apache Spark and Python

3.8 (11)

By: Frank Kane

Overview of this book

Frank Kane’s Taming Big Data with Apache Spark and Python is your companion to learning Apache Spark in a hands-on manner. Frank will start you off by teaching you how to set up Spark on a single system or on a cluster, and you’ll soon move on to analyzing large data sets using Spark RDD, and developing and running effective Spark jobs quickly using Python. Apache Spark has emerged as the next big thing in the Big Data domain – quickly rising from an ascending technology to an established superstar in just a matter of years. Spark allows you to quickly extract actionable insights from large amounts of data, on a real-time basis, making it an essential tool in many modern businesses. Frank has packed this book with over 15 interactive, fun-filled examples relevant to the real world, and he will empower you to understand the Spark ecosystem and implement production-grade real-time Spark projects with ease.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

Getting Started with Spark

Getting Started with Spark

Getting set up - installing Python, a JDK, and Spark and its dependencies

Installing the MovieLens movie rating dataset

Run your first Spark program - the ratings histogram example

Summary

Spark Basics and Spark Examples

Spark Basics and Spark Examples

What is Spark?

The Resilient Distributed Dataset (RDD)

Ratings histogram walk-through

Key/value RDDs and the average friends by age example

Running the average friends by age example

Filtering RDDs and the minimum temperature by location example

Running the minimum temperature example and modifying it for maximums

Running the maximum temperature by location example

Counting word occurrences using flatmap()

Improving the word-count script with regular expressions

Sorting the word count results

Find the total amount spent by customer

Check your results and sort them by the total amount spent

Check your sorted implementation and results against mine

Summary

Advanced Examples of Spark Programs

Advanced Examples of Spark Programs

Finding the most popular movie

Using broadcast variables to display movie names instead of ID numbers

Finding the most popular superhero in a social graph

Running the script - discover who the most popular superhero is

Superhero degrees of separation - introducing the breadth-first search algorithm

Accumulators and implementing BFS in Spark

Superhero degrees of separation - review the code and run it

Item-based collaborative filtering in Spark, cache(), and persist()

Running the similar-movies script using Spark's cluster manager

Improving the quality of the similar movies example

Summary

Running Spark on a Cluster

Running Spark on a Cluster

Introducing Elastic MapReduce

Setting up our Amazon Web Services / Elastic MapReduce account and PuTTY

Partitioning

Creating similar movies from one million ratings - part 1

Creating similar movies from one million ratings - part 2

Creating similar movies from one million ratings – part 3

Troubleshooting Spark on a cluster

More troubleshooting and managing dependencies

Summary

SparkSQL, DataFrames, and DataSets

SparkSQL, DataFrames, and DataSets

Introducing SparkSQL

Executing SQL commands and SQL-style functions on a DataFrame

Using DataFrames instead of RDDs

Summary

Other Spark Technologies and Libraries

Other Spark Technologies and Libraries

Introducing MLlib

Using MLlib to produce movie recommendations

Analyzing the ALS recommendations results

Using DataFrames with MLlib

Spark Streaming and GraphX

Summary

Where to Go From Here? – Learning More About Spark and Data Science

Where to Go From Here? – Learning More About Spark and Data Science

Customer Reviews

3.8 (11)

5 star

54.5%

4 star

9.1%

3 star

9.1%

2 star

18.2%

1 star

9.1%

Executing SQL commands and SQL-style functions on a DataFrame

Alright, open up the sparksql.py file that's included in the download files for this book. Let's take a look at it as a real-world example of using SparkSQL in Spark 2.0. You should see the following code in your editor:

Notice that we're importing a few things here. We're importing the SparkSession object and the Row object. The SparkSession object is basically Spark 2.0's way of creating a context to work with SparkSQL. We'll also import collections here:

from pyspark.sql import SparkSession 
from pyspark.sql import Row 
 
import collections

Earlier, we used to create sparkContext objects, but now, we'll create a SparkSession object:

# Create a SparkSession (Note, the config section is only for Windows!) 
spark = SparkSession.builder.config("spark.sql.warehouse.dir", "file:///C:/temp").appName("SparkSQL").getOrCreate()

So what we're doing here is creating something called spark that's going to be a SparkSession object. We'll use...

Visually different images

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Frank Kane's Taming Big Data with Apache Spark and Python

Search

Your notes and bookmarks