6. Manipulating Your RDD | Fast Data Processing with Spark 2

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Fast Data Processing with Spark 2 - Third Edition

By : Krishna Sankar , Karau

Fast Data Processing with Spark 2

By: Krishna Sankar , Karau

Overview of this book

When people want a way to process big data at speed, Spark is invariably the solution. With its ease of development (in comparison to the relative complexity of Hadoop), it’s unsurprising that it’s becoming popular with data analysts and engineers everywhere. Beginning with the fundamentals, we’ll show you how to get set up with Spark with minimum fuss. You’ll then get to grips with some simple APIs before investigating machine learning and graph processing – throughout we’ll make sure you know exactly how to apply your knowledge. You will also learn how to use the Spark shell, how to load data before finding out how to build and run your own Spark applications. Discover how to manipulate your RDD and get stuck into a range of DataFrame APIs. As if that’s not enough, you’ll also learn some useful Machine Learning algorithms with the help of Spark MLlib and integrating Spark with R. We’ll also make sure you’re confident and prepared for graph processing, as you learn more about the GraphX API.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Installing Spark and Setting Up Your Cluster

1. Installing Spark and Setting Up Your Cluster

Directory organization and convention

Installing the prebuilt distribution

Building Spark from source

Spark topology

A single machine

Running Spark on EC2

Deploying Spark with Chef (Opscode)

Deploying Spark on Mesos

Spark on YARN

Spark standalone mode

References

Summary

2. Using the Spark Shell

2. Using the Spark Shell

The Spark shell

Loading a simple text file

Interactively loading data from S3

Summary

3. Building and Running a Spark Application

3. Building and Running a Spark Application

Building Spark applications

Data wrangling with iPython

Developing Spark with Eclipse

Developing Spark with other IDEs

Building your Spark job with Maven

Building your Spark job with something else

References

Summary

4. Creating a SparkSession Object

4. Creating a SparkSession Object

SparkSession versus SparkContext

Building a SparkSession object

SparkContext - metadata

Shared Java and Scala APIs

Python

iPython

Reference

Summary

5. Loading and Saving Data in Spark

5. Loading and Saving Data in Spark

Spark abstractions

Data modalities

Data modalities and Datasets/DataFrames/RDDs

Loading data into an RDD

Saving your data

References

Summary

6. Manipulating Your RDD

6. Manipulating Your RDD

Manipulating your RDD in Scala and Java

Manipulating your RDD in Python

References

Summary

7. Spark 2.0 Concepts

7. Spark 2.0 Concepts

Code and Datasets for the rest of the book

The data scientist and Spark features

Spark v2.0 and beyond

Apache Spark - evolution

Apache Spark - the full stack

The art of a big data store - Parquet

References

Summary

8. Spark SQL

8. Spark SQL

The Spark SQL architecture

Spark SQL how-to in a nutshell

Spark SQL programming

References

Summary

9. Foundations of Datasets/DataFrames – The Proverbial Workhorse for DataScientists

9. Foundations of Datasets/DataFrames – The Proverbial Workhorse for DataScientists

Datasets - a quick introduction

Dataset APIs - an overview

Dataset interfaces and functions

References

Summary

10. Spark with Big Data

10. Spark with Big Data

Parquet - an efficient and interoperable big data format

HBase

Reference

Summary

11. Machine Learning with Spark ML Pipelines

11. Machine Learning with Spark ML Pipelines

Spark's machine learning algorithm table

Spark machine learning APIs - ML pipelines and MLlib

ML pipelines

Spark ML examples

The API organization

Basic statistics

Linear regression

Classification

Clustering

Recommendation

Hyper parameters

The final thing

References

Summary

12. GraphX

12. GraphX

Graphs and graph processing - an introduction

Spark GraphX

GraphX - computational model

The first example - graph

Building graphs

The GraphX API landscape

Structural APIs

Community, affiliation, and strengths

Algorithms

Partition strategy

Case study - AlphaGo tweets analytics

References

Summary

Manipulating your RDD in Python

Spark has a more limited Python API than Java and Scala, but it supports most of the core functionality.

The hallmark of a MapReduce system lies in two commands: map and reduce. You've seen the map function used in the earlier chapters. The map function works by taking in a function that works on each individual element in the input RDD and produces a new output element. For example, to produce a new RDD where you have added one to every number, you would use rdd.map(lambda x: x+1). It's important to understand that the map function and the other Spark functions do not transform the existing elements; instead, they return a new RDD with new elements. The reduce function takes a function that operates in pairs to combine all of the data. This is returned to the calling program. If you were to sum all the elements, you would use rdd.reduce(lambda x, y: x+y). The flatMap function is a useful utility function that allows you to write a function that returns an...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Fast Data Processing with Spark 2

Search

Your notes and bookmarks