5. Loading and Saving Data in Spark | Fast Data Processing with Spark 2

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Fast Data Processing with Spark 2 - Third Edition

By : Krishna Sankar , Holden Karau

Fast Data Processing with Spark 2

By: Krishna Sankar , Holden Karau

Overview of this book

When people want a way to process big data at speed, Spark is invariably the solution. With its ease of development (in comparison to the relative complexity of Hadoop), it’s unsurprising that it’s becoming popular with data analysts and engineers everywhere. Beginning with the fundamentals, we’ll show you how to get set up with Spark with minimum fuss. You’ll then get to grips with some simple APIs before investigating machine learning and graph processing – throughout we’ll make sure you know exactly how to apply your knowledge. You will also learn how to use the Spark shell, how to load data before finding out how to build and run your own Spark applications. Discover how to manipulate your RDD and get stuck into a range of DataFrame APIs. As if that’s not enough, you’ll also learn some useful Machine Learning algorithms with the help of Spark MLlib and integrating Spark with R. We’ll also make sure you’re confident and prepared for graph processing, as you learn more about the GraphX API.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Installing Spark and Setting Up Your Cluster

1. Installing Spark and Setting Up Your Cluster

Directory organization and convention

Installing the prebuilt distribution

Building Spark from source

Spark topology

A single machine

Running Spark on EC2

Deploying Spark with Chef (Opscode)

Deploying Spark on Mesos

Spark on YARN

Spark standalone mode

References

Summary

2. Using the Spark Shell

2. Using the Spark Shell

The Spark shell

Loading a simple text file

Interactively loading data from S3

Summary

3. Building and Running a Spark Application

3. Building and Running a Spark Application

Building Spark applications

Data wrangling with iPython

Developing Spark with Eclipse

Developing Spark with other IDEs

Building your Spark job with Maven

Building your Spark job with something else

References

Summary

4. Creating a SparkSession Object

4. Creating a SparkSession Object

SparkSession versus SparkContext

Building a SparkSession object

SparkContext - metadata

Shared Java and Scala APIs

Python

iPython

Reference

Summary

5. Loading and Saving Data in Spark

5. Loading and Saving Data in Spark

Spark abstractions

Data modalities

Data modalities and Datasets/DataFrames/RDDs

Loading data into an RDD

Saving your data

References

Summary

6. Manipulating Your RDD

6. Manipulating Your RDD

Manipulating your RDD in Scala and Java

Manipulating your RDD in Python

References

Summary

7. Spark 2.0 Concepts

7. Spark 2.0 Concepts

Code and Datasets for the rest of the book

The data scientist and Spark features

Spark v2.0 and beyond

Apache Spark - evolution

Apache Spark - the full stack

The art of a big data store - Parquet

References

Summary

8. Spark SQL

8. Spark SQL

The Spark SQL architecture

Spark SQL how-to in a nutshell

Spark SQL programming

References

Summary

9. Foundations of Datasets/DataFrames – The Proverbial Workhorse for DataScientists

9. Foundations of Datasets/DataFrames – The Proverbial Workhorse for DataScientists

Datasets - a quick introduction

Dataset APIs - an overview

Dataset interfaces and functions

References

Summary

10. Spark with Big Data

10. Spark with Big Data

Parquet - an efficient and interoperable big data format

HBase

Reference

Summary

11. Machine Learning with Spark ML Pipelines

11. Machine Learning with Spark ML Pipelines

Spark's machine learning algorithm table

Spark machine learning APIs - ML pipelines and MLlib

ML pipelines

Spark ML examples

The API organization

Basic statistics

Linear regression

Classification

Clustering

Recommendation

Hyper parameters

The final thing

References

Summary

12. GraphX

12. GraphX

Graphs and graph processing - an introduction

Spark GraphX

GraphX - computational model

The first example - graph

Building graphs

The GraphX API landscape

Structural APIs

Community, affiliation, and strengths

Algorithms

Partition strategy

Case study - AlphaGo tweets analytics

References

Summary

Saving your data

While distributed computational jobs are a lot of fun, they are much more useful when the results are stored in a useful place. While the methods for loading an RDD are largely found in the SparkContext class, the methods for saving an RDD are defined on the RDD classes. In Scala, implicit conversions exist so that an RDD, which can be saved as a sequence file, could be converted to the appropriate type; in Java, explicit conversions must be used.

Here are the different ways to save an RDD.

Here's the code for Scala:

rddOfStrings.saveAsTextFile("out.txt") 
keyValueRdd.saveAsObjectFile("sequenceOut")

Here's the code for Java:

rddOfStrings.saveAsTextFile("out.txt") 
keyValueRdd.saveAsObjectFile("sequenceOut")

Here's the code for Python:

rddOfStrings.saveAsTextFile("out.txt")

Tip

In addition, users can save the RDD as a compressed text file using the following function: saveAsTextFile(path: String, codec: Class[_ <: CompressionCodec])

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Fast Data Processing with Spark 2

Search

Your notes and bookmarks