Book Image

Apache Spark: Tips, Tricks, & Techniques [Video]

By : Tomasz Lelek
Book Image

Apache Spark: Tips, Tricks, & Techniques [Video]

By: Tomasz Lelek

Overview of this book

Apache Spark has been around for quite some time, but do you really know how to get the most out of Spark? This course aims at giving you new possibilities; you will explore many aspects of Spark, some you may have never heard of and some you never knew existed. In this course you'll learn to implement some practical and proven techniques to improve particular aspects of programming and administration in Apache Spark. You will explore 7 sections that will address different aspects of Spark via 5 specific techniques with clear instructions on how to carry out different Apache Spark tasks with hands-on experience. The techniques are demonstrated using practical examples and best practices. By the end of this course, you will have learned some exciting tips, best practices, and techniques with Apache Spark. You will be able to perform tasks and get the best data out of your databases much faster and with ease. All the code and supporting files for this course are available on Github at https://github.com/PacktPublishing/Apache-Spark-Tips-Tricks-Techniques

Related Content you might be interested in

Current Title:

Small book image
Apache Spark: Tips, Tricks, & Techniques [Video]
Tomasz Lelek
Nov, 2018 | 2 hours 26 minutes
No titles found
Table of Contents (7 chapters)
Free Chapter
1
Transformations and Actions
The Course Overview
Using Spark Transformations to Defer Computations to a Later Time
Avoiding Transformations
Using reduce and reduceByKey to Calculate Results
Performing Actions That Trigger Computations
Reusing the Same RDD for Different Actions
2
Immutable Design
Delve into Spark RDDs Parent/Child Chain
Using RDD in an Immutable Way
Using DataFrame Operations to Transform It
Immutability in the Highly Concurrent Environment
Using Dataset API in an Immutable Way
3
Avoid Shuffle and Reduce Operational Expenses
Detecting a Shuffle in a Processing
Testing Operations That Cause Shuffle in Apache Spark
Changing Design of Jobs with Wide Dependencies
Using keyBy() Operations to Reduce Shuffle
Using Custom Partitioner to Reduce Shuffle
4
Saving Data in the Correct Format
Saving Data in Plain Text
Leveraging JSON as a Data Format
Tabular Formats – CSV
Using Avro with Spark
Columnar Formats – Parquet
5
Working with Spark Key/Value API
Available Transformations on Key/Value Pairs
Using aggregateByKey Instead of groupBy()
Actions on Key/Value Pairs
Available Partitioners on Key/Value Data
Implementing Custom Partitioner
6
Testing Apache Spark Jobs
Separating Logic from Spark Engine – Unit Testing
Integration Testing Using SparkSession
Mocking Data Sources Using Partial Functions
Using ScalaCheck for Property-Based Testing
Testing in Different Versions of Spark
7
Leveraging Spark GraphX API
Creating Graph from Datasource
Using Vertex API
Using Edge API
Calculate Degree of Vertex
Calculate Page Rank
Chapter 6
Testing Apache Spark Jobs
Content Locked
Section 5
Testing in Different Versions of Spark
In the last video of the section, we will learn to apply the test in different versions of spark. - Resisting component to work with Spark pre-2.x - Mock testing pre-2.x - RDD mock testing
Previous Section
End of Chapter 6
Next Chapter