Book Image

Apache Spark Quick Start Guide

By : Shrey Mehrotra, Akash Grade
Book Image

Apache Spark Quick Start Guide

By: Shrey Mehrotra, Akash Grade

Overview of this book

Apache Spark is a ?exible framework that allows processing of batch and real-time data. Its unified engine has made it quite popular for big data use cases. This book will help you to get started with Apache Spark 2.0 and write big data applications for a variety of use cases. It will also introduce you to Apache Spark – one of the most popular Big Data processing frameworks. Although this book is intended to help you get started with Apache Spark, but it also focuses on explaining the core concepts. This practical guide provides a quick start to the Spark 2.0 architecture and its components. It teaches you how to set up Spark on your local machine. As we move ahead, you will be introduced to resilient distributed datasets (RDDs) and DataFrame APIs, and their corresponding transformations and actions. Then, we move on to the life cycle of a Spark application and learn about the techniques used to debug slow-running applications. You will also go through Spark’s built-in modules for SQL, streaming, machine learning, and graph analysis. Finally, the book will lay out the best practices and optimization techniques that are key for writing efficient Spark applications. By the end of this book, you will have a sound fundamental understanding of the Apache Spark framework and you will be able to write and optimize Spark applications.
Table of Contents (10 chapters)

Graph processing

In Chapter 1, Introduction to Apache Spark, we provided a brief introduction to the graph libraries provided by Spark. In this section, we will discuss these libraries in more detail. A graph is one of the data structures that is used in computer science to solve some real-world problems. A graph is represented by a set of vertices and edges. A vertex is an object, and an edge defines a relationship between two vertices. One of the examples of graphs is a social network, where each person is represented by a vertex, and a relationship between two people is represented by an edge. The following figure shows a graph with five vertices and five edges:

A graph with five vertices and five edges

The graph represented here can also be represented as a set, V = {V1, V2, V3, V4, V5} and E = {E1, E2, E3, E4, E5}. A graph can also be categorized as a directed or undirected...