This book is intended to present the GraphX library for Apache Spark and to teach the fundamental techniques and recipes to process graph data at scale. It is intended to be a self-study step-by-step guide for anyone new to Spark with an interest in or need for large-scale graph processing.
The focus of this book is on large-scale graph processing with Apache Spark. The book teaches a variety of graph processing abstractions and algorithms and provides concise and sufficient information about them. You can confidently learn all of it and put it to use in different applications.
Step-by-step guide: Each chapter teaches important techniques for every stage of the pipeline, from loading and transforming graph data to implementing graph-parallel operations and machine learning algorithms.
Hands-on approach: We show how each technique works using the Scala REPL with simple examples and by building standalone Spark applications.
Detailed code: All the Scala code in the book is available for download from the book webpage of Packt Publishing.
Real-world examples: We apply these techniques on open datasets collected from a broad variety of applications ranging from social networks to food science and sports analytics.
This book consists of seven chapters. The first three chapters help you to get started quickly with Spark and GraphX. Then, the next two chapters teach the core techniques and abstractions to manipulate and aggregate graph data. Finally, the last two chapters of this book cover more advanced topics such as graph clustering, implementing graph-parallel iterative algorithms with Pregel, and learning methods from graph data.
Chapter 1, Getting Started with Spark and GraphX, begins with an introduction to the Spark system, its libraries, and the Scala Build Tool. It explains how to install and leverage Spark on the command line and in a standalone Scala program.
Chapter 2, Building and Exploring Graphs, presents the methods for building Spark graphs using illustrative network datasets.
Chapter 3, Graph Analysis and Visualization, walks you through the process of exploring, visualizing, and analyzing different network characteristics.
Chapter 4, Transforming and Shaping Up Graphs to Your Needs, teaches you how to transform raw datasets into a usable form that is appropriate for later analysis.
Chapter 5, Creating Custom Graph Aggregation Operators, teaches you how to create custom graph operations that are tailored to your specific needs with efficiency in mind, using the powerful message-passing aggregation operator in Spark.
Chapter 6, Iterative Graph-Parallel Processing with Pregel, explains the inner workings of the Pregel computational model and describes some use cases.
Chapter 7, Learning Graph Structures, introduces graph clustering, which is useful for detecting communities in graphs and applies it to a social music database.
To learn effectively from this book, it is helpful to have a beginner-level programming experience with Scala. However, intermediate functional constructs or Scala-specific syntax are highlighted and explained as they appear in the book. Prior experience with Spark's core API or with the MapReduce framework is beneficial but not required.
It is also beneficial to follow along with the examples, using a Windows or Unix computer with a Java Development Kit environment. More details on the system requirements are described in the first chapter.
This book is for data engineers, software developers, and data scientists who need to process graph data at a large scale. This book is intended to be self-contained and does not assume any prior experience with Spark. However, its focus is entirely on graph processing.
In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, and user input are shown as follows: When we wish to run a Scala code in the Spark shell, it will start with scala > prompt.
A block of code is set as follows:
val greeting = sc.parallelize("Hello Spark".toList)
greeting.collect foreach {
c => println(c)
}Any command-line input or output is written as follows:
> tar -xf spark-1.4.1-bin-hadoop2.6.tgz
New terms and important words are shown in bold.
Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.
To send us general feedback, simply e-mail <[email protected]>, and mention the book's title in the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.
To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at <[email protected]> with a link to the suspected pirated material.
We appreciate your help in protecting our authors and our ability to bring you valuable content.
If you have a problem with any aspect of this book, you can contact us at <[email protected]>, and we will do our best to address the problem.