YARN is Apache Hadoop's NextGen Resource Manager. The Spark project provides an easy way to schedule jobs on YARN once you have a Spark assembly built. The Spark web page, http://spark.apache.org/docs/latest/running-on-yarn.html, has the configuration details for YARN, which we had built earlier for compiling with the -Pyarn
switch.
Fast Data Processing with Spark 2 - Third Edition
By :
Fast Data Processing with Spark 2 - Third Edition
By:
Overview of this book
When people want a way to process big data at speed, Spark is invariably the solution. With its ease of development (in comparison to the relative complexity of Hadoop), it’s unsurprising that it’s becoming popular with data analysts and engineers everywhere.
Beginning with the fundamentals, we’ll show you how to get set up with Spark with minimum fuss. You’ll then get to grips with some simple APIs before investigating machine learning and graph processing – throughout we’ll make sure you know exactly how to apply your knowledge.
You will also learn how to use the Spark shell, how to load data before finding out how to build and run your own Spark applications. Discover how to manipulate your RDD and get stuck into a range of DataFrame APIs. As if that’s not enough, you’ll also learn some useful Machine Learning algorithms with the help of Spark MLlib and integrating Spark with R. We’ll also make sure you’re confident and prepared for graph processing, as you learn more about the GraphX API.
Table of Contents (18 chapters)
Fast Data Processing with Spark 2 Third Edition
Credits
About the Author
About the Reviewers
www.PacktPub.com
Preface
Free Chapter
Installing Spark and Setting Up Your Cluster
Using the Spark Shell
Building and Running a Spark Application
Creating a SparkSession Object
Loading and Saving Data in Spark
Manipulating Your RDD
Spark 2.0 Concepts
Spark SQL
Foundations of Datasets/DataFrames – The Proverbial Workhorse for DataScientists
Spark with Big Data
Machine Learning with Spark ML Pipelines
GraphX
Customer Reviews