Book Image

Apache Spark 2 for Beginners

By : Rajanarayanan Thottuvaikkatumana
Book Image

Apache Spark 2 for Beginners

By: Rajanarayanan Thottuvaikkatumana

Overview of this book

<p>Spark is one of the most widely-used large-scale data processing engines and runs extremely fast. It is a framework that has tools that are equally useful for application developers as well as data scientists.</p> <p>This book starts with the fundamentals of Spark 2 and covers the core data processing framework and API, installation, and application development setup. Then the Spark programming model is introduced through real-world examples followed by Spark SQL programming with DataFrames. An introduction to SparkR is covered next. Later, we cover the charting and plotting features of Python in conjunction with Spark data processing. After that, we take a look at Spark's stream processing, machine learning, and graph processing libraries. The last chapter combines all the skills you learned from the preceding chapters to develop a real-world Spark application.</p> <p>By the end of this book, you will have all the knowledge you need to develop efficient large-scale applications using Apache Spark.</p>
Table of Contents (15 chapters)
Apache Spark 2 for Beginners
Credits
About the Author
About the Reviewer
www.PacktPub.com
Preface

Understanding custom data processes


The views created here were created to serve various queries and to produce desired outputs. There are some other classes of data processing applications that are often developed to implement real-world use cases. From the Lambda Architecture perspective, this also falls into the serving layer. The reason why these custom data processes fall into the serving layer is mainly because most of these use or process data from the master dataset and create views or outputs. It is also very possible for the custom processed data to remain as a view, and the following use case is one of such cases.

In the SfbMicroBlog microblogging application, it is a very common requirement to see whether a given user A is in some way connected to user B in a direct follower relationship or in a transitive way. This use case can be implemented using a graph data structure to see whether the two users in question are in the same connected component, whether they are connected in...