Before going into the present-day Spark, it might be worthwhile understanding what problems Spark intend to solve, and especially the data movement. Without knowing the background we will not be able to predict the future.

"You have to learn the past to predict the future."

Late 1990s: The world was a much simpler place to live, with proprietary databases being the sole choice of consumers. Data was growing at quite an amazing pace, and some of the biggest databases boasted of maintaining datasets in excess of a Terabyte.

Early 2000s: The dotcom bubble happened, meant companies started going online, and likes of Amazon and eBay leading the revolution. Some of the dotcom start-ups failed, while others succeeded. The commonality among the business models started was a razor-sharp focus on page views, and everything started getting focused on the number of users. A lot of marketing budget was spent on getting people online. This meant more customer behavior data in the form of weblogs. Since the defacto storage was an MPP database, and the value of such weblogs was unknown, more often than not these weblogs were stuffed into archive storage or deleted.

2002: In search for a better search engine, Doug Cutting and Mike Cafarella started work on an open source project called Nutch, the objective of which was to be a web scale crawler. Web-Scale was defined as billions of web pages and Doug and Mike were able to index hundreds of millions of web-pages, running on a handful of nodes and had a knack of falling down.

2004-2006: Google published a paper on the Google File System (GFS) (2003) and MapReduce (2004) demonstrating the backbone of their search engine being resilient to failures, and almost linearly scalable. Doug Cutting took particular interest in this development as he could see that GFS and MapReduce papers directly addressed Nutch’s shortcomings. Doug Cutting added Map Reduce implementation to Nutch which ran on 20 nodes, and was much easier to program. Of course we are talking in comparative terms here.

2006-2008: Cutting went to work with Yahoo in 2006 who had lost the search crown to Google and were equally impressed by the GFS and MapReduce papers. The storage and processing parts of Nutch were spun out to form a separate project named Hadoop under AFS where as Nutch web crawler remained a separate project. Hadoop became a top-level Apache project in 2008. On February 19, 2008 Yahoo announced that its search index is run on a 10000 node Hadoop cluster (truly an amazing feat).

We haven't forget about the proprietary database vendors. the majority of them didn’t expect Hadoop to change anything for them, as database vendors typically focused on relational data, which was smaller in volumes but higher in value. I was talking to a CTO of a major database vendor (will remain unnamed), and discussing this new and upcoming popular elephant (Hadoop of course! Thanks to Doug Cutting’s son for choosing a sane name. I mean he could have chosen anything else, and you know how kids name things these days..). The CTO was quite adamant that the real value is in the relational data, which was the bread and butter of his company, and despite that fact that the relational data had huge volumes, it had less of a business value. This was more of a 80-20 rule for data, where from a size perspective unstructured data was 4 times the size of structured data (80-20), whereas the same structured data had 4 times the value of unstructured data. I would say that the relational database vendors massively underestimated the value of unstructured data back then.

Anyways, back to Hadoop: So, after the announcement by Yahoo, a lot of companies wanted to get a piece of the action. They realised something big was about to happen in the dataspace. Lots of interesting use cases started to appear in the Hadoop space, and the defacto compute engine on Hadoop, MapReduce wasn’t able to meet all those expectations.

The MapReduce Conundrum: The original Hadoop comprised primarily HDFS and Map-Reduce as a compute engine. The original use case of web scale search meant that the architecture was primarily aimed at long-running batch jobs (typically single-pass jobs without iterations), like the original use case of indexing web pages. The core requirement of such a framework was scalability and fault-tolerance, as you don’t want to restart a job that had been running for 3 days, having completed 95% of its work. Furthermore, the objective of MapReduce was to target acyclic data flows.

A typical MapReduce program is composed of a Map() operation and optionally a Reduce() operation, and any workload had to be converted to the MapReduce paradigm before you could get the benefit of Hadoop. Not only that majority of other open source projects on Hadoop also used MapReduce as a way to perform computation. For example: Hive and Pig Latin both generated MapReduce to operate on Big Data sets. The problem with the architecture of MapReduce was that the job output data from each step had to be store in a distributed system before the next step could begin. This meant that each iteration had to reload the data from the disk thus incurring a significant performance penalty. Furthermore, while typically design, for batch jobs, Hadoop has often been used to do exploratory analysis through SQL-like interfaces such as Pig and Hive. Each query incurs significant latency due to initial MapReduce job setup, and initial data read which often means increased wait times for the users.

Beginning of Spark: In June of 2011, Matei Zaharia, Mosharaf Chowdhury, Michael J. Franklin, Scott Shenker and Ion Stoica published a paper in which they proposed a framework that could outperform Hadoop 10 times in iterative machine learning jobs. The framework is now known as Spark. The paper aimed to solve two of the major inadequacies of the Hadoop/MR framework:

The idea that you can plug the gaps of map-reduce from an iterative and interactive analysis point of view, while maintaining its scalability and resilience meant that the platform could be used across a wide variety of use cases.

This created huge interest in Spark, particularly from communities of users who had become frustrated with the relatively slow response from MapReduce, particularly for interactive queries requests. Spark in 2015 became the most active open source project in Big Data, and had tons of new features of improvements during the course of the project. The community grew almost 300%, with attendances at Spark-Summit increasing from just 1,100 in 2014 to almost 4,000 in 2015. The number of meetup groups grew by a factor of 4, and the contributors to the project increased from just over a 100 in 2013 to 600 in 2015.

Spark is today the hottest technology for big data analytics. Numerous benchmarks have confirmed that it is the fastest engine out there. If you go to any Big data conference be it Strata + Hadoop World or Hadoop Summit, Spark is considered to be the technology for future.

Stack Overflow released the results of a 2016 developer survey (http://bit.ly/1MpdIlU) with responses from 56,033 engineers across 173 countries. Some of the facts related to Spark were pretty interesting. Spark was the leader in Trending Tech and the Top-Paying Tech.

Chapter 1, Architecture and Installation, will help you get started on the journey of learning Spark. This will walk you through key architectural components before helping you write your first Spark application.

Chapter 2, Transformations and Actions with Spark RDDs, will help you understand the basic constructs as Spark RDDs and help you understand the difference between transformations, actions, and lazy evaluation, and how you can share data.

Chapter 3, ELT with Spark, will help you with data loading, transformation, and saving it back to external storage systems.

Chapter 4, Spark SQL, will help you understand the intricacies of the DataFrame and Dataset API before a discussion of the under-the-hood power of the Catalyst optimizer and how it ensures that your client applications remain performant irrespective of your client AP.

Chapter 5, Spark Streaming, will help you understand the architecture of Spark Streaming, sliding window operations, caching, persistence, check-pointing, fault-tolerance before discussing structured streaming and how it revolutionizes Stream processing.

Chapter 6, Machine Learning with Spark, is where the rubber hits the road, and where you understand the basics of machine learning before looking at the various types of machine learning, and feature engineering utility functions, and finally looking at the algorithms provided by Spark MLlib API.

Chapter 7, GraphX, will help you understand the importance of Graph in today’s world, before understanding terminology such vertex, edge, Motif etc. We will then look at some of the graph algorithms in GraphX and also talk about GraphFrames.

Chapter 8, Operating in Clustered mode, helps the user understand how Spark can be deployed as standalone, or with YARN or Mesos.

Chapter 9, Building a Recommendation system, will help the user understand the intricacies of a recommendation system before building one with an ALS model.

Chapter 10, Customer Churn Predicting, will help the user understand the importance of Churn prediction before using a random forest classifier to predict churn on a telecommunication dataset.

Appendix, There's More with Spark, is where we cover the topics around performance tuning, sizing your executors, and security before walking the user through setting up PySpark with Jupyter notebook.

You will need Spark 2.0, which you can download from Apache Spark website. We have used few different configurations, but you can essentially run most of these examples inside a virtual machine with 4-8GB of RAM, and 10 GB of available disk space.

This book is for people who have heard of Spark, and want to understand more. This is a beginner-level book for people who want to have some hands-on exercise with the fastest growing open source project. This book provides ample reading and links to exciting YouTube videos for additional exploration of the topics.

In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "We can include other contexts through the use of the include directive."

A block of code is set as follows:

[default]
exten => s,1,Dial(Zap/1|30)
exten => s,2,Voicemail(u100)
exten => s,102,Voicemail(b100)
exten => i,1,Voicemail(s0)

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

[default]
exten => s,1,Dial(Zap/1|30)
exten => s,2,Voicemail(u100)
exten => s,102,Voicemail(b100)
exten => i,1,Voicemail(s0)

Any command-line input or output is written as follows:

# cp /usr/src/asterisk-addons/configs/cdr_mysql.conf.sample
     /etc/asterisk/cdr_mysql.conf

New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: "Clicking the Next button moves you to the next screen."

Feedback from our readers is always welcome. Let us know what you think about this book-what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of. To send us general feedback, simply e-mail [email protected], and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.