Book Image

Mastering Spark for Data Science

By : Andrew Morgan, Antoine Amend, Matthew Hallett, David George
Book Image

Mastering Spark for Data Science

By: Andrew Morgan, Antoine Amend, Matthew Hallett, David George

Overview of this book

Data science seeks to transform the world using data, and this is typically achieved through disrupting and changing real processes in real industries. In order to operate at this level you need to build data science solutions of substance –solutions that solve real problems. Spark has emerged as the big data platform of choice for data scientists due to its speed, scalability, and easy-to-use APIs. This book deep dives into using Spark to deliver production-grade data science solutions. This process is demonstrated by exploring the construction of a sophisticated global news analysis service that uses Spark to generate continuous geopolitical and current affairs insights.You will learn all about the core Spark APIs and take a comprehensive tour of advanced libraries, including Spark SQL, Spark Streaming, MLlib, and more. You will be introduced to advanced techniques and methods that will help you to construct commercial-grade data products. Focusing on a sequence of tutorials that deliver a working news intelligence service, you will learn about advanced Spark architectures, how to work with geographic data in Spark, and how to tune Spark algorithms so they scale linearly.
Table of Contents (22 chapters)
Mastering Spark for Data Science
About the Authors
About the Reviewer
Customer Feedback


The impact of Spark on the world of data science has been startling. It is less than 3 years since Spark 1.0 was released and yet Spark is already accepted as the omni-competent kernel of any big data architecture. We adopted Spark as our core technology at Barclays around this time and this was considered a bold (read ‘rash’) move. Now it is taken as a given that Spark is your starting point for any big data science project.

As data science has developed both as an activity and as an accepted term, there has been much talk about the unicorn data scientist. This is the unlikely character who can do both the maths and the coding. They are apparently hard to find, and harder to keep. My team likes to think more in terms of three data science competencies: pattern recognition, distributed computation, and automation. If data science is about exploiting insights from data in production, then you need to be able to develop applications with these three competencies in mind from the start. There is no point using a machine learning methodology that won’t scale with your data, or building an analytical kernel that needs to be re-coded to be production quality. And so you need either a unicorn or a unicorn-team (my preference) to do the work.

Spark is your unicorn technology. No other language not only expresses analytical concepts elegantly but also moves effortlessly from the small scale to big data, and so naturally facilitates production-ready code as Spark (with the Scala API). We chose Spark because we could compose a model in a few lines, run the same code on the cluster as we had tried out on the laptop, and build robust unit-tested JVM applications that we could be confident would run in business-critical use cases. The combination of functional programming in Scala with the Spark abstractions is uniquely powerful, and choosing it has been a significant cause of the success of the team over the last 3 years.

So here's the conundrum. Why are there no books which present Spark in this way, recognizing that one of the best reasons to work in Spark is its application to production data science? If you scan the bookshelves (or look at tutorials online) all you will find is toy models and a review of the Spark APIs and libs. You will find little or nothing about how Spark fits into the wider architecture, or about how to manage data ETL in a sustainable way.

I think you will find that the practical approach taken by the authors in this book is different. Each chapter takes on a new challenge, and each reads as a voyage of discovery where the outcome was not necessarily known in advance of the exploration. And the value of doing data science properly is set out clearly from the start. This is one of the first books on Spark for grown-ups who want to do real data science that will make an impact on their organisation. I hope you enjoy it.

Harry Powell

Head of Advanced Analytics, Barclays