Book Image

Mastering Spark for Data Science

By : Andrew Morgan, Antoine Amend, Matthew Hallett, David George
Book Image

Mastering Spark for Data Science

By: Andrew Morgan, Antoine Amend, Matthew Hallett, David George

Overview of this book

Data science seeks to transform the world using data, and this is typically achieved through disrupting and changing real processes in real industries. In order to operate at this level you need to build data science solutions of substance –solutions that solve real problems. Spark has emerged as the big data platform of choice for data scientists due to its speed, scalability, and easy-to-use APIs. This book deep dives into using Spark to deliver production-grade data science solutions. This process is demonstrated by exploring the construction of a sophisticated global news analysis service that uses Spark to generate continuous geopolitical and current affairs insights.You will learn all about the core Spark APIs and take a comprehensive tour of advanced libraries, including Spark SQL, Spark Streaming, MLlib, and more. You will be introduced to advanced techniques and methods that will help you to construct commercial-grade data products. Focusing on a sequence of tutorials that deliver a working news intelligence service, you will learn about advanced Spark architectures, how to work with geographic data in Spark, and how to tune Spark algorithms so they scale linearly.
Table of Contents (22 chapters)
Mastering Spark for Data Science
Credits
Foreword
About the Authors
About the Reviewer
www.PacktPub.com
Customer Feedback
Preface

GeoMesa


GeoMesa is an open source product designed to leverage the distributed nature of storage systems, such as Accumulo and Cassandra, to hold a distributed spatio-temporal database. With this design, GeoMesa is capable of running the large-scale geospatial analytics that are required for very large data sets, including GDELT.

We are going to use GeoMesa to store GDELT data and run our analytics across a large proportion of that data; this should give us access to enough data to train our model so that we can predict the future rise and fall of oil prices. Also, GeoMesa will enable us to plot large amounts of points on a map, so that we can visualize GDELT and any other useful data.

Installing

There is a very good tutorial on the GeoMesa website (www.geomesa.org) that guides the user through the installation process. Therefore, it is not our intention here to produce another how-to guide; there are, however, a few points worth noting that may save you time in getting everything up and running...