Book Image

Mastering Spark for Data Science

By : Bifet, Morgan, Amend, Hallett, George
Book Image

Mastering Spark for Data Science

By: Bifet, Morgan, Amend, Hallett, George

Overview of this book

Data science seeks to transform the world using data, and this is typically achieved through disrupting and changing real processes in real industries. In order to operate at this level you need to build data science solutions of substance –solutions that solve real problems. Spark has emerged as the big data platform of choice for data scientists due to its speed, scalability, and easy-to-use APIs. This book deep dives into using Spark to deliver production-grade data science solutions. This process is demonstrated by exploring the construction of a sophisticated global news analysis service that uses Spark to generate continuous geopolitical and current affairs insights.You will learn all about the core Spark APIs and take a comprehensive tour of advanced libraries, including Spark SQL, Spark Streaming, MLlib, and more. You will be introduced to advanced techniques and methods that will help you to construct commercial-grade data products. Focusing on a sequence of tutorials that deliver a working news intelligence service, you will learn about advanced Spark architectures, how to work with geographic data in Spark, and how to tune Spark algorithms so they scale linearly.
Table of Contents (15 chapters)

Studying trends

The dictionary definition of trend is a general direction in which something is developing or changing, but there are other more focused definitions that might be more helpful for guiding data science. Two such definitions are from Salomé Areias, who studies social trends, and Eurostat, the official statistical agency in the European Union:

"A trend is the slow variation over a longer period of time, usually several years, generally associated with the structural causes affecting the phenomenon being measured." - EUROSTAT, official statistical agency in the European Union (http://ec.europa.eu/eurostat/statistics-explained/index.php/Glossary:Trend)

"A Trend is defined by a shift in behavior or mentality that influences a significant amount of people." - Salomé Areias, social trend commentator (https://salomeareias.wordpress.com/what-is-a-trend/)

We generally think of trends as nothing more than a long rise or fall in stock market prices...