Book Image

Learning Spark SQL

By : Aurobindo Sarkar
Book Image

Learning Spark SQL

By: Aurobindo Sarkar

Overview of this book

In the past year, Apache Spark has been increasingly adopted for the development of distributed applications. Spark SQL APIs provide an optimized interface that helps developers build such applications quickly and easily. However, designing web-scale production applications using Spark SQL APIs can be a complex task. Hence, understanding the design and implementation best practices before you start your project will help you avoid these problems. This book gives an insight into the engineering practices used to design and build real-world, Spark-based applications. The book's hands-on examples will give you the required confidence to work on any future projects you encounter in Spark SQL. It starts by familiarizing you with data exploration and data munging tasks using Spark SQL and Scala. Extensive code examples will help you understand the methods used to implement typical use-cases for various types of applications. You will get a walkthrough of the key concepts and terms that are common to streaming, machine learning, and graph applications. You will also learn key performance-tuning details including Cost Based Optimization (Spark 2.2) in Spark SQL applications. Finally, you will move on to learning how such systems are architected and deployed for a successful delivery of your project.
Table of Contents (19 chapters)
Title Page
Credits
About the Author
About the Reviewer
www.PacktPub.com
Customer Feedback
Preface

Munging time series data


Time series data is a sequence of linked to a timestamp. In section, we use Cloudera's spark-ts package for analyzing time-series data.

Note

Refer to Cloudera Engineering Blog, A New Library for Analyzing Time-Series Data with Apache Spark, for more details on time-series data and its processing using spark-ts. This blog is available at: https://github.com/sryza/spark-timeseries.

The spark-ts package can be downloaded and using instructions available at:

https://github.com/sryza/spark-timeseries.

We will attempt to accomplish the following objectives in the following sub-sections:

  • Pre-processing of the time-series Dataset
  • Processing date fields
  • Persisting and loading data
  • Defining a date-time index
  • Using the  TimeSeriesRDD object
  • Handling missing time-series data
  • Computing basic statistics

For this section, specify inclusion of the spark-ts.jar file while starting the Spark shell as shown:

We download Datasets containing pricing and volume data for six stocks over a one year...