Learning Spark SQL

Learning Spark SQL

By : Aurobindo Sarkar

Buy this Book

Learning Spark SQL

By: Aurobindo Sarkar

Buy this Book

Overview of this book

In the past year, Apache Spark has been increasingly adopted for the development of distributed applications. Spark SQL APIs provide an optimized interface that helps developers build such applications quickly and easily. However, designing web-scale production applications using Spark SQL APIs can be a complex task. Hence, understanding the design and implementation best practices before you start your project will help you avoid these problems. This book gives an insight into the engineering practices used to design and build real-world, Spark-based applications. The book's hands-on examples will give you the required confidence to work on any future projects you encounter in Spark SQL. It starts by familiarizing you with data exploration and data munging tasks using Spark SQL and Scala. Extensive code examples will help you understand the methods used to implement typical use-cases for various types of applications. You will get a walkthrough of the key concepts and terms that are common to streaming, machine learning, and graph applications. You will also learn key performance-tuning details including Cost Based Optimization (Spark 2.2) in Spark SQL applications. Finally, you will move on to learning how such systems are architected and deployed for a successful delivery of your project.

Title Page

Credits

About the Author

About the Reviewer

www.PacktPub.com

Customer Feedback

Preface

Free Chapter

Getting Started with Spark SQL

What is Spark SQL?

Introducing SparkSession

Understanding Spark SQL concepts

Using Spark SQL in streaming applications

Summary

Using Spark SQL for Processing Structured and Semistructured Data

Understanding data sources in Spark applications

Using Spark with relational databases

Using Spark with MongoDB (NoSQL database)

Using Spark with JSON data

Using Spark with Avro files

Using Spark with Parquet files

Defining and using custom data sources in Spark

Summary

Using Spark SQL for Data Exploration

Introducing Exploratory Data Analysis (EDA)

Using Spark SQL for basic data analysis

Visualizing data with Apache Zeppelin

Sampling data with Spark SQL APIs

Using Spark SQL for creating pivot tables

Summary

Using Spark SQL for Data Munging

Introducing data munging

Exploring data munging techniques

Munging textual data

Munging time series data

Dealing with variable length records

Preparing data for machine learning

Summary

Using Spark SQL in Streaming Applications

Introducing streaming data applications

Building Spark streaming applications

Using Kafka with Spark Structured Streaming

Writing a receiver for a custom data source

Summary

Using Spark SQL in Machine Learning Applications

Introducing machine learning applications

Introducing feature engineering

Implementing a Spark ML classification model

Introducing Spark ML tools and utilities

Implementing a Spark ML clustering model

Summary

Using Spark SQL in Graph Applications

Introducing large-scale graph applications

Exploring graphs using GraphFrames

Analyzing JSON input modeled as a graph

Processing graphs containing multiple types of relationships

Understanding GraphFrame internals

Summary

Using Spark SQL with SparkR

Introducing SparkR

Understanding the SparkR architecture

Understanding SparkR DataFrames

Using SparkR for EDA and data munging tasks

Using SparkR for computing summary statistics

Using SparkR for data visualization

Using SparkR for machine learning

Summary

Developing Applications with Spark SQL

Introducing Spark SQL applications

Understanding text analysis applications

Understanding themes in document corpuses

Using Naive Bayes classifiers

Developing a machine learning application

Summary

Using Spark SQL in Deep Learning Applications

Introducing neural networks

Introducing deep learning in Spark

Understanding Supervised learning

Using deep neural networks for language processing

Introducing autoencoders

Summary

Tuning Spark SQL Components for Performance

Introducing performance tuning in Spark SQL

Understanding DataFrame/Dataset APIs

Understanding Catalyst optimizations

Visualizing Spark application execution

Cost-based optimizer in Apache Spark 2.2

Understanding multi-way JOIN ordering optimization

Understanding performance improvements using whole-stage code generation

Summary

Spark SQL in Large-Scale Application Architectures

Understanding Spark-based application architectures

Understanding the Lambda architecture

Understanding the Kappa Architecture

Design considerations for building scalable stream processing applications

Building robust ETL pipelines using Spark SQL

Implementing a scalable monitoring solution

Deploying Spark machine learning pipelines

Using cluster managers

Summary

Customer Reviews

5 star

4 star

3 star

2 star

1 star

What is Spark SQL?

Spark SQL is one of the most advanced components of Apache Spark. It has been a part of the core distribution since Spark 1.0 and supports Python, Scala, Java, and R programming APIs. As illustrated in the figure below, Spark SQL components provide the foundation for machine learning applications, streaming applications, graph applications, and many other types of application architectures.

Such applications, typically, use Spark ML pipelines, Structured Streaming, and GraphFrames, which are all based on Spark SQL interfaces (DataFrame/Dataset API). These applications, along with constructs such as SQL, DataFrames, and Datasets API, receive the benefits of the Catalyst optimizer, automatically. This optimizer is also responsible for generating executable query plans based on the lower-level RDD interfaces.

We will explore ML pipelines in more detail in Chapter 6, Using Spark SQL in Machine Learning Applications. GraphFrames will be covered in Chapter 7, Using Spark SQL in Graph Applications. While, we will introduce the key concepts regarding Structured Streaming and the Catalyst optimizer in this chapter, we will get more details about them in Chapter 5, Using Spark SQL in Streaming Applications, and Chapter 11, Tuning Spark SQL Components for Performance.

In Spark 2.0, the DataFrame API has been merged with the Dataset API, thereby unifying data processing capabilities across Spark libraries. This also enables developers to work with a single high-level and type-safe API. However, the Spark software stack does not prevent developers from directly using the low-level RDD interface in their applications. Though the low-level RDD API will continue to be available, a vast majority of developers are expected to (and are recommended to) use the high-level APIs, namely, the Dataset and DataFrame APIs.

Additionally, Spark 2.0 extends Spark SQL capabilities by including a new ANSI SQL parser with support for subqueries and the SQL:2003 standard. More specifically, the subquery support now includes correlated/uncorrelated subqueries, and IN / NOT IN and EXISTS / NOTEXISTS predicates in WHERE / HAVING clauses.

At the core of Spark SQL is the Catalyst optimizer, which leverages Scala's advanced features, such as pattern matching, to provide an extensible query optimizer. DataFrames, Datasets, and SQL queries share the same execution and optimization pipeline; hence, there is no performance of using any one or the other of these constructs (or of using any of the supported programming APIs). The high-level DataFrame-based code written by the developer is converted to Catalyst expressions and then to low-level Java bytecode as it passes through this pipeline.

SparkSession is the entry point into Spark SQL-related functionality and we describe it in more detail in the next section.

Learning Spark SQL

By : Aurobindo Sarkar

Learning Spark SQL

By: Aurobindo Sarkar

Overview of this book

Related Content you might be interested in

Current Title:

Learning Spark SQL

Apache Spark 2.x Cookbook

Learning PySpark

Machine Learning with Spark

What is Spark SQL?