Book Image

Learning PySpark

By : Tomasz Drabas, Denny Lee
Book Image

Learning PySpark

By: Tomasz Drabas, Denny Lee

Overview of this book

Apache Spark is an open source framework for efficient cluster computing with a strong interface for data parallelism and fault tolerance. This book will show you how to leverage the power of Python and put it to use in the Spark ecosystem. You will start by getting a firm understanding of the Spark 2.0 architecture and how to set up a Python environment for Spark. You will get familiar with the modules available in PySpark. You will learn how to abstract data with RDDs and DataFrames and understand the streaming capabilities of PySpark. Also, you will get a thorough overview of machine learning capabilities of PySpark using ML and MLlib, graph processing using GraphFrames, and polyglot persistence using Blaze. Finally, you will learn how to deploy your applications to the cloud using the spark-submit command. By the end of this book, you will have established a firm understanding of the Spark Python API and how it can be used to build data-intensive applications.
Table of Contents (20 chapters)
Learning PySpark
Credits
Foreword
About the Authors
About the Reviewer
www.PacktPub.com
Customer Feedback
Preface
Index

Querying with SQL


Let's run the same queries, except this time, we will do so using SQL queries against the same DataFrame. Recall that this DataFrame is accessible because we executed the .createOrReplaceTempView method for swimmers.

Number of rows

The following is the code snippet to get the number of rows within your DataFrame using SQL:

spark.sql("select count(1) from swimmers").show()

The output is as follows:

Running filter statements using the where Clauses

To run a filter statement using SQL, you can use the where clause, as noted in the following code snippet:

# Get the id, age where age = 22 in SQL
spark.sql("select id, age from swimmers where age = 22").show()

The output of this query is to choose only the id and age columns where age = 22:

As with the DataFrame API querying, if we want to get back the name of the swimmers who have an eye color that begins with the letter b only, we can use the like syntax as well:

spark.sql(
"select name, eyeColor from swimmers where eyeColor like 'b%...