Book Image

Learning PySpark

By : Tomasz Drabas, Denny Lee
Book Image

Learning PySpark

By: Tomasz Drabas, Denny Lee

Overview of this book

Apache Spark is an open source framework for efficient cluster computing with a strong interface for data parallelism and fault tolerance. This book will show you how to leverage the power of Python and put it to use in the Spark ecosystem. You will start by getting a firm understanding of the Spark 2.0 architecture and how to set up a Python environment for Spark. You will get familiar with the modules available in PySpark. You will learn how to abstract data with RDDs and DataFrames and understand the streaming capabilities of PySpark. Also, you will get a thorough overview of machine learning capabilities of PySpark using ML and MLlib, graph processing using GraphFrames, and polyglot persistence using Blaze. Finally, you will learn how to deploy your applications to the cloud using the spark-submit command. By the end of this book, you will have established a firm understanding of the Spark Python API and how it can be used to build data-intensive applications.
Table of Contents (20 chapters)
Learning PySpark
Credits
Foreword
About the Authors
About the Reviewer
www.PacktPub.com
Customer Feedback
Preface
Index

Summary


In this chapter, we walked you through the steps on how to submit applications written in Python to Spark from the command line. The selection of the spark-submit parameters has been discussed. We also showed you how you can package your Python code and submit it alongside your PySpark script. Furthermore, we showed you how you can track the execution of your job.

In addition, we also provided a quick overview of how to run Databricks notebooks using the Databricks Jobs feature. This feature simplifies the transition from development to production, allowing you to take your notebook and execute it as an end-to-end workflow.

This brings us to the end of this book. We hope you enjoyed the journey, and that the material contained herein will help you start working with Spark using Python. Good luck!