Book Image

Distributed Data Systems with Azure Databricks

By : Alan Bernardo Palacio
Book Image

Distributed Data Systems with Azure Databricks

By: Alan Bernardo Palacio

Overview of this book

Microsoft Azure Databricks helps you to harness the power of distributed computing and apply it to create robust data pipelines, along with training and deploying machine learning and deep learning models. Databricks' advanced features enable developers to process, transform, and explore data. Distributed Data Systems with Azure Databricks will help you to put your knowledge of Databricks to work to create big data pipelines. The book provides a hands-on approach to implementing Azure Databricks and its associated methodologies that will make you productive in no time. Complete with detailed explanations of essential concepts, practical examples, and self-assessment questions, you’ll begin with a quick introduction to Databricks core functionalities, before performing distributed model training and inference using TensorFlow and Spark MLlib. As you advance, you’ll explore MLflow Model Serving on Azure Databricks and implement distributed training pipelines using HorovodRunner in Databricks. Finally, you’ll discover how to transform, use, and obtain insights from massive amounts of data to train predictive models and create entire fully working data pipelines. By the end of this MS Azure book, you’ll have gained a solid understanding of how to work with Databricks to create and manage an entire big data pipeline.
Table of Contents (17 chapters)
1
Section 1: Introducing Databricks
4
Section 2: Data Pipelines with Databricks
9
Section 3: Machine and Deep Learning with Databricks

Using the Spark TensorFlow Distributor package

One of the most commonly used frameworks in deep learning is the TensorFlow library, which also supports distributed training on both CPU and GPU clusters. We can use it to train deep learning models in Azure Databricks by using Spark TensorFlow Distributor, which is a library that aims to ease the process of training TensorFlow models with complex architecture and lots of trainable parameters in distributed computing systems with large amounts of data.

Spark was limited to distributed training because of the standard execution mode, which is the Map/Reduce mode. In this mode, jobs are executed independently in each worker without any communication between them. In Spark 3.0, there is a new execution mode named barrier execution that allows us to easily train deep learning models in a distributed way by allowing communication between workers during the execution.

Spark TensorFlow Distributor is a TensorFlow-native package that makes...