Deep Learning on Apache Spark with DeepLearning4j and H2O

Book Overview & Buying
Table Of Contents

Mastering Apache Spark 2.x - Second Edition

By : Romeo Kienzler

4.5 (2)

Buy this Book

Mastering Apache Spark 2.x

4.5 (2)

By: Romeo Kienzler

Buy this Book

Overview of this book

Apache Spark is an in-memory, cluster-based Big Data processing system that provides a wide range of functionalities such as graph processing, machine learning, stream processing, and more. This book will take your knowledge of Apache Spark to the next level by teaching you how to expand Spark’s functionality and build your data flows and machine/deep learning programs on top of the platform. The book starts with a quick overview of the Apache Spark ecosystem, and introduces you to the new features and capabilities in Apache Spark 2.x. You will then work with the different modules in Apache Spark such as interactive querying with Spark SQL, using DataFrames and DataSets effectively, streaming analytics with Spark Streaming, and performing machine learning and deep learning on Spark using MLlib and external tools such as H20 and Deeplearning4j. The book also contains chapters on efficient graph processing, memory management and using Apache Spark on the cloud. By the end of this book, you will have all the necessary information to master Apache Spark, and use it efficiently for Big Data processing and analytics.

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

A First Taste and What’s New in Apache Spark V2

Spark machine learning

Spark Streaming

Spark SQL

Spark graph processing

Extended ecosystem

What's new in Apache Spark V2?

Cluster design

Cluster management

Cloud-based deployments

Performance

Cloud

Summary

Apache Spark SQL

The SparkSession--your gateway to structured data processing

Importing and saving data

Understanding the DataSource API

DataFrames

Using SQL

Using Datasets

User-defined functions

RDDs versus DataFrames versus Datasets

Summary

The Catalyst Optimizer

Understanding the workings of the Catalyst Optimizer

Managing temporary views with the catalog API

The SQL abstract syntax tree

How to go from Unresolved Logical Execution Plan to Resolved Logical Execution Plan

Code generation

Summary

Project Tungsten

Memory management beyond the Java Virtual Machine Garbage Collector

Cache-friendly layout of data in memory

Code generation

Summary

Apache Spark Streaming

Overview

Errors and recovery

Streaming sources

Summary

Structured Streaming

The concept of continuous applications

Windowing

Increased performance with good old friends

How transparent fault tolerance and exactly-once delivery guarantee is achieved

Example - connection to a MQTT message broker

Summary

Apache Spark MLlib

Architecture

Classification with Naive Bayes

Clustering with K-Means

Artificial neural networks

Summary

Apache SparkML

What does the new API look like?

The concept of pipelines

Model evaluation

CrossValidation and hyperparameter tuning

Winning a Kaggle competition with Apache SparkML

Summary

Apache SystemML

Why do we need just another library?

A cost-based optimizer for machine learning algorithms

Performance measurements

Apache SystemML in action

Summary

Deep Learning on Apache Spark with DeepLearning4j and H2O

H2O

Deeplearning4j

Summary

Apache Spark GraphX

Overview

Graph analytics/processing with GraphX

Summary

Apache Spark GraphFrames

Architecture

Examples

Summary

Apache Spark with Jupyter Notebooks on IBM DataScience Experience

Why notebooks are the new standard

Learning by example

Summary

Apache Spark on Kubernetes

Bare metal, virtual machines, and containers

Understanding the core concepts of Docker

Understanding Kubernetes

Using Kubernetes for provisioning containerized Spark applications

Example--Apache Spark on Kubernetes

Summary

Our continuing theme when examining both Apache Hadoop and Spark is that none of these systems stand alone. They need to be integrated to form ETL-based processing systems. Data needs to be sourced and processed in Spark and then passed to the next link in the ETL chain or stored. We hope that this chapter showed you that Spark functionality can be extended with extra libraries and systems such as H2O, DeepLearning4j. Even Apache SystemML supports DeepLearning now and TensorFlow can be run within Apache Spark using TensorFrames and TensorSpark.

Although Apache Spark MLlib and SparkML has a lot of functionality, the combination of H2O Sparkling Water and the Flow web interface provides an extra wealth of data analysis modeling options. Using Flow, you can also visually and interactively process your data. We hope that this chapter shows you, even though it cannot cover...

Tech Concepts

Programming languages

Tech Tools