Book Image

Apache Spark 2.x Machine Learning Cookbook

By : Mohammed Guller, Siamak Amirghodsi, Shuen Mei, Meenakshi Rajendran, Broderick Hall
Book Image

Apache Spark 2.x Machine Learning Cookbook

By: Mohammed Guller, Siamak Amirghodsi, Shuen Mei, Meenakshi Rajendran, Broderick Hall

Overview of this book

Machine learning aims to extract knowledge from data, relying on fundamental concepts in computer science, statistics, probability, and optimization. Learning about algorithms enables a wide range of applications, from everyday tasks such as product recommendations and spam filtering to cutting edge applications such as self-driving cars and personalized medicine. You will gain hands-on experience of applying these principles using Apache Spark, a resilient cluster computing system well suited for large-scale machine learning tasks. This book begins with a quick overview of setting up the necessary IDEs to facilitate the execution of code examples that will be covered in various chapters. It also highlights some key issues developers face while working with machine learning algorithms on the Spark platform. We progress by uncovering the various Spark APIs and the implementation of ML algorithms with developing classification systems, recommendation engines, text analytics, clustering, and learning systems. Toward the final chapters, we’ll focus on building high-end applications and explain various unsupervised methodologies and challenges to tackle when implementing with big data ML systems.
Table of Contents (20 chapters)
Title Page
Credits
About the Authors
About the Reviewer
www.PacktPub.com
Customer Feedback
Preface

Running a sample ML code from Spark


We can verify the setup by simply the sample code from the source tree and importing it into IntelliJ to make sure it runs.

Getting ready

We will first run the logistic regression code from the samples to verify installation. In the next section, we proceed to write our own version of the same program and examine the output in order to understand how it works.

How to do it...

  1. Go to the source directory and pick one of the ML sample code files to run. We've selected the logistic regression example.

Note

If you cannot find the source code in your directory, you can always download the Spark source, unzip, and then extract the examples directory accordingly.

  1. After selecting the example, select Edit Configurations..., as shown in the following screenshot:
  1. In the Configurations tab, define the following options:
    • VM options: The choice shown allows you to run a standalone Spark cluster
    • Program arguments: What we are supposed to pass into the program
  1. Run the logistic regression by going to Run 'LogisticRegressionExample', as shown in the following screenshot:
  1. Verify the exit code and make sure it is as shown in the following screenshot: