Book Image

Machine Learning with Apache Spark Quick Start Guide

By : Jillur Quddus
Book Image

Machine Learning with Apache Spark Quick Start Guide

By: Jillur Quddus

Overview of this book

Every person and every organization in the world manages data, whether they realize it or not. Data is used to describe the world around us and can be used for almost any purpose, from analyzing consumer habits to fighting disease and serious organized crime. Ultimately, we manage data in order to derive value from it, and many organizations around the world have traditionally invested in technology to help process their data faster and more efficiently. But we now live in an interconnected world driven by mass data creation and consumption where data is no longer rows and columns restricted to a spreadsheet, but an organic and evolving asset in its own right. With this realization comes major challenges for organizations: how do we manage the sheer size of data being created every second (think not only spreadsheets and databases, but also social media posts, images, videos, music, blogs and so on)? And once we can manage all of this data, how do we derive real value from it? The focus of Machine Learning with Apache Spark is to help us answer these questions in a hands-on manner. We introduce the latest scalable technologies to help us manage and process big data. We then introduce advanced analytical algorithms applied to real-world use cases in order to uncover patterns, derive actionable insights, and learn from this big data.

Related Content you might be interested in

Current Title:

Small book image
Machine Learning with Apache Spark Quick Start Guide
Jillur Quddus
Dec, 2018 | 240 pages
Small book image
Essential PySpark for Scalable Data Analytics
Sreeram Nudurupati
Oct, 2021 | 322 pages
Small book image
Apache Spark Quick Start Guide
Shrey Mehrotra | Akash Grade
Jan, 2019 | 154 pages
Small book image
Apache Kafka Quick Start Guide
Raúl Estrada
Dec, 2018 | 186 pages
Table of Contents (10 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
The Big Data Ecosystem
The Big Data Ecosystem
A brief history of data
Big data ecosystem
Summary
2
Setting Up a Local Development Environment
Setting Up a Local Development Environment
CentOS Linux 7 virtual machine
Summary
3
Artificial Intelligence and Machine Learning
Artificial Intelligence and Machine Learning
Artificial intelligence
Machine learning
Deep learning
NLP
Cognitive computing
Machine learning pipelines in Apache Spark
Summary
4
Supervised Learning Using Apache Spark
Supervised Learning Using Apache Spark
Linear regression
Logistic regression
Classification and Regression Trees
Summary
5
Unsupervised Learning Using Apache Spark
Unsupervised Learning Using Apache Spark
Clustering
Principal component analysis
Summary
6
Natural Language Processing Using Apache Spark
Natural Language Processing Using Apache Spark
Feature transformers
Feature extractors
Case study – sentiment analysis
Summary
7
Deep Learning Using Apache Spark
Deep Learning Using Apache Spark
Artificial neural networks
Summary
8
Real-Time Machine Learning Using Apache Spark
Real-Time Machine Learning Using Apache Spark
Distributed streaming platform
Distributed stream processing engines
Stream processing pipeline
Summary
9
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Setting Up a Local Development Environment

In this chapter, we will install, configure, and deploy a local analytical development environment by provisioning a self-contained single-node cluster that will allow us to do the following:

  • Prototype and develop machine learning models and pipelines in Python
  • Demonstrate the functionality and usage of Apache Spark's machine learning library, MLlib, via the Spark Python API (PySpark)
  • Develop and test machine learning models on a single-node cluster using small sample datasets, and thereafter scale up to multi-node clusters processing much larger datasets with little or no code changes required

Our single-node cluster will host the following technologies:

Previous Section
End of Section 1
Next Section