6. Machine Learning | Spark for Data Science

Sign In Start Free Trial

Book Overview & Buying
Table Of Contents

Spark for Data Science

By : Duvvuri, Singhal

Spark for Data Science

By: Duvvuri, Singhal

Overview of this book

This is the era of Big Data. The words ‘Big Data’ implies big innovation and enables a competitive advantage for businesses. Apache Spark was designed to perform Big Data analytics at scale, and so Spark is equipped with the necessary algorithms and supports multiple programming languages. Whether you are a technologist, a data scientist, or a beginner to Big Data analytics, this book will provide you with all the skills necessary to perform statistical data analysis, data visualization, predictive modeling, and build scalable data products or solutions using Python, Scala, and R. With ample case studies and real-world examples, Spark for Data Science will help you ensure the successful execution of your data science projects.

Preface

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Free Chapter

1. Big Data and Data Science – An Introduction

1. Big Data and Data Science – An Introduction

Big data overview

Challenges with big data analytics

Evolution of big data analytics

Spark for data analytics

The Spark stack

Summary

References

2. The Spark Programming Model

2. The Spark Programming Model

The programming paradigm

The Spark engine

The RDD API

RDD operations

Summary

References

3. Introduction to DataFrames

3. Introduction to DataFrames

Why DataFrames?

Spark SQL

The DataFrame API

Creating DataFrames

DataFrame operations

Summary

References

4. Unified Data Access

4. Unified Data Access

Data abstractions in Apache Spark

Datasets

Spark SQL

Structured Streaming

Continuous applications

Summary

References

5. Data Analysis on Spark

5. Data Analysis on Spark

Data analytics life cycle

Data acquisition

Data preparation

Basics of statistics

Descriptive statistics

Inferential statistics

Summary

References

6. Machine Learning

6. Machine Learning

Introduction

MLlib and the Pipeline API

Introduction to machine learning

Regression methods

Classification methods

Linear Support Vector Machines (SVM)

Training an SVM

Decision trees

Ensembles

Multilayer perceptron classifier

Clustering techniques

Summary

References

7. Extending Spark with SparkR

7. Extending Spark with SparkR

SparkR basics

Advantages and limitations

Programming with SparkR

SparkR DataFrames

Machine learning

Summary

References

8. Analyzing Unstructured Data

8. Analyzing Unstructured Data

Sources of unstructured data

Processing unstructured data

Text classification

Text clustering

Dimensionality reduction

Singular Value Decomposition

Summary

References:

9. Visualizing Big Data

9. Visualizing Big Data

Why visualize data?

Data visualization tools

Data visualization techniques

Summary

References

10. Putting It All Together

10. Putting It All Together

A quick recap

Introducing a case study

The business problem

Data acquisition and data cleansing

Developing the hypothesis

Data exploration

Data preparation

Model building

Data visualization

Communicating the results to business users

Summary

References

11. Building Data Science Applications

11. Building Data Science Applications

Scope of development

The Scala advantage

Spark development status

The big data trends

Summary

References

Regression methods

Regression methods are a type of supervised learning. If the response variable is quantitative/continuous (takes on numeric values such as age, salary, height, and so on), then the problem can be called a regression problem regardless of the explanatory variables' type. There are various kinds of modeling techniques to address the regression problems. In this section, our focus will be on linear regression techniques and some different variations of it.

Regression methods can be used to predict any real valued outcomes. Following are a few examples:

Predict the salary of an employee based on his educational level, location, type of job, and so on
Predict stock prices
Predict buying potential of a customer
Predict the time a machine would take before failing

Linear regression

Further to what we discussed in the previous section Parametric methods, after the assumption of linearity is made for

(X), we need the training data to fit a model that would describe the relation between...

CONTINUE READING

83

Tech Concepts

36

Programming languages

73

Tech Tools

Unlimited access to the largest independent learning library in tech of over 8,000 expert-authored tech books and videos.

Innovative learning tools, including AI book assistants, code context explainers, and text-to-speech.

50+ new titles added per month and exclusive early access to books as they are being written.

Spark for Data Science

Search

Your notes and bookmarks