Book Image

Spark Cookbook

By : Rishi Yadav
Book Image

Spark Cookbook

By: Rishi Yadav

Overview of this book

Table of Contents (19 chapters)
Spark Cookbook
Credits
About the Author
About the Reviewers
www.PacktPub.com
Preface
Index

Understanding cost function


Cost function or loss function is a very important function in machine learning algorithms. Most algorithms have some form of cost function and the goal is to minimize that. Parameters, which affect cost function, such as stepSize in the last recipe, need to be set by hand. Therefore, understanding the whole concept of cost function is very important.

In this recipe, we are going to analyze cost function for linear regression. Linear regression is a simple algorithm to understand and it will help readers understand the role of cost functions for even complex algorithms.

Let's go back to linear regression. The goal is to find the best-fitting line so that the mean square of error is minimum. Here, we are referring error as the difference between the value as per the best-fitting line and the actual value of the response variable for the training dataset.

For a simple case of single predicate variable, the best-fit line can be written as:

This function is also called...