Book Image

R Data Analysis Cookbook - Second Edition

By : Kuntal Ganguly, Shanthi Viswanathan, Viswa Viswanathan
Book Image

R Data Analysis Cookbook - Second Edition

By: Kuntal Ganguly, Shanthi Viswanathan, Viswa Viswanathan

Overview of this book

Data analytics with R has emerged as a very important focus for organizations of all kinds. R enables even those with only an intuitive grasp of the underlying concepts, without a deep mathematical background, to unleash powerful and detailed examinations of their data. This book will show you how you can put your data analysis skills in R to practical use, with recipes catering to the basic as well as advanced data analysis tasks. Right from acquiring your data and preparing it for analysis to the more complex data analysis techniques, the book will show you how you can implement each technique in the best possible manner. You will also visualize your data using the popular R packages like ggplot2 and gain hidden insights from it. Starting with implementing the basic data analysis concepts like handling your data to creating basic plots, you will master the more advanced data analysis techniques like performing cluster analysis, and generating effective analysis reports and visualizations. Throughout the book, you will get to know the common problems and obstacles you might encounter while implementing each of the data analysis techniques in R, with ways to overcoming them in the easiest possible way. By the end of this book, you will have all the knowledge you need to become an expert in data analysis with R, and put your skills to test in real-world scenarios.
Table of Contents (14 chapters)

Detecting outliers

Outliers in data can distort predictions and affect the accuracy, if you don't detect and handle them appropriately, especially in the data preprocessing stage.

So, identifying the extreme values is important, as it can drastically introduce bias in the analytic pipeline and affect predictions. In this recipe, we will discuss the ways to detect outliers and how to handle them.

Getting ready

Download the files for this chapter and store the ozone.csv file in your R working directory. Read the file using the read.csv() command and save it in a variable:

> ozoneData <- read.csv("ozone.csv", stringsAsFactors=FALSE)

How to do it...

Perform the following steps to detect outliers in the dataset:

  1. Detect outliers in the univariate continuous variable:
>outlier_values <- boxplot.stats(ozoneData$pressure_height)$out

>boxplot(ozoneData$pressure_height, main="Pressure Height", boxwex=0.1)

>mtext(paste("Outliers: ", paste(outlier_values, collapse=", ")), cex=0.6)

The output would be the following screenshot:

  1. Detect outliers in bivariate categorical variables:
> boxplot(ozone_reading ~ Month, data=ozoneData, main="Ozone reading across months") 

The output would be the following screenshot:

How it works...

The most commonly used method to detect outliers is visualization of the data, through boxplot, histogram, or scatterplot.

The boxplot.stats()$out function fetches the values of data points that lie beyond the extremes of the whiskers. The boxwex attribute is a scale factor that is applied to all the boxes; it improves the appearance of the plot by making the boxes narrower. The mtext() function places a text outside the plot area, but within the plot window.

In the case of continuous variables, outliers are those observations that lie outside 1.5 * IQR, where Inter Quartile Range or IQR, is the difference between the 75th and 25th quartiles. The outliers in continuous variables show up as dots outside the whiskers of the boxplot.

In case of bivariate categorical variables, a clear pattern is noticeable and the change in the level of boxes suggests that Month seems to have an impact in ozone_reading. The outliers in respective categorical levels show up as dots outside the whiskers of the boxplot.

There's more...

Detecting and handling outliers depends mostly on your application. Once you have identified the outliers and you have decided to make amends as per the nature of the problem, you may consider one of the following approaches.

Treating the outliers with mean/median imputation

We can handle outliers with mean or median imputation by replacing the observations lower than the 5th percentile with mean and those higher than 95th percentile with median. We can use the same statistics, mean or median, to impute outliers in both directions:

> impute_outliers <- function(x,removeNA = TRUE){
quantiles <- quantile( x, c(.05, .95 ),na.rm = removeNA )
x[ x < quantiles[1] ] <- mean(x,na.rm = removeNA )
x[ x > quantiles[2] ] <- median(x,na.rm = removeNA )

> imputed_data <- impute_outliers(ozoneData$pressure_height)

Validate the imputed data through visualization:

> par(mfrow = c(1, 2))

> boxplot(ozoneData$pressure_height, main="Pressure Height having Outliers", boxwex=0.3)

> boxplot(imputed_data, main="Pressure Height with imputed data", boxwex=0.3)

The output would be the following screenshot:

Handling extreme values with capping

To handle extreme values that lie outside the 1.5 * IQR(Inter Quartile Range) limits, we could cap them by replacing those observations that lie below the lower limit, with the value of 5th percentile and those that lie above the upper limit, with the value of 95th percentile, as shown in the following code:

> replace_outliers <- function(x, removeNA = TRUE) {
pressure_height <- x
qnt <- quantile(pressure_height, probs=c(.25, .75), na.rm = removeNA)
caps <- quantile(pressure_height, probs=c(.05, .95), na.rm = removeNA)
H <- 1.5 * IQR(pressure_height, na.rm = removeNA)
pressure_height[pressure_height < (qnt[1] - H)] <- caps[1]
pressure_height[pressure_height > (qnt[2] + H)] <- caps[2]

> capped_pressure_height <- replace_outliers(ozoneData$pressure_height)

Validate the capped variable capped_pressure_height through visualization:

> par(mfrow = c(1, 2))

> boxplot(ozoneData$pressure_height, main="Pressure Height with Outliers", boxwex=0.1)

> boxplot(capped_pressure_height, main="Pressure Height without Outliers", boxwex=0.1)

The output would be the following screenshot:

Transforming and binning values

Sometimes, transforming variables can also eliminate outliers. The natural log or square root of a value reduces the variation caused by extreme values. Some predictive analytics algorithms, such as decision trees, inherently deal with outliers by using binning techniques (a form of variable transformation).

Outlier detection with LOF

Local Outlier Factor or LOF is an algorithm implemented in DMwR package for identifying density-based local outliers, by comparing the local density of a point with that of its neighbors.

Now we will calculates the local outlier factors using the LOF algorithm using k number of neighbors:

> install.packages("DMwR")
> library(DMwR)
> outlier.scores <- lofactor(ozoneData, k=3)

Finally we will output the top 5 outlier by sorting the outlier score calculated above:

> outliers <- order(outlier.scores, decreasing=T)[1:5]
> print(outliers)