Book Image

Scala Data Analysis Cookbook

By : Arun Manivannan
Book Image

Scala Data Analysis Cookbook

By: Arun Manivannan

Overview of this book

This book will introduce you to the most popular Scala tools, libraries, and frameworks through practical recipes around loading, manipulating, and preparing your data. It will also help you explore and make sense of your data using stunning and insightfulvisualizations, and machine learning toolkits. Starting with introductory recipes on utilizing the Breeze and Spark libraries, get to grips withhow to import data from a host of possible sources and how to pre-process numerical, string, and date data. Next, you’ll get an understanding of concepts that will help you visualize data using the Apache Zeppelin and Bokeh bindings in Scala, enabling exploratory data analysis. iscover how to program quintessential machine learning algorithms using Spark ML library. Work through steps to scale your machine learning models and deploy them into a standalone cluster, EC2, YARN, and Mesos. Finally dip into the powerful options presented by Spark Streaming, and machine learning for streaming data, as well as utilizing Spark GraphX.
Table of Contents (14 chapters)
Scala Data Analysis Cookbook
About the Author
About the Reviewers

Reading and writing CSV files

Reading and writing a CSV file in Breeze is really a breeze. We just have two functions in breeze.linalg package to play with. They are very intuitively named csvread and csvwrite.

In this recipe, you'll see how to:

  1. Read a CSV file into a matrix

  2. Save selected columns of a matrix into a new matrix

  3. Write the newly created matrix into a CSV file

  4. Extract a vector out of the matrix

  5. Write the vector into a CSV

How it works...

There are just two functions that we need to remember in order to read and write data from and to CSV files. The signatures of the functions are pretty straightforward too:

csvread(file, separator, quote, escape, skipLines)
csvwrite(file, mat, separator, quote, escape, skipLines)

Let's look at the parameters by order of importance:

  • file: Represents the file location.

  • separator: Defaults to a comma so as to represent a CSV. Could be overridden when needed.

  • skipLines: This is the number of lines to be skipped while reading the file. Generally, if there is a header, we pass a skipLines=1.

  • mat: While writing, this is the matrix object that is being written.

  • quote: This defaults to double quotes. It is a character that implies that the value inside is one single value.

  • escape: This defaults to a backspace. It is a character used to escape special characters.

Let's see these in action. For the sake of clarity, I have skipped the quote and the escape parameter while calling the csvread and csvwrite functions. For this recipe, we will do three things:

  • Read a CSV file as a matrix

  • Extract a sub-matrix out of the read matrix

  • Write the matrix

Read the CSV as a matrix:

  1. Let's use the csvread function to read a CSV file into a 100*3 matrix. We'll also skip the header while reading and print 5 rows as a sample:

    val usageMatrix=csvread(file=new File("WWWusage.csv"), separator=',', skipLines=1)
    //print first five rows
    println ("Usage matrix \n"+ usageMatrix(0 to 5,::))
    Output :
    1.0  1.0  88.0
    2.0  2.0  84.0
    3.0  3.0  85.0
    4.0  4.0  85.0
    5.0  5.0  84.0
    6.0  6.0  85.0
  2. Extract a sub-matrix out of the read matrix:

    For the sake of generating a submatrix let's skip the first column and save the second and the third column into a new matrix. Let's call it firstColumnSkipped:

    val firstColumnSkipped= usageMatrix(::, 1 to usageMatrix.cols-1)
    //Sample some data so as to ensure we are fine
    println ("First Column skipped \n"+ firstColumnSkipped(0 to 5, ::))
    Output :
    1.0  88.0
    2.0  84.0
    3.0  85.0
    4.0  85.0
    5.0  84.0
    6.0  85.0
  3. Write the matrix:

    As a final step, let's write the firstColumnSkipped matrix to a new CSV file named firstColumnSkipped.csv:

    //Write this modified matrix to a file
    csvwrite(file=new File ("firstColumnSkipped.csv"), mat=firstColumnSkipped, separator=',')