Book Image

Scala Data Analysis Cookbook

By : Arun Manivannan
Book Image

Scala Data Analysis Cookbook

By: Arun Manivannan

Overview of this book

This book will introduce you to the most popular Scala tools, libraries, and frameworks through practical recipes around loading, manipulating, and preparing your data. It will also help you explore and make sense of your data using stunning and insightfulvisualizations, and machine learning toolkits. Starting with introductory recipes on utilizing the Breeze and Spark libraries, get to grips withhow to import data from a host of possible sources and how to pre-process numerical, string, and date data. Next, you’ll get an understanding of concepts that will help you visualize data using the Apache Zeppelin and Bokeh bindings in Scala, enabling exploratory data analysis. iscover how to program quintessential machine learning algorithms using Spark ML library. Work through steps to scale your machine learning models and deploy them into a standalone cluster, EC2, YARN, and Mesos. Finally dip into the powerful options presented by Spark Streaming, and machine learning for streaming data, as well as utilizing Spark GraphX.
Table of Contents (14 chapters)
Scala Data Analysis Cookbook
Credits
About the Author
About the Reviewers
www.PacktPub.com
Preface
Index

Vectors and matrices with randomly distributed values


The breeze.stats.distributions package supplements the random number generator that is built into Scala. Scala's default generator just provides the ability to get the random values one by one using the "next" methods. Random number generators in Breeze provide the ability to build vectors and matrices out of these generators. In this recipe, we'll briefly see three of the most common distributions of random numbers.

In this recipe, we will cover at the following sub-recipes:

  • Creating vectors with uniformly distributed random values

  • Creating vectors with normally distributed random values

  • Creating vectors with random values that have a Poisson distribution

  • Creating a matrix with uniformly random values

  • Creating a matrix with normally distributed random values

  • Creating a matrix with random values that has a Poisson distribution

How it works...

Before we delve into how to create the vectors and matrices out of random numbers, let's create instances of the most common random number distribution. All these generators are under the breeze.stats.distributions package:

//Uniform distribution with low being 0 and high being 10
val uniformDist=Uniform(0,10)

//Gaussian distribution with mean being 5 and Standard deviation being 1
val gaussianDist=Gaussian(5,1)

//Poission distribution with mean being 5
val poissonDist=Poisson(5)

We could actually directly sample from these generators. Given any distribution we created previously, we could sample either a single value or a sequence of values:

//Samples a single value
println (uniformDist.sample())
//eg. 9.151191360491392

//Returns a sample vector of size that is passed in as parameter
println (uniformDist.sample(2))
//eg. Vector(6.001980062275654, 6.210874664967401)

Creating vectors with uniformly distributed random values

With no generator parameter, the DenseVector.rand method accepts a parameter for the length of the vector to be returned. The result is a vector (of length 10) with uniformly distributed values between 0 and 1:

val uniformWithoutSize=DenseVector.rand(10)
println ("uniformWithoutSize \n"+ uniformWithoutSize)
//DenseVector(0.1235038023750481, 0.3120595941786264, 0.3575638744660876, 0.5640844223813524, 0.5336149399548831, 0.1338053814330793, 0.9099684427908603, 0.38690724148973166, 0.22561993631651522, 0.45120359622713657)

The DenseVector.rand method optionally accepts a distribution object and generates random values using that input distribution. The following line generates a vector of 10 uniformly distributed random values that are within the range 0 and 10:

val uniformDist=Uniform(0,10)

val uniformVectInRange=DenseVector.rand(10, uniformDist)
println ("uniformVectInRange \n"+uniformVectInRange)
//DenseVector(1.5545833905907314, 6.172564377264846, 8.45578509265587, 7.683763574965107, 8.018688137742062, 4.5876187984930406, 3.274758584944064, 2.3873947264259954, 2.139988841403757, 8.314112884416943)

Creating vectors with normally distributed random values

In the place of the uniformDist generator, we could also pass the previously created Gaussian generator, which is configured to yield a distribution that has a mean of 5 and standard deviation of 1:

val gaussianVector=DenseVector.rand(10, gaussianDist)
println ("gaussianVector \n"+gaussianVector)
//DenseVector(4.235655596913547, 5.535011377545014, 6.201428236839494, 6.046289604188366, 4.319709374229152,
4.2379652913447154, 2.957868021601233, 3.96371080427211, 4.351274306757224, 5.445022658876723)

Creating vectors with random values that have a Poisson distribution

Similarly, by passing the previously created Poisson random number generator, a vector of values that has a mean of 5 could be generated:

val poissonVector=DenseVector.rand(10, poissonDist)
println ("poissonVector \n"+poissonVector)
//DenseVector(5, 5, 7, 11, 7, 6, 6, 6, 6, 6)

We saw how easy it is to create a vector of random values. Now, let's proceed to create a matrix of random values. Similar to DenseVector.rand to generate vectors with random values, we'll use the DenseMatrix.rand function to generate a matrix of random values.

Creating a matrix with uniformly random values

The DenseMatrix.rand defaults to the uniform distribution and generates a matrix of random values given the row and the column parameter. However, if we would like to have a distribution within a range, then as in vectors, we could use the optional parameter:.

//Uniform distribution, Creates a 3 * 3 Matrix with random values from 0 to 1
val uniformMat=DenseMatrix.rand(3, 3)
println ("uniformMat \n"+uniformMat)

0.4492155777289115  0.9098840386699856    0.8203022252988292
0.0888975848853315  0.009677790736892788  0.6058885905934237
0.6201415814136939  0.7017492438727635    0.08404147915159443

//Creates a 3 * 3 Matrix with uniformly distributed random values with low being 0 and high being 10
val uniformMatrixInRange=DenseMatrix.rand(3,3, uniformDist)
println ("uniformMatrixInRange \n"+uniformMatrixInRange)

7.592014659345548  8.164652560340933    6.966445294464401
8.35949395084735   3.442654641743763    3.6761640240938442
9.42626645215854   0.23658921372298636  7.327120138868571

Creating a matrix with normally distributed random values

Just as in vectors, in place of the uniformDist generator, we could also pass the previously created Gaussian generator to the rand function to generate a matrix of random values that has a mean of 5 and standard deviation of 1:

//Creates a 3 * 3 Matrix with normally distributed random values with mean being 5 and Standard deviation being 1
val gaussianMatrix=DenseMatrix.rand(3, 3,gaussianDist)
println ("gaussianMatrix \n"+gaussianMatrix)

5.724540885605018   5.647051873430568  5.337906135107098
6.2228893721489875  4.799561665187845  5.12469779489833
5.136960834730864   5.176410360757703  5.262707072950913

Creating a matrix with random values that has a Poisson distribution

Similarly, by passing the previously created Poisson random number generator, a matrix of random values that has a mean of 5 could be generated:

//Creates a 3 * 3 Matrix with Poisson distribution with mean being 5
val poissonMatrix=DenseMatrix.rand(3, 3,poissonDist)
println ("poissonMatrix \n"+poissonMatrix)
4  11  3
6  6   5
6  4   2