The logistic regression relies on the sigmoid curve or logistic function is described in the Logistic function section in Chapter 6, Regression and Regularization. The logistic functions are used to segregate training data into classes. The output value of the logistic function ranges from 0 for x = - INFINITY to 1 for x = + INFINITY. Therefore, it makes sense to normalize the input data or observation over [0, 1].

The normalization is a linear transformation of the raw data that can be generalized to any range [l, h].

Note

Linear normalization

M2: [0, 1] Normalization of features {x_i} with minimum x_min and maximum x_max values:

M3: [l, h] Normalization of features {xi}:

The normalization of input data in supervised learning has a specific requirement: the classification and prediction of new observations have to use the normalization parameters (min and max) extracted from the training set, so all the observations share the same scaling factor.

Let's define the MinMax normalization class. The class is immutable: the minimum, min, and maximum, max, values are computed within the constructor. The class takes a time series of a parameterized T type and values as arguments (line 8). The steps of the normalization process are defined as follows:

The class constructor computes the tuple of minimum and maximum values, minMax, using a fold (line 9). The scaleFactors scaling parameters are computed during the normalization of the time series (line 11), which are described as follows. The normalize method initializes the scaling factor parameters (line 12) before normalizing the input data (line 13):

def normalize(low: Double, high: Double): DblVector = 
  setScaleFactors(low, high).map( scale => { //12
    values.map(x =>(x - min)*scale.ratio + scale.low) //13
  }).getOrElse(/* … */)

def setScaleFactors(l: Double, h: Double): Option[ScaleFactors]={
    // .. error handling code
   Some(ScaleFactors(l, h, (h - l)/(max - min))
}

Subsequent observations use the same scaling factors extracted from the input time series in normalize (line 14):

def normalize(value: Double):Double = setScaleFactors.map(scale => 
   if(value < min) scale.low
   else if (value > max) scale.high
   else (value - min)* scale.high + scale.low
).getOrElse( /* … */)

The MinMax class normalizes single variable observations.

The test case with the binomial logistic regression uses a multiple variable normalization, implemented by the MinMaxVector class, which takes observations of the XVSeries[Double] type as inputs:

class MinMaxVector(series: XVSeries[Double]) {
  val minMaxVector: Vector[MinMax[Double]] = //15
      series.transpose.map(new MinMax[Double](_))
  def normalize(low: Double, high: Double): XVSeries[Double]
}

The constructor of the MinMaxVector class transposes the vector of array of observations in order to compute the minimum and maximum value for each dimension (line 15).

At a closer look, a sudden change in price and increase in volume occurs about every three months or so. Experienced investors will undoubtedly recognize that these price-volume patterns are related to the release of quarterly earnings of Cisco. Such a regular but unpredictable pattern can be a source of concern or opportunity if risk can be properly managed. The strong reaction of the stock price to the release of corporate earnings may scare some long-term investors while enticing day traders.

The following graph visualizes the potential correlation between sudden price change (volatility) and heavy trading volume:

Price-volume correlation for the Cisco stock 2012-2013

The next section is not required for the understanding of the test case. It illustrates the capabilities of JFreeChart as a simple visualization and plotting library.

Although charting is not the primary goal of this book, we thought that you will benefit from a brief introduction to JFreeChart.

The dataset (volatility and volume) is converted into internal JFreeChart data structures. The ScatterPlot class implements a simple configurable scatter plot with the following arguments:

The code will be as follows:

class ScatterPlot(config: PlotInfo, theme: PlotTheme) { //16
  def display(xy: Vector[DblPair], width: Int, height) //17
  def display(xt: XVSeries[Double], width: Int, height)
  // ….
}

The PlotTheme class defines a specific theme or preconfiguration of the chart (line 16). The class offers a set of display methods to accommodate a wide range of data structures and configuration (line 17).

The ScatterPlot.display method is used to display the normalized input data used in the binomial logistic regression as follows:

val plot = new ScatterPlot(("CSCO 2012-2013", 
   "Session High - Low", "Session Volume"), new BlackPlotTheme)
plot.display(volatility_vol, 250, 340)

A scatter plot of volatility and volume for the Cisco stock 2012-2013

The scatter plot shows a level of correlation between session volume and session volatility and confirms the initial finding in the stock price and volume chart. We can leverage this information to classify trading sessions by their volatility and volume. The next step is to create a two class model by loading a training set, observations, and expected values, into our logistic regression algorithm. The classes are delimited by a decision boundary (also known as a hyperplane) drawn on the scatter plot.

Visualizing labels—the normalized variation of the stock price between the opening and closing of the trading session is selected as the label for this classifier.

The goal of the training of a model using expected values is to compute the optimal weights that minimizes the error or cost function. We select the batch gradient descent algorithm to minimize the cumulative error between the predicted and expected values for all the observations. Although there are quite a few alternative optimizers, the gradient descent is quite robust and simple enough for this first chapter. The algorithm consists of updating the weights w_i of the regression model by minimizing the cost.

Note

Cost function

M5: Cost (or compound error = predicted – expected):

M6: The batch gradient descent method to update model weights w_i is as follows:

For those interested in learning about of optimization techniques, the Summary of optimization techniques section in the Appendix A, Basic Concepts presents an overview of the most commonly used optimizers. The batch descent gradient method is also used for the training of the multilayer perceptron (refer to The training epoch section under The multilayer perceptron section in Chapter 9, Artificial Neural Networks).

The execution of the batch gradient descent algorithm follows these steps:

The purpose of shuffling the order of the observations between iterations is to avoid the minimization of the cost reaching a local minimum.

The train method consists of iterating through the computation of the weight using a simple descent gradient method. The method computes weights and returns an instance of the LogBinRegressionModel model:

def train: LogBinRegressionModel = {
  val nWeights = obsSet.head.length + 1  //21
  val init = Array.fill(nWeights)(Random.nextDouble )  //22
  val weights = gradientDescent(obsSet.zip(expected),0.0,0,init)
  new LogBinRegressionModel(weights)   //23
}

The train method extracts the number of weights, nWeights, for the regression model as the number of variables in each observation + 1 (line 21). The method initializes weights with random values over [0, 1] (line 22). The weights are computed through the tail recursive gradientDescent method, and the method returns a new model for the binomial logistic regression (line 23).

Let's take a look at the computation for weights through the minimization of the cost function in the gradientDescent method:

type LabelObs = Vector[(DblArray, Int)]

@tailrec
def gradientDescent(
      obsAndLbl: LabelObs, 
      cost: Double, 
      nIters: Int, 
      weights: DblArray): DblArray = {  //24
  
  if(nIters >= maxIters) 
       throw new IllegalStateException("..")//25
  val shuffled = shuffle(obsAndLbl)   //26
  val errorGrad = shuffled.map{ case(x, y) => {  //27
      val error = sigmoid(dot(x, weights)) - y
      (error, x.map( _ * error))  //28
   }}.unzip

   val scale = 0.5/obsAndLbl.size
   val newCost = errorGrad._1   //29
.aggregate(0.0)((s,c) =>s + c*c, _ + _ )*scale
   val relativeError = cost/newCost - 1.0
   
   if( Math.abs(relativeError) < eps)  weights  //30
   else {
     val derivatives = Vector[Double](1.0) ++ 
                 errorGrad._2.transpose.map(_.sum) //31
     val newWeights = weights.zip(derivatives)
                       .map{ case (w, df) => w - eta*df)  //32
     newWeights.copyToArray(weights)
     gradientDescent(shuffled, newCost, nIters+1, newWeights)//33
   }
}

The gradientDescent method recurses on the vector of pairs (observations and expected values), obsAndLbl, cost, and the model weights (line 24). It throws an exception if the maximum number of iterations allowed for the optimization is reached (line 25). It shuffles the order of the observations (line 26) before computing the errorGrad derivatives of the cost over each weights (line 27). The computation of the derivative of the cost (or error = predicted value – expected value) in formula M5 returns a pair of cumulative cost and derivative values using the formula (line 28).

Next, the method computes the overall compound cost using the formula M4 (line 29), converts it to a relative incremental relativeError cost that is compared to the eps convergence criteria (line 30). The method extracts derivatives of cost over weights by transposing the matrix of errors, and then prepends the bias 1.0 value to match the array of weights (line 31).

The formula M6 updates the weights for the next iteration (line 32) before invoking the method with new weights, cost, and iteration count (line 33).

Let's take a look at the shuffling of the order of observations using a random sequence generator. The following implementation is an alternative to the Scala standard library method scala.util.Random.shuffle for shuffling elements of collections. The purpose is to change the order of observations and labels between iterations in order to prevent the optimizer to reach a local minimum. The shuffle method permutes the order in the labelObs vector of observations by partitioning it into segments of random size and reversing the order of the other segment:

val SPAN = 5
def shuffle(labelObs: LabelObs): LabelObs = { 
  shuffle(new ArrayBuffer[Int],0,0).map(labelObs( _ )) //34
}

Once the order of the observations is updated, the vector of pair (observations, labels) is easily built through a map (line 34). The actual shuffling of the index is performed in the following shuffle recursive function:

val maxChunkSize = Random.nextInt(SPAN)+2  //35

@tailrec
def shuffle(indices: ArrayBuffer[Int], count: Int, start: Int): 
      Array[Int] = {
  val end = start + Random.nextInt(maxChunkSize) //36
  val isOdd = ((count & 0x01) != 0x01)
  if(end >= sz) 
    indices.toArray ++ slice(isOdd, start, sz) //37
  else 
    shuffle(indices ++ slice(isOdd, start, end), count+1, end)
}

The maximum size of partition of the maxChunkSize vector observations is randomly computed (line 35). The method extracts the next slice (start, end) (line 36). The slice is either added to the existing indices vector and returned once all the observations have been shuffled (line 37) or passed to the next invocation.

The slice method returns an array of indices over the range (start, end) either in the right order if the number of segments processed is odd, or in reverse order if the number of segment processed is even:

def slice(isOdd: Boolean, start: Int, end: Int): Array[Int] = {
  val r = Range(start, end).toArray
  (if(isOdd) r else r.reverse)
}

In order to train the model, we need to label the input data. The labeling process consists of associating the relative price movement during a session (price at close/price at open – 1) with one of the following two configurations:

The two classes of training observations are segregated by a decision boundary drawn on the scatter plot in the previous section. The labeling process is usually quite cumbersome and should be automated as much as possible.

Once the model is successfully created through training, it is available to classify new observation. The runtime classification of observations using the binomial logistic regression is implemented by the classify method:

def classify(obs: DblArray): Try[(Int, Double)] = 
  val linear = dot(obs, model.weights)  //37
  val prediction = sigmoid(linear)
  (if(linear > 0.0) 1 else 0, prediction) //38
})

The method applies the logistic function to the linear inner product, linear, of the new obs and weights observations of the model (line 37). The method returns the tuple (the predicted class of the observation {0, 1}, prediction value) where the class is defined by comparing the prediction to the boundary value 0.0 (line 38).

The computation of the dot product of weights and observations uses the bias value as follows:

def dot(obs: DblArray, weights: DblArray): Double =
   weights.zip(Array[Double](1.0) ++ obs)
          .aggregate(0.0){case (s, (w,x)) => s + w*x, _ + _ }

The alternative implementation of the dot product of weights and observations consists of extracting the first w.head weight:

def dot(x: DblArray, w: DblArray): Double = 
  x.zip(w.drop(1)).map {case (_x,_w) => _x*_w}.sum + w.head

The dot method is used in the classify method.