Logistic regression relies on the sigmoid curve or logistic function described in the Logistic function section in Chapter 9, Regression and Regularization. The logistic function is used to segregate training data into classes. The output value of the logistic function ranges from 0 for x = - INFINTY to 1 for x = + INFINITY. Therefore, it makes sense to normalize the input data or observation over [0, 1].

Normalization is a linear transformation on the raw data that can be generalized to any range [l, h].

Note

Linear normalization

M2: [0, 1] Normalization features {x_i} with minimum xmin, maximum xmax values:

M3: [l, h] Normalization of features {x_i}:

The normalization of input data in supervised learning has a specific requirement: the classification and prediction of new observations have to use the normalization parameters (min, max) extracted from the training set, so all observations share the same scaling factor.

Let's define the normalization class MinMax. The class is immutable: the minimum, min, and maximum, max, values are computed within the constructor. The class takes a time series of the parameterized type T values as an argument (line 8). The steps of the normalization process are defined as follows:

The class constructor computes the tuple of minimum and maximum values minMax using a fold (line 9). The scaling parameters scaleFactors are computed during the normalization of the time series (line 11), described as follows. The method normalize initializes the scaling factors parameters (line 12) before normalizing the input data (line 13):

def normalize(low: Double, high: Double): Vector[Double] = 
  setScaleFactors(low, high).map( scale => { //12
    values.map(x =>{
      val z = implicitly[ToDouble[T]].apply(x)
      (z - min)*scale.ratio + scale.low //13
    }) 
  }).getOrElse(/* … */)

def setScaleFactors(l: Double, h: Double): Option[ScaleFactors]={
    // .. error handling code
   Some(ScaleFactors(l, h, (h - l)/(max - min))
}

Subsequent observations use the same scaling factors extracted from the input time series in normalize (line 14):

def normalize(value: Double): Double = setScaleFactors.map(
scale => 
   if(value < min) scale.low
   else if (value > max) scale.high
   else (value - min)* scale.high + scale.low
).getOrElse( /* … */)

The class MinMax normalizes single variable observations.

The test case with the binomial logistic regression uses a multiple variable normalization, implemented by the class MinMaxVector which takes observations of type Vector[Array[Double]] as input:

class MinMaxVector(series: Vector[Double]) {
  val minMaxVector: Vector[MinMax[Double]] = //15
      series.transpose.map(new MinMax[Double](_))
  def normalize(low: Double, high: Double): Vector[Double]
}

The constructor of the class MinMaxVector transposes the vector of an array of observations in order to compute the minimum and maximum values for each dimension (line 15).

At a closer look, a sudden change in price and increase in volume occurs about every 3 months or so. Experienced investors will undoubtedly recognize that these price-volume patterns are related to the release of quarterly earnings of Cisco. Such a regular but unpredictable pattern can be a source of concern or opportunity if risk can be properly managed. The strong reaction of the stock price to the release of corporate earnings may scare some long-term investors while enticing day traders.

The following graph visualizes the potential correlation between sudden price change (volatility) and heavy trading volume:

Price-volume correlation for Cisco stock 2012-2013

The next section is not required for the understanding of the test case. It illustrates the capabilities of JFreeChart as a simple visualization and plotting library.

Although charting is not the primary goal of this book, we thought that you would benefit from a brief introduction to JFreeChart.

The dataset (volatility, volume) is converted into internal JFreeChart data structures.

The following code snippet defines the key components of a simple scatter plot:

class ScatterPlot(config: PlotInfo, theme: PlotTheme) {//16
  def display(xy: Vector[DblPair], width: Int, height) //17

  // ….
}

The class ScatterPlot implements a simple configurable scatter plot with the following arguments:

The class PlotTheme defines a specific theme or preconfiguration of the chart (line 16). The class offers a set of methods with the name display to accommodate for a wide range of data structures and configuration (line 17).

Visualizing model features

The ScatterPlot.display method is used to display the normalized input data used in the binomial logistic regression, as follows:

val plot = new ScatterPlot(("CSCO 2012-13 Model features", 
   "Normalized session volatility", "Normalized session Volume"), 
    new BlackPlotTheme)
plot.display(volatilityVolume, 250, 340)

The invocation of the method display generates the following output:

Scatter plot of volatility and volume for Cisco stock 2012-2013

The scatter plot shows some level of correlation between session volume and session volatility and confirms the initial finding in the stock price and volume chart. We can leverage this information to classify trading sessions by their volatility and volume. The next step is to create a two-class model by loading a training set, observations, and expected values into our logistic regression algorithm. The classes are delimited by a decision boundary (also known as a hyperplane) drawn onto the scatter plot.

Visualizing label

The normalized variation of the stock price between the opening and closing of the trading session is selected as the label for this classifier:

Classifier training label: normalized variation of stock price within a trading session

The goal of the training of a model using expected values is to compute the optimal weights that minimize the error or loss function.

We select the Stochastic Gradient Descent (SGD) algorithm to minimize the cumulative error between the predicted and expected values for all the observations. Although there are quite a few alternative optimizers, the SGD is quite robust and simple enough for this first chapter. The algorithm consists of updating the weights wi of the regression model by minimizing the cost.

Note

Cost functions

M5: Logistic loss

M6: SGD method to update model weights at iteration t, wt:

For those interested in learning about about optimization techniques, the Summary of optimization techniques section in the Appendix presents an overview of the most commonly used optimizers. The stochastic descent gradient is used for the training of the multilayer perceptron (refer to the The training epoch subsection in the The multilayer perceptron (MLP) section of Chapter 10, Multilayer Perceptron for more detail).

The execution of the SGD algorithm follows these steps:

The purpose of shuffling the order of the observations between iterations is to avoid the minimization of the cost reaching a local minimum.

The training method, train, consists of iterating through the computation of the weight using a simple descent gradient method. The method train computes the weights, collects the logistic loss, losses, at each iteration and returns an instance of the model LogBinRegressionModel. The code is represented here:

def train: LogBinRegressionModel = {
   val init = Array.fill(nWeights)(Random.nextDouble) //22
   val (weights, losses) = sgd(
      0,init, List[Double]()
   )
   new LogBinRegressionModel(weights, losses.reverse)  //23
}

The method train extracts the number of weights, nWeights, for the regression model as the number of variables in each observation + 1 (line 21). The method initializes the weights with random values over [0, 1] (line 22). The weights are computed through the tail recursive method sgd and the method returns a new model for the binomial logistic regression (line 23).

Let's look at the computation for the weights through the minimization of the loss function in the sgd method:

val shuffled = shuffle(observations.zip(expected)) //24
@tailrec
def sgd(   nIters: Int, 
   weights: Weights,//25
   losses: List[Double]): (Weights, List[Double]
 ) = {  //26 
  if(nIters >= maxIters) 
     (weights, losses)  //27
  else {
     val (x, y) = shuffled(nIters % observations.size)
     val (newLoss, grad) = { 
      val yDot = y * margin(x, weights)
       val gradient = derivativeLoss(y, yDot)
         (logisticLoss(yDot),  // 28
           Array[Double](gradient) ++ x.map(_ *gradient) )//29
    }

     if(newLoss < eps)  //30
       (weights, newLoss :: losses)  //31
     else {
       val newWeights = weights.zip(grad).map{ 
          case (w, df) => w - eta*df //33
       } 
       sgd(
         nIters+1, //34
         newWeights,
         newLoss :: losses)
     }
}

The sgd method recurses on the following arguments:

The method returns the pair of weights and the list of losses computed at each iteration if the maximum number of iterations allowed for the optimization is reached (line 27). The client code evaluates either the size of the losses list or extracts its head value to validate whether SGD converged.

The formula, M4, for the computation of the loss (line 28) and the gradient of the loss over each weight in formula, M5 (line 29), relies on two simple methods: logisticLoss and derivativeLoss. The code is as follows:

def logisticLoss(z: Double): Double = 
  log(1.0 + exp(-z)) / observations.size //30
def derivedLoss(y: Double, yDot: Double):Double = 
 -y / (1.0 + exp(yDot))

The logistic loss is normalized by the number of observations (line 30).

The method evaluates new loss against the convergence criterion eps (line 31) and returns a version of the pair (weights, losses) (line 32) if the SGD converges. The formula M4 that updates the weights is implemented by zipping the weights and the gradient (line 33). The next invocation of SGD selects the next observation in the shuffled sequence of observations using a modulo operator to avoid overflowing (line 34).

Finally, here is an example of implementation of the margin formula:

def margin(observation: Features, weights: Weights):Double =
  weights.drop(1).zip(observation.view)
             .aggregate(weights.head)(dot, _ + _)

This implementation of the margin includes the intercept with its weight associated to the bias, a feature of the value 1.0.

The sequence of observations is randomly shuffled before the SGD is computed. This implementation of shuffling relies on the Scala standard library method, scala.util.Random.shuffle [1:13].

In order to train the model, we need to label input data. The labeling process consists of associating the relative price movement during a session (price at close/price at open – 1) with one of two configurations:

In this particular case, the labeling is automated because the relative price movement is extractable from raw data.

Once the model has been successfully created through training, it is available to classify new observation. The runtime classification of observations using the binomial logistic regression is implemented by the method classify:

def classify(obs: Features): Try[(Int, Double)] = 
  val linear = margin(obs, model.weights) 
           + model.weights(0)  //37
  val prediction = sigmoid(linear)
  (if(linear > 0.0) 1 else 0, prediction) //38
})

The method applies the logistic function to the linear inner product, linear, of the new observation, obs, and the weights of the model (line 37). The method returns the tuple (the predicted class of the observation {0, 1}, prediction value), where the class is defined by comparing the prediction to the boundary value 0.0 (line 38).

The computation of the margin as product of weights and observations is as follows:

def margin(obs: Features, weights: Weights): Double =
   weights.drop(1).zip(obs.view)
          .aggregate(0.0){case (s, (w,x)) => s + w*x, _ + _ }

The margin method is used in the classify method.