Here we will provide a brief introduction to Keras, which is a sublibrary of TensorFlow that provides more high-level functions for implementing deep learning algorithms. Keras uses basic TensorFlow operations, underneath; however, it exposes a higher level, beginner-friendly API for users. To see how to use Keras, we will look at a quick example. We will outline how one might create a CNN using Keras. Full exercise can be found at keras_cnn.ipynb located in the appendix folder.

We will first determine what type of a model we will be defining. Keras has two different APIs: sequential and functional. The sequential API is simpler and allows designing a model, layer by layer. However, the sequential API has limited flexibility in designing the organization and connections between layers of the network. On the other hand, the functional API has much more flexibility and allows the user to design the specific details of the neural network. For demonstration purposes, we will implement a CNN using the sequential API in Keras. A sequential model in this case is a sequence of stack of layers (for example, input layer, convolution layer, and pooling layer):

model = Sequential()

Next, we will define the layers of our CNN one by one. First, we will define a convolution layer with 32 filters, a kernel size of 3 × 3 and ReLU nonlinearity. This layer will be taking an input of size 28 × 28 × 1 (that is, the size of an MNIST image):

model.add(Conv2D(32, 3, activation='relu', input_shape=[28, 28, 1]))

Next, we will define a max-pooling layer. If the kernel size and stride are not defined, they default to 2 (kernel size) and 1 (stride):

model.add(MaxPool2D())

Then we will add a batch normalization layer:

model.add(BatchNormalization())

A batch normalization layer (refer to Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, Ioffe and Szegedy, International Conference on Machine Learning, 2015) normalizes (that is, make activations zero-mean and unit-variance) the outputs of the previous layer. This is an additional step used to improve the performance of the CNN, especially in computer vision applications. Note that we did not use batch normalization in the chapter exercises, as the batch normalization has not been used heavily for NLP tasks, compared to the amount it is used for computer vision applications.

Next, we will add two more convolution layers, followed by a max-pooling layer and a batch normalization layer:

model.add(Conv2D(64, 3, activation='relu'))
model.add(MaxPool2D())
model.add(BatchNormalization())
model.add(Conv2D(128, 3, activation='relu'))
model.add(MaxPool2D())
model.add(BatchNormalization())

Next, we will flatten the input as this is required to feed the output into a fully connected layer:

model.add(Flatten())

Then we will add a fully connected layer with 256 hidden units, a ReLU activation, and a final softmax output layer with 10 softmax units (that is, for the 10 different classes of MNIST):

model.add(Dense(256, activation='relu'))
model.add(Dense(10, activation='softmax'))

Finally, we will compile the model, when we also tell Keras to use Adam as the optimizer and categorical cross-entropy loss and output metric to be the accuracy of the model:

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

Once the model, the loss and an optimizer is defined, we can run the Keras model as follows.

To train the model you can use the following command:

model.fit(x_train, y_train, batch_size = batch_size)

Here, x_train and y_train are the training data. And  batch_size defines the batch size. When you run this, the training progress will be shown below.

Then to evaluate the model, use the following:

test_acc = model.evaluate(x_test, y_test, batch_size=batch_size)  

This line will again output a progress bar as well as the test loss and accuracy of each epoch.