The base of TensorFlow is the computational graph, which we discussed earlier in this chapter, and tensors. A tensor is an n-dimensional vector. Thus, a scalar and a matrix variable is also a tensor. Here, we will try some of the basic computations to start with TensorFlow. Please try to implement this section in a python IDE such as Jupyter Notebook.

For the TensorFlow installation and dependencies please refer to the following link:

https://www.tensorflow.org/install/

Import tensorflow by the following command:

import tensorflow as tf

tf.zeros() and tf.ones() are some of the functions that instantiate basic tensors. The tf.zeros() takes a tensor shape (that is, a tuple) and returns a tensor of that shape with all the values being zero. Similarly, tf.ones() takes a tensor shape but returns a tensor of that shape containing only ones. Try the following commands in python shell to create a tensor:

>>> tf.zeros(3)

<tf.Tensor 'zeros:0' shape=(3,) dtype=float32>

>>>tf.ones(3)

<tf.Tensor 'ones:0' shape=(3,) dtype=float32>

As you can see, TensorFlow returns a reference to the tensor and not the value of the tensor. In order to get the value, we can use eval() or run(), a function of tensor objects by running a session as follows:

>>> a = tf.zeros(3)
>>> with tf.Session() as sess:
        sess.run(a)
        a.eval()

array([0., 0.,0.], dtype=float32)

array([0., 0.,0.], dtype=float32)

Next come the tf.fill() and tf.constant() methods to create a tensor of a certain shape and value:

>>> a = tf.fill((2,2),value=4.)
>>> b = tf.constant(4.,shape=(2,2))
>>> with tf.Session() as sess:
        sess.run(a)
        sess.run(b)

array([[ 4., 4.],
[ 4., 4.]], dtype=float32)

array([[ 4., 4.],
[ 4., 4.]], dtype=float32)

Next, we have functions that can randomly initialize a tensor. Among them, the most frequently used ones are:

>>> a = tf.random_normal((2,2),mean=0,stddev=1)
>>> b = tf.random_uniform((2,2),minval=-3,maxval=3)
>>> with tf.Session() as sess:
        sess.run(a)
        sess.run(b)

array([[-0.31790468, 1.30740941],
[-0.52323157, -0.2980336 ]], dtype=float32)

array([[ 1.38419437, -2.91128755],
[-0.80171156, -0.84285879]], dtype=float32)

Variables in TensorFlow are holders for tensors and are defined by the function tf.Variable():

>>> a = tf.Variable(tf.ones((2,2)))
>>> a

<tf.Variable 'Variable:0' shape=(2, 2) dtype=float32_ref>

The evaluation fails in case of variables because they have to be explicitly initialized by using tf.global_variables_initializer within a session:

>>> a = tf.Variable(tf.ones((2,2)))
>>> with tf.Session() as sess:
        sess.run(tf.global_variables_initializer())
        a.eval()

array([[ 1., 1.],
[ 1., 1.]], dtype=float32)

Next in the queue, we have matrices. Identity matrices are square matrices with ones in the diagonal and zeros elsewhere. This can be done with thefunction tf.eye():

>>> id = tf.eye(4) #size of the square matrix = 4
>>> with tf.Session() as sess:
         sess.run(id)

array([[ 1., 0., 0., 0.],
[ 0., 1., 0., 0.],
[ 0., 0., 1., 0.],
[ 0., 0., 0., 1.]], dtype=float32)

Similarly, there are diagonal matrices, which have values in the diagonal and zeros elsewhere, as shown here:

>>> a = tf.range(1,5,1)
>>> md = tf.diag(a)
>>> mdn = tf.diag([1,2,5,3,2])
>>> with tf.Session() as sess:
        sess.run(md)
        sess.run(mdn)

array([[1, 0, 0, 0],
[0, 2, 0, 0],
[0, 0, 3, 0],
[0, 0, 0, 4]], dtype=int32)

array([[1, 0, 0, 0, 0],
[0, 2, 0, 0, 0],
[0, 0, 5, 0, 0],
[0, 0, 0, 3, 0],
[0, 0, 0, 0, 2]], dtype=int32)

We use the tf.matrix_transpose() function to transpose the given matrix, as shown here:

>>> a = tf.ones((2,3))
>>> b = tf.transpose(a)
>>> with tf.Session() as sess:
        sess.run(a)
        sess.run(b)

array([[ 1., 1., 1.],
[ 1., 1., 1.]], dtype=float32)

array([[ 1., 1.],
[ 1., 1.],
[ 1., 1.]], dtype=float32)

The next matrix operation is the matrix multiplication function as shown here. This is done by the function tf.matmul():

>>> a = tf.ones((3,2))
>>> b = tf.ones((2,4))
>>> c = tf.matmul(a,b)
>>> with tf.Session() as sess:
        sess.run(a)
        sess.run(b)
        sess.run(c)

array([[ 1., 1.],
[ 1., 1.],
[ 1., 1.]], dtype=float32)

array([[ 1., 1., 1., 1.],
[ 1., 1., 1., 1.]], dtype=float32)

array([[ 2., 2., 2., 2.],
[ 2., 2., 2., 2.],
[ 2., 2., 2., 2.]], dtype=float32)

Reshaping of tensors from one to another is done by using the tf.reshape() function, as shown here:

>>> a = tf.ones((2,4)) #initial shape is (2,4)
>>> b = tf.reshape(a,(8,)) # reshaping it to a vector of size 8. Thus shape is (8,)
>>> c = tf.reshape(a,(2,2,2)) #reshaping tensor a to shape (2,2,2)
>>> d = tf.reshape(b,(2,2,2)) #reshaping tensor b to shape (2,2,2) 
#####Thus, tensor 'c' and 'd' will be similar
>>> with tf.Session() as sess:
        sess.run(a)
        sess.run(b)
        sess.run(c)
        sess.run(d)

array([[ 1., 1., 1., 1.],
[ 1., 1., 1., 1.]], dtype=float32)

array([ 1., 1., 1., 1., 1., 1., 1., 1.], dtype=float32)

array([[[ 1., 1.],
[ 1., 1.]],
[[ 1., 1.],
[ 1., 1.]]], dtype=float32)
>
array([[[ 1., 1.],
[ 1., 1.]],
[[ 1., 1.],
[ 1., 1.]]], dtype=float32)

The flow of computation in TensorFlow is represented as a computational graph, which is as instance of tf.Graph. The graph contains tensors and operation objects, and keeps track of a series of operations and tensors involved. The default instance of the graph can be fetched by tf.get_default_graph():

>>> tf.get_default_graph()

<tensorflow.python.framework.ops.Graph object at 0x7fa3e139b550>

We will explore complex operations, the creation of neural networks, and much more in TensorFlow in the coming chapters.

The OpenAI Gym, created by the team at OpenAI is a playground of different environments where you can develop and compare your reinforcement learning algorithms. It is compatible with deep learning libraries such as TensorFlow and Theano.

OpenAI Gym consists of two parts:

For the installation and dependencies, please refer to the following link:

https://gym.openai.com/docs/

With the basics covered, now we can start with the implementation of reinforcement learning using the OpenAI Gym from next Chapter 2, Training Reinforcement Learning Agents using OpenAI Gym.