Book Image

PyTorch 1.x Reinforcement Learning Cookbook

By : Yuxi (Hayden) Liu
Book Image

PyTorch 1.x Reinforcement Learning Cookbook

By: Yuxi (Hayden) Liu

Overview of this book

Reinforcement learning (RL) is a branch of machine learning that has gained popularity in recent times. It allows you to train AI models that learn from their own actions and optimize their behavior. PyTorch has also emerged as the preferred tool for training RL models because of its efficiency and ease of use. With this book, you'll explore the important RL concepts and the implementation of algorithms in PyTorch 1.x. The recipes in the book, along with real-world examples, will help you master various RL techniques, such as dynamic programming, Monte Carlo simulations, temporal difference, and Q-learning. You'll also gain insights into industry-specific applications of these techniques. Later chapters will guide you through solving problems such as the multi-armed bandit problem and the cartpole problem using the multi-armed bandit algorithm and function approximation. You'll also learn how to use Deep Q-Networks to complete Atari games, along with how to effectively implement policy gradients. Finally, you'll discover how RL techniques are applied to Blackjack, Gridworld environments, internet advertising, and the Flappy Bird game. By the end of this book, you'll have developed the skills you need to implement popular RL algorithms and use RL techniques to solve real-world problems.
Table of Contents (11 chapters)

Reviewing the fundamentals of PyTorch

As we’ve already mentioned, PyTorch is the numerical computation library we use to implement reinforcement learning algorithms in this book.

PyTorch is a trendy scientific computing and machine learning (including deep learning) library developed by Facebook. Tensor is the core data structure in PyTorch, which is similar to NumPy's ndarrays. PyTorch and NumPy are comparable in scientific computing. However, PyTorch is faster than NumPy in array operations and array traversing. This is mainly due to the fact that array element access is faster in PyTorch. Hence, more and more people believe PyTorch will replace NumPy.

How to do it...

Let's do a quick review of the basic programming in PyTorch to get more familiar with it:

  1. We created an uninitialized matrix in an earlier recipe. How about a randomly initialized one? See the following commands:
 >>> import torch
>>> x = torch.rand(3, 4)
>>> print(x)
tensor([[0.8052, 0.3370, 0.7676, 0.2442],
[0.7073, 0.4468, 0.1277, 0.6842],
[0.6688, 0.2107, 0.0527, 0.4391]])

Random floats from a uniform distribution in the interval (0, 1) are generated.

  1. We can specify the desired data type of the returned tensor. For example, a tensor of the double type (float64) is returned as follows:
 >>> x = torch.rand(3, 4, dtype=torch.double)
>>> print(x)
tensor([[0.6848, 0.3155, 0.8413, 0.5387],
[0.9517, 0.1657, 0.6056, 0.5794],
[0.0351, 0.3801, 0.7837, 0.4883]], dtype=torch.float64)

By default, float is the returned data type.

  1. Next, let's create a matrix full of zeros and a matrix full of ones:
    >>> x = torch.zeros(3, 4)
>>> print(x)
tensor([[0., 0., 0., 0.],
[0., 0., 0., 0.],
[0., 0., 0., 0.]])
>>> x = torch.ones(3, 4)
>>> print(x)
tensor([[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.]])
  1. To get the size of a tensor, use this code:
 >>> print(x.size())
torch.Size([3, 4])

torch.Size is actually a tuple.

  1. To reshape a tensor, we can use the view() method:
 >>> x_reshaped = x.view(2, 6)
>>> print(x_reshaped)
tensor([[1., 1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1., 1.]])
  1. We can create a tensor directly from data, including a single value, a list, and a nested list:
 >>> x1 = torch.tensor(3)
>>> print(x1)
>>> x2 = torch.tensor([14.2, 3, 4])
>>> print(x2)
tensor([14.2000, 3.0000, 4.0000])
>>> x3 = torch.tensor([[3, 4, 6], [2, 1.0, 5]])
>>> print(x3)
tensor([[3., 4., 6.],
[2., 1., 5.]])
  1. To access the elements in a tensor of more than one element, we can use indexing in a similar way to NumPy:
 >>> print(x2[1])
>>> print(x3[1, 0])
>>> print(x3[:, 1])
tensor([4., 1.])
>>> print(x3[:, 1:])
tensor([[4., 6.],
[1., 5.]])

As with a one-element tensor, we do so by using the item() method:

 >>> print(x1.item())
  1. Tensor and NumPy arrays are mutually convertible. Convert a tensor to a NumPy array using the numpy() method:
 >>> x3.numpy()
array([[3., 4., 6.],
[2., 1., 5.]], dtype=float32)

Convert a NumPy array to a tensor with from_numpy():

>>> import numpy as np
>>> x_np = np.ones(3)
>>> x_torch = torch.from_numpy(x_np)
>>> print(x_torch)
tensor([1., 1., 1.], dtype=torch.float64)
Note that if the input NumPy array is of the float data type, the output tensor will be of the double type. Typecasting may occasionally be needed.

Take a look at the following example, where a tensor of the double type is converted to a float:

 >>> print(x_torch.float())
tensor([1., 1., 1.])
  1. Operations in PyTorch are similar to NumPy as well. Take addition as an example; we can simply do the following:
>>> x4 = torch.tensor([[1, 0, 0], [0, 1.0, 0]])
>>> print(x3 + x4)
tensor([[4., 4., 6.],
[2., 2., 5.]])

Or we can use the add() method as follows:

 >>> print(torch.add(x3, x4))
tensor([[4., 4., 6.],
[2., 2., 5.]])
  1. PyTorch supports in-place operations, which mutate the tensor object. For example, let's run this command:
 >>> x3.add_(x4)
tensor([[4., 4., 6.],
[2., 2., 5.]])

You will see that x3 is changed to the result of the original x3plus x4:

 >>> print(x3)
tensor([[4., 4., 6.],
[2., 2., 5.]])

There's more...

Any method with _ indicates that it is an in-place operation, which updates the tensor with the resulting value.

See also

For the full list of tensor operations in PyTorch, please go to the official docs at This is the best place to search for information if you get stuck on a PyTorch programming problem.