As we’ve already mentioned, PyTorch is the numerical computation library we use to implement reinforcement learning algorithms in this book.

PyTorch is a trendy scientific computing and machine learning (including deep learning) library developed by Facebook. Tensor is the core data structure in PyTorch, which is similar to NumPy's ndarrays. PyTorch and NumPy are comparable in scientific computing. However, PyTorch is faster than NumPy in array operations and array traversing. This is mainly due to the fact that array element access is faster in PyTorch. Hence, more and more people believe PyTorch will replace NumPy.

Let's do a quick review of the basic programming in PyTorch to get more familiar with it:

1. We created an uninitialized matrix in an earlier recipe. How about a randomly initialized one? See the following commands:

 >>> import torch
 >>> x = torch.rand(3, 4)
 >>> print(x)
 tensor([[0.8052, 0.3370, 0.7676, 0.2442],
        [0.7073, 0.4468, 0.1277, 0.6842],
        [0.6688, 0.2107, 0.0527, 0.4391]])

Random floats from a uniform distribution in the interval (0, 1) are generated.

2. We can specify the desired data type of the returned tensor. For example, a tensor of the double type (float64) is returned as follows:

 >>> x = torch.rand(3, 4, dtype=torch.double)
 >>> print(x)
 tensor([[0.6848, 0.3155, 0.8413, 0.5387],
        [0.9517, 0.1657, 0.6056, 0.5794],
        [0.0351, 0.3801, 0.7837, 0.4883]], dtype=torch.float64)

By default, float is the returned data type.

3. Next, let's create a matrix full of zeros and a matrix full of ones:

    >>> x = torch.zeros(3, 4)
    >>> print(x)
    tensor([[0., 0., 0., 0.],
           [0., 0., 0., 0.],
           [0., 0., 0., 0.]])
    >>> x = torch.ones(3, 4)
    >>> print(x)
    tensor([[1., 1., 1., 1.],
           [1., 1., 1., 1.],
           [1., 1., 1., 1.]])

4. To get the size of a tensor, use this code:

 >>> print(x.size())
 torch.Size([3, 4])

torch.Size is actually a tuple.

5. To reshape a tensor, we can use the view() method:

 >>> x_reshaped = x.view(2, 6)
 >>> print(x_reshaped)
 tensor([[1., 1., 1., 1., 1., 1.],
         [1., 1., 1., 1., 1., 1.]])

6. We can create a tensor directly from data, including a single value, a list, and a nested list:

 >>> x1 = torch.tensor(3)
 >>> print(x1)
 tensor(3)
 >>> x2 = torch.tensor([14.2, 3, 4])
 >>> print(x2)
 tensor([14.2000,  3.0000, 4.0000])
 >>> x3 = torch.tensor([[3, 4, 6], [2, 1.0, 5]])
 >>> print(x3)
 tensor([[3., 4., 6.],
         [2., 1., 5.]])

7. To access the elements in a tensor of more than one element, we can use indexing in a similar way to NumPy:

 >>> print(x2[1])
 tensor(3.)
 >>> print(x3[1, 0])
 tensor(2.)
 >>> print(x3[:, 1])
 tensor([4., 1.])
 >>> print(x3[:, 1:])
 tensor([[4., 6.],
         [1., 5.]])

As with a one-element tensor, we do so by using the item() method:

 >>> print(x1.item())
 3

8. Tensor and NumPy arrays are mutually convertible. Convert a tensor to a NumPy array using the numpy() method:

 >>> x3.numpy()
 array([[3., 4., 6.],
        [2., 1., 5.]], dtype=float32)

Convert a NumPy array to a tensor with from_numpy():

>>> import numpy as np
>>> x_np = np.ones(3)
>>> x_torch = torch.from_numpy(x_np)
>>> print(x_torch)
tensor([1., 1., 1.], dtype=torch.float64)

Take a look at the following example, where a tensor of the double type is converted to a float:

 >>> print(x_torch.float())
 tensor([1., 1., 1.])

9. Operations in PyTorch are similar to NumPy as well. Take addition as an example; we can simply do the following:

>>> x4 = torch.tensor([[1, 0, 0], [0, 1.0, 0]])
>>> print(x3 + x4)
tensor([[4., 4., 6.],
         [2., 2., 5.]])

Or we can use the add() method as follows:

 >>> print(torch.add(x3, x4))
 tensor([[4., 4., 6.],
         [2., 2., 5.]])

10. PyTorch supports in-place operations, which mutate the tensor object. For example, let's run this command:

 >>> x3.add_(x4)
 tensor([[4., 4., 6.],
         [2., 2., 5.]])

You will see that x3 is changed to the result of the original x3plus x4:

 >>> print(x3)
 tensor([[4., 4., 6.],
         [2., 2., 5.]])

Any method with _ indicates that it is an in-place operation, which updates the tensor with the resulting value.

For the full list of tensor operations in PyTorch, please go to the official docs at https://pytorch.org/docs/stable/torch.html. This is the best place to search for information if you get stuck on a PyTorch programming problem.