Hardware and software requirements
The examples in this book were implemented and tested using Python version 3.7. I assume that you're already familiar with the language and common concepts such as virtual environments, so I won't cover in detail how to install the package and how to do this in an isolated way. The examples will use the previously mentioned Python type annotations, which will allow us to provide type signatures for functions and class methods.
The external libraries that we will use in this book are open source software, and they include the following:
- NumPy: This is a library for scientific computing, and implementing matrix operations and common functions.
- OpenCV Python bindings: This is a computer vision library and provides many functions for image processing.
- Gym: This is an RL framework that has various environments that can be communicated with in a unified way.
- PyTorch: This is a flexible and expressive deep learning (DL) library. A short crash course on it will be given in Chapter 3, Deep Learning with PyTorch.
- PyTorch Ignite: This is a set of high-level tools on top of PyTorch used to reduce boilerplate code. It will be covered briefly in Chapter 3. The full documentation is available here: https://pytorch.org/ignite/.
- PTAN (https://github.com/Shmuma/ptan): This is an open source extension to Gym that I created to support the modern deep RL methods and building blocks. All classes used will be described in detail together with the source code.
Other libraries will be used for specific chapters; for example, we will use Microsoft TextWorld for solving text-based games, PyBullet for robotic simulations, OpenAI Universe for browser-based automation problems, and so on. Those specialized chapters will include installation instructions for those libraries.
A significant portion of this book (parts two, three, and four) is focused on the modern deep RL methods that have been developed over the past few years. The word "deep" in this context means that DL is heavily used. You may be aware that DL methods are computationally hungry. One modern graphics processing unit (GPU) can be 10 to 100 times faster than even the fastest multiple central processing unit (CPU) systems. In practice, this means that the same code that takes one hour to train on a system with a GPU could take from half a day to one week even on the fastest CPU system. It doesn't mean that you can't try the examples from this book without having access to a GPU, but it will take longer. To experiment with the code on your own (the most useful way to learn anything), it is better to get access to a machine with a GPU. This can be done in various ways:
- Buying a modern GPU suitable for CUDA
- Using cloud instances. Both Amazon Web Services and Google Cloud can provide you with GPU-powered instances
- Google Colab offers free GPU access to its Jupyter notebooks
The instructions on how to set up the system are beyond the scope of this book, but there are plenty of manuals available on the Internet. In terms of an operating system (OS), you should use Linux or macOS. Windows is supported by PyTorch and Gym, but the examples in the book were not fully tested under Windows OS.
To give you the exact versions of the external dependencies that we will use throughout the book, here is an output of the pip freeze command (it may be useful for the potential troubleshooting of examples in the book, as open source software and DL toolkits are evolving extremely quickly):
atari-py==0.2.6
gym==0.15.3
numpy==1.17.2
opencv-python==4.1.1.26
tensorboard==2.0.1
torch==1.3.0
torchvision==0.4.1
pytorch-ignite==0.2.1
tensorboardX==1.9
tensorflow==2.0.0
ptan==0.6
All the examples in the book were written and tested with PyTorch 1.3, which can be installed by following the instructions on the http://pytorch.org website (normally, that's just the conda install pytorch torchvision -c pytorch command).
Now, let's go into the details of the OpenAI Gym API, which provides us with tons of environments, from trivial to challenging ones.