Book Image

Deep Reinforcement Learning Hands-On - Second Edition

By : Maxim Lapan
5 (2)
Book Image

Deep Reinforcement Learning Hands-On - Second Edition

5 (2)
By: Maxim Lapan

Overview of this book

Deep Reinforcement Learning Hands-On, Second Edition is an updated and expanded version of the bestselling guide to the very latest reinforcement learning (RL) tools and techniques. It provides you with an introduction to the fundamentals of RL, along with the hands-on ability to code intelligent learning agents to perform a range of practical tasks. With six new chapters devoted to a variety of up-to-the-minute developments in RL, including discrete optimization (solving the Rubik's Cube), multi-agent methods, Microsoft's TextWorld environment, advanced exploration techniques, and more, you will come away from this book with a deep understanding of the latest innovations in this emerging field. In addition, you will gain actionable insights into such topic areas as deep Q-networks, policy gradient methods, continuous control problems, and highly scalable, non-gradient methods. You will also discover how to build a real hardware robot trained with RL for less than $100 and solve the Pong environment in just 30 minutes of training using step-by-step code optimization. In short, Deep Reinforcement Learning Hands-On, Second Edition, is your companion to navigating the exciting complexities of RL as it helps you attain experience and knowledge through real-world examples.

Related Content you might be interested in

Current Title:

Small book image
Deep Reinforcement Learning Hands-On - Second Edition
Maxim Lapan
Jan, 2020 | 826 pages
Small book image
TensorFlow Reinforcement Learning Quick Start Guide
Kaushik Balakrishnan
Mar, 2019 | 184 pages
Small book image
Reinforcement Learning Algorithms with Python
Andrea Lonza
Oct, 2019 | 366 pages
Small book image
Hands-On Reinforcement Learning for Games
Micheal Lanham
Jan, 2020 | 432 pages
Table of Contents (28 chapters)
Preface
Preface
Why I wrote this book
The approach
Who this book is for
What this book covers
To get the most out of this book
Get in touch
1
What Is Reinforcement Learning?
What Is Reinforcement Learning?
Supervised learning
Unsupervised learning
Reinforcement learning
RL's complications
RL formalisms
The theoretical foundations of RL
Summary
Free Chapter
2
OpenAI Gym
OpenAI Gym
The anatomy of the agent
Hardware and software requirements
The OpenAI Gym API
The random CartPole agent
Extra Gym functionality – wrappers and monitors
Summary
3
Deep Learning with PyTorch
Deep Learning with PyTorch
Tensors
Gradients
NN building blocks
Custom layers
The final glue – loss functions and optimizers
Monitoring with TensorBoard
Example – GAN on Atari images
PyTorch Ignite
Summary
4
The Cross-Entropy Method
The Cross-Entropy Method
The taxonomy of RL methods
The cross-entropy method in practice
The cross-entropy method on CartPole
The cross-entropy method on FrozenLake
The theoretical background of the cross-entropy method
Summary
5
Tabular Learning and the Bellman Equation
Tabular Learning and the Bellman Equation
Value, state, and optimality
The Bellman equation of optimality
The value of the action
The value iteration method
Value iteration in practice
Q-learning for FrozenLake
Summary
6
Deep Q-Networks
Deep Q-Networks
Real-life value iteration
Tabular Q-learning
Deep Q-learning
DQN on Pong
Things to try
Summary
7
Higher-Level RL Libraries
Higher-Level RL Libraries
Why RL libraries?
The PTAN library
The PTAN CartPole solver
Other RL libraries
Summary
8
DQN Extensions
DQN Extensions
Basic DQN
N-step DQN
Double DQN
Noisy networks
Prioritized replay buffer
Dueling DQN
Categorical DQN
Combining everything
Summary
References
9
Ways to Speed up RL
Ways to Speed up RL
Why speed matters
The baseline
The computation graph in PyTorch
Several environments
Play and train in separate processes
Tweaking wrappers
Benchmark summary
Going hardcore: CuLE
Summary
References
10
Stocks Trading Using RL
Stocks Trading Using RL
Trading
Data
Problem statements and key decisions
The trading environment
Models
Training code
Results
Things to try
Summary
11
Policy Gradients – an Alternative
Policy Gradients – an Alternative
Values and policy
The REINFORCE method
REINFORCE issues
Policy gradient methods on CartPole
Policy gradient methods on Pong
Summary
12
The Actor-Critic Method
The Actor-Critic Method
Variance reduction
CartPole variance
Actor-critic
A2C on Pong
A2C on Pong results
Tuning hyperparameters
Summary
13
Asynchronous Advantage Actor-Critic
Asynchronous Advantage Actor-Critic
Correlation and sample efficiency
Adding an extra A to A2C
Multiprocessing in Python
A3C with data parallelism
A3C with gradients parallelism
Summary
14
Training Chatbots with RL
Training Chatbots with RL
An overview of chatbots
Chatbot training
The deep NLP basics
Seq2seq training
Chatbot example
Dataset exploration
Training: cross-entropy
Training: SCST
Models tested on data
Telegram bot
Summary
15
The TextWorld Environment
The TextWorld Environment
Interactive fiction
The environment
Baseline DQN
The command generation model
Summary
16
Web Navigation
Web Navigation
Web navigation
OpenAI Universe
The simple clicking approach
Human demonstrations
Adding text descriptions
Things to try
Summary
17
Continuous Action Space
Continuous Action Space
Why a continuous space?
The A2C method
Deterministic policy gradients
Distributional policy gradients
Things to try
Summary
18
RL in Robotics
RL in Robotics
Robots and robotics
The first training objective
The emulator and the model
DDPG training and results
Controlling the hardware
Policy experiments
Summary
19
Trust Regions – PPO, TRPO, ACKTR, and SAC
Trust Regions – PPO, TRPO, ACKTR, and SAC
Roboschool
The A2C baseline
PPO
TRPO
ACKTR
SAC
Summary
20
Black-Box Optimization in RL
Black-Box Optimization in RL
Black-box methods
Evolution strategies
Genetic algorithms
Summary
References
21
Advanced Exploration
Advanced Exploration
Why exploration is important
What's wrong with ε-greedy?
Alternative ways of exploration
MountainCar experiments
Atari experiments
Summary
References
22
Beyond Model-Free – Imagination
Beyond Model-Free – Imagination
Model-based methods
The imagination-augmented agent
I2A on Atari Breakout
Experiment results
Summary
References
23
AlphaGo Zero
AlphaGo Zero
Board games
The AlphaGo Zero method
The Connect 4 bot
Connect 4 results
Summary
References
24
RL in Discrete Optimization
RL in Discrete Optimization
RL's reputation
The Rubik's Cube and combinatorial optimization
Optimality and God's number
Approaches to cube solving
The training process
The model application
The paper's results
The code outline
The experiment results
Further improvements and experiments
Summary
25
Multi-agent RL
Multi-agent RL
Multi-agent RL explained
The MAgent environment
Deep Q-network for tigers
Collaboration by the tigers
Training both tigers and deer
The battle between equal actors
Summary
26
Other Books You May Enjoy
Other Books You May Enjoy
27
Index
Index

CartPole variance

To check this theoretical conclusion in practice, let's plot our policy gradient variance during the training for both the baseline version and the version without the baseline. The complete example is in Chapter12/01_cartpole_pg.py, and most of the code is the same as in Chapter 11, Policy Gradients – an Alternative. The differences in this version are the following:

  • It now accepts the command-line option --baseline, which enables the mean subtraction from the reward. By default, no baseline is used.
  • On every training loop, we gather the gradients from the policy loss and use this data to calculate the variance.

To gather only the gradients from the policy loss and exclude the gradients from the entropy bonus added for exploration, we need to calculate the gradients in two stages. Luckily, PyTorch allows this to be done easily. In the following code, only the relevant part of the training loop is included to illustrate the idea:

...
Previous Section
End of Section 3
Next Section