Book Image

Hands-On Reinforcement Learning with Python

By : Sudharsan Ravichandiran
Book Image

Hands-On Reinforcement Learning with Python

By: Sudharsan Ravichandiran

Overview of this book

Reinforcement Learning (RL) is the trending and most promising branch of artificial intelligence. Hands-On Reinforcement learning with Python will help you master not only the basic reinforcement learning algorithms but also the advanced deep reinforcement learning algorithms. The book starts with an introduction to Reinforcement Learning followed by OpenAI Gym, and TensorFlow. You will then explore various RL algorithms and concepts, such as Markov Decision Process, Monte Carlo methods, and dynamic programming, including value and policy iteration. This example-rich guide will introduce you to deep reinforcement learning algorithms, such as Dueling DQN, DRQN, A3C, PPO, and TRPO. You will also learn about imagination-augmented agents, learning from human preference, DQfD, HER, and many more of the recent advancements in reinforcement learning. By the end of the book, you will have all the knowledge and experience needed to implement reinforcement learning and deep reinforcement learning in your projects, and you will be all set to enter the world of artificial intelligence.

Related Content you might be interested in

Current Title:

Small book image
Hands-On Reinforcement Learning with Python
Sudharsan Ravichandiran
Jun, 2018 | 318 pages
Small book image
PyTorch 1.x Reinforcement Learning Cookbook
Yuxi Hayden Liu
Oct, 2019 | 340 pages
Small book image
Reinforcement Learning with TensorFlow
Sayon Dutta
Apr, 2018 | 334 pages
Small book image
Hands-On Deep Learning Algorithms with Python
Sudharsan Ravichandiran
Jul, 2019 | 512 pages
Table of Contents (16 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Introduction to Reinforcement Learning
Introduction to Reinforcement Learning
What is RL?
RL algorithm
How RL differs from other ML paradigms
Elements of RL
Agent environment interface
Types of RL environment
RL platforms
Applications of RL
Summary
Questions
Further reading
2
Getting Started with OpenAI and TensorFlow
Getting Started with OpenAI and TensorFlow
Setting up your machine
OpenAI Gym
OpenAI Universe
TensorFlow
Summary
Questions
Further reading
3
The Markov Decision Process and Dynamic Programming
The Markov Decision Process and Dynamic Programming
The Markov chain and Markov process
Markov Decision Process
The Bellman equation and optimality
Solving the Bellman equation
Solving the frozen lake problem
Summary
Questions
Further reading
4
Gaming with Monte Carlo Methods
Gaming with Monte Carlo Methods
Monte Carlo methods
Monte Carlo prediction
Monte Carlo control
Summary
Questions
Further reading
5
Temporal Difference Learning
Temporal Difference Learning
TD learning
TD prediction
TD control
The difference between Q learning and SARSA
Summary
Questions
Further reading
6
Multi-Armed Bandit Problem
Multi-Armed Bandit Problem
The MAB problem
Applications of MAB
Identifying the right advertisement banner using MAB
Contextual bandits
Summary
Questions
Further reading
7
Deep Learning Fundamentals
Deep Learning Fundamentals
Artificial neurons
ANNs
Deep diving into ANN
Neural networks in TensorFlow
RNN
Long Short-Term Memory RNN
Convolutional neural networks
Classifying fashion products using CNN
Summary
Questions
Further reading
8
Atari Games with Deep Q Network
Atari Games with Deep Q Network
What is a Deep Q Network?
Architecture of DQN
Building an agent to play Atari games
Double DQN
Prioritized experience replay
Dueling network architecture
Summary
Questions
Further reading
9
Playing Doom with a Deep Recurrent Q Network
Playing Doom with a Deep Recurrent Q Network
DRQN
Training an agent to play Doom
DARQN
Summary
Questions
Further reading
10
The Asynchronous Advantage Actor Critic Network
The Asynchronous Advantage Actor Critic Network
The Asynchronous Advantage Actor Critic
Driving up a mountain with A3C
Summary
Questions
Further reading
11
Policy Gradients and Optimization
Policy Gradients and Optimization
Policy gradient
Deep deterministic policy gradient
Trust Region Policy Optimization
Proximal Policy Optimization
Summary
Questions
Further reading
12
Capstone Project – Car Racing Using DQN
Capstone Project – Car Racing Using DQN
Environment wrapper functions
Dueling network
Replay memory
Training the network
Car racing
Summary
Questions
Further reading
13
Recent Advancements and Next Steps
Recent Advancements and Next Steps
Imagination augmented agents
Learning from human preference
Deep Q learning from demonstrations
Hindsight experience replay
Hierarchical reinforcement learning
Inverse reinforcement learning
Summary
Questions
Further reading
14
Assessments
Assessments
Chapter 1
Chapter 2
Chapter 3
Chapter 4
Chapter 5
Chapter 6
Chapter 7
Chapter 8
Chapter 9
Chapter 10
Chapter 11
Chapter 12
Chapter 13
15
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Proximal Policy Optimization

Now we will look at another policy optimization algorithm called Proximal Policy Optimization (PPO). It acts as an improvement to TRPO and has become the default RL algorithm of choice in solving many complex RL problems due to its performance. It was proposed by researchers at OpenAI for overcoming the shortcomings of TRPO. Recall the surrogate objective function of TRPO. It is a constraint optimization problem where we impose a constraint—that average KL divergence between the old and new policy should be less than . But the problem with TRPO is that it requires a lot of computing power for computing conjugate gradients to perform constrained optimization.

So, PPO modifies the objective function of TRPO by changing the constraint to a penalty term so that we don't want to perform conjugate gradient. Now let's see how PPO works....

Previous Section
End of Section 5
Next Section