Book Image

Reinforcement Learning Algorithms with Python

By : Andrea Lonza
Book Image

Reinforcement Learning Algorithms with Python

By: Andrea Lonza

Overview of this book

Reinforcement Learning (RL) is a popular and promising branch of AI that involves making smarter models and agents that can automatically determine ideal behavior based on changing requirements. This book will help you master RL algorithms and understand their implementation as you build self-learning agents. Starting with an introduction to the tools, libraries, and setup needed to work in the RL environment, this book covers the building blocks of RL and delves into value-based methods, such as the application of Q-learning and SARSA algorithms. You'll learn how to use a combination of Q-learning and neural networks to solve complex problems. Furthermore, you'll study the policy gradient methods, TRPO, and PPO, to improve performance and stability, before moving on to the DDPG and TD3 deterministic algorithms. This book also covers how imitation learning techniques work and how Dagger can teach an agent to drive. You'll discover evolutionary strategies and black-box optimization techniques, and see how they can improve RL algorithms. Finally, you'll get to grips with exploration approaches, such as UCB and UCB1, and develop a meta-algorithm called ESBAS. By the end of the book, you'll have worked with key RL algorithms to overcome challenges in real-world applications, and be part of the RL research community.

Related Content you might be interested in

Current Title:

Small book image
Reinforcement Learning Algorithms with Python
Andrea Lonza
Oct, 2019 | 366 pages
Small book image
Mastering Reinforcement Learning with Python
Enes Bilgin
Dec, 2020 | 544 pages
Small book image
TensorFlow Reinforcement Learning Quick Start Guide
Kaushik Balakrishnan
Mar, 2019 | 184 pages
Small book image
Hands-On Intelligent Agents with OpenAI Gym
Palanisamy P
Jul, 2018 | 254 pages
Table of Contents (19 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Section 1: Algorithms and Environments
Section 1: Algorithms and Environments
2
The Landscape of Reinforcement Learning
The Landscape of Reinforcement Learning
An introduction to RL
Elements of RL
Applications of RL
Summary
Questions
Further reading
3
Implementing RL Cycle and OpenAI Gym
Implementing RL Cycle and OpenAI Gym
Setting up the environment
OpenAI Gym and RL cycles
Development of ML models using TensorFlow
Introducing TensorBoard
Types of RL environments
Summary
Questions
Further reading
4
Solving Problems with Dynamic Programming
Solving Problems with Dynamic Programming
MDP
Categorizing RL algorithms
Dynamic programming
Summary
Questions
Further reading
5
Section 2: Model-Free RL Algorithms
Section 2: Model-Free RL Algorithms
6
Q-Learning and SARSA Applications
Q-Learning and SARSA Applications
Learning without a model
TD learning
SARSA
Applying SARSA to Taxi-v2
Q-learning
Applying Q-learning to Taxi-v2
Summary
Questions
7
Deep Q-Network
Deep Q-Network
Deep neural networks and Q-learning
DQN
DQN applied to Pong
DQN variations
Summary
Questions
Further reading
8
Learning Stochastic and PG Optimization
Learning Stochastic and PG Optimization
Policy gradient methods
Understanding the REINFORCE algorithm
REINFORCE with baseline
Learning the AC algorithm
Summary
Questions
Further reading
9
TRPO and PPO Implementation
TRPO and PPO Implementation
Roboschool
Natural policy gradient
Trust region policy optimization
Proximal Policy Optimization
Summary
Questions
Further reading
10
DDPG and TD3 Applications
DDPG and TD3 Applications
Combining policy gradient optimization with Q-learning
Deep deterministic policy gradient
Twin delayed deep deterministic policy gradient (TD3)
Summary
Questions
Further reading
11
Section 3: Beyond Model-Free Algorithms and Improvements
Section 3: Beyond Model-Free Algorithms and Improvements
12
Model-Based RL
Model-Based RL
Model-based methods
Combining model-based with model-free learning
ME-TRPO applied to an inverted pendulum
Summary
Questions
Further reading
13
Imitation Learning with the DAgger Algorithm
Imitation Learning with the DAgger Algorithm
Technical requirements
The imitation approach
Playing Flappy Bird
Understanding the dataset aggregation algorithm
IRL
Summary
Questions
Further reading
14
Understanding Black-Box Optimization Algorithms
Understanding Black-Box Optimization Algorithms
Beyond RL
The core of EAs
Scalable evolution strategies
Applying scalable ES to LunarLander
Summary
Questions
Further reading
15
Developing the ESBAS Algorithm
Developing the ESBAS Algorithm
Exploration versus exploitation
Approaches to exploration
Epochal stochastic bandit algorithm selection
Summary
Questions
Further reading
16
Practical Implementation for Resolving RL Challenges
Practical Implementation for Resolving RL Challenges
Best practices of deep RL
Challenges in deep RL
Advanced techniques
RL in the real world
Future of RL and its impact on society
Summary
Questions
Further reading
17
Assessments
Assessments
18
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Natural policy gradient

REINFORCE and Actor-Critic are very intuitive methods that work well on small to medium-sized RL tasks. However, they present some problems that need to be addressed so that we can adapt policy gradient algorithms so that they work on much larger and complex tasks. The main problems are as follows:

  • Difficult to choose a correct step size: This comes from the nature of RL being non-stationary, meaning that the distribution of the data changes continuously over time and as the agent learns new things, it explores a different state space. Finding an overall stable learning rate is very tricky.
  • Instability: The algorithms aren't aware of the amount by which the policy will change. This is also related to the problem we stated previously. A single, not controlled update could induce a substantial shift of the policy that will drastically change the action...
Previous Section
End of Section 3
Next Section