Book Image

Reinforcement Learning Algorithms with Python

By : Andrea Lonza

Book Image

Reinforcement Learning Algorithms with Python

By: Andrea Lonza

Overview of this book

Reinforcement Learning (RL) is a popular and promising branch of AI that involves making smarter models and agents that can automatically determine ideal behavior based on changing requirements. This book will help you master RL algorithms and understand their implementation as you build self-learning agents. Starting with an introduction to the tools, libraries, and setup needed to work in the RL environment, this book covers the building blocks of RL and delves into value-based methods, such as the application of Q-learning and SARSA algorithms. You'll learn how to use a combination of Q-learning and neural networks to solve complex problems. Furthermore, you'll study the policy gradient methods, TRPO, and PPO, to improve performance and stability, before moving on to the DDPG and TD3 deterministic algorithms. This book also covers how imitation learning techniques work and how Dagger can teach an agent to drive. You'll discover evolutionary strategies and black-box optimization techniques, and see how they can improve RL algorithms. Finally, you'll get to grips with exploration approaches, such as UCB and UCB1, and develop a meta-algorithm called ESBAS. By the end of the book, you'll have worked with key RL algorithms to overcome challenges in real-world applications, and be part of the RL research community.

Preface

Who this book is for

What this book covers

To get the most out of this book

Free Chapter

Section 1: Algorithms and Environments

Section 1: Algorithms and Environments

The Landscape of Reinforcement Learning

The Landscape of Reinforcement Learning

An introduction to RL

Applications of RL

Further reading

Implementing RL Cycle and OpenAI Gym

Implementing RL Cycle and OpenAI Gym

Setting up the environment

OpenAI Gym and RL cycles

Development of ML models using TensorFlow

Introducing TensorBoard

Types of RL environments

Further reading

Solving Problems with Dynamic Programming

Solving Problems with Dynamic Programming

Categorizing RL algorithms

Dynamic programming

Further reading

Section 2: Model-Free RL Algorithms

Section 2: Model-Free RL Algorithms

Q-Learning and SARSA Applications

Q-Learning and SARSA Applications

Learning without a model

Applying SARSA to Taxi-v2

Applying Q-learning to Taxi-v2

Deep Q-Network

Deep neural networks and Q-learning

DQN applied to Pong

Further reading

Learning Stochastic and PG Optimization

Learning Stochastic and PG Optimization

Policy gradient methods

Understanding the REINFORCE algorithm

REINFORCE with baseline

Learning the AC algorithm

Further reading

TRPO and PPO Implementation

TRPO and PPO Implementation

Natural policy gradient

Trust region policy optimization

Proximal Policy Optimization

Further reading

DDPG and TD3 Applications

DDPG and TD3 Applications

Combining policy gradient optimization with Q-learning

Deep deterministic policy gradient

Twin delayed deep deterministic policy gradient (TD3)

Further reading

Section 3: Beyond Model-Free Algorithms and Improvements

Section 3: Beyond Model-Free Algorithms and Improvements

Model-Based RL

Model-based methods

Combining model-based with model-free learning

ME-TRPO applied to an inverted pendulum

Further reading

Imitation Learning with the DAgger Algorithm

Imitation Learning with the DAgger Algorithm

Technical requirements

The imitation approach

Playing Flappy Bird

Understanding the dataset aggregation algorithm

Further reading

Understanding Black-Box Optimization Algorithms

Understanding Black-Box Optimization Algorithms

The core of EAs

Scalable evolution strategies

Applying scalable ES to LunarLander

Further reading

Developing the ESBAS Algorithm

Developing the ESBAS Algorithm

Exploration versus exploitation

Approaches to exploration

Epochal stochastic bandit algorithm selection

Further reading

Practical Implementation for Resolving RL Challenges

Practical Implementation for Resolving RL Challenges

Best practices of deep RL

Challenges in deep RL

Advanced techniques

RL in the real world

Future of RL and its impact on society

Further reading

Assessments

Other Books You May Enjoy

Other Books You May Enjoy

Leave a review - let other readers know what you think

Customer Reviews

5 star

0

4 star

0

3 star

0

2 star

0

1 star

0

Setting up the environment

The following are the three main tools required to create deep RL algorithms:

Programming language: Python is the first choice for the development of machine learning algorithms on account of its simplicity and the third-party libraries that are built around it.
Deep learning framework: In this book, we use TensorFlow because, as we'll see in the TensorFlow section, it is scalable, flexible, and very expressive. Despite this, many other frameworks can be used in its place, including PyTorch and Caffe.
Environment: Throughout the book, we'll use many different environments to demonstrate how to deal with different types of problems and to highlight the strengths of RL algorithms.

In this book, we use Python 3.7, but all versions above 3.5 should work. We also assume that you've already installed numpy and matplotlib.

If you haven&apos...