Book Image

TensorFlow Reinforcement Learning Quick Start Guide

By : Kaushik Balakrishnan
Book Image

TensorFlow Reinforcement Learning Quick Start Guide

By: Kaushik Balakrishnan

Overview of this book

Advances in reinforcement learning algorithms have made it possible to use them for optimal control in several different industrial applications. With this book, you will apply Reinforcement Learning to a range of problems, from computer games to autonomous driving. The book starts by introducing you to essential Reinforcement Learning concepts such as agents, environments, rewards, and advantage functions. You will also master the distinctions between on-policy and off-policy algorithms, as well as model-free and model-based algorithms. You will also learn about several Reinforcement Learning algorithms, such as SARSA, Deep Q-Networks (DQN), Deep Deterministic Policy Gradients (DDPG), Asynchronous Advantage Actor-Critic (A3C), Trust Region Policy Optimization (TRPO), and Proximal Policy Optimization (PPO). The book will also show you how to code these algorithms in TensorFlow and Python and apply them to solve computer games from OpenAI Gym. Finally, you will also learn how to train a car to drive autonomously in the Torcs racing car simulator. By the end of the book, you will be able to design, build, train, and evaluate feed-forward neural networks and convolutional neural networks. You will also have mastered coding state-of-the-art algorithms and also training agents for various control problems.

Related Content you might be interested in

Current Title:

Small book image
TensorFlow Reinforcement Learning Quick Start Guide
Kaushik Balakrishnan
Mar, 2019 | 184 pages
Small book image
Reinforcement Learning Algorithms with Python
Andrea Lonza
Oct, 2019 | 366 pages
Small book image
Hands-On Reinforcement Learning with Python
Sudharsan Ravichandiran
Jun, 2018 | 318 pages
Small book image
Hands-On Intelligent Agents with OpenAI Gym
Palanisamy P
Jul, 2018 | 254 pages
Table of Contents (11 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Up and Running with Reinforcement Learning
Up and Running with Reinforcement Learning
Why RL?
The relationship between an agent and its environment
Identifying episodes
Identifying reward functions and the concept of discounted rewards
Learning the Markov decision process
Defining the Bellman equation
On-policy versus off-policy learning
Model-free and model-based training
Algorithms covered in this book
Summary
Questions
Further reading
2
Temporal Difference, SARSA, and Q-Learning
Temporal Difference, SARSA, and Q-Learning
Technical requirements
Understanding TD learning
Understanding SARSA and Q-Learning
Cliff walking and grid world problems
Summary
Further reading
3
Deep Q-Network
Deep Q-Network
Technical requirements
Learning the theory behind a DQN
Understanding target networks
Learning about replay buffer
Getting introduced to the Atari environment
Summary
Questions
Further reading
4
Double DQN, Dueling Architectures, and Rainbow
Double DQN, Dueling Architectures, and Rainbow
Technical requirements
Understanding Double DQN
Understanding dueling network architectures
Understanding Rainbow networks
Running a Rainbow network on Dopamine
Summary
Questions
Further reading
5
Deep Deterministic Policy Gradient
Deep Deterministic Policy Gradient
Technical requirements
Actor-Critic algorithms and policy gradients
Deep Deterministic Policy Gradient
Training and testing the DDPG on Pendulum-v0
Summary
Questions
Further reading
6
Asynchronous Methods - A3C and A2C
Asynchronous Methods - A3C and A2C
Technical requirements
The A3C algorithm
The A3C algorithm applied to CartPole
The A3C algorithm applied to LunarLander
The A2C algorithm
Summary
Questions
Further reading
7
Trust Region Policy Optimization and Proximal Policy Optimization
Trust Region Policy Optimization and Proximal Policy Optimization
Technical requirements
Learning TRPO
Learning PPO
Using PPO to solve the MountainCar problem
Evaluating the performance
Summary
Questions
Further reading
8
Deep RL Applied to Autonomous Driving
Deep RL Applied to Autonomous Driving
Technical requirements
Car driving simulators
Learning to use TORCS
Training a DDPG agent to learn to drive
Training a PPO agent
Summary
Questions
Further reading
9
Assessment
Assessment
Chapter 1
Chapter 3
Chapter 4
Chapter 5
Chapter 6
Chapter 7
Chapter 8
10
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Identifying episodes

We mentioned earlier that the agent explores the environment in numerous trials-and-errors before it can learn to maximize its goals. Each such trial from start to finish is called an episode. The start location may or may not always be from the same location. Likewise, the finish or end of the episode can be a happy or sad ending.

A happy, or good, ending can be when the agent accomplishes its pre-defined goal, which could be successfully navigating to a final destination for a mobile robot, or successfully picking up a peg and placing it in a hole for an industrial robot arm, and so on. Episodes can also have a sad ending, where the agent crashes into obstacles or gets trapped in a maze, unable to get out of it, and so on.

In many RL problems, an upper bound in the form of a fixed number of time steps is generally specified for terminating an episode, although in others, no such bound exists and the episode can last for a very long time, ending with the accomplishment of a goal or by crashing into obstacles or falling off a cliff, or something similar. The Voyager spacecraft was launched by NASA in 1977, and has traveled outside our solar system – this is an example of a system with an infinite time episode.

We will next find out what a reward function is and why we need to discount future rewards. This reward function is the key, as it is the signal for the agent to learn.

Previous Section
End of Section 4
Next Section