Book Image

The Reinforcement Learning Workshop

By : Alessandro Palmas, Emanuele Ghelfi, Dr. Alexandra Galina Petre, Mayur Kulkarni, Anand N.S., Quan Nguyen, Aritra Sen, Anthony So, Saikat Basak
Book Image

The Reinforcement Learning Workshop

By: Alessandro Palmas, Emanuele Ghelfi, Dr. Alexandra Galina Petre, Mayur Kulkarni, Anand N.S., Quan Nguyen, Aritra Sen, Anthony So, Saikat Basak

Overview of this book

Various intelligent applications such as video games, inventory management software, warehouse robots, and translation tools use reinforcement learning (RL) to make decisions and perform actions that maximize the probability of the desired outcome. This book will help you to get to grips with the techniques and the algorithms for implementing RL in your machine learning models. Starting with an introduction to RL, youÔÇÖll be guided through different RL environments and frameworks. YouÔÇÖll learn how to implement your own custom environments and use OpenAI baselines to run RL algorithms. Once youÔÇÖve explored classic RL techniques such as Dynamic Programming, Monte Carlo, and TD Learning, youÔÇÖll understand when to apply the different deep learning methods in RL and advance to deep Q-learning. The book will even help you understand the different stages of machine-based problem-solving by using DARQN on a popular video game Breakout. Finally, youÔÇÖll find out when to use a policy-based method to tackle an RL problem. By the end of The Reinforcement Learning Workshop, youÔÇÖll be equipped with the knowledge and skills needed to solve challenging problems using reinforcement learning.

Related Content you might be interested in

Current Title:

Small book image
The Reinforcement Learning Workshop
Alessandro Palmas | Emanuele Ghelfi | Dr. Alexandra Galina Petre | Mayur Kulkarni | Anand N.S. | Quan Nguyen | Aritra Sen | Anthony So | Saikat Basak
Aug, 2020 | 822 pages
Small book image
Hands-On Reinforcement Learning with Python
Sudharsan Ravichandiran
Jun, 2018 | 318 pages
Small book image
Reinforcement Learning Algorithms with Python
Andrea Lonza
Oct, 2019 | 366 pages
Small book image
PyTorch 1.x Reinforcement Learning Cookbook
Yuxi Hayden Liu
Oct, 2019 | 340 pages
Table of Contents (14 chapters)
Preface
About the Book
1
1. Introduction to Reinforcement Learning
1. Introduction to Reinforcement Learning
Introduction
Learning Paradigms
Fundamentals of Reinforcement Learning
Reinforcement Learning Frameworks
Applications of Reinforcement Learning
Summary
Free Chapter
2
2. Markov Decision Processes and Bellman Equations
2. Markov Decision Processes and Bellman Equations
Introduction
Markov Processes
3
3. Deep Learning in Practice with TensorFlow 2
3. Deep Learning in Practice with TensorFlow 2
Introduction
An Introduction to TensorFlow and Keras
How to Implement a Neural Network Using TensorFlow
Simple Regression Using TensorFlow
Simple Classification Using TensorFlow
TensorBoard – How to Visualize Data Using TensorBoard
Summary
4
4. Getting Started with OpenAI and TensorFlow for Reinforcement Learning
4. Getting Started with OpenAI and TensorFlow for Reinforcement Learning
Introduction
OpenAI Gym
OpenAI Universe – Complex Environment
TensorFlow for Reinforcement Learning
OpenAI Baselines
Training an RL Agent to Solve a Classic Control Problem
Summary
5
5. Dynamic Programming
5. Dynamic Programming
Introduction
Solving Dynamic Programming Problems
Identifying Dynamic Programming Problems
Dynamic Programming in RL
Summary
6
6. Monte Carlo Methods
6. Monte Carlo Methods
Introduction
The Workings of Monte Carlo Methods
Understanding Monte Carlo with Blackjack
Types of Monte Carlo Methods
Exploration versus Exploitation Trade-Off
Importance Sampling
Solving Frozen Lake Using Monte Carlo
Summary
7
7. Temporal Difference Learning
7. Temporal Difference Learning
Introduction to TD Learning
TD(0) – SARSA and Q-Learning
N-Step TD and TD(λ) Algorithms
The Relationship between DP, Monte-Carlo, and TD Learning
Summary
8
8. The Multi-Armed Bandit Problem
8. The Multi-Armed Bandit Problem
Introduction
Formulation of the MAB Problem
The Python Interface
The Greedy Algorithm
The Explore-then-Commit Algorithm
The ε-Greedy Algorithm
The UCB algorithm
Thompson Sampling
Contextual Bandits
Summary
9
9. What Is Deep Q-Learning?
9. What Is Deep Q-Learning?
Introduction
Basics of Deep Learning
Basics of PyTorch
The Action-Value Function (Q Value Function)
Deep Q Learning
Challenges in DQN
Summary
10
10. Playing an Atari Game with Deep Recurrent Q-Networks
10. Playing an Atari Game with Deep Recurrent Q-Networks
Introduction
Understanding the Breakout Environment
CNNs in TensorFlow
Combining a DQN with a CNN
RNNs in TensorFlow
Building a DRQN
Introduction to the Attention Mechanism and DARQN
Summary
11
11. Policy-Based Methods for Reinforcement Learning
11. Policy-Based Methods for Reinforcement Learning
Introduction
Policy Gradients
Deep Deterministic Policy Gradients
Improving Policy Gradients
Summary
12
12. Evolutionary Strategies for RL
12. Evolutionary Strategies for RL
Introduction
Problems with Gradient-Based Methods
Introduction to Genetic Algorithms
Summary
Appendix
1. Introduction to Reinforcement Learning
2. Markov Decision Processes and Bellman Equations
3. Deep Learning in Practice with TensorFlow 2
4. Getting started with OpenAI and TensorFlow for Reinforcement Learning
5. Dynamic Programming
6. Monte Carlo Methods
7. Temporal Difference Learning
8. The Multi-Armed Bandit Problem
9. What Is Deep Q-Learning?
10. Playing an Atari Game with Deep Recurrent Q-Networks
11. Policy-Based Methods for Reinforcement Learning
12. Evolutionary Strategies for RL

Deep Deterministic Policy Gradients

In this section, we will apply the DDPG technique to understand the continuous action space. Moreover, we will learn how to code a moon lander simulation to understand DDPGs.

Note

We suggest that you type all the code given in this section into your Jupyter notebook as we will be using it later, in Exercise 11.02, Creating a Learning Agent.

We are going to use the OpenAI Gym Lunar Lander environment for continuous action spaces here. Let's start by importing the essentials:

import os
import gym
import torch as T
import numpy as np

Now, we will learn how to define some classes, such as the OUActionNoise class, the ReplayBuffer class, the ActorNetwork class, and the CriticNetwork class, which will help us to implement the DDGP technique. At the end of this section, you'll have the complete code base that applies the DDPG within our OpenAI Gym game environment.

Ornstein-Uhlenbeck Noise

First, we will define...

Previous Section
End of Section 4
Next Section