Book Image

The Reinforcement Learning Workshop

By : Alessandro Palmas, Emanuele Ghelfi, Dr. Alexandra Galina Petre, Mayur Kulkarni, Anand N.S., Quan Nguyen, Aritra Sen, Anthony So, Saikat Basak
Book Image

The Reinforcement Learning Workshop

By: Alessandro Palmas, Emanuele Ghelfi, Dr. Alexandra Galina Petre, Mayur Kulkarni, Anand N.S., Quan Nguyen, Aritra Sen, Anthony So, Saikat Basak

Overview of this book

Various intelligent applications such as video games, inventory management software, warehouse robots, and translation tools use reinforcement learning (RL) to make decisions and perform actions that maximize the probability of the desired outcome. This book will help you to get to grips with the techniques and the algorithms for implementing RL in your machine learning models. Starting with an introduction to RL, youÔÇÖll be guided through different RL environments and frameworks. YouÔÇÖll learn how to implement your own custom environments and use OpenAI baselines to run RL algorithms. Once youÔÇÖve explored classic RL techniques such as Dynamic Programming, Monte Carlo, and TD Learning, youÔÇÖll understand when to apply the different deep learning methods in RL and advance to deep Q-learning. The book will even help you understand the different stages of machine-based problem-solving by using DARQN on a popular video game Breakout. Finally, youÔÇÖll find out when to use a policy-based method to tackle an RL problem. By the end of The Reinforcement Learning Workshop, youÔÇÖll be equipped with the knowledge and skills needed to solve challenging problems using reinforcement learning.

Related Content you might be interested in

Current Title:

Small book image
The Reinforcement Learning Workshop
Alessandro Palmas | Emanuele Ghelfi | Dr. Alexandra Galina Petre | Mayur Kulkarni | Anand N.S. | Quan Nguyen | Aritra Sen | Anthony So | Saikat Basak
Aug, 2020 | 822 pages
Small book image
Hands-On Reinforcement Learning with Python
Sudharsan Ravichandiran
Jun, 2018 | 318 pages
Small book image
Reinforcement Learning Algorithms with Python
Andrea Lonza
Oct, 2019 | 366 pages
Small book image
PyTorch 1.x Reinforcement Learning Cookbook
Yuxi Hayden Liu
Oct, 2019 | 340 pages
Table of Contents (14 chapters)
Preface
About the Book
1
1. Introduction to Reinforcement Learning
1. Introduction to Reinforcement Learning
Introduction
Learning Paradigms
Fundamentals of Reinforcement Learning
Reinforcement Learning Frameworks
Applications of Reinforcement Learning
Summary
Free Chapter
2
2. Markov Decision Processes and Bellman Equations
2. Markov Decision Processes and Bellman Equations
Introduction
Markov Processes
3
3. Deep Learning in Practice with TensorFlow 2
3. Deep Learning in Practice with TensorFlow 2
Introduction
An Introduction to TensorFlow and Keras
How to Implement a Neural Network Using TensorFlow
Simple Regression Using TensorFlow
Simple Classification Using TensorFlow
TensorBoard – How to Visualize Data Using TensorBoard
Summary
4
4. Getting Started with OpenAI and TensorFlow for Reinforcement Learning
4. Getting Started with OpenAI and TensorFlow for Reinforcement Learning
Introduction
OpenAI Gym
OpenAI Universe – Complex Environment
TensorFlow for Reinforcement Learning
OpenAI Baselines
Training an RL Agent to Solve a Classic Control Problem
Summary
5
5. Dynamic Programming
5. Dynamic Programming
Introduction
Solving Dynamic Programming Problems
Identifying Dynamic Programming Problems
Dynamic Programming in RL
Summary
6
6. Monte Carlo Methods
6. Monte Carlo Methods
Introduction
The Workings of Monte Carlo Methods
Understanding Monte Carlo with Blackjack
Types of Monte Carlo Methods
Exploration versus Exploitation Trade-Off
Importance Sampling
Solving Frozen Lake Using Monte Carlo
Summary
7
7. Temporal Difference Learning
7. Temporal Difference Learning
Introduction to TD Learning
TD(0) – SARSA and Q-Learning
N-Step TD and TD(λ) Algorithms
The Relationship between DP, Monte-Carlo, and TD Learning
Summary
8
8. The Multi-Armed Bandit Problem
8. The Multi-Armed Bandit Problem
Introduction
Formulation of the MAB Problem
The Python Interface
The Greedy Algorithm
The Explore-then-Commit Algorithm
The ε-Greedy Algorithm
The UCB algorithm
Thompson Sampling
Contextual Bandits
Summary
9
9. What Is Deep Q-Learning?
9. What Is Deep Q-Learning?
Introduction
Basics of Deep Learning
Basics of PyTorch
The Action-Value Function (Q Value Function)
Deep Q Learning
Challenges in DQN
Summary
10
10. Playing an Atari Game with Deep Recurrent Q-Networks
10. Playing an Atari Game with Deep Recurrent Q-Networks
Introduction
Understanding the Breakout Environment
CNNs in TensorFlow
Combining a DQN with a CNN
RNNs in TensorFlow
Building a DRQN
Introduction to the Attention Mechanism and DARQN
Summary
11
11. Policy-Based Methods for Reinforcement Learning
11. Policy-Based Methods for Reinforcement Learning
Introduction
Policy Gradients
Deep Deterministic Policy Gradients
Improving Policy Gradients
Summary
12
12. Evolutionary Strategies for RL
12. Evolutionary Strategies for RL
Introduction
Problems with Gradient-Based Methods
Introduction to Genetic Algorithms
Summary
Appendix
1. Introduction to Reinforcement Learning
2. Markov Decision Processes and Bellman Equations
3. Deep Learning in Practice with TensorFlow 2
4. Getting started with OpenAI and TensorFlow for Reinforcement Learning
5. Dynamic Programming
6. Monte Carlo Methods
7. Temporal Difference Learning
8. The Multi-Armed Bandit Problem
9. What Is Deep Q-Learning?
10. Playing an Atari Game with Deep Recurrent Q-Networks
11. Policy-Based Methods for Reinforcement Learning
12. Evolutionary Strategies for RL

Introduction

In the previous chapter, we studied the main elements of Reinforcement Learning (RL). We described an agent as an entity that can perceive an environment's state and act by modifying the environment state in order to achieve a goal. An agent acts through a policy that represents its behavior, and the way the agent selects an action is based on the environment state. In the second half of the previous chapter, we introduced Gym and Baselines, two Python libraries that simplify the environment representation and the algorithm implementation, respectively.

We mentioned that RL considers problems as Markov Decision Processes (MDPs), without entering into the details and without giving a formal definition.

In this chapter, we will formally describe what an MDP is, its properties, and its characteristics. When facing a new problem in RL, we have to ensure that the problem can be formalized as an MDP; otherwise, applying RL techniques is impossible.

Before presenting a formal definition of MDPs, we need to understand Markov Chains (MCs) and Markov Reward Processes (MRPs). MCs and MRPs are specific cases (simplified) of MDPs. An MC only focuses on state transitions without modeling rewards and actions. Consider the example of the game of snakes and ladders, where the next action is completely dependent on the number displayed on the dice. MRPs also include the reward component in the state transition. MRPs and MCs are useful in understanding the characteristics of MDPs gradually. We will be looking at specific examples of MCs and MRPs later in the chapter.

Along with MDPs, this chapter also presents the concepts of the state-value function and the action-value function, which are used to evaluate how good a state is for an agent and how good an action taken in a given state is. State-value functions and action-value functions are the building blocks of the algorithms used to solve real-world problems. The concepts of state-value functions and action-value functions are highly related to the agent's policy and the environment dynamics, as we will learn later in this chapter.

The final part of this chapter presents two Bellman equations, namely the Bellman expectation equation and the Bellman optimality equation. These equations are helpful in the context of RL in order to evaluate the behavior of an agent and find a policy that maximizes the agent's performance in an MDP.

In this chapter, we will practice with some MDP examples, such as the student MDP and Gridworld. We will implement the solution methods and equations explained in this chapter using Python, SciPy, and NumPy.

Previous Section
End of Section 2
Next Section