Book Image

PyTorch 1.x Reinforcement Learning Cookbook

By : Yuxi (Hayden) Liu

Book Image

PyTorch 1.x Reinforcement Learning Cookbook

By: Yuxi (Hayden) Liu

Overview of this book

Reinforcement learning (RL) is a branch of machine learning that has gained popularity in recent times. It allows you to train AI models that learn from their own actions and optimize their behavior. PyTorch has also emerged as the preferred tool for training RL models because of its efficiency and ease of use. With this book, you'll explore the important RL concepts and the implementation of algorithms in PyTorch 1.x. The recipes in the book, along with real-world examples, will help you master various RL techniques, such as dynamic programming, Monte Carlo simulations, temporal difference, and Q-learning. You'll also gain insights into industry-specific applications of these techniques. Later chapters will guide you through solving problems such as the multi-armed bandit problem and the cartpole problem using the multi-armed bandit algorithm and function approximation. You'll also learn how to use Deep Q-Networks to complete Atari games, along with how to effectively implement policy gradients. Finally, you'll discover how RL techniques are applied to Blackjack, Gridworld environments, internet advertising, and the Flappy Bird game. By the end of this book, you'll have developed the skills you need to implement popular RL algorithms and use RL techniques to solve real-world problems.

Preface

Who this book is for

What this book covers

To get the most out of this book

Free Chapter

Getting Started with Reinforcement Learning and PyTorch

Getting Started with Reinforcement Learning and PyTorch

Setting up the working environment

Installing OpenAI Gym

Simulating Atari environments

Simulating the CartPole environment

Reviewing the fundamentals of PyTorch

Implementing and evaluating a random search policy

Developing the hill-climbing algorithm

Developing a policy gradient algorithm

Markov Decision Processes and Dynamic Programming

Markov Decision Processes and Dynamic Programming

Technical requirements

Creating a Markov chain

Creating an MDP

Performing policy evaluation

Simulating the FrozenLake environment

Solving an MDP with a value iteration algorithm

Solving an MDP with a policy iteration algorithm

Solving the coin-flipping gamble problem

Monte Carlo Methods for Making Numerical Estimations

Monte Carlo Methods for Making Numerical Estimations

Calculating Pi using the Monte Carlo method

Performing Monte Carlo policy evaluation

Playing Blackjack with Monte Carlo prediction

Performing on-policy Monte Carlo control

Developing MC control with epsilon-greedy policy

Performing off-policy Monte Carlo control

Developing MC control with weighted importance sampling

Temporal Difference and Q-Learning

Temporal Difference and Q-Learning

Setting up the Cliff Walking environment playground

Developing the Q-learning algorithm

Setting up the Windy Gridworld environment playground

Developing the SARSA algorithm

Solving the Taxi problem with Q-learning

Solving the Taxi problem with SARSA

Developing the Double Q-learning algorithm

Solving Multi-armed Bandit Problems

Solving Multi-armed Bandit Problems

Creating a multi-armed bandit environment

Solving multi-armed bandit problems with the epsilon-greedy policy

Solving multi-armed bandit problems with the softmax exploration

Solving multi-armed bandit problems with the upper confidence bound algorithm

Solving internet advertising problems with a multi-armed bandit

Solving multi-armed bandit problems with the Thompson sampling algorithm

Solving internet advertising problems with contextual bandits

Scaling Up Learning with Function Approximation

Scaling Up Learning with Function Approximation

Setting up the Mountain Car environment playground

Estimating Q-functions with gradient descent approximation

Developing Q-learning with linear function approximation

Developing SARSA with linear function approximation

Incorporating batching using experience replay

Developing Q-learning with neural network function approximation

Solving the CartPole problem with function approximation

Deep Q-Networks in Action

Deep Q-Networks in Action

Developing deep Q-networks

Improving DQNs with experience replay

Developing double deep Q-Networks

Tuning double DQN hyperparameters for CartPole

Developing Dueling deep Q-Networks

Applying Deep Q-Networks to Atari games

Using convolutional neural networks for Atari games

Implementing Policy Gradients and Policy Optimization

Implementing Policy Gradients and Policy Optimization

Implementing the REINFORCE algorithm

Developing the REINFORCE algorithm with baseline

Implementing the actor-critic algorithm

Solving Cliff Walking with the actor-critic algorithm

Setting up the continuous Mountain Car environment

Solving the continuous Mountain Car environment with the advantage actor-critic network

Playing CartPole through the cross-entropy method

Capstone Project – Playing Flappy Bird with DQN

Capstone Project – Playing Flappy Bird with DQN

Setting up the game environment

Building a Deep Q-Network to play Flappy Bird

Training and tuning the network

Deploying the model and playing the game

Other Books You May Enjoy

Other Books You May Enjoy

Leave a review - let other readers know what you think

Customer Reviews

5 star

0

4 star

0

3 star

0

2 star

0

1 star

0

Developing SARSA with linear function approximation

We've just solved the Mountain Car problem using the off-policy Q-learning algorithm in the previous recipe. Now, we will do so with the on-policy State-Action-Reward-State-Action (SARSA) algorithm (the FA version of course).

In general, the SARSA algorithm updates the Q-function based on the following equation:

Here, s' is the resulting state after taking action, a, in state s; r is the associated reward; α is the learning rate; and γ is the discount factor. We simply pick up the next action, a', by also following an epsilon-greedy policy to update the Q value. And the action, a', is taken in the next step. Accordingly, SARSA with FA has the following error term:

Our learning goal is to minimize the error term to zero, which means that the estimated V(st) should satisfy the following equation...