Book Image

PyTorch 1.x Reinforcement Learning Cookbook

By : Yuxi (Hayden) Liu
Book Image

PyTorch 1.x Reinforcement Learning Cookbook

By: Yuxi (Hayden) Liu

Overview of this book

Reinforcement learning (RL) is a branch of machine learning that has gained popularity in recent times. It allows you to train AI models that learn from their own actions and optimize their behavior. PyTorch has also emerged as the preferred tool for training RL models because of its efficiency and ease of use. With this book, you'll explore the important RL concepts and the implementation of algorithms in PyTorch 1.x. The recipes in the book, along with real-world examples, will help you master various RL techniques, such as dynamic programming, Monte Carlo simulations, temporal difference, and Q-learning. You'll also gain insights into industry-specific applications of these techniques. Later chapters will guide you through solving problems such as the multi-armed bandit problem and the cartpole problem using the multi-armed bandit algorithm and function approximation. You'll also learn how to use Deep Q-Networks to complete Atari games, along with how to effectively implement policy gradients. Finally, you'll discover how RL techniques are applied to Blackjack, Gridworld environments, internet advertising, and the Flappy Bird game. By the end of this book, you'll have developed the skills you need to implement popular RL algorithms and use RL techniques to solve real-world problems.

Related Content you might be interested in

Current Title:

Small book image
PyTorch 1.x Reinforcement Learning Cookbook
Yuxi (Hayden) Liu
Oct, 2019 | 340 pages
Small book image
Hands-On Reinforcement Learning with Python
Sudharsan Ravichandiran
Jun, 2018 | 318 pages
Small book image
Python Machine Learning by Example,
Yuxi Hayden Liu
Oct, 2020 | 526 pages
Small book image
Hands-On Q-Learning with Python
Nazia Habib
Apr, 2019 | 212 pages
Table of Contents (11 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Sections
Get in touch
Free Chapter
1
Getting Started with Reinforcement Learning and PyTorch
Getting Started with Reinforcement Learning and PyTorch
Setting up the working environment
Installing OpenAI Gym
Simulating Atari environments
Simulating the CartPole environment
Reviewing the fundamentals of PyTorch
Implementing and evaluating a random search policy
Developing the hill-climbing algorithm
Developing a policy gradient algorithm
2
Markov Decision Processes and Dynamic Programming
Markov Decision Processes and Dynamic Programming
Technical requirements
Creating a Markov chain
Creating an MDP
Performing policy evaluation
Simulating the FrozenLake environment
Solving an MDP with a value iteration algorithm
Solving an MDP with a policy iteration algorithm
Solving the coin-flipping gamble problem
3
Monte Carlo Methods for Making Numerical Estimations
Monte Carlo Methods for Making Numerical Estimations
Calculating Pi using the Monte Carlo method
Performing Monte Carlo policy evaluation
Playing Blackjack with Monte Carlo prediction
Performing on-policy Monte Carlo control
Developing MC control with epsilon-greedy policy
Performing off-policy Monte Carlo control
Developing MC control with weighted importance sampling
4
Temporal Difference and Q-Learning
Temporal Difference and Q-Learning
Setting up the Cliff Walking environment playground
Developing the Q-learning algorithm
Setting up the Windy Gridworld environment playground
Developing the SARSA algorithm
Solving the Taxi problem with Q-learning
Solving the Taxi problem with SARSA
Developing the Double Q-learning algorithm
5
Solving Multi-armed Bandit Problems
Solving Multi-armed Bandit Problems
Creating a multi-armed bandit environment
Solving multi-armed bandit problems with the epsilon-greedy policy
Solving multi-armed bandit problems with the softmax exploration
Solving multi-armed bandit problems with the upper confidence bound algorithm
Solving internet advertising problems with a multi-armed bandit
Solving multi-armed bandit problems with the Thompson sampling algorithm
Solving internet advertising problems with contextual bandits
6
Scaling Up Learning with Function Approximation
Scaling Up Learning with Function Approximation
Setting up the Mountain Car environment playground
Estimating Q-functions with gradient descent approximation
Developing Q-learning with linear function approximation
Developing SARSA with linear function approximation
Incorporating batching using experience replay
Developing Q-learning with neural network function approximation
Solving the CartPole problem with function approximation
7
Deep Q-Networks in Action
Deep Q-Networks in Action
Developing deep Q-networks
Improving DQNs with experience replay
Developing double deep Q-Networks
Tuning double DQN hyperparameters for CartPole
Developing Dueling deep Q-Networks
Applying Deep Q-Networks to Atari games
Using convolutional neural networks for Atari games
8
Implementing Policy Gradients and Policy Optimization
Implementing Policy Gradients and Policy Optimization
Implementing the REINFORCE algorithm
Developing the REINFORCE algorithm with baseline
Implementing the actor-critic algorithm
Solving Cliff Walking with the actor-critic algorithm
Setting up the continuous Mountain Car environment
Solving the continuous Mountain Car environment with the advantage actor-critic network
Playing CartPole through the cross-entropy method
9
Capstone Project – Playing Flappy Bird with DQN
Capstone Project – Playing Flappy Bird with DQN
Setting up the game environment
Building a Deep Q-Network to play Flappy Bird
Training and tuning the network
Deploying the model and playing the game
10
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Solving multi-armed bandit problems with the epsilon-greedy policy

Instead of exploring solely with random policy, we can do better with a combination of exploration and exploitation. Here comes the well-known epsilon-greedy policy.

Epsilon-greedy for multi-armed bandits exploits the best action the majority of the time and also keeps exploring different actions from time to time. Given a parameter, ε, with a value from 0 to 1, the probabilities of performing exploration and exploitation are ε and 1 - ε, respectively:

  • Epsilon: Each action is taken with a probability calculated as follows:

Here, |A| is the number of possible actions.

  • Greedy: The action with the highest state-action value is favored, and its probability of being chosen is increased by 1 - ε:
...
Previous Section
End of Section 3
Next Section