Book Image

Hands-On Q-Learning with Python

By : Nazia Habib
Book Image

Hands-On Q-Learning with Python

By: Nazia Habib

Overview of this book

Q-learning is a machine learning algorithm used to solve optimization problems in artificial intelligence (AI). It is one of the most popular fields of study among AI researchers. This book starts off by introducing you to reinforcement learning and Q-learning, in addition to helping you become familiar with OpenAI Gym as well as libraries such as Keras and TensorFlow. A few chapters into the book, you will gain insights into model-free Q-learning and use deep Q-networks and double deep Q-networks to solve complex problems. This book will guide you in exploring use cases such as self-driving vehicles and OpenAI Gym’s CartPole problem. You will also learn how to tune and optimize Q-networks and their hyperparameters. As you progress, you will understand the reinforcement learning approach to solving real-world problems. You will also explore how to use Q-learning and related algorithms in scientific research. Toward the end, you’ll gain insight into what’s in store for reinforcement learning. By the end of this book, you will be equipped with the skills you need to solve reinforcement learning problems using Q-learning algorithms with OpenAI Gym, Keras, and TensorFlow.

Related Content you might be interested in

Current Title:

Small book image
Hands-On Q-Learning with Python
Nazia Habib
Apr, 2019 | 212 pages
Small book image
PyTorch 1.x Reinforcement Learning Cookbook
Yuxi Hayden Liu
Oct, 2019 | 340 pages
Small book image
Reinforcement Learning Algorithms with Python
Andrea Lonza
Oct, 2019 | 366 pages
Small book image
Hands-On Reinforcement Learning for Games
Micheal Lanham
Jan, 2020 | 432 pages
Table of Contents (14 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Section 1: Q-Learning: A Roadmap
Section 1: Q-Learning: A Roadmap
2
Brushing Up on Reinforcement Learning Concepts
Brushing Up on Reinforcement Learning Concepts
What is RL?
States, actions, and rewards
Key concepts in RL
SARSA versus Q-learning – on-policy or off?
Summary
Questions
3
Getting Started with the Q-Learning Algorithm
Getting Started with the Q-Learning Algorithm
Technical requirements
Demystifying MDPs
Your Q-learning agent in its environment
Fine-tuning your model – learning, discount, and exploration rates
MABP – a classic exploration versus exploitation problem
Optimal versus safe paths – revisiting SARSA
Summary
Questions
4
Setting Up Your First Environment with OpenAI Gym
Setting Up Your First Environment with OpenAI Gym
Technical requirements
Getting started with OpenAI Gym
Exploring the Taxi-v2 environment
Creating a baseline agent
Summary
Questions
5
Teaching a Smartcab to Drive Using Q-Learning
Teaching a Smartcab to Drive Using Q-Learning
Technical requirements
Getting to know your learning agent
Implementing your agent
The learning parameters – alpha, gamma, and epsilon
Model-tuning and tracking your agent's long-term performance
Summary
Questions
6
Section 2: Building and Optimizing Q-Learning Agents
Section 2: Building and Optimizing Q-Learning Agents
7
Building Q-Networks with TensorFlow
Building Q-Networks with TensorFlow
Technical requirements
A brief overview of neural networks
Taking a closer look
Implementing a neural network with NumPy
Neural networks and Q-learning
Building your first Q-network
Summary
Questions
Further reading
8
Digging Deeper into Deep Q-Networks with Keras and TensorFlow
Digging Deeper into Deep Q-Networks with Keras and TensorFlow
Technical requirements
Introducing CartPole-v1
Getting started with the CartPole task
Building a DQN to solve the CartPole problem
Testing and results
Adding in experience replay
Building further on DQNs
Summary
Questions
Further reading
9
Section 3: Advanced Q-Learning Challenges with Keras, TensorFlow, and OpenAI Gym
Section 3: Advanced Q-Learning Challenges with Keras, TensorFlow, and OpenAI Gym
10
Decoupling Exploration and Exploitation in Multi-Armed Bandits
Decoupling Exploration and Exploitation in Multi-Armed Bandits
Technical requirements
Probability distributions and ongoing knowledge
Revisiting a simple bandit problem
Multi-armed bandit strategy overview
Contextual bandits and state diagrams
Thompson sampling and the Bayesian control rule
Solving a multi-armed bandit problem in Python – user advertisement clicks
Multi-armed bandits in experimental design
Summary
Questions
Further reading
11
Further Q-Learning Research and Future Projects
Further Q-Learning Research and Future Projects
Google's DeepMind and the future of Q-learning
OpenAI Gym and RL research
More OpenAI Gym environments
Contextual bandits and probability distributions
Summary
Questions
Further reading
12
Assessments
Assessments
Chapter 1, Brushing Up on Reinforcement Learning Concepts
Chapter 2, Getting Started with the Q-Learning Algorithm
Chapter 3, Setting Up Your First Environment with OpenAI Gym
Chapter 4, Teaching a Smartcab to Drive Using Q-Learning
Chapter 5, Building Q-Networks with TensorFlow
Chapter 6, Digging Deeper into Deep Q-Networks with Keras and TensorFlow
Chapter 7, Decoupling Exploration and Exploitation in Multi-Armed Bandits
Chapter 8, Further Q-Learning Research and Future Projects
13
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

The learning parameters – alpha, gamma, and epsilon

Here's an updated version of the Bellman equation:

Compare it to the version we used in the last section:

In the new version, we've added in an alpha term, which means we need to include the current Q-value of the state-action pair and discount it by the alpha value.

The first equation is telling us that the new Q-value (the right side of the equation) of our state-action pair is equal to the old Q-value plus the current reward and the discounted future reward, minus the old Q-value multiplied by the alpha term. Because the alpha value is relatively small, more of the current Q-value is incorporated into the new Q-value. In both versions of the equation, because the gamma value is also relatively small, current rewards are valued more highly than future rewards.

Notice that, if the alpha value is 1, the first...

Previous Section
End of Section 5
Next Section