Book Image

Hands-On Q-Learning with Python

By : Nazia Habib
Book Image

Hands-On Q-Learning with Python

By: Nazia Habib

Overview of this book

Q-learning is a machine learning algorithm used to solve optimization problems in artificial intelligence (AI). It is one of the most popular fields of study among AI researchers. This book starts off by introducing you to reinforcement learning and Q-learning, in addition to helping you become familiar with OpenAI Gym as well as libraries such as Keras and TensorFlow. A few chapters into the book, you will gain insights into model-free Q-learning and use deep Q-networks and double deep Q-networks to solve complex problems. This book will guide you in exploring use cases such as self-driving vehicles and OpenAI Gym’s CartPole problem. You will also learn how to tune and optimize Q-networks and their hyperparameters. As you progress, you will understand the reinforcement learning approach to solving real-world problems. You will also explore how to use Q-learning and related algorithms in scientific research. Toward the end, you’ll gain insight into what’s in store for reinforcement learning. By the end of this book, you will be equipped with the skills you need to solve reinforcement learning problems using Q-learning algorithms with OpenAI Gym, Keras, and TensorFlow.

Related Content you might be interested in

Current Title:

Small book image
Hands-On Q-Learning with Python
Nazia Habib
Apr, 2019 | 212 pages
Small book image
PyTorch 1.x Reinforcement Learning Cookbook
Yuxi Hayden Liu
Oct, 2019 | 340 pages
Small book image
Reinforcement Learning Algorithms with Python
Andrea Lonza
Oct, 2019 | 366 pages
Small book image
Hands-On Reinforcement Learning for Games
Micheal Lanham
Jan, 2020 | 432 pages
Table of Contents (14 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Section 1: Q-Learning: A Roadmap
Section 1: Q-Learning: A Roadmap
2
Brushing Up on Reinforcement Learning Concepts
Brushing Up on Reinforcement Learning Concepts
What is RL?
States, actions, and rewards
Key concepts in RL
SARSA versus Q-learning – on-policy or off?
Summary
Questions
3
Getting Started with the Q-Learning Algorithm
Getting Started with the Q-Learning Algorithm
Technical requirements
Demystifying MDPs
Your Q-learning agent in its environment
Fine-tuning your model – learning, discount, and exploration rates
MABP – a classic exploration versus exploitation problem
Optimal versus safe paths – revisiting SARSA
Summary
Questions
4
Setting Up Your First Environment with OpenAI Gym
Setting Up Your First Environment with OpenAI Gym
Technical requirements
Getting started with OpenAI Gym
Exploring the Taxi-v2 environment
Creating a baseline agent
Summary
Questions
5
Teaching a Smartcab to Drive Using Q-Learning
Teaching a Smartcab to Drive Using Q-Learning
Technical requirements
Getting to know your learning agent
Implementing your agent
The learning parameters – alpha, gamma, and epsilon
Model-tuning and tracking your agent's long-term performance
Summary
Questions
6
Section 2: Building and Optimizing Q-Learning Agents
Section 2: Building and Optimizing Q-Learning Agents
7
Building Q-Networks with TensorFlow
Building Q-Networks with TensorFlow
Technical requirements
A brief overview of neural networks
Taking a closer look
Implementing a neural network with NumPy
Neural networks and Q-learning
Building your first Q-network
Summary
Questions
Further reading
8
Digging Deeper into Deep Q-Networks with Keras and TensorFlow
Digging Deeper into Deep Q-Networks with Keras and TensorFlow
Technical requirements
Introducing CartPole-v1
Getting started with the CartPole task
Building a DQN to solve the CartPole problem
Testing and results
Adding in experience replay
Building further on DQNs
Summary
Questions
Further reading
9
Section 3: Advanced Q-Learning Challenges with Keras, TensorFlow, and OpenAI Gym
Section 3: Advanced Q-Learning Challenges with Keras, TensorFlow, and OpenAI Gym
10
Decoupling Exploration and Exploitation in Multi-Armed Bandits
Decoupling Exploration and Exploitation in Multi-Armed Bandits
Technical requirements
Probability distributions and ongoing knowledge
Revisiting a simple bandit problem
Multi-armed bandit strategy overview
Contextual bandits and state diagrams
Thompson sampling and the Bayesian control rule
Solving a multi-armed bandit problem in Python – user advertisement clicks
Multi-armed bandits in experimental design
Summary
Questions
Further reading
11
Further Q-Learning Research and Future Projects
Further Q-Learning Research and Future Projects
Google's DeepMind and the future of Q-learning
OpenAI Gym and RL research
More OpenAI Gym environments
Contextual bandits and probability distributions
Summary
Questions
Further reading
12
Assessments
Assessments
Chapter 1, Brushing Up on Reinforcement Learning Concepts
Chapter 2, Getting Started with the Q-Learning Algorithm
Chapter 3, Setting Up Your First Environment with OpenAI Gym
Chapter 4, Teaching a Smartcab to Drive Using Q-Learning
Chapter 5, Building Q-Networks with TensorFlow
Chapter 6, Digging Deeper into Deep Q-Networks with Keras and TensorFlow
Chapter 7, Decoupling Exploration and Exploitation in Multi-Armed Bandits
Chapter 8, Further Q-Learning Research and Future Projects
13
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Solving a multi-armed bandit problem in Python – user advertisement clicks

In this section we'll be solving a multi-armed bandit problem using a simulated set of ad-click data. We'll generate a set of clicks for 5 different advertisements. Each ad will either be clicked or not clicked when it is shown to a user. Our goal is to determine which ad to show next based on how each ad is performing at any given point in the simulation.

We start with a baseline loop that chooses a random advertisement from the selection each time. This model does not learn from its actions and always chooses a random action. If the user clicks on the ad, we get a reward of 1; if not, we get a reward of 0.

We import the necessary packages and generate a data frame of simulated data using random numbers. We will specify a distribution of 90% zero values and 10% 1 values for this example...

Previous Section
End of Section 8
Next Section