Book Image

Hands-On Reinforcement Learning for Games

By : Micheal Lanham
Book Image

Hands-On Reinforcement Learning for Games

By: Micheal Lanham

Overview of this book

With the increased presence of AI in the gaming industry, developers are challenged to create highly responsive and adaptive games by integrating artificial intelligence into their projects. This book is your guide to learning how various reinforcement learning techniques and algorithms play an important role in game development with Python. Starting with the basics, this book will help you build a strong foundation in reinforcement learning for game development. Each chapter will assist you in implementing different reinforcement learning techniques, such as Markov decision processes (MDPs), Q-learning, actor-critic methods, SARSA, and deterministic policy gradient algorithms, to build logical self-learning agents. Learning these techniques will enhance your game development skills and add a variety of features to improve your game agent’s productivity. As you advance, you’ll understand how deep reinforcement learning (DRL) techniques can be used to devise strategies to help agents learn from their actions and build engaging games. By the end of this book, you’ll be ready to apply reinforcement learning techniques to build a variety of projects and contribute to open source applications.

Related Content you might be interested in

Current Title:

Small book image
Hands-On Reinforcement Learning for Games
Micheal Lanham
Jan, 2020 | 432 pages
Small book image
Hands-On Deep Learning for Games
Micheal Lanham
Mar, 2019 | 392 pages
Small book image
Learn Unity ML-Agents - Fundamentals of Unity Machine Learning
Micheal Lanham
Jun, 2018 | 204 pages
Small book image
Hands-On Reinforcement Learning with Python
Sudharsan Ravichandiran
Jun, 2018 | 318 pages
Table of Contents (19 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
1
Section 1: Exploring the Environment
Section 1: Exploring the Environment
Free Chapter
2
Understanding Rewards-Based Learning
Understanding Rewards-Based Learning
Technical requirements
Understanding rewards-based learning
Introducing the Markov decision process
Using value learning with multi-armed bandits
Exploring Q-learning with contextual bandits
Summary
Questions
3
Dynamic Programming and the Bellman Equation
Dynamic Programming and the Bellman Equation
Introducing DP
Understanding the Bellman equation
Building policy iteration
Building value iteration
Playing with policy versus value iteration
Exercises
Summary
4
Monte Carlo Methods
Monte Carlo Methods
Understanding model-based and model-free learning
Introducing the Monte Carlo method
Adding RL
Playing the FrozenLake game
Using prediction and control
Exercises
Summary
5
Temporal Difference Learning
Temporal Difference Learning
Understanding the TCA problem
Introducing TDL
Applying TDL to Q-learning
Exploring TD(0) in Q-learning
Running off- versus on-policy
Exercises
Summary
6
Exploring SARSA
Exploring SARSA
Exploring SARSA on-policy learning
Using continuous spaces with SARSA
Extending continuous spaces
Working with TD (λ) and eligibility traces
Understanding SARSA (λ)
Exercises
Summary
7
Section 2: Exploiting the Knowledge
Section 2: Exploiting the Knowledge
8
Going Deep with DQN
Going Deep with DQN
DL for RL
Using PyTorch for DL
Building neural networks with Torch
Understanding DQN in PyTorch
Exercising DQN
Exercises
Summary
9
Going Deeper with DDQN
Going Deeper with DDQN
Understanding visual state
Introducing CNNs
Working with a DQN on Atari
Introducing DDQN
Extending replay with prioritized experience replay
Exercises
Summary
10
Policy Gradient Methods
Policy Gradient Methods
Understanding policy gradient methods
Introducing REINFORCE
Using advantage actor-critic
Building a deep deterministic policy gradient
Exploring trust region policy optimization
Exercises
Summary
11
Optimizing for Continuous Control
Optimizing for Continuous Control
Understanding continuous control with Mujoco
Introducing proximal policy optimization
Using PPO with recurrent networks
Deciding on synchronous and asynchronous actors
Building actor-critic with experience replay
Exercises
Summary
12
All about Rainbow DQN
All about Rainbow DQN
Rainbow – combining improvements in deep reinforcement learning
Using TensorBoard
Introducing distributional RL
Understanding noisy networks
Unveiling Rainbow DQN
Exercises
Summary
13
Exploiting ML-Agents
Exploiting ML-Agents
Installing ML-Agents
Building a Unity environment
Training a Unity environment with Rainbow
Creating a new environment
Advancing RL with ML-Agents
Exercises
Summary
14
DRL Frameworks
DRL Frameworks
Choosing a framework
Introducing Google Dopamine
Playing with Keras-RL
Exploring RL Lib
Using TF-Agents
Exercises
Summary
15
Section 3: Reward Yourself
Section 3: Reward Yourself
16
3D Worlds
3D Worlds
Reasoning on 3D worlds
Training a visual agent
Generalizing 3D vision
Challenging the Unity Obstacle Tower Challenge
Exploring Habitat – embodied agents by FAIR
Exercises
Summary
17
From DRL to AGI
From DRL to AGI
Learning meta learning
Introducing meta reinforcement learning
Using hindsight experience replay
Imagination and reasoning in RL
Understanding imagination-augmented agents
Exercises
Summary
18
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

To get the most out of this book

A working knowledge of Python and game development is essential. A good PC with a GPU would be beneficial.

Download the example code files

You can download the example code files for this book from your account at www.packt.com. If you purchased this book elsewhere, you can visit www.packtpub.com/support and register to have the files emailed directly to you.

You can download the code files by following these steps:

  1. Log in or register at www.packt.com.
  2. Select the Support tab.
  3. Click on Code Downloads.
  4. Enter the name of the book in the Search box and follow the onscreen instructions.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

  • WinRAR/7-Zip for Windows
  • Zipeg/iZip/UnRarX for Mac
  • 7-Zip/PeaZip for Linux

The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Hands-On-Reinforcement-Learning-for-Games. In case there's an update to the code, it will be updated on the existing GitHub repository.

We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out!

Download the color images

Conventions used

There are a number of text conventions used throughout this book.

CodeInText: Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "The three functions make_atari, wrap_deepmind, and wrap_pytorch are all located in the new wrappers.py file we imported earlier."

A block of code is set as follows:

env_id = 'PongNoFrameskip-v4'
env = make_atari(env_id)
env = wrap_deepmind(env)
env = wrap_pytorch(env)

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:

epsilon_start = 1.0
epsilon_final = 0.01
epsilon_decay = 30000

epsilon_by_episode = lambda episode: epsilon_final + (epsilon_start - epsilon_final) * math.exp(-1. * episode / epsilon_decay)

plt.plot([epsilon_by_episode(i) for i in range(1000000)])
plt.show()

Any command-line input or output is written as follows:

pip install mujoco

Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "Building on that, we'll look at a variant of the DQN called the DDQN, or double (dueling) DQN."

Warnings or important notes appear like this.
Tips and tricks appear like this.
Previous Section
Next Section