Book Image

Deep Reinforcement Learning Hands-On

By : Maxim Lapan
Book Image

Deep Reinforcement Learning Hands-On

By: Maxim Lapan

Overview of this book

Deep Reinforcement Learning Hands-On is a comprehensive guide to the very latest DL tools and their limitations. You will evaluate methods including Cross-entropy and policy gradients, before applying them to real-world environments. Take on both the Atari set of virtual games and family favorites such as Connect4. The book provides an introduction to the basics of RL, giving you the know-how to code intelligent learning agents to take on a formidable array of practical tasks. Discover how to implement Q-learning on 'grid world' environments, teach your agent to buy and trade stocks, and find out how natural language models are driving the boom in chatbots.

Related Content you might be interested in

Current Title:

Small book image
Deep Reinforcement Learning Hands-On
Maxim Lapan
Jun, 2018 | 546 pages
Small book image
Reinforcement Learning Algorithms with Python
Andrea Lonza
Oct, 2019 | 366 pages
Small book image
TensorFlow Reinforcement Learning Quick Start Guide
Kaushik Balakrishnan
Mar, 2019 | 184 pages
Small book image
Hands-On Reinforcement Learning for Games
Micheal Lanham
Jan, 2020 | 432 pages
Table of Contents (23 chapters)
Deep Reinforcement Learning Hands-On
Deep Reinforcement Learning Hands-On
Contributors
Contributors
Preface
Preface
Other Books You May Enjoy
Other Books You May Enjoy
Free Chapter
1
What is Reinforcement Learning?
What is Reinforcement Learning?
Learning – supervised, unsupervised, and reinforcement
RL formalisms and relations
Markov decision processes
Summary
2
OpenAI Gym
OpenAI Gym
The anatomy of the agent
Hardware and software requirements
OpenAI Gym API
The random CartPole agent
The extra Gym functionality – wrappers and monitors
Summary
3
Deep Learning with PyTorch
Deep Learning with PyTorch
Tensors
Gradients
NN building blocks
Custom layers
Final glue – loss functions and optimizers
Monitoring with TensorBoard
Example – GAN on Atari images
Summary
4
The Cross-Entropy Method
The Cross-Entropy Method
Taxonomy of RL methods
Practical cross-entropy
Cross-entropy on CartPole
Cross-entropy on FrozenLake
Theoretical background of the cross-entropy method
Summary
5
Tabular Learning and the Bellman Equation
Tabular Learning and the Bellman Equation
Value, state, and optimality
The Bellman equation of optimality
Value of action
The value iteration method
Value iteration in practice
Q-learning for FrozenLake
Summary
6
Deep Q-Networks
Deep Q-Networks
Real-life value iteration
Tabular Q-learning
Deep Q-learning
DQN on Pong
Summary
7
DQN Extensions
DQN Extensions
The PyTorch Agent Net library
Basic DQN
N-step DQN
Double DQN
Noisy networks
Prioritized replay buffer
Dueling DQN
Categorical DQN
Combining everything
Summary
References
8
Stocks Trading Using RL
Stocks Trading Using RL
Trading
Data
Problem statements and key decisions
The trading environment
Models
Training code
Results
Things to try
Summary
9
Policy Gradients – An Alternative
Policy Gradients – An Alternative
Values and policy
The REINFORCE method
REINFORCE issues
PG on CartPole
PG on Pong
Summary
10
The Actor-Critic Method
The Actor-Critic Method
Variance reduction
CartPole variance
Actor-critic
A2C on Pong
A2C on Pong results
Tuning hyperparameters
Summary
11
Asynchronous Advantage Actor-Critic
Asynchronous Advantage Actor-Critic
Correlation and sample efficiency
Adding an extra A to A2C
Multiprocessing in Python
A3C – data parallelism
A3C – gradients parallelism
Summary
12
Chatbots Training with RL
Chatbots Training with RL
Chatbots overview
Deep NLP basics
Training of seq2seq
The chatbot example
Summary
13
Web Navigation
Web Navigation
Web navigation
OpenAI Universe
Simple clicking approach
Human demonstrations
Adding text description
Things to try
Summary
14
Continuous Action Space
Continuous Action Space
Why a continuous space?
Action space
Environments
The Actor-Critic (A2C) method
Deterministic policy gradients
Distributional policy gradients
Things to try
Summary
15
Trust Regions – TRPO, PPO, and ACKTR
Trust Regions – TRPO, PPO, and ACKTR
Introduction
Roboschool
A2C baseline
Proximal Policy Optimization
Trust Region Policy Optimization
A2C using ACKTR
Summary
16
Black-Box Optimization in RL
Black-Box Optimization in RL
Black-box methods
Evolution strategies
ES on CartPole
ES on HalfCheetah
Genetic algorithms
GA on CartPole
GA tweaks
GA on Cheetah
Summary
References
17
Beyond Model-Free – Imagination
Beyond Model-Free – Imagination
Model-based versus model-free
Model imperfections
Imagination-augmented agent
I2A on Atari Breakout
Experiment results
Summary
References
18
AlphaGo Zero
AlphaGo Zero
Board games
The AlphaGo Zero method
Connect4 bot
Connect4 results
Summary
References
Book summary
Index
Index

Deep Reinforcement Learning Hands-On

Copyright © 2018 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information.

Acquisition Editors: Frank Pohlmann, Suresh Jain

Project Editor: Kishor Rit

Technical Editor: Nidhisha Shetty

Proofreader: Tom Jacob

Indexer: Tejal Daruwale Soni

Graphics: Sandip Tadge

Production Coordinator: Shantanu Zagade

First published: June 2018

Production reference: 1150618

Published by Packt Publishing Ltd.

Livery Place

35 Livery Street

Birmingham B3 2PB, UK.

ISBN 978-1-78883-424-7

www.packtpub.com

mapt.io

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website.

Why subscribe?

  • Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals

  • Learn better with Skill Plans built especially for you

  • Get a free eBook or video every month

  • Mapt is fully searchable

  • Copy and paste, print, and bookmark content

PacktPub.com

Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at  for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

Previous Section
Next Chapter