Book Image

Mastering Reinforcement Learning with Python

By : Enes Bilgin
Book Image

Mastering Reinforcement Learning with Python

By: Enes Bilgin

Overview of this book

Reinforcement learning (RL) is a field of artificial intelligence (AI) used for creating self-learning autonomous agents. Building on a strong theoretical foundation, this book takes a practical approach and uses examples inspired by real-world industry problems to teach you about state-of-the-art RL. Starting with bandit problems, Markov decision processes, and dynamic programming, the book provides an in-depth review of the classical RL techniques, such as Monte Carlo methods and temporal-difference learning. After that, you will learn about deep Q-learning, policy gradient algorithms, actor-critic methods, model-based methods, and multi-agent reinforcement learning. Then, you'll be introduced to some of the key approaches behind the most successful RL implementations, such as domain randomization and curiosity-driven learning. As you advance, you’ll explore many novel algorithms with advanced implementations using modern Python libraries such as TensorFlow and Ray’s RLlib package. You’ll also find out how to implement RL in areas such as robotics, supply chain management, marketing, finance, smart cities, and cybersecurity while assessing the trade-offs between different approaches and avoiding common pitfalls. By the end of this book, you’ll have mastered how to train and deploy your own RL agents for solving RL problems.

Related Content you might be interested in

Current Title:

Small book image
Mastering Reinforcement Learning with Python
Enes Bilgin
Dec, 2020 | 544 pages
Small book image
Reinforcement Learning Algorithms with Python
Andrea Lonza
Oct, 2019 | 366 pages
Small book image
Hands-On Intelligent Agents with OpenAI Gym
Palanisamy P
Jul, 2018 | 254 pages
Small book image
TensorFlow Reinforcement Learning Quick Start Guide
Kaushik Balakrishnan
Mar, 2019 | 184 pages
Table of Contents (24 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Reviews
1
Section 1: Reinforcement Learning Foundations
Section 1: Reinforcement Learning Foundations
Free Chapter
2
Chapter 1: Introduction to Reinforcement Learning
Chapter 1: Introduction to Reinforcement Learning
Why reinforcement learning?
The three paradigms of ML
RL application areas and success stories
Elements of a RL problem
Setting up your RL environment
Summary
References
3
Chapter 2: Multi-Armed Bandits
Chapter 2: Multi-Armed Bandits
Exploration-Exploitation Trade-Off
What is a MAB?
Case study: Online advertising
A/B/n testing
ε-greedy actions
Action selection using upper confidence bounds
Thompson (Posterior) sampling
Summary
References
4
Chapter 3: Contextual Bandits
Chapter 3: Contextual Bandits
Why we need function approximations
Using function approximation for context
Using function approximation for action
Other applications of multi-armed and contextual bandits
Summary
References
5
Chapter 4: Makings of a Markov Decision Process
Chapter 4: Makings of a Markov Decision Process
Starting with Markov chains
Introducing the reward: Markov reward process
Bringing the action in: Markov decision process
Partially observable Markov decision process
Summary
Exercises
References
6
Chapter 5: Solving the Reinforcement Learning Problem
Chapter 5: Solving the Reinforcement Learning Problem
Exploring dynamic programming
Training your agent with Monte Carlo methods
Temporal-difference learning
Understanding the importance of the simulation in reinforcement learning
Summary
References
7
Section 2: Deep Reinforcement Learning
Section 2: Deep Reinforcement Learning
8
Chapter 6: Deep Q-Learning at Scale
Chapter 6: Deep Q-Learning at Scale
From tabular Q-learning to deep Q-learning
Deep Q-networks
Extensions to DQN: Rainbow
Distributed deep Q-learning
Implementing scalable deep Q-learning algorithms using Ray
RLlib: Production-grade deep reinforcement learning
Summary
References
9
Chapter 7: Policy-Based Methods
Chapter 7: Policy-Based Methods
Need for policy-based methods
Vanilla policy gradient
Actor-critic methods
Trust-region methods
Revisiting off-policy Methods
Comparison of the policy-based methods in Lunar Lander
How to pick the right algorithm?
Open source implementations of policy-gradient methods
Summary
References
10
Chapter 8: Model-Based Methods
Chapter 8: Model-Based Methods
Introducing model-based methods
Planning through a model
Learning a world model
Unifying model-based and model-free approaches
Summary
References
11
Chapter 9: Multi-Agent Reinforcement Learning
Chapter 9: Multi-Agent Reinforcement Learning
Introducing multi-agent reinforcement learning
Exploring the challenges in multi-agent reinforcement learning
Training policies in multi-agent settings
Training tic-tac-toe agents through self-play
Summary
References
12
Section 3: Advanced Topics in RL
Section 3: Advanced Topics in RL
13
Chapter 10: Introducing Machine Teaching
Chapter 10: Introducing Machine Teaching
Introduction to machine teaching
Engineering the reward function
Curriculum learning
Warm starts with demonstrations
Action masking
Summary
References
14
Chapter 11: Achieving Generalization and Overcoming Partial Observability
Chapter 11: Achieving Generalization and Overcoming Partial Observability
Focusing on generalization in reinforcement learning
Enriching agent experience via domain randomization
Using memory to overcome partial observability
Quantifying generalization via CoinRun
Summary
References
15
Chapter 12: Meta-Reinforcement Learning
Chapter 12: Meta-Reinforcement Learning
Introducing meta-reinforcement learning
Meta-reinforcement learning with recurrent policies
Gradient-based meta-reinforcement learning
Meta-reinforcement learning as partially observed reinforcement learning
Challenges in meta-reinforcement learning
Conclusion
References
16
Chapter 13: Exploring Advanced Topics
Chapter 13: Exploring Advanced Topics
Diving deeper into distributed reinforcement learning
Exploring curiosity-driven reinforcement learning
Offline reinforcement learning
Summary
References
17
Section 4: Applications of RL
Section 4: Applications of RL
18
Chapter 14: Solving Robot Learning
Chapter 14: Solving Robot Learning
Introducing PyBullet
Getting familiar with the Kuka environment
Developing strategies to solve the Kuka environment
Using curriculum learning to train the Kuka robot
Going beyond PyBullet into autonomous driving
Summary
References
19
Chapter 15: Supply Chain Management
Chapter 15: Supply Chain Management
Optimizing inventory procurement decisions
Modeling routing problems
Summary
References
20
Chapter 16: Personalization, Marketing, and Finance
Chapter 16: Personalization, Marketing, and Finance
Going beyond bandits for personalization
Developing effective marketing strategies using reinforcement learning
Applying reinforcement learning in finance
Summary
References
21
Chapter 17: Smart City and Cybersecurity
Chapter 17: Smart City and Cybersecurity
Controlling traffic lights to optimize vehicle flow
Providing ancillary service to power grid
Detecting cyberattacks in a smart grid
Summary
References
22
Chapter 18: Challenges and Future Directions in Reinforcement Learning
Chapter 18: Challenges and Future Directions in Reinforcement Learning
What you have achieved with this book
Challenges and future directions
Suggestions for aspiring reinforcement learning experts
Final words
References
23
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Why reinforcement learning?

Creating intelligent machines that make decisions at or superior to human level is a dream of many scientist and engineers, and one which is gradually becoming closer to reality. In the seven decades since the Turing test, AI research and development has been on a roller coaster. The expectations were very high initially: In the 1960s, for example, Herbert Simon (who later received the Nobel Prize in Economics) predicted that machines would be capable of doing any work humans can do within twenty years. It was this excitement that attracted big government and corporate funding flowing into AI research, only to be followed by big disappointments and a period called the "AI winter." Decades later, thanks to the incredible developments in computing, data, and algorithms, humankind is again very excited, more than ever before, in its pursuit of the AI dream. 

Note

If you're not familiar with Alan Turing's instrumental work on the foundations of AI in 1950, it's worth learning more about the Turing Test here: https://youtu.be/3wLqsRLvV-c

The AI dream is certainly one of grandiosity. After all, the potential in intelligent autonomous systems is enormous. Think about how we are limited in terms of specialist medical doctors in the world. It takes years and significant intellectual and financial resources to educate them, which many countries don't have at sufficient levels. In addition, even after years of education, it is nearly impossible for a specialist to stay up-to-date with all of the scientific developments in her field, learn from the outcomes of the tens of thousands of treatments around the world, and effectively incorporate all this knowledge into practice.

Conversely, an AI model could process and learn from all this data and combine it with a rich set of information about a patient (medical history, lab results, presenting symptoms, health profile) to make diagnosis and suggest treatments. Such a model could serve even in the most rural parts of the world (as far as an internet connection and computer are available) and direct the local health personnel about the treatment. No doubt that it would revolutionize international healthcare and improve the lives of millions of people.

Note

AI is already transforming the healthcare industry. In a recent article, Google published results from an AI system surpassing human experts in breast cancer prediction using mammography readings (McKinney et al. 2020). Microsoft is collaborating with one of India's largest healthcare providers to detect cardiac illnesses using AI (Agrawal, 2018). IBM Watson for Clinical Trial Matching uses natural language processing to recommend potential treatments for patients from medical databases (https://youtu.be/grDWR7hMQQQ).

On our quest to develop AI systems that are at or superior to human level, which is -sometimes controversially- called Artificial General Intelligence (AGI), it makes sense to develop a model that can learn from its own experience - without necessarily needing a supervisor. RL is the computational framework that enables us to create such intelligent agents. To better understand the value of RL, it is important to compare it with the other ML paradigms, which we'll look into next.

Previous Section
End of Section 2
Next Section