In our previous discussion on the history of reinforcement learning, we covered the two main threads, trial and error and Dynamic Programming (DP), which came together to derive current modern Reinforcement Learning (RL). As we mentioned in earlier chapters, there is also a third thread that arrived late called Temporal Difference Learning (TDL). In this chapter, we will explore TDL and how it solves the Temporal Credit Assignment (TCA) problem. From there, we will explore how TD differs from Monte Carlo (MC) and how it evolves to full Q-learning. After that, we will explore the differences between on-policy and off-policy learning and then, finally, work on a new example RL environment.
For this chapter, we will introduce TDL and how it improves on the previous techniques we looked at in previous chapters. Here are the main topics we will cover in...