We have seen that background subtraction can be an effective technique for detecting moving objects; however, we know that it has some inherent limitations. Notably, it assumes that the current background can be predicted based on past frames. This assumption is fragile. For example, if the camera moves, the entire background model could suddenly become outdated. Thus, in a robust tracking system, it is important to build some kind of model of foreground objects rather than just the background.
We have already seen various ways of detecting objects in Chapter 5, Detecting and Recognizing Faces, Chapter 6, Retrieving Images and Searching Using Image Descriptors, and Chapter 7, Building Custom Object Detectors. For object detection, we favored algorithms that could deal with a lot of variation within a class of objects, so that...