Word tagging gets much more interesting when you can create your own models. The realm of annotating part-of-speech tagging corpora is a bit too much for a mere recipe book—annotation of the part-of-speech data is very difficult because it requires considerable linguistic knowledge to do well. This recipe will directly address the machine-learning component of the HMM-based sentence detector.

As this is a recipe book, we will minimally explain what an HMM is. The token language models that we have been working with do their previous context calculations on some number of words/tokens that precede the current word being estimated. HMMs take into account some length of the previous tags while calculating estimates for the current token's tag. This allows for seemingly disparate neighbors such as of and in to be similar, because they are both prepositions.

In the Sentence detection recipe, from Chapter 5, Finding Spans in Text – Chunking, a useful but not very flexible sentence...