POS tagging is a powerful technique for identifying the grammatical parts of a sentence. It provides useful processing for downstream tasks, such as question analysis and analyzing the sentiment of text. We will return to this subject when we address parsing in Chapter 7, Information Retrieval.

Tagging is not an easy process, due to the ambiguities found in most languages. The increasing use of textese only makes the process more difficult. Fortunately, there are models that can do a good job of identifying this type of text. However, as new terms and slang are introduced, these models need to be kept up to date.

We investigated the use of OpenNLP, the Stanford API, and LingPipe in support of tagging. These libraries used several different approaches to tagging words, including both rule-based and model-based approaches. We saw how dictionaries can be used to enhance the tagging process.

We briefly touched on the model training process. Pretagged sample texts are used as input to the...