Summary
In this chapter, we have looked at the issues that arise when you try to split a piece of text into words by looking at how to split text into tokens, how to find the basic components of words, and how to identify compound words. These are all useful steps for assigning emotions to informal texts. We also looked at what happens when we try to take the next step and assign grammatical relations to the words that make up an informal text, concluding that this is an extremely difficult task that provides comparatively little benefit for our overall task. We had to look quite carefully at this step, even though we believe it is not all that useful, since we need to understand why it is so hard and why the results of even the best parsers cannot be relied on.