This section tries to scratch the surface of the complex field of NLP. The previous chapters have mentioned some of the basics that are necessary for dealing with textual data (for example, tokenization) without going too much into the details. Here, we'll try to go one step further into the basic understanding of this discipline. Due to its complexity and many aspects, we're taking a pragmatic approach and only scratching the surface of the theoretical foundations in favor of practical examples.
An essential part of any NLP system is the preprocessing pipeline. Before we can perform any interesting task on a piece of text, we must first convert it in a useful representation.
In the previous chapters, we already performed some analysis on the textual data without digging into the details of text preprocessing, but instead using the common tools with a pragmatic approach. In this section, we'll highlight some of the common preprocessing steps and discuss their role...