This chapter covered a lot of new ground. We started by performing linguistic processing on our text. We met spaCy, which we will continue to dive deeper into as we move on in this book. We covered the following foundational ideas from linguistics, tokenization doing this with and without spaCy, stop word removal, case standardization, lemmatization (we skipped stemming) – using spaCy and its peculiarities such as-PRON-
But what do we do with spaCy, other than text cleaning? Can we build something? Yes!
Not only can we extend our simple linguistics based text cleaning using spaCy pipelines but also do parts of speech tagging, named entity recognition, and other common tasks. We will look at this in the next chapter.
We looked at spelling correction or the closest word match problem. We discussed FuzzyWuzzy and Jellyfish in this context. To ensure that we can scale...