There are several well-known text cleaning ideas. They have all made their way into the most popular tools today such as NLTK, Stanford CoreNLP, and spaCy. I like spaCy for two main reasons:
- It's an industry-grade NLP, unlike NLTK, which is mainly meant for teaching.
- It has good speed-to-performance trade-off. spaCy is written in Cython, which gives it C-like performance with Python code.
spaCy is actively maintained and developed, and incorporates the best methods available for most challenges.
By the end of this section, you will be able to do the following:
- Understand tokenization and do it manually yourself using spaCy
- Understand why stop word removal and case standardization works, with spaCy examples
- Differentiate between stemming and lemmatization, with spaCy lemmatization examples