Lemmatization is a more methodical way of converting all the grammatical/inflected forms of the root of the word. Lemmatization uses context and part of speech to determine the inflected form of the word and applies different normalization rules for each part of speech to get the root word (lemma):
>>>from nltk.stem import WordNetLemmatizer >>>wlem = WordNetLemmatizer() >>>wlem.lemmatize("ate") eat
Here, WordNetLemmatizer is using wordnet, which takes a word and searches wordnet, a semantic dictionary. It also uses a morph analysis to cut to the root and search for the specific lemma (variation of the word). Hence, in our example it is possible to get eat for the given variation ate, which was never possible with stemming.
Can you explain what the difference is between Stemming and lemmatization?
Can you come up with a Porter stemmer (Rule-based) for your native language?
Why would it be harder to implement a stemmer for languages like Chinese?