Chunking is the process used to perform entity detection. It is used for the segmentation and labeling of multiple sequences of tokens in a sentence.
To design a chunker, a chunk grammar should be defined. A chunk grammar holds the rules of how chunking should be done.
Let's consider the example that performs Noun Phrase Chunking by forming the chunk rules:
>>> import nltk >>> sent=[("A","DT"),("wise", "JJ"), ("small", "JJ"),("girl", "NN"), ("of", "IN"), ("village", "N"), ("became", "VBD"), ("leader", "NN")] >>> sent=[("A","DT"),("wise", "JJ"), ("small", "JJ"),("girl", "NN"), ("of", "IN"), ("village", "NN"), ("became", "VBD"), ("leader", "NN")] >>> grammar = "NP: {<DT>?<JJ>*<NN><IN>?<NN>*}" >>> find = nltk.RegexpParser(grammar) >>> res = find.parse(sent) >>> print(res) (S (NP A/DT wise/JJ small/JJ girl/NN of/IN village/NN) became/VBD (NP leader/NN...