Tokenization is the process of dividing text into a set of meaningful pieces. These pieces are called tokens. For example, we can divide a chunk of text into words, or we can divide it into sentences. Depending on the task at hand, we can define our own conditions to divide the input text into meaningful tokens. Let's take a look at how to do this.
Create a new Python file and add the following lines. Let's define some sample text for analysis:
text = "Are you curious about tokenization? Let's see how it works! We need to analyze a couple of sentences with punctuations to see it in action."
Let's start with sentence tokenization. NLTK provides a sentence tokenizer, so let's import this:
# Sentence tokenization from nltk.tokenize import sent_tokenize
Run the sentence tokenizer on the input text and extract the tokens:
sent_tokenize_list = sent_tokenize(text)
Print the list of sentences to see whether it works correctly:
print "\nSentence tokenizer...