Book Image

Artificial Intelligence with Python

Book Image

Artificial Intelligence with Python

Overview of this book

Artificial Intelligence is becoming increasingly relevant in the modern world. By harnessing the power of algorithms, you can create apps which intelligently interact with the world around you, building intelligent recommender systems, automatic speech recognition systems and more. Starting with AI basics you'll move on to learn how to develop building blocks using data mining techniques. Discover how to make informed decisions about which algorithms to use, and how to apply them to real-world scenarios. This practical book covers a range of topics including predictive analytics and deep learning.
Table of Contents (23 chapters)
Artificial Intelligence with Python
Credits
About the Author
About the Reviewer
www.PacktPub.com
Customer Feedback
Preface

Dividing text data into chunks


Text data usually needs to be divided into pieces for further analysis. This process is known as chunking. This is used frequently in text analysis. The conditions that are used to divide the text into chunks can vary based on the problem at hand. This is not the same as tokenization where we also divide text into pieces. During chunking, we do not adhere to any constraints and the output chunks need to be meaningful.

When we deal with large text documents, it becomes important to divide the text into chunks to extract meaningful information. In this section, we will see how to divide the input text into a number of pieces.

Create a new python file and import the following packages:

import numpy as np 
from nltk.corpus import brown 

Define a function to divide the input text into chunks. The first parameter is the text and the second parameter is the number of words in each chunk:

# Split the input text into chunks, where 
# each chunk contains N...