Document analysis using Hadoop and Mahout

In this section, we will take an example of document analysis to illustrate analytics using Hadoop and Mahout. We will be using Pig as the higher-level abstraction for Hadoop MapReduce. We will be calculating the distance between documents using a score called Tf-idf. This distance metric is very popular in the field of information retrieval and text analytics. It is based on the statistics of words occurring in a document.

Tf-idf is used to rank documents based on query terms. It is extensively used in text search scenarios. The distance between the query terms and the document terms determines how close the query is with respect to the document. This distance can be used to rank documents.

For this particular example, we will be using the NSF grants abstracts that are available at http://kdd.ics.uci.edu/databases/nsfabs/nsfawards.html. The dataset consists of about 120,000 abstracts and comes in three parts. Each grant abstract is a separate text...