Book Image

Rapid - Apache Mahout Clustering designs

Book Image

Rapid - Apache Mahout Clustering designs

Overview of this book

Related Content you might be interested in

Current Title:

Small book image
Rapid - Apache Mahout Clustering designs
Oct, 2015 | 130 pages
No titles found
Table of Contents (16 chapters)
Apache Mahout Clustering Designs
Apache Mahout Clustering Designs
Credits
Credits
About the Author
About the Author
About the Reviewers
About the Reviewers
www.PacktPub.com
www.PacktPub.com
Preface
Preface
Free Chapter
1
Understanding Clustering
Understanding Clustering
The clustering concept
Understanding distance measures
Understanding different clustering techniques
Algorithm support in Mahout
Clustering algorithms in Mahout
Installing Mahout
Preparing data for use with clustering techniques
Summary
2
Understanding K-means Clustering
Understanding K-means Clustering
Learning K-means
Visualizing clusters
Summary
3
Understanding Canopy Clustering
Understanding Canopy Clustering
Running Canopy clustering on Mahout
Visualizing clusters
Working with CSV files
Summary
4
Understanding the Fuzzy K-means Algorithm Using Mahout
Understanding the Fuzzy K-means Algorithm Using Mahout
Learning Fuzzy K-means clustering
Visualizing clusters
Summary
5
Understanding Model-based Clustering
Understanding Model-based Clustering
Learning model-based clustering
Running LDA using Mahout
Summary
6
Understanding Streaming K-means
Understanding Streaming K-means
Learning Streaming K-means
Using Mahout for streaming K-means
Summary
7
Spectral Clustering
Spectral Clustering
Understanding spectral clustering
Mahout implementation of spectral clustering
Summary
8
Improving Cluster Quality
Improving Cluster Quality
Evaluating clusters
Using DistanceMeasure interface
Summary
9
Creating a Cluster Model for Production
Creating a Cluster Model for Production
Preparing the dataset
Launching the Mahout job on the cluster
Performance tuning for the job
Summary
Index
Index

Running LDA using Mahout

To run LDA using Mahout, we will use a 20newsgroups dataset. We will convert the corpus into vectors, run LDA on those vectors, and get the resultant topics.

Let's run this example to view how topic modeling works in Mahout.

Dataset selection

We will use 20newsgroups dataset for this exercise. Download the dataset 20news-bydate.tar.gz from http://qwone.com/~jason/20Newsgroups/.

Steps to execute CVB (LDA)

  1. Create a directory 20newsdata and unzip the data here:

    mkdir /tmp/20newsdata
cdtmp/20newsdata
tar-xzvf /tmp/20news-bydate.tar.gz

  2. There are two folders under 20newsdata, 20news-bydate-test, and 20news-bydate-train. Now, create another directory 20newsdataall and merge both training and test data of the group.

  3. Now, move to the home directory and execute the following command:

    mkdir /tmp/20newsdataall
cp –R /20newsdata/*/* /tmp/20newsdataall

  4. Create a directory in Hadoop and save this data in HDFS:

    hadoopfs –mkdir /usr/hue/20newsdata
hadoopfs –put /tmp/20newsdataall /usr/hue...
Previous Section
End of Section 3
Next Section