The WordCount sample uses MapReduce to count the number of word occurrences within a set of input documents. The sample code is available in the chapter1/Wordcount.java file of the source folder of this chapter. The code has three parts—Mapper, Reducer, and the main program.

The Mapper extends from the org.apache.hadoop.mapreduce.Mapper interface. Hadoop InputFormat provides each line in the input files as an input key-value pair to the map function. The map function breaks each line into substrings using whitespace characters such as the separator, and for each token (word) emits (word,1) as the output.

public void map(Object key, Text value, Context context) throws IOException, InterruptedException {
  // Split the input text value to words
  StringTokenizer itr = new StringTokenizer(value.toString());

  // Iterate all the words in the input text value
  while (itr.hasMoreTokens()) {
    word.set(itr.nextToken());
    context.write(word, new IntWritable(1));
  }
}

Each reduce function invocation receives a key and all the values of that key as the input. The reduce function outputs the key and the number of occurrences of the key as the output.

public void reduce(Text key, Iterable<IntWritable>values, Context context) throws IOException, InterruptedException
{
  int sum = 0;
  // Sum all the occurrences of the word (key)
  for (IntWritableval : values) {
    sum += val.get();
  }
  result.set(sum);
  context.write(key, result);
}

The main driver program configures the MapReduce job and submits it to the Hadoop YARN cluster:

Configuration conf = new Configuration();
……
// Create a new job
Job job = Job.getInstance(conf, "word count");
// Use the WordCount.class file to point to the job jar
job.setJarByClass(WordCount.class);

job.setMapperClass(TokenizerMapper.class);
job.setReducerClass(IntSumReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);

// Setting the input and output locations
FileInputFormat.addInputPath(job, new Path(otherArgs[0]));
FileOutputFormat.setOutputPath(job, newPath(otherArgs[1]));
// Submit the job and wait for it's completion
System.exit(job.waitForCompletion(true) ? 0 : 1);

Compile the sample using the Gradle build as mentioned in the introduction of this chapter by issuing the gradle build command from the chapter1 folder of the sample code repository. Alternatively, you can also use the provided Apache Ant build file by issuing the ant compile command.

Run the WordCount sample using the following command. In this command, chapter1.WordCount is the name of the main class. wc-input is the input data directory and wc-output is the output path. The wc-input directory of the source repository contains a sample text file. Alternatively, you can copy any text file to the wc-input directory.

$ $HADOOP_HOME/bin/hadoop jar \
hcb-c1-samples.jar \
chapter1.WordCount wc-input wc-output

The output directory (wc-output) will have a file named part-r-XXXXX, which will have the count of each word in the document. Congratulations! You have successfully run your first MapReduce program.

$ cat wc-output/part*

Hadoop reads the input, breaks it using new line characters as the separator and then runs the map function passing each line as an argument with the line number as the key and the line contents as the value.

The map function tokenizes the line, and for each token (word), emits a key-value pair (word,1).

Hadoop collects all the (word,1) pairs, sorts them by the word, groups all the values emitted against each unique key, and invokes the reduce function once for each unique key passing the key and values for that key as an argument.

The reduce function counts the number of occurrences of each word using the values and emits it as a key-value pair.

Hadoop writes the final output to the output directory.