It is important to set the Platform environment variable to x64 or Win32 depending on the Hadoop deployment desired. This can be done using the following command:

set Platform=x64 

For Win32, use the following command:

set Platform=Win32

It is very important to ensure that the environment variable has the right name. This variable is case sensitive and instructs the Visual Studio project files to use the appropriate build configuration.

The next step is to actually issue the Maven build command. The command mvn package -Pdist,native-win -DskipTests –Dtar is used to start the build. Using the newer JDK can cause some parse issues when generating Javadocs. This can be solved by either using an older JDK such as 1.7 or skipping Javadocs generation. The latter is done by adding the –Dmaven.javadocs.skip=true option in the Maven package command.

The following screenshot shows the end of the build process. The summary of the standard output shows the status of each build step. Once a failure is encountered, the rest of the steps are skipped. It is also important to have the computer connected to the Internet during the build process. Maven automatically downloads dependencies from configured binary repositories during the build process:

The build yields a target directory. Inside the target directory, Hadoop binaries, samples, and configuration files are bundled in a zipped TAR file. In this example, a hadoop-2.5.0.tar.gz file is generated. We extract the contents of the file to the C:\hdp\hdp path.

The Hadoop-env.cmd file is present in the etc\hadoop directory at the root of the Hadoop installation. This is the configuration directory for Hadoop. The hadoop-env.cmd command file needs to be modified to set the right environment to execute the Hadoop daemons correctly. The most important configuration is the setting of the JAVA_HOME environment variable. We also set HADOOP_HOME to the root of the Hadoop installation, that is, the path from where we extracted the Hadoop binaries and configuration files. The HADOOP_CONF_DIR and YARN_CONF_DIR environment variables are set to the configuration directories of Hadoop and YARN respectively. The YARN configuration directory is the same as the Hadoop configuration directory in our example. We also add the Hadoop directories to the Path variable. The following script snippet is a sample hadoop-env.cmd script file:

@rem The java implementation to use.  Required.
set JAVA_HOME=%JAVA_HOME%

set HADOOP_HOME=c:\hdp\hdp

@rem The jsvc implementation to use. Jsvc is required to run secure datanodes.
@rem set JSVC_HOME=%JSVC_HOME%

set HADOOP_CONF_DIR=%HADOOP_HOME%\etc\hadoop

set YARN_CONF_DIR=%HADOOP_CONF_DIR%
set PATH=%PATH%;%HADOOP_HOME%\bin

@rem Extra Java CLASSPATH elements.  Automatically insert capacity-scheduler.
if exist %HADOOP_HOME%\contrib\capacity-scheduler (
  if not defined HADOOP_CLASSPATH (
    set HADOOP_CLASSPATH=%HADOOP_HOME%\contrib\capacity-scheduler\*.jar
  ) else (
    set HADOOP_CLASSPATH=%HADOOP_CLASSPATH%;%HADOOP_HOME%\contrib\capacity-scheduler\*.jar
  )
)

@rem The maximum amount of heap to use, in MB. Default is 1000.
@rem set HADOOP_HEAPSIZE=
@rem set HADOOP_NAMENODE_INIT_HEAPSIZE=""

@rem Extra Java runtime options.  Empty by default.
@rem set HADOOP_OPTS=%HADOOP_OPTS% -Djava.net.preferIPv4Stack=true

@rem Command specific options appended to HADOOP_OPTS when specified
if not defined HADOOP_SECURITY_LOGGER (
  set HADOOP_SECURITY_LOGGER=INFO,RFAS
)
if not defined HDFS_AUDIT_LOGGER (
  set HDFS_AUDIT_LOGGER=INFO,NullAppender
)

set HADOOP_NAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_NAMENODE_OPTS%
set HADOOP_DATANODE_OPTS=-Dhadoop.security.logger=ERROR,RFAS %HADOOP_DATANODE_OPTS%
set HADOOP_SECONDARYNAMENODE_OPTS=-Dhadoop.security.logger=%HADOOP_SECURITY_LOGGER% -Dhdfs.audit.logger=%HDFS_AUDIT_LOGGER% %HADOOP_SECONDARYNAMENODE_OPTS%

@rem The following applies to multiple commands (fs, dfs, fsck, distcp etc)
set HADOOP_CLIENT_OPTS=-Xmx512m %HADOOP_CLIENT_OPTS%
@rem set HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData %HADOOP_JAVA_PLATFORM_OPTS%"

@rem On secure datanodes, user to run the datanode as after dropping privileges
set HADOOP_SECURE_DN_USER=%HADOOP_SECURE_DN_USER%

@rem Where log files are stored.  %HADOOP_HOME%/logs by default.
@rem set HADOOP_LOG_DIR=%HADOOP_LOG_DIR%\%USERNAME%

@rem Where log files are stored in the secure data environment.
set HADOOP_SECURE_DN_LOG_DIR=%HADOOP_LOG_DIR%\%HADOOP_HDFS_USER%

@rem The directory where pid files are stored. /tmp by default.
@rem NOTE: this should be set to a directory that can only be written to by 
@rem       the user that will run the hadoop daemons. Otherwise there is the
@rem       potential for a symlink attack.
set HADOOP_PID_DIR=%HADOOP_PID_DIR%
set HADOOP_SECURE_DN_PID_DIR=%HADOOP_PID_DIR%

@rem A string representing this instance of hadoop. %USERNAME% by default.
set HADOOP_IDENT_STRING=%USERNAME%

Next we configure the core-site.xml file. The most important configuration is setting the fs.default.name property to the HDFS NameNode host and port. In our case, since it is a single node deployment, it points to localhost on port 19000. The following configuration snippet illustrates this setting:

<configuration>
    <property>
        <name>fs.default.name</name>
       <value>hdfs://0.0.0.0:19000</value>
    </property>
 </configuration>

We then configure the hdfs-site.xml file. Here we set the replication factor to 1 as we are doing a single-node deployment of Hadoop. The following configuration snippet illustrates this setting:

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>   
    </property> 
</configuration>

The mapred-site.xml file needs to be configured and pointed to YARN in Hadoop 2.X. The %USERNAME% element can be replaced by the username of the entity submitting the jobs. The following configuration snippet illustrates a sample mapred-site.xml file. If the file is not present, it can be copied from the mapred-site.xml.template file present in the configuration directory:

<configuration>
    <property>
        <name>mapreduce.job.user.name</name>
        <value>%USERNAME%</value>
    </property>     
    <property>      
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
    <property>
        <name>yarn.apps.stagingDir</name>
        <value>/user/%USERNAME%/staging</value>
    </property>
    <property>
        <name>mapreduce.jobtracker.address</name>
        <value>local</value>
    </property>  
</configuration>

The yarn-site.xml file is configured for the settings on the ResourceManager and NodeManager daemons. The configurations include setting the daemon endpoints and the log directories, and specifying the shuffle handlers. The following configuration snippet illustrates a sample configuration for the YARN daemons:

<configuration>
    <property>
        <name>yarn.server.resourcemanager.address</name>
        <value>0.0.0.0:8020</value>
    </property>
    <property>
        <name>yarn.server.resourcemanager.application.expiry.interval</name>
        <value>60000</value>
    </property>
     <property>
        <name>yarn.server.nodemanager.address</name>
        <value>0.0.0.0:45454</value>
    </property>
     <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
        <value>org.apache.hadoop.mapred.ShuffleHandler</value>
    </property>
    <property>
        <name>yarn.server.nodemanager.remote-app-log-dir</name>
     <value>/app-logs</value>
    </property>
    <property>
        <name>yarn.nodemanager.log-dirs</name>
        <value>/dep/logs/userlogs</value>
    </property>
    <property>
        <name>yarn.server.mapreduce-appmanager.attempt-listener.bindAddress</name>
        <value>0.0.0.0</value>
    </property>
    <property>
        <name>yarn.server.mapreduce-appmanager.client-service.bindAddress</name>
        <value>0.0.0.0</value>
    </property>
    <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
    </property>
    <property>
        <name>yarn.log-aggregation.retain-seconds</name>
        <value>-1</value>   </property>
    <property>
        <name>yarn.application.classpath</name>
        <value>%HADOOP_CONF_DIR%,%HADOOP_COMMON_HOME%/share/hadoop/common/*,%HADOOP_COMMON_HOME%/share/hadoop/common/lib/*,%HADOOP_HDFS_HOME%/share/hadoop/hdfs/*,%HADOOP_HDFS_HOME%/share/hadoop/hdfs/lib/*,%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/*,%HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/lib/*,%HADOOP_YARN_HOME%/share/hadoop/yarn/*,%HADOOP_YARN_HOME%/share/hadoop/yarn/lib/*</value>
    </property>
</configuration>

Before starting the daemons, we can format the NameNode by issuing the following command:

hdfs namenode –format

The following screenshot shows the output of the format command. Now the HDFS is formatted and ready to use. Since we have not specified a particular directory name, the NameNode uses the C:\tmp directory to store all of the metadata.

We then start the HDFS daemons, the NameNode, and the DataNode. This is done by issuing start-dfs.cmd. This command script is present in the %HADOOP_HOME%\sbin folder. The Windows firewall may pop up a notification asking the user to allow the daemons to open a listening port in the firewall. It is important to allow the firewall to be reconfigured so that the DataNode and NameNode can communicate with each other. The following screenshot shows the Windows Firewall screen offering to allow access:

Once access has been granted, the NameNode and DataNode start in two separate command windows, as shown in the following screenshot. The standard output of each HDFS operation can be examined in these two windows. Issuing HDFS commands on the filesystem once both the daemons are up and running can conduct tests to validate HDFS. You might have to create the user directories using the mkdir command before starting off though.

The NameNode and DataNode command windows

Next we need to start the YARN to run MapReduce jobs. This can be done by the start-yarn.cmd file present in the sbin folder. Again, the ResourceManager and the NodeManager start off in two separate command windows as illustrated in the following screenshot. The standard output can be examined to see the trace on the ResourceManager and NodeManager.

The ResourceManager and NodeManager command windows

By navigating to localhost:50070 on the browser, the user should now be able to see the web endpoint for HDFS. The home page is shown in the following screenshot. It gives an overview of the health of HDFS and the different parameters that were used to configure it.

Selecting the Datanodes link on the top bar gives the different DataNodes present in HDFS and the health of each DataNode, as shown in the following screenshot:

The startup progress link in the top bar shows the health of HDFS during startup. This includes statistics about the fsimage and edits file during NameNode startup. It also indicates whether HDFS went into safe mode or not.

The utilities link gives two options: one to browse HDFS and the other to view the logfiles. The browse functionality is based on a search box that can be used to search the HDFS directory structure. Each listing for a file is similar to executing the hdfs dfs –ls command on the directory. It also gives statistics about the block size and a deep link to peek into the contents of the file.