There will always be failures in clusters, such as hardware issues or a need to upgrade nodes. This should be done in a graceful manner, without any data loss.
When the Datanode daemon is stopped on a Datanode, it takes approximately ten minutes for the Namenode to remove that node. This has to do with the heartbeat retry interval. At any time, we can abruptly remove the Datanode, but it can result in data loss.
It is recommended that you opt for the graceful removal of the node from the cluster, as this ensures that all the data on that node is drained.
For the following steps, we assume that the cluster that is up and running with Datanodes is in a healthy state and the one with the Datanode dn1.cluster1.com needs maintenance and must be removed from the cluster. We will login to the Namenode and make changes there.
ssh to Namenode and edit the file
hdfs-site.xmlby adding the following property to it:<property> <name>dfs.hosts.exclude</name> <value>/home/hadoop/excludes</value> <final>true</final> </property>
Make sure the file
excludesis readable by the userhadoop.Restart the Namenode daemon for the property to take effect:
$ hadoop-daemons.sh stop namenode $ hadoop-daemons.sh start namenode
A restart of Namenode is required only when any property is changed in the file. Once the property is in place, Namenode can read the changes to the contents of the file
excludesby simply refreshing nodes.Add the
dn1.cluster1.comnode to the fileexcludes:$ cat excludes dn1.cluster1.com
After adding the node to the file, we just need to reload the file by doing the following:
$ hadoop dfsadmin -refreshNodesAfter sometime, the node will be decommissioned. The time will vary according to the data the particular Datanode had. We can see the decommissioned nodes using the following:
$ hdfs dfsadmin -reportThe preceding command will list the nodes in the cluster, and against the
dn1.cluster1.comnode we can see that its status will either be decommissioning or decommissioned.
Let's have a look at what we did throughout this recipe.
In steps 1 through 6, we added the new property to the hdfs-site.xml file and then restarted Namenode to make it aware of the changes. Once the property is in place, the Namenode is aware of the excludes file, and it can be asked to re-read by simply refreshing the node list, as done in step 6.
With these steps, the data on the Datanode dn1.cluster1.com will be moved to other nodes in the cluster, and once the data has been drained, the Datanode daemon on the node will be shutdown. During the process, the node will change the status from normal to decommissioning and then to decommissioned.
Care must be taken while decommissioning nodes in the cluster. The user should not decommission multiple nodes at a time as this will generate lot of network traffic and cause congestion and data loss.