The network is the backbone of a Hadoop cluster. Its stability is critical for the performance of the cluster. In this recipe, we will outline a few general rules for designing a Hadoop cluster network.
The network architecture for a small to medium-sized cluster can be as simple as connecting the cluster nodes with one or more switches. Connection redundancy can add reliability to the network.
Tip
Warning!
Computing nodes in a Hadoop cluster should be configured within the same network segment (Local Area Network (LAN)). Advanced features such as VLANs that can cause overhead are not recommended. Connecting nodes with a router is also not recommended.
The network architecture for the Hadoop clusters with hundreds or thousands of nodes is much more complex. In a large cluster, the physical nodes are usually so small, for example, a blade server, that they can be mounted on racks. Each rack has a local switch that interconnects nodes on the same rack...