The following diagram is a basic representation of Proxmox and the Ceph cluster. Note that both clusters are on a separate subnet on separate switches.

A Ceph cluster should be set up with a separate subnet on a separate switch to keep it isolated from the Proxmox public subnet. The advantage of this practice is to keep Ceph's internal traffic isolated so that it does not interfere with the traffic of running virtual machines. On a healthy Ceph cluster with the active+clean state, this is not a big issue. However, when Ceph goes into self-healing mode due to OSD or a node failure, it rebalances itself by moving PGs around, which causes very high bandwidth consumption. On a bad day, separating two clusters ensures that the cluster does not slow down significantly due to the shortage of network bandwidth.

This also provides added security since the Ceph cluster network is completely hidden from any public access using a separate switch. In our previous example, we have three...