In the unlikely event that you lose all of your monitors, all is not lost. You can rebuild the monitor database from the contents of the OSDs with the use of ceph-objectstore-tool.
To set the scenario, we will assume that an event has occurred and has corrupted all three monitors, effectively leaving the Ceph cluster inaccessible. To recover the cluster, we will shut down two of the monitors and leave a single failed monitor running. We will then rebuild the monitor database, overwrite the corrupted copy, and restart the monitor to bring the Ceph cluster back online.
The objectstore tool needs to be able to access every OSD in the cluster to rebuild the monitor database; in this example, we will use a script, which will connect via ssh to access the OSD data. As the OSD data is not accessible by every user, we will use the root user to...