This section of the chapter will cover the scenario where a number of OSDs may have gone offline in a short period of time, leaving some objects with no valid replica copies. It's important to note that there is a difference between an object that has no remaining copies and an object that has a remaining copy, but it is known that another copy has had more recent writes. The latter is normally seen when running the cluster with min_size set to 1.
To demonstrate how to recover an object that has an out-of-date copy of data, let's perform a series of steps to break the cluster:
- First, let's set
min_sizeto 1; hopefully, by the end of this example, you will see why you don't ever want to do this in real life:
sudo ceph osd pool set rbd min_size 1
- Create a test object that we will make later make Ceph believe is lost:
sudo rados -p rbd put lost_object logo.png sudo ceph osd set norecover sudo ceph osd set nobackfill
These two flags make sure that when the OSDS come back...