Although often described as cache tiering, it's better to think of the functionality in Ceph as a tiering technology rather than a cache. It's important that you take this into consideration before reading any further as it's vital to understand the difference between the two.
A cache is typically designed to accelerate access to a set of data unless it's a writeback cache; it will not hold the only copy of the data, and normally there is little overhead to promoting data to cache. Cache tends to operate over a shorter timeframe, quite often everything that is accessed is promoted into cache.
A tiering solution is also designed to accelerate access to a set of data; however, it's promotion strategy normally works over a longer period of time and is more selective about what data is promoted, mainly due to the promotion action having a small impact on overall storage performance. Also, it is quite common with tiering technologies that only a single tier may hold the valid...