Unlike the majority of Cephs features, which by default perform well for a large number of workloads, Cephs tiering functionality requires careful configuration of its various parameters to ensure good performance. You should also have a basic understanding of your workloads I/O profile; tiering will only work well if your data has a small percentage of hot data. Workloads that are uniformly random or involve lots of sequential access patterns will either show no improvement or in some cases may actually be slower.
The main tuning options should be looked at first are the ones that define the size limit to the top tier, when it should flush and when it should evict.
The following two configuration options configure the maximum size of the data to be stored in the top tier pool:
target_max_bytes target_max_objects
The size is either specified in bytes or number objects and does not have to be the same size as the actual pool, but it cannot be larger...