As you might know, the Lucene index is divided into smaller pieces called segments, and each segment is stored on disk. Depending on the indexing and merge policy settings, Lucene, from time to time, merges two or more segments into a new one. This operation requires reading the old segments and writing a new one with the information from the old segments. The merges can happen at the same time when Solr indexes data and queries are run. The same goes for writing the segments; it can be pretty expensive when it comes to I/O usage. It is because of this that Solr allows us to configure the limits for I/O usage. This recipe will show you how to do this.
Before continuing further with this recipe, read the Choosing the proper directory configuration recipe of this chapter to see what directories are available and how to configure them.
Let's assume that we want to limit the I/O usage for our use case that uses solr.MMapDirectoryFactory. So, in the solrconfig.xml file, we will have the following configuration present:
<directoryFactory name="DirectoryFactory" class="solr.MMapDirectoryFactory"> </directoryFactory>
Now, let's introduce the following limits:
We allow Solr to write a maximum of 20 MB per second during segment writes
We allow Solr to write a maximum of 10 MB per second during segment merges
We allow Solr to read a maximum of 50 MB per second
To do this, we change our previous configuration to the following:
<directoryFactory name="DirectoryFactory" class="solr.MMapDirectoryFactory"> <double name="maxWriteMBPerSecFlush">20</double> <double name="maxWriteMBPerSecMerge">10</double> <double name="maxWriteMBPerSecRead">50</double> </directoryFactory>
After altering the configuration, all we need to do is restart Solr and the limits will be taken into consideration.
The logic behind setting the limits is very simple. All directories that extend the Solr CachingDirectoryFactory class allow us to set the maxWriteMBPerSecFlush, maxWriteMBPerSecMerge and maxWriteMBPerSecRead properties. The mentioned directory implementations are all the directory implementations that were mentioned in the Choosing the proper directory configuration recipe of this chapter.
The maxWriteMBPerSecFlush property allows us to tell Solr how many megabytes per second can be written by Solr during segment flush (so, during the write operation that is not triggered by segment merging). The maxWriteMBPerSecMerge property allows us to specify how many megabytes per second can be written by Solr during segment merge. Finally, the maxWriteMBPerSecRead property specifies the amount of megabytes allowed to be read per second. One thing to remember is that the values are approximated, not exact.
Limiting I/O usage can be very handy, especially in deployments where I/O usage is at its maximum. During query peak hours, when we want to solve server queries as fast as we can, we need to minimize the indexing and merging impact. With proper configuration that is adjusted to our needs, we can just limit the I/O usage and still serve queries with the latency we want.