As the data grows, it impacts the time taken for both search, as well as creating new indexes to keep up with the increasing size of the repository. The simplest way to preserve the same search performance while scaling your data is to keep increasing your hardware, which includes higher processing power and higher memory size. However, this is not a cost-effective alternative. So, instead we will want to look for optimizing the running of the big data search instance. We have also covered different architectures of Solr in Chapter 4, Big Data Search Using Hadoop and Its Ecosystem, among which the most suitable architecture can be chosen on the basis of the requirements and the usage patterns.

The overall optimization of the technology stack, which includes Apache Hadoop and Apache Solr, helps you maintain more data with reasonable performance. The optimization is most important while scaling your instance for big data with Hadoop and Solr. We are going...