Book Image

Mastering Elasticsearch 5.x - Third Edition

By : Bharvi Dixit
Book Image

Mastering Elasticsearch 5.x - Third Edition

By: Bharvi Dixit

Overview of this book

Elasticsearch is a modern, fast, distributed, scalable, fault tolerant, and open source search and analytics engine. Elasticsearch leverages the capabilities of Apache Lucene, and provides a new level of control over how you can index and search even huge sets of data. This book will give you a brief recap of the basics and also introduce you to the new features of Elasticsearch 5. We will guide you through the intermediate and advanced functionalities of Elasticsearch, such as querying, indexing, searching, and modifying data. We’ll also explore advanced concepts, including aggregation, index control, sharding, replication, and clustering. We’ll show you the modules of monitoring and administration available in Elasticsearch, and will also cover backup and recovery. You will get an understanding of how you can scale your Elasticsearch cluster to contextualize it and improve its performance. We’ll also show you how you can create your own analysis plugin in Elasticsearch. By the end of the book, you will have all the knowledge necessary to master Elasticsearch and put it to efficient use.
Table of Contents (13 chapters)

The changed default text scoring in Lucene - BM25


Scoring is the most important part of Apache Lucene. It is the process of calculating the score property of a document in a scope of a given query. A score is a factor that describes how well the document matches the query. For score calculation, Lucene supports many algorithms, but since the beginning of Lucene, TF-IDF (term frequency-inverse document frequency) has been the default scoring algorithm. With the release of Apache Lucene 6.0, one of the major changes in Lucene is the changed default scoring algorithm. The default algorithm is now BM25 (Best Matching). In this section, we will also cover two fundamental concepts of search relevancy: precision and recall, and after that, we'll look at the new default Apache Lucene scoring mechanism and how it differs from TF-IDF.

Precision versus recall

After executing a search query, an obvious question comes to mind: Have I found the most relevant documents or am I missing important documents...