Book Image

Monitoring Elasticsearch

By : Dan Noble, Pulkit Agrawal, Mahmoud Lababidi
Book Image

Monitoring Elasticsearch

By: Dan Noble, Pulkit Agrawal, Mahmoud Lababidi

Overview of this book

ElasticSearch is a distributed search server similar to Apache Solr with a focus on large datasets, a schema-less setup, and high availability. This schema-free architecture allows ElasticSearch to index and search unstructured content, making it perfectly suited for both small projects and large big data warehouses with petabytes of unstructured data. This book is your toolkit to teach you how to keep your cluster in good health, and show you how to diagnose and treat unexpected issues along the way. You will start by getting introduced to ElasticSearch, and look at some common performance issues that pop up when using the system. You will then see how to install and configure ElasticSearch and the ElasticSearch monitoring plugins. Then, you will proceed to install and use the Marvel dashboard to monitor ElasticSearch. You will find out how to troubleshoot some of the common performance and reliability issues that come up when using ElasticSearch. Finally, you will analyze your cluster’s historical performance, and get to know how to get to the bottom of and recover from system failures. This book will guide you through several monitoring tools, and utilizes real-world cases and dilemmas faced when using ElasticSearch, showing you how to solve them simply, quickly, and cleanly.
Table of Contents (15 chapters)
Monitoring Elasticsearch
About the Author
About the Reviewers


Welcome to Monitoring Elasticsearch!

There are many books and online tutorials that cover the Elasticsearch API and how to configure a cluster. But, until now, there hasn't been a thorough, accessible resource for monitoring and troubleshooting purposes. We've found that Elasticsearch monitoring tools drastically improve our ability to solve cluster issues and greatly increase cluster reliability and performance as a result. We wrote this book to share those use cases and the insights that came out of them.

This book covers how to use several popular open source and commercial Elasticsearch monitoring tools, namely, Elasticsearch-head, Bigdesk, Marvel, Kopf, and Kibana. There's also a section on the Elasticsearch cat API and how to use Nagios to perform general system monitoring. Moreover, we will discuss several case studies with real-world examples of troubleshooting Elasticsearch issues using these tools.

We believe that the best way to learn is to do. In this book, we'll go over how to set up a sample Elasticsearch cluster and load it with data. At times, we'll deliberately introduce problems into the cluster so that we can see how the errors are tracked using our various monitoring tools. Following along with these examples in your own cluster will help you learn both how to use the monitoring tools and how to tackle new and unknown issues that may arise.

After reading this book, we hope that you will be better equipped to run and maintain an Elasticsearch cluster. You will also be more prepared to diagnose and solve cluster issues, such as a node going down, the Elasticsearch process dying, configuration errors, shard errors, OutOfMemoryError exceptions, slow queries, and slow indexing performance.

What this book covers

Chapter 1, Introduction to Monitoring Elasticsearch, gives an overview of Elasticsearch and talks about some things to keep in mind when monitoring a cluster or troubleshooting a problem.

Chapter 2, Installation and the Requirements for Elasticsearch, covers how to install Elasticsearch and several Elasticsearch monitoring tools.

Chapter 3, Elasticsearch-head and Bigdesk, demonstrates how to configure a multinode Elasticsearch cluster and how to use the monitoring tools Elasticsearch-head and Bigdesk to examine the health and status of a cluster.

Chapter 4, Marvel Dashboard, goes over Marvel, a commercial monitoring tool created by the makers of Elasticsearch.

Chapter 5, System Monitoring, covers the Elasticsearch utilities Kopf, Kibana, the Elasticsearch cat API, and several Unix command-line utilities. This chapter also demonstrates how to use Nagios for general system monitoring.

Chapter 6, Troubleshooting Performance and Reliability Issues, covers how to tackle some of the common performance and reliability issues that arise when using Elasticsearch. It also contains case studies with some real-world examples of troubleshooting.

Chapter 7, Node Failure and Post-Mortem Analysis, dives into analyzing your cluster's historical performance and how to get to the bottom of and recover from system failures. It also contains some case studies with real-world examples.

Chapter 8, Looking Forward, concludes the book by discussing what is to come with Elasticsearch 5, the next major software release, and some new monitoring tools that will be available for the release.

What you need for this book

To follow along with the examples in this book, you'll need a real or virtualized three-node Elasticsearch cluster. You may optionally want two other nodes to run Marvel and Nagios, covered in Chapter 4, Marvel Dashboard, and Chapter 5, System Monitoring, respectively. It is possible to run Marvel and Nagios on the same host as a node in your Elasticsearch cluster, but you shouldn't do this in a production cluster. Check out VMWare Player ( and VirtualBox ( for standing up your own virtual five-node environment or Amazon EC2 ( for building a cluster in the cloud.

For your Elasticsearch nodes, you'll need a 64-bit version of Windows, Mac OS X, or Linux and a recent distribution of the Java Runtime Environment. The CPU speed doesn't matter as much on these hosts, but we recommend that you have at least 512 MB of memory per node. We use Ubuntu 14.04 and Oracle Java 7 for all examples in this book, but any modern operating system and either OpenJDK or Oracle Java 7 and 8 will work for running through the examples. The only exception is Nagios, which needs to run on Linux.

You will need the following software packages:

All of these software packages are free and open source except for Marvel, which is only free for use in development.

Finally, several examples in this book use the curl ( command-line utility for making REST calls to Elasticsearch and, optionally, Python 2.7 for pretty-printing the results.

Who this book is for

This book is for software developers, DevOps engineers, and system administrators who use Elasticsearch. We'll cover the basics of Elasticsearch to get a simple cluster installed and configured, but we will avoid going into detail about the Elasticsearch API. Thus, a basic understanding of the Elasticsearch API may be helpful, though not required, to understand this book.


In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "Now we'll install Marvel on the elasticsearch-marvel-01."

A block of code is set as follows: my_elasticsearch_cluster "elasticsearch-node-01" false ["elasticsearch-node-02", "elasticsearch-node-03"]

When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold: my_elasticsearch_cluster "elasticsearch-node-01" false ["elasticsearch-node-02","elasticsearch-node-03"]

Any command-line input or output is written as follows:

# sudo service elasticsearch start

New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: "Clicking the Next button moves you to the next screen."


Warnings or important notes appear in a box like this.


Tips and tricks appear like this.

Reader feedback

Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of.

To send us general feedback, simply e-mail , and mention the book's title in the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at

Customer support

Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.

Downloading the example code

You can download the example code files for this book from your account at If you purchased this book elsewhere, you can visit and register to have the files e-mailed directly to you.

You can download the code files by following these steps:

  1. Log in or register to our website using your e-mail address and password.

  2. Hover the mouse pointer on the SUPPORT tab at the top.

  3. Click on Code Downloads & Errata.

  4. Enter the name of the book in the Search box.

  5. Select the book for which you're looking to download the code files.

  6. Choose from the drop-down menu where you purchased this book from.

  7. Click on Code Download.

You can also download the code files by clicking on the Code Files button on the book's webpage at the Packt Publishing website. This page can be accessed by entering the book's name in the Search box. Please note that you need to be logged in to your Packt account.

Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of:

  • WinRAR / 7-Zip for Windows

  • Zipeg / iZip / UnRarX for Mac

  • 7-Zip / PeaZip for Linux

The code bundle for the book is also hosted on GitHub at We also have other code bundles from our rich catalog of books and videos available at Check them out!

Downloading the color images of this book

We also provide you with a PDF file that has color images of the screenshots/diagrams used in this book. The color images will help you better understand the changes in the output. You can download this file from


Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title.

To view the previously submitted errata, go to and enter the name of the book in the search field. The required information will appear under the Errata section.


Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.

Please contact us at with a link to the suspected pirated material.

We appreciate your help in protecting our authors and our ability to bring you valuable content.


If you have a problem with any aspect of this book, you can contact us at , and we will do our best to address the problem.