One of the main requirements of today applications is the search capability. In the market we can find a lot of solutions to answer this need, both in the commercial and in the open source world. One of the frequently used libraries for searching is Apache Lucene. This library is the base of a large number of search solutions such as Apache Solr, Indextank, and ElasticSearch.
ElasticSearch is one of the younger solutions, written with the cloud, and distributed computing in mind. Its main author, Shay Banon, famous for having developed Compass (http://www.compass-project.org), released the first version of ElasticSearch in March 2010.
Thus the main scope of ElasticSearch is to be a search engine; it also provides a lot of features that allows it to be used also as data store and analytic engine via facets.
ElasticSearch contains a lot of innovative features: JSON REST-based, natively distributed in a map/reduce approach, easy to set up, and extensible with plugins. In this book, we will study in depth about these features and many others available in ElasticSearch.
Before ElasticSearch, only Apache Solr was able to provide some of these functionalities, but it was not designed for the cloud and it is not using JSON REST API. In the last year, this situation has changed a bit with the release of Solr Cloud in 2012. For users who want to have a deeper comparison between these two products, I suggest to read posts by Rafal Kuc available at http://blog.sematext.com/2012/08/23/solr-vs-elasticsearch-part-1-overview/.
ElasticSearch is also a product in continuous evolution and new functionalities are released both by the ElasticSearch Company (the company founded by Shay Banon to provide commercial support for ElasticSearch) and by ElasticSearch users as a plugin (mainly available on GitHub).
In my opinion, ElasticSearch is probably one of the most powerful and easy-to-use search solutions in the market. In writing this book and these recipes, the book reviewers and I have tried to transmit our knowledge, our passion, and the best practices to manage it in a better way.
Chapter 1, Getting Started, gives the reader an overview of the basic concepts of ElasticSearch and the ways to communicate with it.
Chapter 2, Downloading and Setting Up ElasticSearch, covers the basic steps to start using ElasticSearch from the simple install to cloud ones.
Chapter 3, Managing Mapping, covers the correct definition of the data fields to improve both indexing and searching quality.
Chapter 4, Standard Operations, teaches the most common actions that are required to ingest data in ElasticSearch and to manage it.
Chapter 5, Search, Queries, and Filters, talks about Search DSL—the core of the search functionalities of ElasticSearch. It is the only way to execute queries in ElasticSearch.
Chapter 6, Facets, covers another capability of ElasticSearch—the possibility to execute analytics on search results to improve both user experience and to drill down the information contained in ElasticSearch.
Chapter 7, Scripting, shows how to customize ElasticSearch with scripting in different languages.
Chapter 8, Rivers, extends ElasticSearch giving the ability to pull data from different sources such as databases, NoSQL solutions, or data streams.
Chapter 9, Cluster and Nodes Monitoring, shows how to analyze the behavior of a cluster/node to understand common pitfalls.
Chapter 10, Java Integration, describes how to integrate ElasticSearch in Java application using both REST and Native protocols.
Chapter 11, Python Integration, covers the usage of the official ElasticSearch Python client and the Pythonic PyES library.
Chapter 12, Plugin Development, describes how to create the different types of plugins: site and native. Some examples show the plugin skeletons, the setup process, and their building.
For this book you will need a computer, of course. In terms of the software required, you don't have to be worried, all the components we use are open source and available for every platform.
For all the REST examples the cURL software (http://curl.haxx.se/) is used to simulate a command from the command line. It's commonly preinstalled in Linux and Mac OS X operative systems. For Windows, it can be downloaded from its site and put in a path that can be called from a command line.
For Chapter 10, Java Integration and Chapter 12, Plugin Development, the Maven built tool (http://maven.apache.org/) is required, which is a standard for managing build, packaging, and deploy in Java. It is natively supported in Java IDEs such as Eclipse and IntelliJ IDEA.
Chapter 11, Python Integration, requires the Python interpreter installed. By default it's available on Linux and Mac OS X. For Windows it can be downloaded from the official Python site (http//www.python.org). For the current examples Version 2.X is used.
This book is for developers who want to start using both ElasticSearch and at the same time improve their ElasticSearch knowledge. The book covers all aspects of using ElasticSearch and provides solutions and hints for everyday usage. The recipes are reduced in complexity to easily focus the reader on the discussed ElasticSearch aspect and to easily memorize the ElasticSearch functionalities.
The latter chapters that discuss the ElasticSearch integration in JAVA and Python, shows the user how to integrate the power of ElasticSearch in their applications.
The last chapter talks about advanced usage of ElasticSearch and its core extension, so some skilled Java know-how is required.
In this book, you will find a number of styles of text that distinguish between different kinds of information. Here are some examples of these styles, and an explanation of their meaning.
Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "Open the config/elasticsearch.yml file with an editor of your choice."
A block of code is set as follows:
path.conf: /opt/data/es/conf path.data: /opt/data/es/data1,/opt2/data/data2 path.work: /opt/data/work path.logs: /opt/data/logs path.plugins: /opt/data/plugins
When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold:
{
"order": {
"_uid": {
"store": "yes"
},
"_id": {
"path": "order_id"
},
"properties": {
"order_id": {
"type": "string",
"store": "yes",
"index": "not_analyzed"
},
Any command-line input or output is written as follows:
bin/plugin -install elasticsearch/elasticsearch-mapper-attachments/1.9.0
New terms and important words are shown in bold. Words that you see on the screen, in menus or dialog boxes for example, appear in the text like this: " The Any Request [+] tab allows executing custom query. On the left-hand side there are the following options:".
Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or may have disliked. Reader feedback is important for us to develop titles that you really get the most out of.
To send us general feedback, simply send an e-mail to <[email protected]>, and mention the book title via the subject of your message.
If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide on www.packtpub.com/authors.
Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
You can download the example code files for all Packt books you have purchased from your account at http://www.packtpub.com. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you would report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the errata submission form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded on our website, or added to any list of existing errata, under the Errata section of that title. Any existing errata can be viewed by selecting your title from http://www.packtpub.com/support.
Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works, in any form, on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy.
Please contact us at <[email protected]> with a link to the suspected pirated material.
We appreciate your help in protecting our authors, and our ability to bring you valuable content.
You can contact us at <[email protected]> if you are having a problem with any aspect of the book, and we will do our best to address it.