Elasticsearch 5.x Cookbook

Elasticsearch 5.x Cookbook - Third Edition

By : Alberto Paro

Buy this Book

Elasticsearch 5.x Cookbook - Third Edition

By: Alberto Paro

Buy this Book

Overview of this book

Elasticsearch is a Lucene-based distributed search server that allows users to index and search unstructured content with petabytes of data. This book is your one-stop guide to master the complete Elasticsearch ecosystem. We’ll guide you through comprehensive recipes on what’s new in Elasticsearch 5.x, showing you how to create complex queries and analytics, and perform index mapping, aggregation, and scripting. Further on, you will explore the modules of Cluster and Node monitoring and see ways to back up and restore a snapshot of an index. You will understand how to install Kibana to monitor a cluster and also to extend Kibana for plugins. Finally, you will also see how you can integrate your Java, Scala, Python, and Big Data applications such as Apache Spark and Pig with Elasticsearch, and add enhanced functionalities with custom plugins. By the end of this book, you will have an in-depth knowledge of the implementation of the Elasticsearch architecture and will be able to manage data efficiently and effectively with Elasticsearch.

Credits

About the Author

About the Reviewer

www.PacktPub.com

Customer Feedback

Dedication

Preface

Free Chapter

Getting Started

Introduction

Understanding node and cluster

Understanding node services

Managing your data

Understanding cluster, replication, and sharding

Communicating with Elasticsearch

Using the HTTP protocol

Using the native protocol

Downloading and Setup

Introduction

Downloading and installing Elasticsearch

Setting up networking

Setting up a node

Setting up for Linux systems

Setting up different node types

Setting up a client node

Setting up an ingestion node

Installing plugins in Elasticsearch

Installing plugins manually

Removing a plugin

Changing logging settings

Setting up a node via Docker

Managing Mappings

Introduction

Using explicit mapping creation

Using dynamic templates in document mapping

Managing nested objects

Managing child document

Adding a field with multiple mapping

Mapping a GeoPoint field

Mapping a GeoShape field

Mapping an IP field

Mapping an attachment field

Adding metadata to a mapping

Specifying a different analyzer

Mapping a completion field

Basic Operations

Introduction

Creating an index

Deleting an index

Opening/closing an index

Putting a mapping in an index

Checking if an index or type exists

Managing index settings

Speeding up atomic operations (bulk operations)

Speeding up GET operations (multi GET)

Executing a scrolling query

Using the search_after functionality

Returning inner hits in results

Suggesting a correct query

Counting matched results

Matching all the documents

Using a boolean query

Text and Numeric Queries

Using a wildcard query

Using a regexp query

Using span queries

Using a match query

Using a query string query

Using a simple query string query

Using the range query

The common terms query

Using IDs query

Using the function score query

Using the exists query

Using the template query

Relationships and Geo Queries

Introduction

Using the has_child query

Using the has_parent query

Using nested queries

Using the geo_bounding_box query

Using the geo_polygon query

Using the geo_distance query

Using the geo_distance_range query

Aggregations

Introduction

Executing an aggregation

Executing stats aggregations

Executing terms aggregation

Executing significant terms aggregation

Executing range aggregations

Executing histogram aggregations

Executing date histogram aggregations

Executing filter aggregations

Executing filters aggregations

Executing global aggregations

Executing geo distance aggregations

Executing children aggregations

Executing nested aggregations

Executing top hit aggregations

Executing a matrix stats aggregation

Executing geo bounds aggregations

Executing geo centroid aggregations

Scripting

Introduction

Painless scripting

Installing additional script plugins

Managing scripts

Sorting data using scripts

Computing return fields with scripting

Filtering a search via scripting

Using scripting in aggregations

Updating a document using scripts

Reindexing with a script

Managing Clusters and Nodes

Introduction

Controlling cluster health via an API

Controlling cluster state via an API

Getting nodes information via API

Getting node statistics via the API

Using the task management API

Hot thread API

Managing the shard allocation

Monitoring segments with the segment API

Cleaning the cache

Backup and Restore

Introduction

Managing repositories

Executing a snapshot

Restoring a snapshot

Setting up a NFS share for backup

Reindexing from a remote cluster

User Interfaces

Introduction

Installing and using Cerebro

Installing Kibana and X-Pack

Managing Kibana dashboards

Monitoring with Kibana

Using Kibana dev-console

Visualizing data with Kibana

Installing Kibana plugins

Generating graph with Kibana

Ingest

Introduction

Pipeline definition

Put an ingest pipeline

Get an ingest pipeline

Delete an ingest pipeline

Simulate an ingest pipeline

Built-in processors

Grok processor

Using the ingest attachment plugin

Using the ingest GeoIP plugin

Java Integration

Introduction

Creating a standard Java HTTP client

Creating an HTTP Elasticsearch client

Creating a native client

Managing indices with the native client

Managing mappings

Managing documents

Managing bulk actions

Building a query

Executing a standard search

Executing a search with aggregations

Executing a scroll search

Scala Integration

Introduction

Creating a client in Scala

Managing indices

Managing mappings

Managing documents

Executing a standard search

Executing a search with aggregations

Python Integration

Introduction

Creating a client

Managing indices

Managing mappings include the mapping

Managing documents

Executing a standard search

Executing a search with aggregations

Plugin Development

Introduction

Creating a plugin

Creating an analyzer plugin

Creating a REST plugin

Creating a cluster action

Creating an ingest plugin

Big Data Integration

Introduction

Installing Apache Spark

Indexing data via Apache Spark

Indexing data with meta via Apache Spark

Reading data with Apache Spark

Reading data using SparkSQL

Indexing data with Apache Pig

Customer Reviews

5 star

4 star

3 star

2 star

1 star

Managing your data

If you'll be using Elasticsearch as a search engine or a distributed data store, it's important to understand concepts on how Elasticsearch stores and manages your data.

Getting ready

To work with Elasticsearch data, a user must have basic knowledge of data management and JSON (https://en.wikipedia.org/wiki/JSON) data format that is the lingua franca for working with Elasticsearch data and services.

How it works...

Our main data container is called index (plural indices) and it can be considered similar to a database in the traditional SQL world. In an index, the data is grouped in data types called mappings in Elasticsearch. A mapping describes how the records are composed (fields). Every record, that must be stored in Elasticsearch, must be a JSON object.

Natively, Elasticsearch is a schema-less data store: when you put records in it, during insert it processes the records, splits it in fields, and updates the schema to manage the inserted data.

To manage huge volumes of records, Elasticsearch uses the common approach to split an index into multiple parts (shards) so that they can be spread on several nodes. The shard management is transparent to user usage: all common record operations are managed automatically in Elasticsearch's application layer.

Every record is stored in only a shard; the sharding algorithm is based on record ID, so many operations, that require loading and changing of records/objects, can be achieved without hitting all the shards, but only the shard (and their replicas) that contains your object.

The following schema compares Elasticsearch structure with SQL and MongoDB ones:

Elasticsearch	SQL	MongoDB
Index (indices)	Database	Database
Shard	Shard	Shard
Mapping/Type	Table	Collection
Field	Column	Field
Object (JSON object)	Record (tuples)	Record (BSON object)

The following screenshot is a conceptual representation of an Elasticsearch cluster with three nodes, one index with four shards and replica set to 1 (primary shards are in bold):

There's more...

Elasticsearch, to ensure safe operations on index/mapping/objects, internally has rigid rules about how to execute operations.

In Elasticsearch the operations are divided into:

Cluster/Index operations: All write actions are locking, first they are applied to the master node and then to the secondary one. The read operations are typically broadcasted to all the nodes.
Document operations: All write actions are locking only for the single hit shard. The read operations are balanced on all the shard replicas.

When a record is saved in Elasticsearch, the destination shard is chosen based on:

The unique identifier (ID) of the record. If the ID is missing, it is auto generated by Elasticsearch
If routing or parent (we'll see it in the parent/child mapping) parameters are defined, the correct shard is chosen by the hash of these parameters

Splitting an index in a shard allows you to store your data in different nodes, because Elasticsearch tries to balance the shard distribution on all the available nodes.

Every shard can contain up to 2³² records (about 4.9 Billions), so the real limit to shard size it is the storage size.

Shards contain your data, and during the search process all the shards are used to calculate and retrieve results: so Elasticsearch performance in big data scales horizontally with the number of shards.

All native records operations (that is, index, search, update, and delete) are managed in shards.

The shard management is completely transparent to the user. Only advanced users tend to change the default shard routing and management to cover their custom scenarios, for example, if there is a requirement to put customer data in the same shard to speed up his operations (search/index/analytics).

Best practices

It's best practice not to have too big in size shard (over 10Gb) to avoid poor performance in indexing due to continuous merging and resizing of index segments.

While indexing (a record update is equal to indexing a new element) Lucene, the Elasticsearch engine, writes the indexed documents in blocks (segments/files) to speed up the write process. Over time the small segments are deleted and their sum up is written as a new fragment. Having big fragments due to big shards with a lot of data slows down the indexing performance.

It is not good to over-allocate the number of shards to avoid poor search performance because Elasticsearch works in a map and reduce way due to native distribute search. Shards consist of the worker that does the job of indexing/searching and the master/client nodes do the redux part (collect the results from shards and compute the result to be sent to the user). Having a huge number of empty shards in indices consumes only memory and increases search times due to an overhead on network and results aggregation phases.

Elasticsearch 5.x Cookbook - Third Edition

By : Alberto Paro

Elasticsearch 5.x Cookbook - Third Edition

By: Alberto Paro

Overview of this book

Related Content you might be interested in

Current Title:

Elasticsearch 5.x Cookbook - Third Edition

Mastering Elasticsearch 5.x

Learning Elasticsearch

Elasticsearch 7 Quick Start Guide

Managing your data

Getting ready

How it works...

There's more...

Best practices

See also