Book Image

Elasticsearch 8.x Cookbook - Fifth Edition

By : Alberto Paro
Book Image

Elasticsearch 8.x Cookbook - Fifth Edition

By: Alberto Paro

Overview of this book

Elasticsearch is a Lucene-based distributed search engine at the heart of the Elastic Stack that allows you to index and search unstructured content with petabytes of data. With this updated fifth edition, you'll cover comprehensive recipes relating to what's new in Elasticsearch 8.x and see how to create and run complex queries and analytics. The recipes will guide you through performing index mapping, aggregation, working with queries, and scripting using Elasticsearch. You'll focus on numerous solutions and quick techniques for performing both common and uncommon tasks such as deploying Elasticsearch nodes, using the ingest module, working with X-Pack, and creating different visualizations. As you advance, you'll learn how to manage various clusters, restore data, and install Kibana to monitor a cluster and extend it using a variety of plugins. Furthermore, you'll understand how to integrate your Java, Scala, Python, and big data applications such as Apache Spark and Pig with Elasticsearch and create efficient data applications powered by enhanced functionalities and custom plugins. By the end of this Elasticsearch cookbook, you'll have gained in-depth knowledge of implementing the Elasticsearch architecture and be able to manage, search, and store data efficiently and effectively using Elasticsearch.
Table of Contents (20 chapters)

Using index components and templates

Real-world index mapping can be very complex and often, parts of it can be reused between different indices types. To be able to simplify this management, mappings can be divided into the following:

  • Components: These will collect the reusable parts of the mapping.
  • Index templates: These aggregate the components in a single template.

Using components is the most manageable way to scale on large index mappings because they can simplify large template management.

Getting ready

You will need an up-and-running Elasticsearch installation, as we described in the Downloading and installing Elasticsearch recipe of Chapter 1Getting Started.

To execute the commands in this recipe, you can use any HTTP client, such as curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest using the Kibana console, which provides code completion and better character escaping for Elasticsearch.

How to do it…

We want to build an index mapping composed of two reusable components. To achieve this, follow these steps:

  1. First, we will create three components for the timestamp, order, and items. These will store parts of our index mapping:
    PUT _component_template/timestamp-management
    { "template": {
        "mappings": {
          "properties": {
            "@timestamp": { "type": "date"  } } } } }
    PUT _component_template/order-data
    { "template": {
        "mappings": {
          "properties": {
            "id": { "type": "keyword" },
            "date": { "type": "date" },
            "customer_id": { "type": "keyword" },
            "sent": { "type": "boolean" } } } } }
    PUT _component_template/items-data
    { "template": {
        "mappings": {
          "properties": {
            "item": {
              "type": "object",
              "properties": {
                "name": { "type": "keyword" },
                "quantity": { "type": "long" },
                "cost": { "type": "alias", "path": "item.price" },
                "price": { "type": "double" },
                "vat": { "type": "double" } } } } } } }
  2. Now, we can create an index template that can sum them up:
    PUT _index_template/order
    {
      "index_patterns": ["order*"],
      "template": {
        "settings": { "number_of_shards": 1 },
        "mappings": {
          "properties": { "id": { "type": "keyword" } } 
         },
        "aliases": { "order": { } }
      },
      "priority": 200,
      "composed_of": ["timestamp-management", "order-data", "items-data"],
      "version": 1,
      "_meta": { "description": "My order index template" } }

How it works…

The process of using index components to build indices templates is very simple: you can register as many components as you wish (Steps 1 and 2 in this recipe) and then aggregate them when you define the template (Step 3). By using this approach, your template is divided into blocks, and the index template is simpler to manage and easily reusable.

For simple use cases, using components to build indices template is too verbose. This approach shines when you need to manage different logs or documents in Elasticsearch that have common parts because you can refactorize them very quickly and reuse them.

Components are simple partial templates that are merged in an index template. Here, the parameters are as follows:

  • index_patterns: This is a list of index glob patterns. When an index is created, if its name matches the glob patterns, the template is applied when the index is created.
  • aliases: This is an optional alias definition to be applied to the created index.
  • template: This is the template to be applied to the index.
  • priority: This is an optional order of priority for applying this template. The standard priority of ELK components is 100, so if the value is set below 100, a custom template can override an ELK one.
  • version: This is an optional incremental number that is managed by the user to keep track of the updates that are made to the template.
  • _meta: This is an optional JSON object that contains metadata for the index.
  • composed_of: This is an optional list of index components that are merged to build the final index mapping.

    Note

    This functionality is available from Elasticsearch version 7.8 and above.

See also

The Adding metadata to a mapping recipe in this chapter about using the _meta field.