Using dynamic templates in document mapping

In the Using explicit mapping creation recipe, we saw how Elasticsearch can guess the field type using reflection. In this recipe, we'll see how we can help it improve its guessing capabilities via dynamic templates.

The dynamic template feature is very useful. For example, it may be useful in situations where you need to create several indices with similar types because it allows you to move the need to define mappings from coded initial routines to automatic index-document creation. Typical usage is to define types for Logstash log indices.

Getting ready

You will need an up-and-running Elasticsearch installation, as we described in the Downloading and installing Elasticsearch recipe of Chapter 1, Getting Started.

To execute the commands in this recipe, you can use any HTTP client, such as curl (https://curl.haxx.se/), Postman (https://www.getpostman.com/), or similar. I suggest using the Kibana console, which provides code completion and better character escaping for Elasticsearch.

How to do it…

We can extend the previous mapping by adding document-related settings, as follows:

PUT test/_mapping
{
    "dynamic_date_formats":["yyyy-MM-dd", "dd-MM-yyyy"],\
    "date_detection": true,
    "numeric_detection": true,
    "dynamic_templates":[
      {"template1":{
        "match":"*",
        "match_mapping_type": "long",
        "mapping": {"type":" {dynamic_type}", "store": true}
      }}    ],
    "properties" : {...}
}

How it works…

The root object (document) controls the behavior of its fields and all its children object fields. In document mapping, we can define the following:

• date_detection: This allows you to extract a date from a string (true is the default).
• dynamic_date_formats: This is a list of valid date formats. This is used if date_detection is active.
• numeric_detection: This enables you to convert strings into numbers, if possible (false is the default).
• dynamic_templates: This is a list of templates that are used to change the explicit mapping inference. If one of these templates is matched, the rules that have been defined in it are used to build the final mapping.

A dynamic template is composed of two parts: the matcher and the mapping.

To match a field to activate the template, you can use several types of matchers, such as the following:

• match: This allows you to define a match on the field name. The expression is a standard GLOB pattern (http://en.wikipedia.org/wiki/Glob_(programming)).
• unmatch: This allows you to define the expression to be used to exclude matches (optional).
• match_mapping_type: This controls the types of the matched fields; for example, string, integer, and so on (optional).
• path_match: This allows you to match the dynamic template against the full dot notation of the field; for example, obj1.*.value (optional).
• path_unmatch: This will do the opposite of path_match, excluding the matched fields (optional).
• match_pattern: This allows you to switch the matchers to regex (regular expression); otherwise, the glob pattern match is used (optional).

The dynamic template mapping part is a standard one but can use special placeholders, such as the following:

• {name}: This will be replaced with the actual dynamic field name.
• {dynamic_type}: This will be replaced with the type of the matched field.

The order of the dynamic templates is very important; only the first one that is matched is executed. It is good practice to order the ones with more strict rules first, and then the others.

There's more...

Dynamic templates are very handy when you need to set a mapping configuration to all the fields. This can be done by adding a dynamic template, similar to this one:

"dynamic_templates" : [
  { "store_generic" : {
      "match" : "*", "mapping" : { "store" : true }
} } ]  

In this example, all the new fields, which will be added with explicit mapping, will be stored.

See also

• You can find the default Elasticsearch behavior for creating a mapping in the Using explicit mapping creation recipe and the base way of defining a mapping in the Mapping a document recipe.
• The glob pattern is available at http://en.wikipedia.org/wiki/Glob_pattern.