Book Image

Learning Elastic Stack 7.0 - Second Edition

By : Pranav Shukla, Sharath Kumar M N
Book Image

Learning Elastic Stack 7.0 - Second Edition

By: Pranav Shukla, Sharath Kumar M N

Overview of this book

The Elastic Stack is a powerful combination of tools that help in performing distributed search, analytics, logging, and visualization of data. Elastic Stack 7.0 encompasses new features and capabilities that will enable you to find unique insights into analytics using these techniques. This book will give you a fundamental understanding of what the stack is all about, and guide you in using it efficiently to build powerful real-time data processing applications. The first few sections of the book will help you understand how to set up the stack by installing tools and exploring their basic configurations. You’ll then get up to speed with using Elasticsearch for distributed search and analytics, Logstash for logging, and Kibana for data visualization. As you work through the book, you will discover the technique of creating custom plugins using Kibana and Beats. This is followed by coverage of the Elastic X-Pack, a useful extension for effective security and monitoring. You’ll also find helpful tips on how to use Elastic Cloud and deploy Elastic Stack in production environments. By the end of this book, you’ll be well-versed with fundamental Elastic Stack functionalities and the role of each component in the stack to solve different data processing problems.

Related Content you might be interested in

Current Title:

Small book image
Learning Elastic Stack 7.0 - Second Edition
Pranav Shukla | Sharath Kumar M N
May, 2019 | 474 pages
Small book image
Mastering Elastic Stack
Yuvraj Gupta | Ravi Kumar Gupta
Feb, 2017 | 526 pages
Small book image
Mastering Kibana 6.x
Anurag Srivastava
Jul, 2018 | 376 pages
Small book image
Elasticsearch 7 Quick Start Guide
Douglas Miller | Anurag Srivastava
Oct, 2019 | 186 pages
Table of Contents (17 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Section 1: Introduction to Elastic Stack and Elasticsearch
Section 1: Introduction to Elastic Stack and Elasticsearch
2
Introducing Elastic Stack
Introducing Elastic Stack
What is Elasticsearch, and why use it?
Exploring the components of the Elastic Stack
Use cases of Elastic Stack
Downloading and installing
Summary
3
Getting Started with Elasticsearch
Getting Started with Elasticsearch
Using the Kibana Console UI
Core concepts of Elasticsearch
CRUD operations
Creating indexes and taking control of mapping
REST API overview
Summary
4
Section 2: Analytics and Visualizing Data
Section 2: Analytics and Visualizing Data
5
Searching - What is Relevant
Searching - What is Relevant
The basics of text analysis
Searching from structured data
Searching from the full text
Writing compound queries
Modeling relationships
Summary
6
Analytics with Elasticsearch
Analytics with Elasticsearch
The basics of aggregations
Preparing data for analysis
Metric aggregations
Bucket aggregations
Pipeline aggregations
Summary
7
Analyzing Log Data
Analyzing Log Data
Log analysis challenges
Using Logstash
The Logstash architecture
Overview of Logstash plugins
Ingest node
Summary
8
Building Data Pipelines with Logstash
Building Data Pipelines with Logstash
Parsing and enriching logs using Logstash
Introducing Beats
Filebeat
Summary
9
Visualizing Data with Kibana
Visualizing Data with Kibana
Downloading and installing Kibana
Preparing data
Kibana UI
Timelion
Using plugins
Summary
10
Section 3: Elastic Stack Extensions
Section 3: Elastic Stack Extensions
11
Elastic X-Pack
Elastic X-Pack
Installing Elasticsearch and Kibana with X-Pack
Configuring X-Pack
Securing Elasticsearch and Kibana
Monitoring Elasticsearch
Alerting
Summary
12
Section 4: Production and Server Infrastructure
Section 4: Production and Server Infrastructure
13
Running Elastic Stack in Production
Running Elastic Stack in Production
Hosting Elastic Stack on a managed cloud
Hosting Elastic Stack on your own
Backing up and restoring
Setting up index aliases
Setting up index templates
Modeling time series data
Summary
14
Building a Sensor Data Analytics Application
Building a Sensor Data Analytics Application
Introduction to the application
Modeling data in Elasticsearch
Setting up the metadata database
Building the Logstash data pipeline
Sending data to Logstash over HTTP
Visualizing the data in Kibana
Summary
15
Monitoring Server Infrastructure
Monitoring Server Infrastructure
Metricbeat
Configuring Metricbeat
Capturing system metrics
Deployment architecture
Summary
16
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Building the Logstash data pipeline

Having set up the mechanism to automatically create the Elasticsearch index and the metadata database, we can now focus on building the data pipeline using Logstash. What should our data pipeline do? It should perform the following steps:

  • Accept JSON requests over the web (over HTTP).
  • Enrich the JSON with the metadata we have in the MySQL database.
  • Store the resulting documents in Elasticsearch.

These three main functions that we want to perform correspond exactly with the Logstash data pipeline's input, filter, and output plugins, respectively. The full Logstash configuration file for this data pipeline is in the code base at https://github.com/pranav-shukla/learningelasticstack/tree/v7.0/chapter-10/files/logstash_sensor_data_http.conf.

Let us look at how to achieve the end goal of our data pipeline by following the aforementioned steps...

Previous Section
End of Section 5
Next Section