Book Image

Machine Learning with the Elastic Stack

By : Rich Collier, Bahaaldine Azarmi
Book Image

Machine Learning with the Elastic Stack

By: Rich Collier, Bahaaldine Azarmi

Overview of this book

Machine Learning with the Elastic Stack is a comprehensive overview of the embedded commercial features of anomaly detection and forecasting. The book starts with installing and setting up Elastic Stack. You will perform time series analysis on varied kinds of data, such as log files, network flows, application metrics, and financial data. As you progress through the chapters, you will deploy machine learning within the Elastic Stack for logging, security, and metrics. In the concluding chapters, you will see how machine learning jobs can be automatically distributed and managed across the Elasticsearch cluster and made resilient to failure. By the end of this book, you will understand the performance aspects of incorporating machine learning within the Elastic ecosystem and create anomaly detection jobs and view results from Kibana directly.

Related Content you might be interested in

Current Title:

Small book image
Machine Learning with the Elastic Stack
Rich Collier | Bahaaldine Azarmi
Jan, 2019 | 304 pages
Small book image
Learning Kibana 7
Bahaaldine Azarmi | Anurag Srivastava
Jul, 2019 | 280 pages
Small book image
Learning Kibana 5.0
Bahaaldine Azarmi
Feb, 2017 | 284 pages
Small book image
Mastering Kibana 6.x
Anurag Srivastava
Jul, 2018 | 376 pages
Table of Contents (12 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
Machine Learning for IT
Machine Learning for IT
Overcoming the historical challenges
Theory of operation
Operationalization
Supporting indices
The orchestration
Summary
2
Installing the Elastic Stack with Machine Learning
Installing the Elastic Stack with Machine Learning
Installing the Elastic Stack
A guided tour of Elastic ML features
Summary
3
Event Change Detection
Event Change Detection
How to understand the normal rate of occurrence
Exploring count functions
Counting in population analysis
Detecting things that rarely occur
Counting message-based logs via categorization
Summary
4
IT Operational Analytics and Root Cause Analysis
IT Operational Analytics and Root Cause Analysis
Holistic application visibility
Data organization
Bringing it all together for root cause analysis
Summary
5
Security Analytics with Elastic Machine Learning
Security Analytics with Elastic Machine Learning
Security in the field
Threat hunting architecture
Investigation analytics
Summary
6
Alerting on ML Analysis
Alerting on ML Analysis
Results presentation
The results index
Alerts from the Machine Learning UI in Kibana
Creating ML alerts manually
Summary
7
Using Elastic ML Data in Kibana Dashboards
Using Elastic ML Data in Kibana Dashboards
Visualization options in Kibana
Preparing data for anomaly detection analysis
Building the visualizations
Summary
8
Using Elastic ML with Kibana Canvas
Using Elastic ML with Kibana Canvas
Introduction to Canvas
Building Elastic ML Canvas slides
Summary
9
Forecasting
Forecasting
Forecasting versus prophesying
Forecasting use cases
Forecasting – theory of operation
Single time series forecasting
Forecast results
Multiple time series forecasting
Summary
10
ML Tips and Tricks
ML Tips and Tricks
Job groups
Influencers in split versus non-split jobs
Using ML on scripted fields
Using one-sided ML functions to your advantage
Ignoring time periods
Don't over-engineer the use case
ML job throughput considerations
Top-down alerting by leveraging custom rules
Sizing ML deployments
Summary
11
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think

Event Change Detection

As shown in the previous chapter, Chapter 2, Installing the Elastic Stack with Machine Learning, tracking metrics and their potential abnormalities over time is certainly an extremely important application of anomaly detection to IT data. This affords a broad, proactive coverage of many key indicators of performance and availability.

However, there are many important use cases that revolve around the idea of event change detection. These include the following:

  • Discovering a flood of error messages suddenly cropping up in a log file
  • Detecting a sudden drop in the amount of orders processed by an online system
  • Determining a sudden excessive number of attempts at accessing something (for example, brute-force authentication or reconnaissance scanning)

In this chapter, we'll discuss the concepts of determining anomalies based on the occurrence rates of...

Previous Section
End of Section 1
Next Section