Book Image

Machine Learning with the Elastic Stack - Second Edition

By : Rich Collier, Camilla Montonen, Bahaaldine Azarmi
5 (1)
Book Image

Machine Learning with the Elastic Stack - Second Edition

5 (1)
By: Rich Collier, Camilla Montonen, Bahaaldine Azarmi

Overview of this book

Elastic Stack, previously known as the ELK stack, is a log analysis solution that helps users ingest, process, and analyze search data effectively. With the addition of machine learning, a key commercial feature, the Elastic Stack makes this process even more efficient. This updated second edition of Machine Learning with the Elastic Stack provides a comprehensive overview of Elastic Stack's machine learning features for both time series data analysis as well as for classification, regression, and outlier detection. The book starts by explaining machine learning concepts in an intuitive way. You'll then perform time series analysis on different types of data, such as log files, network flows, application metrics, and financial data. As you progress through the chapters, you'll deploy machine learning within Elastic Stack for logging, security, and metrics. Finally, you'll discover how data frame analysis opens up a whole new set of use cases that machine learning can help you with. By the end of this Elastic Stack book, you'll have hands-on machine learning and Elastic Stack experience, along with the knowledge you need to incorporate machine learning in your distributed search and data analysis platform.
Table of Contents (19 chapters)
1
Section 1 – Getting Started with Machine Learning with Elastic Stack
4
Section 2 – Time Series Analysis – Anomaly Detection and Forecasting
11
Section 3 – Data Frame Analysis

Anomaly detection job throughput considerations

Elastic ML is awesome and is no doubt very fast and scalable, but there will still be a practical upper bound of events/second processed to any anomaly detection job, depending on a couple of different factors:

  • The speed at which data can be delivered to the algorithms (that is, query performance)
  • The speed at which the algorithms can chew through the data, given the desired analysis

For the latter, much of the performance is based upon the following:

  • The function(s) chosen for the analysis, that is, count is faster than lat_long
  • The bucket_span value chosen (longer bucket spans are faster than smaller bucket spans because more buckets analyzed per unit of time compound the per-bucket processing overhead, which is writing results and so on)

However, if you have a defined analysis set up and can't change it for other reasons, then there's not that much you can do unless you get creative and...