Book Image

Machine Learning for Streaming Data with Python

By : Joos Korstanje
Book Image

Machine Learning for Streaming Data with Python

By: Joos Korstanje

Overview of this book

Streaming data is the new top technology to watch out for in the field of data science and machine learning. As business needs become more demanding, many use cases require real-time analysis as well as real-time machine learning. This book will help you to get up to speed with data analytics for streaming data and focus strongly on adapting machine learning and other analytics to the case of streaming data. You will first learn about the architecture for streaming and real-time machine learning. Next, you will look at the state-of-the-art frameworks for streaming data like River. Later chapters will focus on various industrial use cases for streaming data like Online Anomaly Detection and others. As you progress, you will discover various challenges and learn how to mitigate them. In addition to this, you will learn best practices that will help you use streaming data to generate real-time insights. By the end of this book, you will have gained the confidence you need to stream data in your machine learning models.

Related Content you might be interested in

Current Title:

Small book image
Machine Learning for Streaming Data with Python
Joos Korstanje
Jul, 2022 | 258 pages
Small book image
Machine Learning for Time-Series with Python
Ben Auffarth
Oct, 2021 | 370 pages
Table of Contents (17 chapters)
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Conventions used
Get in touch
Share Your Thoughts
1
Part 1: Introduction and Core Concepts of Streaming Data
Part 1: Introduction and Core Concepts of Streaming Data
Free Chapter
2
Chapter 1: An Introduction to Streaming Data
Chapter 1: An Introduction to Streaming Data
Technical requirements
A short history of data science
Working with streaming data
Real-time data formats and importing an example dataset in Python
Summary
Further reading
3
Chapter 2: Architectures for Streaming and Real-Time Machine Learning
Chapter 2: Architectures for Streaming and Real-Time Machine Learning
Technical requirements
Defining your analytics as a function
Understanding microservices architecture
Communicating between services through APIs
Demystifying the HTTP protocol
Building a simple API on AWS
Big data tools for real time streaming
Summary
Further reading
4
Chapter 3: Data Analysis on Streaming Data
Chapter 3: Data Analysis on Streaming Data
Technical requirements
Descriptive statistics on streaming data
Introduction to sampling theory
Overview of the main descriptive statistics
Real-time visualizations
Building basic alerting systems
Summary
Further reading
5
Part 2: Exploring Use Cases for Data Streaming
Part 2: Exploring Use Cases for Data Streaming
6
Chapter 4: Online Learning with River
Chapter 4: Online Learning with River
Technical requirements
What is online machine learning?
Using River for online learning
Summary
Further reading
7
Chapter 5: Online Anomaly Detection
Chapter 5: Online Anomaly Detection
Technical requirements
Defining anomaly detection
Exploring use cases of anomaly detection
Comparing anomaly detection and imbalanced classification
Algorithms for detecting anomalies in River
Going further with anomaly detection
Summary
Further reading
8
Chapter 6: Online Classification
Chapter 6: Online Classification
Technical requirements
Defining classification
Identifying use cases of classification
Overview of classification algorithms in River
Summary
Further reading
9
Chapter 7: Online Regression
Chapter 7: Online Regression
Technical requirements
Defining regression
Use cases of regression
Overview of regression algorithms in River
Summary
Further reading
10
Chapter 8: Reinforcement Learning
Chapter 8: Reinforcement Learning
Technical requirements
Defining reinforcement learning
The main steps of a reinforcement learning model
Exploring Q-learning
Deep Q-learning
Using reinforcement learning for streaming data
Use cases of reinforcement learning
Implementing reinforcement learning in Python
Summary
Further reading
11
Part 3: Advanced Concepts and Best Practices around Streaming Data
Part 3: Advanced Concepts and Best Practices around Streaming Data
12
Chapter 9: Drift and Drift Detection
Chapter 9: Drift and Drift Detection
Technical requirements
Defining drift
Introducing model explicability
Measuring drift
Measuring drift in Python
Counteracting drift
Summary
Further reading
13
Chapter 10: Feature Transformation and Scaling
Chapter 10: Feature Transformation and Scaling
Technical requirements
Challenges of data preparation with streaming data
Scaling data for streaming
Transforming features in a streaming context
Summary
Further reading
14
Chapter 11: Catastrophic Forgetting
Chapter 11: Catastrophic Forgetting
Technical requirements
Introducing catastrophic forgetting
Catastrophic forgetting in online models
Detecting catastrophic forgetting
Model explicability versus catastrophic forgetting
Summary
Further reading
15
Chapter 12: Conclusion and Best Practices
Chapter 12: Conclusion and Best Practices
Going further
Summary
Why subscribe?
16
Other Books You May Enjoy
Other Books You May Enjoy
Packt is searching for authors like you
Share Your Thoughts

Measuring drift in Python

When measuring drift, the first thing to do is to make sure that your model is writing out logs or results in some way. For the following example, you'll use a dataset in which each prediction was logged so that we have for each prediction the input variables, the prediction, the ground truth, and the absolute differences between prediction and ground truth as an indicator of error.

Logging your model's behavior is an absolute prerequisite if you want to work on drift detection. Let's start with some basic measurements that could help you to detect drift using Python.

A basic intuitive approach to measuring drift

In this section, you will discover an intuitive approach to measuring drift. Here are the steps we'll follow:

  1. To get started measuring drift on our logged results data, we start by importing the data as a pandas DataFrame. This is done in the following code block:

Code block 9-1

import pandas as pd
data...
Previous Section
End of Section 6
Next Section