Book Image

Python for Secret Agents - Volume II - Second Edition

By : Steven F. Lott, Steven F. Lott
Book Image

Python for Secret Agents - Volume II - Second Edition

By: Steven F. Lott, Steven F. Lott

Overview of this book

Python is easy to learn and extensible programming language that allows any manner of secret agent to work with a variety of data. Agents from beginners to seasoned veterans will benefit from Python's simplicity and sophistication. The standard library provides numerous packages that move beyond simple beginner missions. The Python ecosystem of related packages and libraries supports deep information processing. This book will guide you through the process of upgrading your Python-based toolset for intelligence gathering, analysis, and communication. You'll explore the ways Python is used to analyze web logs to discover the trails of activities that can be found in web and database servers. We'll also look at how we can use Python to discover details of the social network by looking at the data available from social networking websites. Finally, you'll see how to extract history from PDF files, which opens up new sources of data, and you’ll learn about the ways you can gather data using an Arduino-based sensor device.
Table of Contents (12 chapters)
Python for Secret Agents Volume II
About the Author
About the Reviewer

Reducing noise with a simple filter

Is there a way that we can reduce the variability in the output? One possibility to use a moving average of the raw values. Using an Exponentially Weighted Moving Average (EWMA) algorithm will tend to damp out small perturbations in the data, providing a more stable reading.

This moving average is called exponentially weighted because the weights given to previous values fall off exponentially. The immediately previous value is weighted more heavily than the value before that. All values figure into the current value, but as we go back in time, the weights for those old values become very, very small.

The core calculation for a weighted data point, , from the raw data point, , looks like this:

We used a weighting value, w, that expresses the influence of the previous data point on the current data point. If w is one, previous values have no influence. If w is zero, the initial value is the only one that matters and new values are ignored.

The very first data...