Book Image

Python for Secret Agents - Volume II - Second Edition

By : Steven F. Lott
Book Image

Python for Secret Agents - Volume II - Second Edition

By: Steven F. Lott

Overview of this book

Python is easy to learn and extensible programming language that allows any manner of secret agent to work with a variety of data. Agents from beginners to seasoned veterans will benefit from Python's simplicity and sophistication. The standard library provides numerous packages that move beyond simple beginner missions. The Python ecosystem of related packages and libraries supports deep information processing. This book will guide you through the process of upgrading your Python-based toolset for intelligence gathering, analysis, and communication. You'll explore the ways Python is used to analyze web logs to discover the trails of activities that can be found in web and database servers. We'll also look at how we can use Python to discover details of the social network by looking at the data available from social networking websites. Finally, you'll see how to extract history from PDF files, which opens up new sources of data, and you’ll learn about the ways you can gather data using an Arduino-based sensor device.
Table of Contents (7 chapters)
6
Index

Background briefing – web servers and logs


At its heart, the World Wide Web is a vast collection of computers that handle the HTTP protocol. The HTTP protocol defines a request message and a response. A web server handles these requests, creating appropriate responses. This activity is written to a log, and we're interested in that log.

When we interact with a complex web site for a company that conducts e-business—buying or selling on the web—it can seem a lot more sophisticated than this simplistic request and reply protocol. This apparent complexity arises from an HTML web page, which includes JavaScript programming. This extra layer of code can make requests and process replies in ways that aren't obvious to the user of the site.

All web site processing begins with some initial request for an HTML web page. Other requests from JavaScript programs will be data requests that don't lead to a complete HTML page being sent from the server. It's common for JavaScript programs to request JSON...