Book Image

Python for Secret Agents - Volume II - Second Edition

By : Steven F. Lott, Steven F. Lott
Book Image

Python for Secret Agents - Volume II - Second Edition

By: Steven F. Lott, Steven F. Lott

Overview of this book

Python is easy to learn and extensible programming language that allows any manner of secret agent to work with a variety of data. Agents from beginners to seasoned veterans will benefit from Python's simplicity and sophistication. The standard library provides numerous packages that move beyond simple beginner missions. The Python ecosystem of related packages and libraries supports deep information processing. This book will guide you through the process of upgrading your Python-based toolset for intelligence gathering, analysis, and communication. You'll explore the ways Python is used to analyze web logs to discover the trails of activities that can be found in web and database servers. We'll also look at how we can use Python to discover details of the social network by looking at the data available from social networking websites. Finally, you'll see how to extract history from PDF files, which opens up new sources of data, and you’ll learn about the ways you can gather data using an Arduino-based sensor device.
Table of Contents (12 chapters)
Python for Secret Agents Volume II
Credits
About the Author
About the Reviewer
www.PacktPub.com
Preface
Index

Index

A

  • analog data
    • collecting / Collecting analog data
  • Apache CLF
    • URL / Looking at the CLF
  • Apache logs
    • URL / Getting a web server log
  • Arduino
    • defining / Background briefing: Arduino basics
    • references / Background briefing: Arduino basics
    • shopping list, organizing / Organizing a shopping list
    • flexible and responsive, to fluid situation / Getting it right the first time
    • bulk data, collecting with / Collecting bulk data with the Arduino
  • Arduino IDE
    • obtaining / Getting the Arduino IDE
    • URL / Getting the Arduino IDE
  • Arduino programming language
    • mastering / Mastering the Arduino programming language
    • arithmetic operators, using / Using the arithmetic and comparison operators
    • comparison operators, using / Using the arithmetic and comparison operators
    • common processing statements, using / Using common processing statements
    • Edit-Compile-Test-Break cycle, defining / Hacking and the edit, download, test and break cycle
  • Arduino sensor data feed
    • about / Simple Arduino sensor data feed

B

  • Beautiful Soup
    • upgrading / Mission One – upgrade Beautiful Soup
    • reference / Mission One – upgrade Beautiful Soup
    • HTML page, obtaining / Getting an HTML page
    • HTML structure, navigating / Navigating the HTML structure
  • blinking light
    • about / Seeing a better blinking light
  • blocks of text
    • displaying / Displaying blocks of text
  • bulk data
    • collecting, with Arduino / Collecting bulk data with the Arduino
  • Busy Waiting
    • about / Seeing a better blinking light
  • Busy Waiting design
    • about / Simple Arduino sensor data feed

C

  • CLF
    • definition / Looking at the CLF
    • fields / Looking at the CLF
    • pattern / Looking at the CLF
  • collected data
    • formatting / Formatting the collected data
  • Common Log Format (CLF)
    • about / Understanding the variety of formats
  • complex layouts
    • defining / Understanding tables and complex layouts
  • content filter
    • writing / Writing a content filter
  • cooked strings
    • versus raw strings / Finding a pattern in a file
  • CSV output
    • emitting / Emitting CSV output

D

  • data
    • scraping, from PDF files / Scraping data from PDF files
    • collecting, from serial port / Collecting data from the serial port
  • data collection
    • controlling / Controlling data collection
  • data modeling and analysis
    • with Python / Data modeling and analysis with Python
  • design principles, for methods
    • about / Extending the resource manager
  • digital output pins
    • starting with / Starting with the digital output pins
    • external LED, designing / Designing an external LED
    • working prototype, assembling / Assembling a working prototype
  • diode
    • about / Designing an external LED
  • document
    • text data, obtaining from / Getting text data from a document
  • downloads
    • tracking / What are they downloading?

E

  • EARTH character
    • about / Using some simple statements
  • entities
    • URL / Starting with someone we know
    • about / Starting with someone we know
  • Expandable Cloud Computing (EC2)
    • about / Getting a web server log

F

  • filter
    • noise, reducing with / Reducing noise with a simple filter
  • Flickr
    • URL, for photos / Background briefing – images and social media
  • Fritzing application
    • URL / Getting the Arduino IDE
  • Fritzing tool
    • URL / Getting the Arduino IDE
  • ftplib
    • solution, for obtaining logs from server / Building a more complete solution

G

  • gadgets
    • building / Building our own gadgets
  • generator expressions
    • Eager / Using generator expressions
    • Lazy / Using generator expressions
  • Google Cloud Platform
    • URL / Who's doing the talking?
  • grid
    • exposing / Exposing the grid
  • gzip compressed file
    • reading / Reading a gzip compressed file

H

  • http.client
    • web services, accessing with / Accessing web services with urllib or http.client
  • Hypertext Transfer Protocol (HTTP)
    • about / Accessing web services with urllib or http.client

I

  • image processing
    • with PIL package / Background briefing – images and social media

J

  • JavaScript Object Notation (JSON)
    • about / Navigating the HTML structure, Background briefing – web servers and logs

L

  • language analysis
    • about / Deep Under Cover – NLTK and language analysis
  • Latin America or Caribbean (LAC)
    • about / Processing whois queries
  • linear model
    • creating / Creating a linear model

N

  • Network Information Center (NIC)
    • about / Processing whois queries
  • NLTK
    • about / Deep Under Cover – NLTK and language analysis
    • reference link / Deep Under Cover – NLTK and language analysis
  • noise
    • reducing, with simple filter / Reducing noise with a simple filter
  • numbers
    • defining / Crunching the numbers

O

  • other upgrades
    • defining / Doing other upgrades

P

  • page iterator
    • filtering / Filtering the page iterator
  • PDF
    • URL / Background briefing–Portable Document Format
  • PDF content
    • extracting / Extracting PDF content
    • generator expressions, using / Using generator expressions
    • generator functions, writing / Writing generator functions
    • bad data, filtering / Filtering bad data
    • context manager, writing / Writing a context manager
    • PDF parser resource manager, writing / Writing a PDF parser resource manager
    • resource manager, extending / Extending the resource manager
  • PDF document
    • URL / Extracting PDF content
    • about / Extracting PDF content
  • PDFDocument
    • about / Extracting PDF content
  • PDF files
    • data, scraping from / Scraping data from PDF files
  • PDF Miner
    • references / Extracting PDF content
  • PDF Miner 3k
    • URL / Scraping data from PDF files
  • pdf package
    • URL / Scraping data from PDF files
  • PDFPageInterpreter
    • about / Extracting PDF content
  • PDFParser
    • about / Extracting PDF content
  • PDFResourceManager
    • about / Extracting PDF content
  • Pillow
    • references / Doing other upgrades
  • PIL package
    • used, for processing image / Background briefing – images and social media
  • pip3.4 program
    • about / Background briefing – images and social media
  • pip application
    • using / Mission One – upgrade Beautiful Soup
  • ply package
    • references / Scraping data from PDF files
    • defining / Sidebar on the ply package
  • Portable Document Format
    • defining / Background briefing–Portable Document Format
  • PostScript
    • about / Background briefing–Portable Document Format
  • PyPI
    • URL / Mission One – upgrade Beautiful Soup
  • PySerial
    • URL / Getting a Python serial interface
  • Python
    • URL / Doing a Python upgrade
    • about / Sidebar on the ply package
    • used, for running other programs / Using Python to run other programs
  • Python 3 compatibility
    • URL / Mission One – upgrade Beautiful Soup
  • Python Enhancement Proposals (PEP)
    • about / Background briefing on tools
    • URL / Background briefing on tools
  • Python language
    • reviewing / Background briefing: review of the Python language
    • variables used, for saving results / Using variables to save results
    • strings, using / Using the sequence collections: strings
    • tuples and lists, using / Using other common sequences: tuples and lists
    • dictionary mapping, using / Using the dictionary mapping
    • data, comparing / Comparing data and using the logic operators
    • logic operators, using / Comparing data and using the logic operators
    • simple statements, using / Using some simple statements
    • if statement, using / Using compound statements for conditions: if
    • for statement, using / Using compound statements for repetition: for and while
    • while statement, using / Using compound statements for repetition: for and while
    • functions, defining / Defining functions
    • script files, creating / Creating script files
  • Python serial interface
    • obtaining / Getting a Python serial interface

R

  • raw data
    • reading / Reading and understanding the raw data
    • Physical Format / Reading and understanding the raw data
    • Logical Layout / Reading and understanding the raw data
    • Conceptual Content / Reading and understanding the raw data
    • gzip compressed file, reading / Reading a gzip compressed file
  • raw strings
    • versus cooked strings / Finding a pattern in a file
  • Read Eval Print Loop (REPL)
    • about / Background briefing: review of the Python language
  • referrer URL
    • parsing / Trails of activity
  • regular expression
    • writing, for parsing / Writing a regular expression for parsing
    • rules and patterns / Introducing some regular expression rules and patterns
    • pattern, searching in file / Finding a pattern in a file
    • suffix operators, using / Using regular expression suffix operators
    • characters, capturing by name / Capturing characters by name
    • CLF / Looking at the CLF
  • Regular Expression Strings
    • about / Writing a regular expression for parsing
  • remote files
    • reading / Reading remote files
  • Representational State Transfer (REST)
    • about / Background briefing – web servers and logs
  • resistors
    • adding, for LEDs / Solving problems adding an audible alarm
  • RFC 3912
    • URL / Processing whois queries

S

  • semiconductor
    • about / Designing an external LED
  • Spinning
    • about / Seeing a better blinking light
  • suffix operators
    • using / Using regular expression suffix operators

T

  • tables
    • defining / Understanding tables and complex layouts
  • text block recognition tweaks
    • creating / Making some text block recognition tweaks
  • text data
    • obtaining, from document / Getting text data from a document
  • toolkit
    • expanding / Mission to expand our toolkit
  • tools
    • defining / Background briefing on tools
    • reasons / Background briefing on tools
    • features / Background briefing on tools
    • performance / Background briefing on tools
    • security / Background briefing on tools
    • housecleaning / Background briefing on tools
    • Python, upgrading / Doing a Python upgrade
    • pip, upgrading / Preliminary mission to upgrade pip
  • Twitter API project, on PyPI
    • URL / Mission to expand our toolkit
  • Twitter project from sixohsix
    • URL / Mission to expand our toolkit
  • Twitter social network
    • user information, gathering / Who's doing the talking?
    • profile information, obtaining / Starting with someone we know
    • URL / Starting with someone we know
    • followers, searching / Finding our followers
    • conversation, examining / What do they seem to be talking about?
    • images being posted, gathering / What are they posting?

U

  • urllib
    • web services, accessing with / Accessing web services with urllib or http.client

W

  • weather forecasts
    • URL / Mission One – upgrade Beautiful Soup
  • web server logs
    • overview / Background briefing – web servers and logs
    • formats / Understanding the variety of formats
    • obtaining / Getting a web server log
    • studying / Studying a log in more detail
    • obtaining, from server with ftplib / Getting logs from a server with ftplib
  • web services
    • accessing, with urllib / Accessing web services with urllib or http.client
    • accessing, with http.client / Accessing web services with urllib or http.client
  • Whois program
    • about / Who is this person?
    • URL / Who is this person?
    • reference link / Who is this person?
    • Python, using / Using Python to run other programs
    • whois queries, processing / Processing whois queries
    • request, decomposing / Breaking a request into stanzas and lines
    • stanza-finding algorithm / Alternate stanza-finding algorithm
    • bulk requests, creating / Making bulk requests
  • World Wide Web (WWW)
    • about / Accessing web services with urllib or http.client

Y

  • Yacc (Yet Another Compiler Compiler)
    • about / Sidebar on the ply package