We could write a custom spider script to crawl the target website to discover sufficient information about the web application. However, there are often lots of configuration files, leftover development files, backup files, debugging scripts, and many other files that can provide sensitive information about the web application or expose some functionality that the developer of the application did not intend to expose.

The method to discover this type of content is to use brute-forcing to trace common filenames and directories. It is always far superior to have our own custom scripts, which will help us to customize the target files and to filter the results according to our requirements.

First, as usual we import the required modules. Here we use threading to run multiple requests in parallel. But make sure to keep the threads low; a large number of threads may cause denial of service:

import urllib 
import urllib2 
import threading 
...