Nmap is an amazing tool for information gathering, and the variety of tasks that can be done with the Nmap Scripting Engine is simply remarkable. The popular service ShodanHQ (https://www.shodan.io/) offers a database of HTTP banners, which is useful for analyzing the impact of vulnerabilities. Its users can find out the number of devices that are online by country, which are identified by their service banners. ShodanHQ uses its own built-in house tools to gather its data, but Nmap can easily be used for this task.

In the following recipe, we will see how to scan indefinitely for web servers, and collect their HTTP headers with Nmap.

Open your terminal and enter the following command:

$ nmap -p80,443 -Pn -T4 --open --script http-headers,http-title,ssl-cert --script-args http.useragent="A friendly web crawler (http://calderonpale.com)",http-headers.useget -oX random-webservers.xml -iR 0

This command will launch an instance of Nmap that will run indefinitely, looking for web servers in port 80 and 443 and then save the output to random-webservers.xml. Each host that has port 80 or 443 open will return something like the following:

   Nmap scan report for XXXX 
   Host is up (0.23s latency). 
   PORT   STATE SERVICE 
   80/tcp open  http 
   |_http-title: Protected Object 
   | http-headers: 
   |   WWW-Authenticate: Basic realm="TD-8840T" 
   |   Content-Type: text/html 
   |   Transfer-Encoding: chunked 
   |   Server: RomPager/4.07 UPnP/1.0 
   |   Connection: close 
   |   EXT: 
   | 
   |_  (Request type: GET) 

The following command will tell Nmap to only check port 80 or 443 (-p80,443), without ping (-Pn), and to use the aggressive timing template (-T4). If port 80 or 443 is open, Nmap will run the NSE scripts http-title, http-headers, and ssl-cert(--script http-headers,http-title,ssl-cert) to collect server headers and web server title; if HTTPS is detected, we will also extract information from SSL certificates if available:

$nmap -p80 -Pn -T4 --open --script http-headers,http-title --script-args http.useragent="A friendly web crawler (http://calderonpale.com)",http-headers.useget -oX random-webservers.xml -iR 0  

The script arguments that are passed are used to set the HTTP user agent in the requests (--script-args http.useragent="A friendly web crawler (http://calderonpale.com)") and use a GET request to retrieve the HTTP headers (--script-args http-headers.useget).

Finally, the argument -iR 0 tell Nmap to generate external IP addresses indefinitely and save the results in a file in XML format (-oX random-webservers.xml).

Nmap's HTTP library has cache support, but if you are planning to scan many hosts, you need to consider your cache file. The cache is stored in a temporary file that grows with each new request. If this file starts to get too big, cache lookups start to take a considerable amount of time.

You can disable the cache system of the HTTP library by setting the http-max-cache-size=0 library argument, as shown in the following command:

$ nmap -p80 --script http-headers --script-args http-max-cache-size=0  -iR 0