After we define the portrule, the only step left is to define the action that executes when the portrule returns true. In our case, we want to check whether robots.txt exists on the web server we're scanning.

In order to determine whether the server exists, there's a little bit about the Hypertext Transfer Protocol (HTTP) that we need to learn. First of all, the way to request a page is through an HTTP GET request. For example, if we wanted to go to http://google.com/images, our browser would send a request containing GET /images to the server at Google.com.

If the status of the GET request is OK, the web server returns the status code 200. If there is a server-side error, a 500 error will return. If the page is moved, an error in the range of 300 will return. Lastly (for our purposes), if there is an authorization error or file-not-found error, the server will return 403 or 404 respectively.

In order to define our action function, we need to perform the following...