Web servers and applications are exposed to the internet more than most other enterprise applications: they have to be available and serve their end customers. Because of this, defenders have been taught to view user traffic (surfing the site, interacting with the dynamic content, and so on) as normal, so long as it follows behavioral norms. Their defenses will focus on broad-based interactions while letting the slow trickle of normal user activity slide. Effective pen testers will mimic this behavior whenever possible to learn as much as they can about their target before launching later, more intrusive stages of the Kill Chain.
As we noted in Chapter 4, Scanning for Vulnerabilities with Arachni, specialized scanning tools can be a double-edged sword. For one thing, most scanners, Arachni included, specialize in looking for vulnerabilities...