When an HTTP client makes a request to a web server, they identify who they are. This holds true for web scrapers and normal browsers alike. Have you ever wondered why a website knows that you are a Windows or a Mac user? This information is contained inside your User-Agent string. Here is an example User-Agent string for a Firefox browser on a Linux computer:
Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0
You can see that this string identifies the family, name, and version of the web browser, as well as the operating system. This string will be sent with every request from this browser inside of a request header, such as the following:
GET /index.html HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0
Not all User-Agent strings contain this much information. HTTP clients...
