Even though some sites offer APIs, most websites are designed mainly for human eyes and only provide HTML pages formatted for humans. If we want a program to fetch some data from such a website, we have to parse the markup to get the information we need. Web scraping is the method of using a computer program to analyze a web page and get the data needed.
There are many methods to fetch the content from the site with Python modules:
Use
urllib/urllib2to create an HTTP request that will fetch the webpage, and usingBeautifulSoupto parse the HTMLTo parse an entire website we can use Scrapy (http://scrapy.org), which helps to create web spiders
Use requests module to fetch and lxml to parse
Urllib is a high-level module that allows us to script different services such as HTTP, HTTPS, and FTP.