-
Book Overview & Buying
-
Table Of Contents
Python Automation Cookbook - Third Edition
By :
Given the nature of hyperlink pages, starting from a known place and following links to other pages is a very important tool in your arsenal when scraping the web.
To do so, we crawl a page looking for a short phrase, and we print any paragraph that contains it. We will search only in pages that belong to a single site, for example: only URLs starting with www.somesite.com. We won't follow links to external sites.
This recipe talks about consistent crawling of a site, which can be relevant when we are interested, for example, to know whether something on a page has changed by crawling it once a day. For example, we may be interested in what offers are available or in the evolution of prices. Crawling can also be used to download full information, for example, collecting all the reviews for a movie on a movie review site.
Right now GenAI can be very useful for crawling and condensing parts of the web to collect information in a less strict, but faster and more flexible...