Selecting the right libraries and frameworks does depend on the project scope. Users are free to choose libraries and experience the online process.
In this chapter, we have used and explored various aspects of traversing web documents using Beautiful Soup and have explored a framework built for crawling activities using Spiders: Scrapy. Scrapy provides a complete framework to develop a crawler and is effective using XPath and CSS Selectors with support for the data export. Scrapy projects can also be deployed using Scraping hub to experience the live performance of the deployed Spider and enjoy features provided by the Scrapings hub (Scrapy Cloud) at https://scrapinghub.com/scrapy-cloud.
In the next chapter, we will explore more information regarding scraping data from the web.
