Now that we've had an in-depth look at both ThreadPoolExecutors
and ProcessPoolExecutors
, it's time to actually put these newly learned concepts into practice. In Chapter 5, Communication between Threads, we started developing a multithreaded web crawler that was able to crawl every available link on a given website.
Note
The full source code for this Python web crawler can be found at this link: https://github.com/elliotforbes/python-crawler.
It didn't, however, output the results in the most readable format, and the code could be improved using ThreadPoolExecutors
. So, let's have a look at implementing both more readable code and more readable results.