In Chapter 7, Scraping with Concurrency, about concurrency, we saw how defining a clear separation of roles between the worker goroutines and the main goroutine helped mitigate issues in the program. By clearly giving the main goroutine the responsibility of maintaining the state of the target URLs, and allowing the scraper threads to focus on scraping, we laid the groundwork for making a modular system which can easily scale components independently. This separation of concerns is the foundation for building large-scale systems of any kind.
There are a few main components that make up a web scraper. Each of these components should be able to scale without affecting other parts of the system, if they are properly decoupled. You will know if this decoupling is solid if you can break this system into its own package and reuse it for other projects...