In this chapter, we covered the basics of how to navigate your web scraper through a website. We looked into the anatomy of a web link, and how to use HTTP GET requests to simulate following a link. We looked at how HTTP forms, such as search boxes, generate HTTP requests. We also saw the difference between HTTP GET and POST requests, and how to send POST requests in Go. We also covered how to avoid loops by tracking your history. Finally, the differences between breadth-first and depth-first web crawling, and their respective trade-offs were covered.
In Chapter 6, Protecting Your Web Scraper, we will look at ways to ensure your safety as you crawl the web.