What this book covers
Chapter 1, Web Scraping Fundamentals, provides an introduction to web scraping and also explains the latest core web technologies and data-finding techniques.
Chapter 2, Python Programming for Data and Web, provides an overview of choosing and using Python for web scraping. The chapter also explores and explains the World Wide Web (WWW) and URL-based operations by setting up and using the necessary Python libraries, tools, and virtual environments.
Chapter 3, Searching and Processing Web Documents, provides an overview of and introduction to identifying, traversing, and processing web documents using XPath and CSS selectors. The chapter also explains scraping using lxml, collecting data in a file, parsing information from robots.txt, and exploring sitemaps.
Chapter 4, Scraping Using Pyquery, a jQuery-Like Library for Python, provides an introduction to a jQuery-like Python library: pyquery. This chapter provides information on installing and exploring pyquery’s features on web documents. Examples of scraping using pyquery and writing data to JSON and CSV are also covered.
Chapter 5, Scraping the Web with Scrapy and Beautiful Soup, provides an overview and examples of using and deploying a popular web-crawling framework: Scrapy. It also introduces parsing and scraping using BeautifulSoup.
Chapter 6, Working with the Secure Web, provides an overview of dealing with secure web content, using sessions and cookies. The chapter also guides you through and explores scraping content by processing HTML form- and authentication-related issues, as well as providing a guide with examples of how to use proxies during HTTP communication.
Chapter 7, Data Extraction Using Web APIs, provides a detailed overview of the web API, its benefits when used with HTTP content, along with the data formats and patterns available in the API. The chapter also provides a few examples of scraping the web API.
Chapter 8, Using Selenium to Scrape the Web, introduces Selenium WebDriver, which helps automate actions in web browsers, and also covers how to use Selenium to scrape data.
Chapter 9, Using Regular Expressions and PDFs, provides a detailed overview of regular expressions and their usage and implementation using Python. The chapter also provides examples of data extraction using regular expressions and PDF documents using the pypdf2 Python library.
Chapter 10, Data Mining, Analysis, and Visualization, provides an introduction to and detailed overview of data mining and data analysis using the pandas Python library and visualization using Plotly. The chapter also introduces the concept of exploratory data analysis using the ydata_profiling Python library.
Chapter 11, Machine Learning and Web Scraping, provides a detailed introduction to machine learning, a branch of artificial intelligence. The chapter also provides examples of a few machine learning topics using the scikit-learn Python library, along with conducting sentiment analysis from scraped and collected data.
Chapter 12, After Scraping – Next Steps and Data Analysis, provides an overview of and introduction to the next steps related to growing technologies, covering topics such as web requests and data processing in more detail. The chapter also provides information on and guides developers in exploring prospective careers and jobs relating to scraping and data.