Book Image

R Web Scraping Quick Start Guide

By : Olgun Aydin
Book Image

R Web Scraping Quick Start Guide

By: Olgun Aydin

Overview of this book

Web scraping is a technique to extract data from websites. It simulates the behavior of a website user to turn the website itself into a web service to retrieve or introduce new data. This book gives you all you need to get started with scraping web pages using R programming. You will learn about the rules of RegEx and Xpath, key components for scraping website data. We will show you web scraping techniques, methodologies, and frameworks. With this book's guidance, you will become comfortable with the tools to write and test RegEx and XPath rules. We will focus on examples of dynamic websites for scraping data and how to implement the techniques learned. You will learn how to collect URLs and then create XPath rules for your first web scraping script using rvest library. From the data you collect, you will be able to calculate the statistics and create R plots to visualize them. Finally, you will discover how to use Selenium drivers with R for more sophisticated scraping. You will create AWS instances and use R to connect a PostgreSQL database hosted on AWS. By the end of the book, you will be sufficiently confident to create end-to-end web scraping systems using R.
Table of Contents (7 chapters)

Introduction to XPath (XML Path)

An XPath represents a path, and when evaluated on a tree, the result is the node set at the end of any path in the tree. HTML, the formatting language used to configure the data in web pages, aims to create a visually appealing interface.

In particular, XML Path Language (XPath) provides powerful syntax for handling specific elements of an XML document and, to the same extent, HTML web pages, in a simple way. XPath is defined as a DOM by the World Wide Web Consortium.

There are two ways to use XPath:

  • To identify a single item in the document tree
  • To address multiple instances of the same item

The main weakness of XPath is its lack of flexibility. Each XPath expression is strictly related to the structure of the web page you are defining.

However, this limitation has been partially reduced since relative road expressions have been added in recent releases. In general, even small changes to the structure of a web page can cause an XPath expression that was defined in an earlier version of the page to not work correctly. In the following screenshot, you can see one XPath rule and its response: